Develop Reference

Compress JSON to GZIP and Upload to S3

I'm trying to pull in JSON from a lambda, compress it to gzip format and upload to s3. I can do all of this except compress it to gzip. I have pulled various bit of code from here (S.O.) the first code but does not seem to work correctly. Here is what I have tried and the outcome:
this first method seems to make the file much smaller and is gzip format:
public void compressAndUpload(AmazonS3 s3, InputStream in) throws IOException {
Path tmpPath = Files.createTempFile("atest", ".json.gz");
OutputStream out = Files.newOutputStream(tmpPath);
GzipCompressorOutputStream gzOut = new GzipCompressorOutputStream(out);
IOUtils.copy(in, gzOut);
InputStream fileIn = Files.newInputStream(tmpPath);
long size = Files.size(tmpPath);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("application/x-gzip");
metadata.setContentLength(size);
s3.putObject(bucketName, "atest.json.gz", fileIn, metadata);
}
However, when I pull it to my local machine, but when I try to use 'gunzip' on it i get the following error message:
gzip: atest.json.gz: unexpected end of file
this next method when is not actually compressing the file and when i pull it down locally it says "not in gzip format"
public String handleRequest(Input input, Context context) {
try {
byte[] btArr = compress(input.getMessage());
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("application/x-gzip");
metadata.setContentLength(btArr.length);
AmazonS3ClientBuilder.defaultClient().putObject(new PutObjectRequest(bucketName, "test22.json.gz",
new ByteArrayInputStream(btArr), metadata));
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
public static byte[] compress(String str) throws Exception {
if (str == null || str.length() == 0) {
return null;
}
System.out.println("String length : " + str.length());
ByteArrayOutputStream obj=new ByteArrayOutputStream();
GzipCompressorOutputStream gzip = new GzipCompressorOutputStream(obj);
gzip.write(str.getBytes("UTF-8"));
gzip.flush(); <-------******Update: This was missing.. caused it to fail.
gzip.close();
return obj.toByteArray();
}
Am I missing a step here? I feel like this should be a fairly straight forward thing...

How to correctly encode and decode a string in Base64?

I want to encode a string in Base64 for later decoding it. I encode it doing this:
public static String encryptString(String string) {
byte[] bytesEncoded = Base64.getEncoder().encode(string.getBytes());
return (new String(bytesEncoded));
}
Then, the encoded string is stored on disk using UTF-8. After restarting the application, the encoded string is readed from disk and I'm trying to decode the string using this:
public static String decryptString(String string) {
byte[] valueDecoded = Base64.getDecoder().decode(string);
return (new String(valueDecoded));
}
Something is wrong because it is giving me this exception:
java.lang.IllegalArgumentException: Illegal base64 character d
at java.base/java.util.Base64$Decoder.decode0(Base64.java:743)
at java.base/java.util.Base64$Decoder.decode(Base64.java:535)
at java.base/java.util.Base64$Decoder.decode(Base64.java:558)
This is a TRACE step by step
1º i encode this: {"configuration":{"shop":{"name":"","addressLine1":"","addressLine2":"","postalCode":"","city":"","country":"","phoneNumber":""}},"jointBets":[],"groups":[{"name":"Test","members":[]}]}
into this: eyJjb25maWd1cmF0aW9uIjp7InNob3AiOnsibmFtZSI6IiIsImFkZHJlc3NMaW5lMSI6IiIsImFkZHJlc3NMaW5lMiI6IiIsInBvc3RhbENvZGUiOiIiLCJjaXR5IjoiIiwiY291bnRyeSI6IiIsInBob25lTnVtYmVyIjoiIn19LCJqb2ludEJldHMiOltdLCJncm91cHMiOlt7Im5hbWUiOiJUZXN0IiwibWVtYmVycyI6W119XX0=
2º i store it on disk in utf8
3º i retreive it from disk and it's this string:
eyJjb25maWd1cmF0aW9uIjp7InNob3AiOnsibmFtZSI6IiIsImFkZHJlc3NMaW5lMSI6IiIsImFkZHJlc3NMaW5lMiI6IiIsInBvc3RhbENvZGUiOiIiLCJjaXR5IjoiIiwiY291bnRyeSI6IiIsInBob25lTnVtYmVyIjoiIn19LCJqb2ludEJldHMiOltdLCJncm91cHMiOlt7Im5hbWUiOiJUZXN0IiwibWVtYmVycyI6W119XX0=
4º i decode it and get the exception.

The old Base64 utility add linebreaks every 76 characters in Java8.
The result looks like that:
/9j/4AAQSkZJRgABAgAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0a
HBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIy
MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCABkAGQDASIA
AhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQA
AAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3
ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWm
...
It seems that this behaviour changed with some version. At least with Java11 the decoder is not accepting line-breaks anymore.
To avoid the problem you could change you method
public static String decryptString(String string) {
byte[] valueDecoded = Base64.getDecoder().decode(string.replace("\n","").replace("\r","");
return new String(valueDecoded);
}

Then, the encoded string is stored on disk using UTF-8. After
restarting the application, the encoded string is readed from disk and
I'm trying to decode the string using this:
This seems to be a point of failure. Most likely your problem is OS/JDK dependent Apparently the following code seems to work well for me (Win 7, latest JDK 1.8):
public static void main(String[] args) throws IOException {
String source = "{\"configuration\":{\"shop\":{\"name\":\"España\",\"addressLine1\":\"\",\"addressLine2\":\"\"," +
"\"postalCode\":\"\",\"city\":\"\",\"country\":\"\",\"phoneNumber\":\"\"}},\"jointBets\":[]," +
"\"groups\":[{\"name\":\"Test\",\"members\":[]}]}";
// Encode string
String encoded = encryptString(source);
System.out.println("Base64 encoded: " + encoded);
// Temp Dir
String tempDir = System.getProperty("java.io.tmpdir");
// Write to File
try (BufferedWriter writer = new BufferedWriter(new FileWriter(tempDir + "data.txt"))) {
writer.write(encoded);
}
// Read from File
Path path = Paths.get(tempDir + "data.txt");
Stream<String> lines = Files.lines(path);
String dataFromFile = lines.collect(Collectors.joining("\n"));
lines.close();
// Compare content
assert encoded.equals(dataFromFile);
// Decode string
String decoded = decryptString(dataFromFile);
System.out.println("Base64 decoded: " + decoded);
}
public static String encryptString(String string) {
byte[] bytesEncoded = Base64.getEncoder().encode(string.getBytes(StandardCharsets.UTF_8));
return new String(bytesEncoded);
}
public static String decryptString(String string) {
byte[] valueDecoded = Base64.getDecoder().decode(string);
return new String(valueDecoded);
}
Base64 encoded:
eyJjb25maWd1cmF0aW9uIjp7InNob3AiOnsibmFtZSI6IkVzcGHDsWEiLCJhZGRyZXNzTGluZTEiOiIiLCJhZGRyZXNzTGluZTIiOiIiLCJwb3N0YWxDb2RlIjoiIiwiY2l0eSI6IiIsImNvdW50cnkiOiIiLCJwaG9uZU51bWJlciI6IiJ9fSwiam9pbnRCZXRzIjpbXSwiZ3JvdXBzIjpbeyJuYW1lIjoiVGVzdCIsIm1lbWJlcnMiOltdfV19
Base64 decoded:
{"configuration":{"shop":{"name":"España","addressLine1":"","addressLine2":"","postalCode":"","city":"","country":"","phoneNumber":""}},"jointBets":[],"groups":[{"name":"Test","members":[]}]}

My guess is that you are not specifying a charset. Try running the below maybe with and without the charset specified for the String constructor to verify.
#Test
public void base64Test() throws Exception{
String string = "ABCDF";
byte[] bytesEncoded = Base64.getEncoder().encode(string.getBytes());
String encodedStr = (new String(bytesEncoded,Charset.forName("ISO-8859-1")));
System.out.println(encodedStr);
byte[] valueDecoded = Base64.getDecoder().decode(encodedStr);
String decodedStr = (new String(valueDecoded,Charset.forName("ISO-8859-1")));
System.out.println(decodedStr);
}

Sending pdf data through rest api inside json

I have made a webservice that send multiple pdfs as response to client using mutlipart/formdata but as it happens one of the client is salesforce which does not support mutlipart/ formdata.
They want a json in response like -
{ "filename": xyzname,
"fileContent": fileContent
}
I tried encoding data in Base64 using apache codec library but pdf on client side seems to get corrupted and I am unable to open it using acrobat.
Please find code below -
import org.apache.commons.io.FileUtils;
//------Server side ----------------
#POST
#Consumes(MULTIPART_FORM_DATA)
#Produces(MediaType.APPLICATION_JSON)
#Path("somepath")
public Response someMethod(someparam) throws Exception
{
....
JSONArray filesJson = new JSONArray();
String base64EncodedData = Base64.encodeBase64URLSafeString(loadFileAsBytesArray(tempfile));
JSONObject fileJSON = new JSONObject();
fileJSON.put("fileName",somename);
fileJSON.put("fileContent", base64EncodedData);
filesJson.put(fileJSON);
.. so on ppopulate jsonArray...
//sending reponse
responseBuilder = Response.ok().entity(filesJson.toString()).type(MediaType.APPLICATION_JSON_TYPE) ;
response = responseBuilder.build();
}
//------------Client side--------------
Response clientResponse = webTarget.request()
.post(Entity.entity(entity,MediaType.MULTIPART_FORM_DATA));
String response = clientResponse.readEntity((String.class));
JSONArray fileList = new JSONArray(response);
for(int count= 0 ;count< fileList.length();count++)
{
JSONObject fileJson = fileList.getJSONObject(count);
byte[] decodedBytes = Base64.decodeBase64(fileJson.get("fileContent").toString());
outputFile = new File("somelocation/" + fileJson.get("fileName").toString() + ".pdf");
FileUtils.writeByteArraysToFile(outputFile, fileJson.get("fileContent").toString().getBytes());
}
-------------------------------
Kindly advise.

Yes so the problem was with the client.
while decoding we should use
byte[] decodedBytes = Base64.decodeBase64(fileJson.getString("fileContent"));
rather than
byte[] decodedBytes = Base64.decodeBase64(fileJson.get("fileContent").toString());
Since encoded data.toString() yields some think else
Also replaced encodeBase64URLSafeString with encodeBase64String
Well quite a simple solution :)

We are doing the same, basically sending PDF as JSON to Android/iOS and Web-Client (so Java and Swift).
The JSON Object:
public class Attachment implements Serializable {
private String name;
private String content;
private Type contentType; // enum: PDF, RTF, CSV, ...
// Getters and Setters
}
And then from byte[] content it is set the following way:
public Attachment createAttachment(byte[] content, String name, Type contentType) {
Attachment attachment = new Attachment();
attachment.setContentType(contentType);
attachment.setName(name);
attachment.setContent(new String(Base64.getMimeEncoder().encode(content), StandardCharsets.UTF_8));
}
Client Side Java we create our own file type object first before mapping to java.io.File:
public OurFile getAsFile(String content, String name, Type contentType) {
OurFile file = new OurFile();
file.setContentType(contentType);
file.setName(name);
file.setContent(Base64.getMimeDecoder().decode(content.getBytes(StandardCharsets.UTF_8)));
return file;
}
And finally:
public class OurFile {
//...
public File getFile() {
if (content == null) {
return null;
}
try {
File tempDir = Files.createTempDir();
File tmpFile = new File(tempDir, name + contentType.getFileEnding());
tempDir.deleteOnExit();
FileUtils.copyInputStreamToFile(new ByteArrayInputStream(content), tmpFile);
return tmpFile;
} catch (IOException e) {
throw new RuntimeException(e);
}
}

In my REST application with PHP:
  1. Send the data encoded in base64 $data = base64_encode($data) to REST.
  2. Before writing the file, I decode $data = base64_decode($data).
  3. Therefore, when the file is downloaded, it is already in the correct format.

I would change from using "Safe" to just using "string". So change:
encodeBase64URLSafeString(...)
to:
encodeBase64String(...)
The reason is that the "safe" versions actually change the content to preserve URLs before encrypting - I'm totally uncertain what that would do to a PDF, but suspect it is the source of your problem.
If that doesn't do it for you, I suggest encrypting/decrypting right on the server (or a separate test app) and comparing the results while you try to work it out. That way you can see if what you are doing is working, but don't have to go through the whole "start the server, start the client, connect..." process each time, and it will speed your debugging.

Is it possible to change the filename of a google cloud storage file using java?

It's time to change the downloaded file name in the Google Cloud Storage using java(BlobstoreService). Is there any provision in the BlobstoreService to change the file name before downloading that file? Is there any useful API for changing the filename? Here the thing that happens is, when I save a file in the GCS, it will generate a blob key. And the file type also changes in the Google cloud storage. Now I just want to change the file name before it downloads, and also the type of the file.

You can rename a file by reading the file in and saving it with the new name. Here is some sample code to get you started, you will likely need to add your own delete function in order to delete the old file:
public void moveFile(String fileName, String bucket, String newFilename, String contentType) throws IOException {
byte[] bytes = loadFile(bucket, fileName);
ByteArrayInputStream in = new ByteArrayInputStream(bytes);
saveToGcs(bucket, fileName, in, contentType);
}
public byte[] loadFile(String bucket, String fileName) throws IOException {
GcsFilename gcsFileName = new GcsFilename(bucket, fileName);
GcsInputChannel readChannel = gcsService.openReadChannel(gcsFileName, 0);
InputStream in = Channels.newInputStream(readChannel);
return IOUtils.toByteArray(in);
}
private void saveToGcs(String bucket, String filename, InputStream inputStream, String mimeType) throws IOException {
GcsFilename gcsFilename = new GcsFilename(bucket, filename);
GcsFileOptions options = new GcsFileOptions.Builder().mimeType(mimeType).acl("public-read").build();
GcsOutputChannel writeChannel = gcsService.createOrReplace(gcsFilename, options);
BufferedOutputStream outputStream = new BufferedOutputStream(Channels.newOutputStream(writeChannel));
IOUtils.copy(inputStream, outputStream);
outputStream.close();
writeChannel.close();
}

Read here : https://cloud.google.com/storage/docs/copying-renaming-moving-objects#storage-rename-object-java
Create a copyWriter and copy the original blob to the renamed (new) address. Then delete the original

HtmlUnit use WebClient to download an image as base64 encoded DATA Uri

I want to use an existing instance of WebClient to download an image. The reason for this is because I want the cookies to be passed with the request.
How can I download an image using an existing instance of WebClient?
Also, how can I base64 encode the image to be able to view it using data:image/jpeg;base64,...
Current code:
WebClient client = new WebClient(BrowserVersion.FIREFOX_3_6);
UnexpectedPage imagePage = client.getPage("http://...");
String imageString = imagePage.getWebResponse().getContentAsString();
BASE64Encoder encoder = new BASE64Encoder();
String base64data = encoder.encode(imageString.getBytes());
So now I have base64 data of the image, but I still can't view the image using data:image/jpeg;base64,....

A couple of things to consider:
The BASE64Encoder() generates a string that has a line break every 77 chars. Take that out using .replaceAll("\r?\n","").
For that method also, it is better to retrieve the web page InputStream rather than the string. Also, to convert that to a byte array, I used a utility method (source and other options can be found here).
Working source code:
public static void main (String args[]) throws IOException {
WebClient client = new WebClient(BrowserVersion.FIREFOX_3_6);
UnexpectedPage imagePage = client.getPage("http://i.stack.imgur.com/9DdHc.jpg");
BASE64Encoder encoder = new BASE64Encoder();
String base64data = encoder.encode(inputStreamToByteArray(imagePage.getWebResponse().getContentAsStream()));
System.out.println("<img src=\"data:image/png;base64,"+base64data.replaceAll("\r?\n","")+"\" />");
}
private static byte[] inputStreamToByteArray(InputStream is) throws IOException {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[16384];
while ((nRead = is.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
return buffer.toByteArray();
}
Source image:
Output base64 image here.