Develop Reference

Compress JSON to GZIP and Upload to S3

I'm trying to pull in JSON from a lambda, compress it to gzip format and upload to s3. I can do all of this except compress it to gzip. I have pulled various bit of code from here (S.O.) the first code but does not seem to work correctly. Here is what I have tried and the outcome:
this first method seems to make the file much smaller and is gzip format:
public void compressAndUpload(AmazonS3 s3, InputStream in) throws IOException {
Path tmpPath = Files.createTempFile("atest", ".json.gz");
OutputStream out = Files.newOutputStream(tmpPath);
GzipCompressorOutputStream gzOut = new GzipCompressorOutputStream(out);
IOUtils.copy(in, gzOut);
InputStream fileIn = Files.newInputStream(tmpPath);
long size = Files.size(tmpPath);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("application/x-gzip");
metadata.setContentLength(size);
s3.putObject(bucketName, "atest.json.gz", fileIn, metadata);
}
However, when I pull it to my local machine, but when I try to use 'gunzip' on it i get the following error message:
gzip: atest.json.gz: unexpected end of file
this next method when is not actually compressing the file and when i pull it down locally it says "not in gzip format"
public String handleRequest(Input input, Context context) {
try {
byte[] btArr = compress(input.getMessage());
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("application/x-gzip");
metadata.setContentLength(btArr.length);
AmazonS3ClientBuilder.defaultClient().putObject(new PutObjectRequest(bucketName, "test22.json.gz",
new ByteArrayInputStream(btArr), metadata));
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
public static byte[] compress(String str) throws Exception {
if (str == null || str.length() == 0) {
return null;
}
System.out.println("String length : " + str.length());
ByteArrayOutputStream obj=new ByteArrayOutputStream();
GzipCompressorOutputStream gzip = new GzipCompressorOutputStream(obj);
gzip.write(str.getBytes("UTF-8"));
gzip.flush(); <-------******Update: This was missing.. caused it to fail.
gzip.close();
return obj.toByteArray();
}
Am I missing a step here? I feel like this should be a fairly straight forward thing...

Lengthy string compression/decompression in Java

I am looking for String length compression to avoid lengthy filename as below. The string contains UTF-8 characters as well.
"dt=20200623_isValid=valid_module_name=A&B&C_data_source=internet_part-00001-1234-9d12-1234-123d-1234567890a1.b001.json"
Tried Huffman compression from GitHub here, it reduces the size but not much on the String length.
Size before compression: 944
Size after compression: 569
Compressed string:
01011111001111100011101000111011101011001000111110001101000011011001000110001111010001010111111001010110001111010001010001101101010000101101110001110000000110101011010110100000111111001101011111100111101111110100000010101011011110011000010011001000101110010011101001000001111101001010111110000001001101010000111100001110101001100100111110001011101110111011101001001010011000111110111000101100000101100110000010100110001111101110001010011000111110101001010011000111110111011010111011001101100110110111000011100110100111000111011101110111010011100011101111001100100010101
Please advise how to achieve length compression in Java? (The decompressed file Name value is needed for further processing).

You should try ZLIB/GZ Compression. You can find GZ compression snippet here compression and decompression of string data in java
ZLIB compression implementation is also fairly easy. You can use the below code as a starter and improve upon it.
Detailed explanation on compressions How are zlib, gzip and zip related? What do they have in common and how are they different?
Read Deflator strategies before proceeding ahead: Java Deflater strategies - DEFAULT_STRATEGY, FILTERED and HUFFMAN_ONLY
public void compressFile(String originalFileName, String compressedFileName) {
try (FileInputStream fileInputStream = new FileInputStream(originalFileName);
FileOutputStream fileOutputStream = new FileOutputStream(compressedFileName);
DeflaterOutputStream deflaterOutputStream = new DeflaterOutputStream(fileOutputStream))
{
int data;
while ((data = fileInputStream.read()) != -1) {
deflaterOutputStream.write(data);
}
} catch (IOException e) {
e.printStackTrace();
}
}
You can decompress using Inflator.
public void decompressFile(String fileTobeDecomporessed, String outputfile) {
try (
FileInputStream fileInputStream = new FileInputStream(fileTobeDecomporessed);
FileOutputStream fileOutputStream = new FileOutputStream(outputfile);
InflaterInputStream inflaterInputStream = new InflaterInputStream(fileInputStream)) {
int data;
while ((data = inflaterInputStream.read()) != -1) {
fileOutputStream.write(data);
}
} catch (IOException e) {
e.printStackTrace();
}
}
Refer: http://cr.openjdk.java.net/~iris/se/11/latestSpec/api/java.base/java/util/zip/Deflater.html

Of course using one character per binary digit is going to use up a lot of space. That library is using 16 bits (the size of a char) to represent a single bit, so it is literally making its result 16 times larger than it needs to be.
A far more compact way to represent binary data is by converting it to hexadecimal.
byte[] compressedBytes = new BigInteger(compressedString, 2).toByteArray();
Formatter formatter = new Formatter();
for (byte b : compressedBytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
Then the result is 142 bytes:
BE7C7477591F1A1B231E8AFCAC7A28DA85B8E0356B41F9AFCF7E8156F30991727483E95F026A1E1D4C9F17777494C7DC582CC14C7DC531F5298FBB5D9B36E1CD38EEEE9C779915
You could even go a step farther and Base64 encode it, reducing the result to 96 bytes:
String s = Base64.getEncoder().encodeToString(compressedBytes);
Result:
AL58dHdZHxobIx6K/Kx6KNqFuOA1a0H5r89+gVbzCZFydIPpXwJqHh1Mnxd3dJTH3FgswUx9xTH1KY+7XZs24c047u6cd5kV

Java - Reading UTF8 bytes from File into String in a system independent way

How to read a UTF8 encoded file in Java into a String accurately?
When i change the encoding of this .java file to UTF-8 (Eclipse > Rightclick on App.java > Properties > Resource > Text file encoding) , it works fine from within Eclipse but not command line. Seems like eclipse is setting file.encoding parameter when running App.
Why should the encoding of the source file have any impact on creating String from bytes. What is the fool-proof way to create String from bytes when encoding is known?
I may have files with different encodings. Once encoding of a file is known, I must be able to read into string, regardless of value of file.encoding?
The content of utf8 file is below
English Hello World.
Korean 안녕하세요.
Japanese 世界こんにちは。
Russian Привет мир.
German Hallo Welt.
Spanish Hola mundo.
Hindi हैलो वर्ल्ड।
Gujarati હેલો વર્લ્ડ.
Thai สวัสดีชาวโลก.
-end of file-
The code is below. MY observations are in the comments within.
public class App {
public static void main(String[] args) {
String slash = System.getProperty("file.separator");
File inputUtfFile = new File("C:" + slash + "sources" + slash + "TestUtfRead" + slash + "utf8text.txt");
File outputUtfFile = new File("C:" + slash + "sources" + slash + "TestUtfRead" + slash + "utf8text_out.txt");
File outputUtfByteWrittenFile = new File(
"C:" + slash + "sources" + slash + "TestUtfRead" + slash + "utf8text_byteout.txt");
outputUtfFile.delete();
outputUtfByteWrittenFile.delete();
try {
/*
* read a utf8 text file with internationalized strings into bytes.
* there should be no information loss here, when read into raw bytes.
* We are sure that this file is UTF-8 encoded.
* Input file created using Notepad++. Text copied from Google translate.
*/
byte[] fileBytes = readBytes(inputUtfFile);
/*
* Create a string from these bytes. Specify that the bytes are UTF-8 bytes.
*/
String str = new String(fileBytes, StandardCharsets.UTF_8);
/*
* The console is incapable of displaying this string.
* So we write into another file. Open in notepad++ to check.
*/
ArrayList<String> list = new ArrayList<>();
list.add(str);
writeLines(list, outputUtfFile);
/*
* Works fine when I read bytes and write bytes.
* Open the other output file in notepad++ and check.
*/
writeBytes(fileBytes, outputUtfByteWrittenFile);
/*
* I am using JDK 8u60.
* I tried running this on command line instead of eclipse. Does not work.
* I tried using apache commons io library. Does not work.
*
* This means that new String(bytes, charset); does not work correctly.
* There is no real effect of specifying charset to string.
*/
} catch (IOException e) {
e.printStackTrace();
}
}
public static void writeLines(List<String> lines, File file) throws IOException {
BufferedWriter writer = null;
OutputStreamWriter osw = null;
OutputStream fos = null;
try {
fos = new FileOutputStream(file);
osw = new OutputStreamWriter(fos);
writer = new BufferedWriter(osw);
String lineSeparator = System.getProperty("line.separator");
for (int i = 0; i < lines.size(); i++) {
String line = lines.get(i);
writer.write(line);
if (i < lines.size() - 1) {
writer.write(lineSeparator);
}
}
} catch (IOException e) {
throw e;
} finally {
close(writer);
close(osw);
close(fos);
}
}
public static byte[] readBytes(File file) {
FileInputStream fis = null;
byte[] b = null;
try {
fis = new FileInputStream(file);
b = readBytesFromStream(fis);
} catch (Exception e) {
e.printStackTrace();
} finally {
close(fis);
}
return b;
}
public static void writeBytes(byte[] inBytes, File file) {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(file);
writeBytesToStream(inBytes, fos);
fos.flush();
} catch (Exception e) {
e.printStackTrace();
} finally {
close(fos);
}
}
public static void close(InputStream inStream) {
try {
inStream.close();
} catch (IOException e) {
e.printStackTrace();
}
inStream = null;
}
public static void close(OutputStream outStream) {
try {
outStream.close();
} catch (IOException e) {
e.printStackTrace();
}
outStream = null;
}
public static void close(Writer writer) {
if (writer != null) {
try {
writer.close();
} catch (IOException e) {
e.printStackTrace();
}
writer = null;
}
}
public static long copy(InputStream readStream, OutputStream writeStream) throws IOException {
int bytesread = -1;
byte[] b = new byte[4096]; //4096 is default cluster size in Windows for < 2TB NTFS partitions
long count = 0;
bytesread = readStream.read(b);
while (bytesread != -1) {
writeStream.write(b, 0, bytesread);
count += bytesread;
bytesread = readStream.read(b);
}
return count;
}
public static byte[] readBytesFromStream(InputStream readStream) throws IOException {
ByteArrayOutputStream writeStream = null;
byte[] byteArr = null;
writeStream = new ByteArrayOutputStream();
try {
copy(readStream, writeStream);
writeStream.flush();
byteArr = writeStream.toByteArray();
} finally {
close(writeStream);
}
return byteArr;
}
public static void writeBytesToStream(byte[] inBytes, OutputStream writeStream) throws IOException {
ByteArrayInputStream bis = null;
bis = new ByteArrayInputStream(inBytes);
try {
copy(bis, writeStream);
} finally {
close(bis);
}
}
};
Edit: For #JB Nizet, And Everyone :)
//writeLines(list, outputUtfFile, StandardCharsets.UTF_16BE); //does not work
//writeLines(list, outputUtfFile, Charset.defaultCharset()); //does not work.
writeLines(list, outputUtfFile, StandardCharsets.UTF_16LE); //works
I need to specify encoding of bytes when reading bytes into String.
I need to specify encoding of bytes when I am writing bytes from String into file.
Once I have a String in JVM, I do not need to remember the source byte encoding, am I right?
When I write to file, it should convert the String into the default Charset of my machine (be it UTF8 or ASCII or cp1252). That is failing.
It fails for UTF16 BE too. Why does it fail for some Charsets?

The Java source file encoding is indeed irrelevant. And the reading part of your code is correct (although inefficient). What is incorrect is the writing part:
osw = new OutputStreamWriter(fos);
should be changed to
osw = new OutputStreamWriter(fos, StandardCharsets.UTF_8);
Otherwise, you use the default encoding (which doesn't seem to be UTF8 on your system) instead of using UTF8.
Note that Java allows using forward slashes in file paths, even on Windows. You could simply write
File inputUtfFile = new File("C:/sources/TestUtfRead/utf8text.txt");
EDIT:
Once I have a String in JVM, I do not need to remember the source byte encoding, am I right?
Yes, you're right.
When I write to file, it should convert the String into the default Charset of my machine (be it UTF8 or ASCII or cp1252). That is failing.
If you don't specify any encoding, Java will indeed use the platform default encoding to transform the characters into bytes. If you specify an encoding (as suggested in the beginning of this answer), then it uses the encoding you tell it to use.
But all encodings can't, like UTF8, represent all the unicode characters. ASCII for example only supports 128 different characters. Cp1252, AFAIK, only supports 256 characters. So, the encoding succeeds, but it replaces unencodable characters with a special one (I can't remember which one) which means: I can't encode this Thai or Russian character because it's not part of my supported character set.
UTF16 encoding should be fine. But make sure to also configure your text editor to use UTF16 when reading and displaying the content of the file. If it's configured to use another encoding, the displayed content won't be correct.

images courrupted

I am creating httpServer and I have done writing file server part.
But I am having problems when I download images.
FileInputStream fis = new FileInputStream(file_path);
output = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int n = 0;
while (-1 != (n = fis.read(buffer))) {
output.write(buffer, 0, n);
}
data = output.toByteArray();
body = new String(data);
return body
I return the body of response to my original method.
// body is return value from above code, header is also another String return value from
// makeHeader method
String response = header + body;
byte[] Response = null;
try{
Response = response.getBytes("US-ASCII");
}catch (UnsupportedEncodingException e) {}
return Response
My server is working when it comes to text files, .html, .css but not with images.
Can you please point me out what did I do wrong

If you mix text and binary you are sure to corrupt the data. For example US-ASCII is only 7 bit and any byte with the top bit set will be corrupted.
You should attempt to send the image without using String or text to avoid corruption.

Solr Encoding/Decoding Data

I am trying to send a encoded string to Solr and then decode it on retrieval. My encode looks like:
public static String compress(String inputString) {
try {
if (inputString == null || inputString.length() == 0) {
return null;
}
return new String(compress(inputString.getBytes("UTF-8")));
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
return null;
}
private static byte[] compress(byte[] input) {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
GZIPOutputStream gzip = new GZIPOutputStream(out);
gzip.write(input);
gzip.close();
return out.toByteArray();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
Then I send the to SOLR, and when I try to get it back (ignoring decoding for now because it fails here)
SolrDocument resultDoc = iter.next();
String content = (String) resultDoc.getFieldValue("source");
System.out.println(content);
If I send a string such as "Hello my name is Chris" the encoded will look like (ignoring what stack overflow changed);
ã�������ÛHÕ……W»≠T»KÃMU»,VpŒ( ,�ìùùG���
Yet what I get back from SOLR is
#31;ã#8;#0;#0;#0;#0;#0;#0;#0;ÛHÕ……W»≠T»KÃMU»,VpŒ( ,#6;#0;ìùùG#22;#0;#0;#0;
which will obviously make decoding fail. I have tried using the Jetty install and Tomcat both with the same issue.

See this entry from the example schema.xml file that comes with the Solr distribution.
<!--Binary data type. The data should be sent/retrieved in as Base64 encoded Strings -->
<fieldtype name="binary" class="solr.BinaryField"/>
Make sure that the field you are using to store your encoded value in the index is using the binary fieldType and that you are using base64 encoded strings.