I am using LZ4 for compressing and decompressing a string.I have tried the following way
public class CompressionDemo {
public static byte[] compressLZ4(LZ4Factory factory, String data) throws IOException {
final int decompressedLength = data.getBytes().length;
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressedLength);
byte[] compressed = new byte[maxCompressedLength];
compressor.compress(data.getBytes(), 0, decompressedLength, compressed, 0, maxCompressedLength);
return compressed;
}
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[data.length];
decompressor.decompress(data,0,restored, 0,data.length);
return new String(restored);
}
public static void main(String[] args) throws IOException, DataFormatException {
String string = "kjshfhshfashfhsakjfhksjafhkjsafhkjashfkjhfjkfhhjdshfhhjdfhdsjkfhdshfdskjfhksjdfhskjdhfkjsdhfk";
LZ4Factory factory = LZ4Factory.fastestInstance();
byte[] arr = compressLZ4(factory, string);
System.out.println(arr.length);
System.out.println(deCompressLZ4(factory, arr) + "decom");
}
}
it is giving following excpetion
Exception in thread "main" net.jpountz.lz4.LZ4Exception: Error decoding offset 92 of input buffer
The problem here is that decompressing is working only if i pass the actual String byte[] length i.e
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[data.length];
decompressor.decompress(data,0,restored, 0,"kjshfhshfashfhsakjfhksjafhkjsafhkjashfkjhfjkfhhjdshfhhjdfhdsjkfhdshfdskjfhksjdfhskjdhfkjsdhfk".getBytes().length);
return new String(restored);
}
It is expecting the actual string byte[] size.
Can someone help me with this
As the compression and decompressions may happen on different machines, or the machine default character encoding is not one of the Unicode formats, one should indicate the encoding too.
For the rest it is using the actual compression and decompression lengths, and better store the size of the uncompressed data too, in plain format, so it may be extracted prior to decompressing.
public static byte[] compressLZ4(LZ4Factory factory, String data) throws IOException {
byte[] decompressed = data.getBytes(StandardCharsets.UTF_8).length;
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressed.length);
byte[] compressed = new byte[4 + maxCompressedLength];
int compressedSize = compressor.compress(decompressed, 0, decompressed.length,
compressed, 4, maxCompressedLength);
ByteBuffer.wrap(compressed).putInt(decompressed.length);
return Arrays.copyOf(compressed, 0, 4 + compressedSize);
}
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
int decrompressedLength = ByteBuffer.wrap(data).getInt();
byte[] restored = new byte[decrompressedLength];
decompressor.decompress(data, 4, restored, 0, decrompressedLength);
return new String(restored, StandardCharsets.UTF_8);
}
It should be told, that String is not suited for binary data, and your compression/decompression is for text handling only. (String contains Unicode text in the form of UTF-16 two-byte chars. Conversion to binary data always involves a conversion with the encoding of the binary data. That costs in memory, speed and possible data corruption.)
I just faced the same error on Android and resolved it based on issue below:
https://github.com/lz4/lz4-java/issues/68
In short make sure you are using the same factory for both operations (compression + decompression) and use Arrays.copyOf() as below:
byte[] compress(final byte[] data) {
LZ4Factory lz4Factory = LZ4Factory.safeInstance();
LZ4Compressor fastCompressor = lz4Factory.fastCompressor();
int maxCompressedLength = fastCompressor.maxCompressedLength(data.length);
byte[] comp = new byte[maxCompressedLength];
int compressedLength = fastCompressor.compress(data, 0, data.length, comp, 0, maxCompressedLength);
return Arrays.copyOf(comp, compressedLength);
}
byte[] decompress(final byte[] compressed) {
LZ4Factory lz4Factory = LZ4Factory.safeInstance();
LZ4SafeDecompressor decompressor = lz4Factory.safeDecompressor();
byte[] decomp = new byte[compressed.length * 4];//you might need to allocate more
decomp = decompressor.decompress(Arrays.copyOf(compressed, compressed.length), decomp.length);
return decomp;
Hope this will help.
restored byte[] length is to small, you should not use compressed data.length, instead you should use data[].length * 3 or more than 3.
I resoved like this:
public static byte[] decompress( byte[] finalCompressedArray,String ... extInfo) {
int len = finalCompressedArray.length * 3;
int i = 5;
while (i > 0) {
try {
return decompress(finalCompressedArray, len);
} catch (Exception e) {
len = len * 2;
i--;
if (LOGGER.isInfoEnabled()) {
LOGGER.info("decompress Error: extInfo ={} ", extInfo, e);
}
}
}
throw new ItemException(1, "decompress error");
}
/**
* 解压一个数组
*
* #param finalCompressedArray 压缩后的数据
* #param length 原始数据长度, 精确的长度,不能大,也不能小。
* #return
*/
private static byte[] decompress(byte[] finalCompressedArray, int length) {
byte[] desc = new byte[length ];
int decompressLen = decompressor.decompress(finalCompressedArray, desc);
byte[] result = new byte[decompressLen];
System.arraycopy(desc,0,result,0,decompressLen);
return result;
}
Related
Why I am getting the following exception:
Exception in thread "main" java.io.IOException: Push back buffer is full
at java.io.PushbackInputStream.unread(PushbackInputStream.java:232)
at java.io.PushbackInputStream.unread(PushbackInputStream.java:252)
at org.tests.io.PushBackStream_FUN.read(PushBackStream_FUN.java:32)
at org.tests.io.PushBackStream_FUN.main(PushBackStream_FUN.java:43)
In this code:
public class PushBackStream_FUN {
public int write(String outFile) throws Exception {
FileOutputStream outputStream = new FileOutputStream(new File(outFile));
String str = new String("Hello World");
byte[] data = str.getBytes();
outputStream.write(data);
outputStream.close();
return data.length;
}
public void read(String inFile, int ln) throws Exception {
PushbackInputStream inputStream = new PushbackInputStream(new FileInputStream(new File(inFile)));
byte[] data = new byte[ln];
String str;
// read
inputStream.read(data);
str = new String(data);
System.out.println("MSG_0 = "+str);
// unread
inputStream.unread(data);
// read
inputStream.read(data);
str = new String(data);
System.out.println("MSG_1 = "+str);
}
public static void main(final String[] args) throws Exception {
PushBackStream_FUN fun = new PushBackStream_FUN();
String path = "aome/path/output_PushBack_FUN";
int ln = fun.write(path);
fun.read(path, ln);
}
}
UPDATE
Think this is the solution. Java sources to the rescue. I have made some "experiments". It turns out that when I specify PushbackInputStream with a specified buffer size it works. The java sources tells me this:
public PushbackInputStream(InputStream in) {
this(in, 1);
}
public PushbackInputStream(InputStream in, int size) {
super(in);
if (size <= 0) {
throw new IllegalArgumentException("size <= 0");
}
this.buf = new byte[size];
this.pos = size;
}
I think that if you use PushbackInputStream with constrcutor that uses default buffer size, you can only unread single byte. I am unreading more than single byte, thus the exception.
By default, PushbackInputStream only allocates enough space to be able to unread() for a single character. If you want to be able to push back more than that you must specify the capacity at construction time.
In your case it'd look something like this:
final PushbackInputStream pis = new PushbackInputStream( inputStream, ln );
I have a class TouchPoint which implements Serializable and because it contains Bitmap I wrote writeObject and readObject for that class:
private void writeObject(ObjectOutputStream oos) throws IOException {
long t1 = System.currentTimeMillis();
oos.defaultWriteObject();
if(_bmp!=null){
int bytes = _bmp.getWidth()*_bmp.getHeight()*4;
ByteBuffer buffer = ByteBuffer.allocate(bytes);
_bmp.copyPixelsToBuffer(buffer);
byte[] array = buffer.array();
oos.writeObject(array);
}
Log.v("PaintFX","Elapsed Time: "+(System.currentTimeMillis()-t1));
}
private void readObject(ObjectInputStream ois) throws IOException, ClassNotFoundException{
ois.defaultReadObject();
byte[] data = (byte[]) ois.readObject();
if(data != null && data.length > 0){
_bmp = BitmapFactory.decodeByteArray(data, 0, data.length);
}
}
The problem is that I get
SkImageDecoder::Factory returned null
So how can I fix it. I know that possible solution is to change writeObject() to
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
_bmp.compress(Bitmap.CompressFormat.PNG, 100, byteStream);
oos.writeObject(byteStream.toByteArray);
BUT this method is slower almost 10+ times.
copyPixelsToBuffer ~14ms for writing image
_bmp.compress ~ 160ms
UPDATE
Find out that the actual problem is that after
buffer.array();
All byte[] array elements are: 0
Finally I find a way to make it work and be faster at the same time. I was encountered two issues using this method:
I should pass the Bitmap.Config param also, without that I can't decode the byte array
_bmp.compress and _bmp.copyPixelsToBuffer give different arrays so I couldn't use decodeByteArray.
I solved them this way
private void writeObject(ObjectOutputStream oos) throws IOException {
oos.defaultWriteObject();
if(_bmp!=null){
int bytes = _bmp.getWidth()*_bmp.getHeight()*4;
ByteBuffer buffer = ByteBuffer.allocate(bytes);
_bmp.copyPixelsToBuffer(buffer);
byte[] array = new byte[bytes]; // looks like this is extraneous memory allocation
if (buffer.hasArray()) {
try{
array = buffer.array();
} catch (BufferUnderflowException e) {
e.printStackTrace();
}
}
String configName = _bmp.getConfig().name();
oos.writeObject(array);
oos.writeInt(_bmp.getWidth());
oos.writeInt(_bmp.getHeight());
oos.writeObject(configName);
} else {
oos.writeObject(null);
}
}
private void readObject(ObjectInputStream ois) throws IOException, ClassNotFoundException{
ois.defaultReadObject();
byte[] data = (byte[]) ois.readObject();
if (data != null) {
int w = ois.readInt();
int h = ois.readInt();
String configName = (String) ois.readObject();
Bitmap.Config configBmp = Bitmap.Config.valueOf(configName);
Bitmap bitmap_tmp = Bitmap.createBitmap(w, h, configBmp);
ByteBuffer buffer = ByteBuffer.wrap(data);
bitmap_tmp.copyPixelsFromBuffer(buffer);
_bmp = bitmap_tmp.copy(configBmp,true);
bitmap_tmp.recycle();
} else {
_bmp = null;
}
}
This is enough fast for me - about 15x faster then the bmp.compress way. hope this helps :)
Bitmap to byte[]:
Bitmap bmp; // your bitmap
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bmp.compress(Bitmap.CompressFormat.PNG, 100, stream);
byte[] byteArray = stream.toByteArray();
Use Bufferedstreams for better performance.
We have a class which wraps BouncyCastle (actually SpongyCastle for Android) Blowfish to encrypt data to stream:
public class BlowfishOutputStream extends OutputStream
{
private final OutputStream os;
private final PaddedBufferedBlockCipher bufferedCipher;
Our original code encrypted a whole byte array before writing to the output stream in a single operation
public void write(byte[] raw, int offset, int length) throws IOException
{
byte[] out = new byte[bufferedCipher.getOutputSize(length)];
int result = this.bufferedCipher.processBytes(raw, 0, length, out, 0);
if (result > 0)
{
this.os.write(out, 0, result);
}
}
When sending images (ie large amount of data at once) it results in two copies being retained in memory at once.
The following code is meant to be equivalent but is not, and I do not know why. I can verify that data is being sent (sum of c2 is equivalent to the length) but an intermediate process when it is received on our server discards the image before we get to see what arrives. All I know at this stage is that when the initial code is used, the response is received and the included images can be extracted, when the replacement code is used the response is received (and accepted) but images do not appear to be extracted.
public void write(byte[] raw, int offset, int length) throws IOException
{
// write to the output stream as we encrypt, not all at once.
final byte[] inBuffer = new byte[Constants.ByteBufferSize];
final byte[] outBuffer = new byte[Constants.ByteBufferSize];
ByteArrayInputStream bis = new ByteArrayInputStream(raw);
// read into inBuffer, encrypt into outBuffer and write to output stream
for (int len; (len = bis.read(inBuffer)) != -1;)
{
int c2 = this.bufferedCipher.processBytes(inBuffer, 0, len, outBuffer, 0);
this.os.write(outBuffer, 0, c2);
}
}
Note that the issue is not due to a missing call to doFinal, as this is called when the stream is closed.
public void close() throws IOException
{
byte[] out = new byte[bufferedCipher.getOutputSize(0)];
int result = this.bufferedCipher.doFinal(out, 0);
if (result > 0)
{
this.os.write(out, 0, result);
}
*nb try/catch omitted*
}
Confirmed, although ironically the issue was not with the images but in previous data, but that data was writing the complete raw byte array and not just the range specified. The equivalent code for encrypting the byte array on the fly is:
#Override
public void write(byte[] raw, int offset, int length) throws IOException
{
// write to the stream as we encrypt, not all at once.
final byte[] inBuffer = new byte[Constants.ByteBufferSize];
final byte[] outBuffer = new byte[Constants.ByteBufferSize];
int readStart = offset;
// read into inBuffer, encrypt into outBuffer and write to output stream
while(readStart<length)
{
int readAmount = Math.min(length-readStart, inBuffer.length);
System.arraycopy(raw, readStart, inBuffer, 0, readAmount);
readStart+=readAmount;
int c2 = this.bufferedCipher.processBytes(inBuffer, 0, readAmount, outBuffer, 0);
this.os.write(outBuffer, 0, c2);
}
}
I want to compress/decompress and serialize/deserialize String content. I'm using the following two static functions.
/**
* Compress data based on the {#link Deflater}.
*
* #param pToCompress
* input byte-array
* #return compressed byte-array
* #throws NullPointerException
* if {#code pToCompress} is {#code null}
*/
public static byte[] compress(#Nonnull final byte[] pToCompress) {
checkNotNull(pToCompress);
// Compressed result.
byte[] compressed = new byte[] {};
// Create the compressor.
final Deflater compressor = new Deflater();
compressor.setLevel(Deflater.BEST_SPEED);
// Give the compressor the data to compress.
compressor.setInput(pToCompress);
compressor.finish();
/*
* Create an expandable byte array to hold the compressed data.
* You cannot use an array that's the same size as the orginal because
* there is no guarantee that the compressed data will be smaller than
* the uncompressed data.
*/
try (ByteArrayOutputStream bos = new ByteArrayOutputStream(pToCompress.length)) {
// Compress the data.
final byte[] buf = new byte[1024];
while (!compressor.finished()) {
final int count = compressor.deflate(buf);
bos.write(buf, 0, count);
}
// Get the compressed data.
compressed = bos.toByteArray();
} catch (final IOException e) {
LOGWRAPPER.error(e.getMessage(), e);
throw new RuntimeException(e);
}
return compressed;
}
/**
* Decompress data based on the {#link Inflater}.
*
* #param pCompressed
* input string
* #return compressed byte-array
* #throws NullPointerException
* if {#code pCompressed} is {#code null}
*/
public static byte[] decompress(#Nonnull final byte[] pCompressed) {
checkNotNull(pCompressed);
// Create the decompressor and give it the data to compress.
final Inflater decompressor = new Inflater();
decompressor.setInput(pCompressed);
byte[] decompressed = new byte[] {};
// Create an expandable byte array to hold the decompressed data.
try (final ByteArrayOutputStream bos = new ByteArrayOutputStream(pCompressed.length)) {
// Decompress the data.
final byte[] buf = new byte[1024];
while (!decompressor.finished()) {
try {
final int count = decompressor.inflate(buf);
bos.write(buf, 0, count);
} catch (final DataFormatException e) {
LOGWRAPPER.error(e.getMessage(), e);
throw new RuntimeException(e);
}
}
// Get the decompressed data.
decompressed = bos.toByteArray();
} catch (final IOException e) {
LOGWRAPPER.error(e.getMessage(), e);
}
return decompressed;
}
Yet, compared to non-compressed values it's orders of magnitudes slower even if I'm caching the decompressed-result and the values are only decompressed if the content is really needed.
That is, it's used for a DOM-like persistable tree-structure and XPath-queries which force the decompression of the String-values are about 50 times if not even more slower (not really benchmarked, just executed unit tests). My laptop even freezes after some unit tests (everytime, checked it about 5-times), because Eclipse isn't responding anymore due to heavy disk I/O and what not. I've even set the compression level to Deflater.BEST_SPEED, whereas other compression levels might be better, maybe I'm providing a configuration option parameter which can be set for resources. Maybe I've messed something up as I haven't used the deflater before. I'm even only compressing content where the String lenght is > 10.
Edit: After considering to extract the Deflater instantiation to a static field it seems creating an instance of deflater and inflater is very costly as the performance bottleneck is gone and perhaps without microbenchmarks or the like I can't see any performance loss :-) I'm just resetting the deflater/inflater before using a new input.
How you considered using the higher level api like Gzip.
Here is an example for compressing:
public static byte[] compressToByte(final String data, final String encoding)
throws IOException
{
if (data == null || data.length == 0)
{
return null;
}
else
{
byte[] bytes = data.getBytes(encoding);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream os = new GZIPOutputStream(baos);
os.write(bytes, 0, bytes.length);
os.close();
byte[] result = baos.toByteArray();
return result;
}
}
Here is an example for uncompressing:
public static String unCompressString(final byte[] data, final String encoding)
throws IOException
{
if (data == null || data.length == 0)
{
return null;
}
else
{
ByteArrayInputStream bais = new ByteArrayInputStream(data);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
GZIPInputStream is = new GZIPInputStream(bais);
byte[] tmp = new byte[256];
while (true)
{
int r = is.read(tmp);
if (r < 0)
{
break;
}
buffer.write(tmp, 0, r);
}
is.close();
byte[] content = buffer.toByteArray();
return new String(content, 0, content.length, encoding);
}
}
We get very good performance and compression ratio with this.
The zip api is also an option.
Your comments are the correct answer.
In general, if a method is going to be used frequently, you want to eliminate any allocations and copying of data. This often means removing instance initialization and other setup to either static variables or to the constructor.
Using statics is easier, but you may run into lifetime issues (as in how do you know when to clean up the statics - do they exist forever?).
Doing the setup and initialization in the constructor allows the user of the class to determine the lifetime of the object and clean up appropriately. You could instantiate it once before going into a processing loop and GC it after exiting.
The following code is based on the example given in the javadocs for java.util.zip.Deflater. The only changes I have made is to create a byte array called dict and then set the dictionary on both the Deflater and Inflater instances using the setDictionary(byte[]) method.
The problem I'm seeing is that when I call Inflater.setDictionary() with the exact same array as I used for the Deflater, I get an IllegalArgumentException.
Here is the code in question:
import java.util.zip.Deflater;
import java.util.zip.Inflater;
public class DeflateWithDictionary {
public static void main(String[] args) throws Exception {
String inputString = "blahblahblahblahblah??";
byte[] input = inputString.getBytes("UTF-8");
byte[] dict = "blah".getBytes("UTF-8");
// Compress the bytes
byte[] output = new byte[100];
Deflater compresser = new Deflater();
compresser.setInput(input);
compresser.setDictionary(dict);
compresser.finish();
int compressedDataLength = compresser.deflate(output);
// Decompress the bytes
Inflater decompresser = new Inflater();
decompresser.setInput(output, 0, compressedDataLength);
decompresser.setDictionary(dict); //IllegalArgumentExeption thrown here
byte[] result = new byte[100];
int resultLength = decompresser.inflate(result);
decompresser.end();
// Decode the bytes into a String
String outputString = new String(result, 0, resultLength, "UTF-8");
System.out.println("Decompressed String: " + outputString);
}
}
If I try deflating the same compressed bytes without setting the dictionary, I get no error but the result returned is zero bytes.
Is there anything special I need to do in order to use a custom dictionary with Deflater/Inflater?
I actually figured this out while formulating the question but thought I should post the question anyway so others might benefit from my struggles.
It turns out you have to call inflate() once after setting the input but before setting the dictionary. The value returned will be 0, and a call to needsDictionary() will then return true. After that you can set the dictionary and call inflate again.
The amended code is as follows:
import java.util.zip.Deflater;
import java.util.zip.Inflater;
public class DeflateWithDictionary {
public static void main(String[] args) throws Exception {
String inputString = "blahblahblahblahblah??";
byte[] input = inputString.getBytes("UTF-8");
byte[] dict = "blah".getBytes("UTF-8");
// Compress the bytes
byte[] output = new byte[100];
Deflater compresser = new Deflater();
compresser.setInput(input);
compresser.setDictionary(dict);
compresser.finish();
int compressedDataLength = compresser.deflate(output);
// Decompress the bytes
Inflater decompresser = new Inflater();
decompresser.setInput(output, 0, compressedDataLength);
byte[] result = new byte[100];
decompresser.inflate(result);
decompresser.setDictionary(dict);
int resultLength = decompresser.inflate(result);
decompresser.end();
// Decode the bytes into a String
String outputString = new String(result, 0, resultLength, "UTF-8");
System.out.println("Decompressed String: " + outputString);
}
}
This seems very counter intuitive and clunky from an API design perspective, so please enlighten me if there are any better alternatives.