We have a class which wraps BouncyCastle (actually SpongyCastle for Android) Blowfish to encrypt data to stream:
public class BlowfishOutputStream extends OutputStream
{
private final OutputStream os;
private final PaddedBufferedBlockCipher bufferedCipher;
Our original code encrypted a whole byte array before writing to the output stream in a single operation
public void write(byte[] raw, int offset, int length) throws IOException
{
byte[] out = new byte[bufferedCipher.getOutputSize(length)];
int result = this.bufferedCipher.processBytes(raw, 0, length, out, 0);
if (result > 0)
{
this.os.write(out, 0, result);
}
}
When sending images (ie large amount of data at once) it results in two copies being retained in memory at once.
The following code is meant to be equivalent but is not, and I do not know why. I can verify that data is being sent (sum of c2 is equivalent to the length) but an intermediate process when it is received on our server discards the image before we get to see what arrives. All I know at this stage is that when the initial code is used, the response is received and the included images can be extracted, when the replacement code is used the response is received (and accepted) but images do not appear to be extracted.
public void write(byte[] raw, int offset, int length) throws IOException
{
// write to the output stream as we encrypt, not all at once.
final byte[] inBuffer = new byte[Constants.ByteBufferSize];
final byte[] outBuffer = new byte[Constants.ByteBufferSize];
ByteArrayInputStream bis = new ByteArrayInputStream(raw);
// read into inBuffer, encrypt into outBuffer and write to output stream
for (int len; (len = bis.read(inBuffer)) != -1;)
{
int c2 = this.bufferedCipher.processBytes(inBuffer, 0, len, outBuffer, 0);
this.os.write(outBuffer, 0, c2);
}
}
Note that the issue is not due to a missing call to doFinal, as this is called when the stream is closed.
public void close() throws IOException
{
byte[] out = new byte[bufferedCipher.getOutputSize(0)];
int result = this.bufferedCipher.doFinal(out, 0);
if (result > 0)
{
this.os.write(out, 0, result);
}
*nb try/catch omitted*
}
Confirmed, although ironically the issue was not with the images but in previous data, but that data was writing the complete raw byte array and not just the range specified. The equivalent code for encrypting the byte array on the fly is:
#Override
public void write(byte[] raw, int offset, int length) throws IOException
{
// write to the stream as we encrypt, not all at once.
final byte[] inBuffer = new byte[Constants.ByteBufferSize];
final byte[] outBuffer = new byte[Constants.ByteBufferSize];
int readStart = offset;
// read into inBuffer, encrypt into outBuffer and write to output stream
while(readStart<length)
{
int readAmount = Math.min(length-readStart, inBuffer.length);
System.arraycopy(raw, readStart, inBuffer, 0, readAmount);
readStart+=readAmount;
int c2 = this.bufferedCipher.processBytes(inBuffer, 0, readAmount, outBuffer, 0);
this.os.write(outBuffer, 0, c2);
}
}
Related
I want to implement a custom InputStream, that calculates Hashes (MD5, SHA1 etc.) and is then wrapped inside another InputStream, that encrypts the Data. The goal is to retrieve the encrypted data later and throw an Exception, if the Hashes do not match anymore.
The use case is pretty simple:
upload file -> stream contents and -> calculate hashes and -> encrypt and -> save the encrypted data
After some digging, I found this simple implementation, that also uses an internal buffer (which seems to be required when I want to achieve somewhat of a performance. My first attempt was to do this in a single InputStream implementation, but I failed horribly.
So instead I am just calculating the Hashes on-the-fly, but even that does not work properly. This is my current implementation:
public class HashingInputStream extends InputStream {
private final static int BUFFER_LENGTH = 16;
private final List<MessageDigest> digests;
private final InputStream inputStream;
private final byte[] buffer = new byte[BUFFER_LENGTH];
private int writeIndex, readIndex;
private boolean eof = false;
public HashingInputStream(InputStream inputStream, List<MessageDigest> digests) {
this.inputStream = inputStream;
this.digests = digests;
}
#Override
public int read() throws IOException {
if(eof)
return -1; // why??
if (readIndex == writeIndex) {
if (writeIndex == buffer.length) {
writeIndex = readIndex = 0;
}
// read bytes into buffer
int bytesRead = 0;
while (bytesRead == 0) {
bytesRead = readBytesIntoBuffer();
}
// if no more data could be read in, return -1 and mark stream as finished
if (bytesRead == -1) {
eof = true;
return -1;
} else {
// update hashes
for (MessageDigest digest : digests) {
digest.update(buffer, 0, bytesRead);
}
}
}
return 255 & buffer[readIndex++];
}
private int readBytesIntoBuffer() throws IOException {
int bytesRead = inputStream.read(buffer, writeIndex, buffer.length - writeIndex);
writeIndex += bytesRead;
return bytesRead;
}
#Override
public void close() throws IOException {
inputStream.close();
}
}
When bytesRead is set to -1 and thus the stream is finished, this stream itself should be finished, but it somehow executes read once more and therefore needed the eof property. If that's not bugging enough, the calculated Hashes differ.
The calling side is straight foreward:
MessageDigest md5 = DigestUtils.getMd5Digest();
MessageDigest sha1 = DigestUtils.getSha1Digest();
try(HashingInputStream uploadInputStream = new HashingInputStream(file.getInputStream(), List.of(md5, sha1))) {
storage.persist(uploadInputStream, upload.getId());
upload.setMd5Hash(DigestUtils.md5Hex(md5.digest()));
upload.setSha1Hash(DigestUtils.sha1Hex(sha1.digest()));
}
Thanks for helping out! Is there any chance I completely misunderstood something?
I'm trying to convert this java snippet code in c# but I'm a bit confused about it.
This is the java code:
My try is the following, but there are some errors in gis.Read, because it wants a char* and not a byte[] and in the String constructor for the same reason.
public static String decompress(InputStream input) throws IOException
{
final int BUFFER_SIZE = 32;
GZIPInputStream gis = new GZIPInputStream(input, BUFFER_SIZE);
StringBuilder string = new StringBuilder();
byte[] data = new byte[BUFFER_SIZE];
int bytesRead;
while ((bytesRead = gis.read(data)) != -1) {
string.append(new String(data, 0, bytesRead));
}
gis.close();
// is.close();
return string.toString();
}
I expected to get a readable string.
You need to transform the bytes to characters first. For that, you need to know the encoding.
In your code, you could have replaced new String(data, 0, bytesRead) with Encoding.UTF8.GetString(data, 0, bytesRead) to do that. However, I would handle this slightly differently.
StreamReader is a useful class to read bytes as text in C#. Just wrap it around your GZipStream and let it do its magic.
public static string Decompress(Stream input)
{
// note this buffer size is REALLY small.
// You could stick with the default buffer size of the StreamReader (1024)
const int BUFFER_SIZE = 32;
string result = null;
using (var gis = new GZipStream(input, CompressionMode.Decompress, leaveOpen: true))
using (var reader = new StreamReader(gis, Encoding.UTF8, true, BUFFER_SIZE))
{
result = reader.ReadToEnd();
}
return result;
}
I am using LZ4 for compressing and decompressing a string.I have tried the following way
public class CompressionDemo {
public static byte[] compressLZ4(LZ4Factory factory, String data) throws IOException {
final int decompressedLength = data.getBytes().length;
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressedLength);
byte[] compressed = new byte[maxCompressedLength];
compressor.compress(data.getBytes(), 0, decompressedLength, compressed, 0, maxCompressedLength);
return compressed;
}
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[data.length];
decompressor.decompress(data,0,restored, 0,data.length);
return new String(restored);
}
public static void main(String[] args) throws IOException, DataFormatException {
String string = "kjshfhshfashfhsakjfhksjafhkjsafhkjashfkjhfjkfhhjdshfhhjdfhdsjkfhdshfdskjfhksjdfhskjdhfkjsdhfk";
LZ4Factory factory = LZ4Factory.fastestInstance();
byte[] arr = compressLZ4(factory, string);
System.out.println(arr.length);
System.out.println(deCompressLZ4(factory, arr) + "decom");
}
}
it is giving following excpetion
Exception in thread "main" net.jpountz.lz4.LZ4Exception: Error decoding offset 92 of input buffer
The problem here is that decompressing is working only if i pass the actual String byte[] length i.e
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[data.length];
decompressor.decompress(data,0,restored, 0,"kjshfhshfashfhsakjfhksjafhkjsafhkjashfkjhfjkfhhjdshfhhjdfhdsjkfhdshfdskjfhksjdfhskjdhfkjsdhfk".getBytes().length);
return new String(restored);
}
It is expecting the actual string byte[] size.
Can someone help me with this
As the compression and decompressions may happen on different machines, or the machine default character encoding is not one of the Unicode formats, one should indicate the encoding too.
For the rest it is using the actual compression and decompression lengths, and better store the size of the uncompressed data too, in plain format, so it may be extracted prior to decompressing.
public static byte[] compressLZ4(LZ4Factory factory, String data) throws IOException {
byte[] decompressed = data.getBytes(StandardCharsets.UTF_8).length;
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressed.length);
byte[] compressed = new byte[4 + maxCompressedLength];
int compressedSize = compressor.compress(decompressed, 0, decompressed.length,
compressed, 4, maxCompressedLength);
ByteBuffer.wrap(compressed).putInt(decompressed.length);
return Arrays.copyOf(compressed, 0, 4 + compressedSize);
}
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
int decrompressedLength = ByteBuffer.wrap(data).getInt();
byte[] restored = new byte[decrompressedLength];
decompressor.decompress(data, 4, restored, 0, decrompressedLength);
return new String(restored, StandardCharsets.UTF_8);
}
It should be told, that String is not suited for binary data, and your compression/decompression is for text handling only. (String contains Unicode text in the form of UTF-16 two-byte chars. Conversion to binary data always involves a conversion with the encoding of the binary data. That costs in memory, speed and possible data corruption.)
I just faced the same error on Android and resolved it based on issue below:
https://github.com/lz4/lz4-java/issues/68
In short make sure you are using the same factory for both operations (compression + decompression) and use Arrays.copyOf() as below:
byte[] compress(final byte[] data) {
LZ4Factory lz4Factory = LZ4Factory.safeInstance();
LZ4Compressor fastCompressor = lz4Factory.fastCompressor();
int maxCompressedLength = fastCompressor.maxCompressedLength(data.length);
byte[] comp = new byte[maxCompressedLength];
int compressedLength = fastCompressor.compress(data, 0, data.length, comp, 0, maxCompressedLength);
return Arrays.copyOf(comp, compressedLength);
}
byte[] decompress(final byte[] compressed) {
LZ4Factory lz4Factory = LZ4Factory.safeInstance();
LZ4SafeDecompressor decompressor = lz4Factory.safeDecompressor();
byte[] decomp = new byte[compressed.length * 4];//you might need to allocate more
decomp = decompressor.decompress(Arrays.copyOf(compressed, compressed.length), decomp.length);
return decomp;
Hope this will help.
restored byte[] length is to small, you should not use compressed data.length, instead you should use data[].length * 3 or more than 3.
I resoved like this:
public static byte[] decompress( byte[] finalCompressedArray,String ... extInfo) {
int len = finalCompressedArray.length * 3;
int i = 5;
while (i > 0) {
try {
return decompress(finalCompressedArray, len);
} catch (Exception e) {
len = len * 2;
i--;
if (LOGGER.isInfoEnabled()) {
LOGGER.info("decompress Error: extInfo ={} ", extInfo, e);
}
}
}
throw new ItemException(1, "decompress error");
}
/**
* 解压一个数组
*
* #param finalCompressedArray 压缩后的数据
* #param length 原始数据长度, 精确的长度,不能大,也不能小。
* #return
*/
private static byte[] decompress(byte[] finalCompressedArray, int length) {
byte[] desc = new byte[length ];
int decompressLen = decompressor.decompress(finalCompressedArray, desc);
byte[] result = new byte[decompressLen];
System.arraycopy(desc,0,result,0,decompressLen);
return result;
}
How to get a byte array from ByteBuf efficiently in the code below? I need to get the array and then serialize it.
package testingNetty;
import io.netty.buffer.ByteBuf;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelInboundHandlerAdapter;
public class ServerHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
System.out.println("Message receive");
ByteBuf buff = (ByteBuf) msg;
// There is I need get bytes from buff and make serialization
byte[] bytes = BuffConvertor.GetBytes(buff);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
// Close the connection when an exception is raised.
cause.printStackTrace();
ctx.close();
}
}
ByteBuf buf = ...
byte[] bytes = new byte[buf.readableBytes()];
buf.readBytes(bytes);
If you don't want the readerIndex to change:
ByteBuf buf = ...
byte[] bytes = new byte[buf.readableBytes()];
int readerIndex = buf.readerIndex();
buf.getBytes(readerIndex, bytes);
If you want to minimize the memory copy, you can use the backing array of the ByteBuf, if it's available:
ByteBuf buf = ...
byte[] bytes;
int offset;
int length = buf.readableBytes();
if (buf.hasArray()) {
bytes = buf.array();
offset = buf.arrayOffset();
} else {
bytes = new byte[length];
buf.getBytes(buf.readerIndex(), bytes);
offset = 0;
}
Please note that you can't simply use buf.array(), because:
Not all ByteBufs have backing array. Some are off-heap buffers (i.e. direct memory)
Even if a ByteBuf has a backing array (i.e. buf.hasArray() returns true), the following isn't necessarily true because the buffer might be a slice of other buffer or a pooled buffer:
buf.array()[0] == buf.getByte(0)
buf.array().length == buf.capacity()
Another option is ByteBufUtil.getBytes(ByteBuf buf, int start, int length, boolean copy)
See ByteBufUtil
So I have some code that reads a certain amount of bytes from a file and returns the resulting byte array (this is basically used for chunking up files to send over the network as (eventually) base64-encoded ascii text).
It works fine, except that when the last chunk of the file is generated, it isnt a full chunk. Therefore, the resulting byte array isnt full. However, it is a constant size, which means that the file is reassembled there is a whole bunch of extra data (0's maybe) appended to the end.
How can I make it so that the byte[] for the last chunk of the file really only contains the data it needs to? The code looks like this:
private byte[] readData(File f, int startByte, int chunkSize) throws Exception {
RandomAccessFile raf = new RandomAccessFile(f, "r");
raf.seek(startByte);
byte[] data = new byte[chunkSize];
raf.read(data);
raf.close();
return data;
}
So if chunkSize is bigger than the remaining bytes in the file, a full sized byte[] gets returned but its only half-full with data.
You'll have to check the return value of RandomAccessFile.read() to determine the number of bytes read. If it's different than the chunkSize, you'll have to copy the array over to a smaller one and return that.
private byte[] readData(File f, int startByte, int chunkSize) throws Exception {
RandomAccessFile raf = new RandomAccessFile(f, "r");
raf.seek(startByte);
byte[] data = new byte[chunkSize];
int bytesRead = raf.read(data);
if (bytesRead != chunkSize) {
byte[] smallerData = new byte[bytesRead];
System.arraycopy(data, 0, smallerData, 0, bytesRead);
data = smallerData;
}
raf.close();
return data;
}
RandomAccessFile.read() returns the number of bytes read, so you can do copy the array if needed:
private byte[] readData(File f, int startByte, int chunkSize) throws Exception {
RandomAccessFile raf = new RandomAccessFile(f, "r");
raf.seek(startByte);
byte[] data = new byte[chunkSize];
int read = raf.read(data);
raf.close();
if (read == data.length) return data;
else
return Arrays.copyOf(data, read);
}
If you are using Java pre-6, then you need to implement Arrays.copyOf yourself:
byte[] r = new byte[read];
System.arraycopy(data, 0, r, 0, read);
return r;
You could also use the size of the file to calculate the remaining number of bytes.
private byte[] readData(File f, int startByte, int chunkSize) throws Exception {
RandomAccessFile raf = new RandomAccessFile(f, "r");
raf.seek(startByte);
int size = (int) Math.min(chunkSize, raf.length()-startByte);
byte[] data = new byte[size];
raf.read(data);
// TODO check the value returned by read (throw Exception or loop)
raf.close();
return data;
}
This way you don't create an additional Array and do not need the copy. Probably not a big impact.
One important point IMO: check the value returned by read, I think it can be less than the remaining bytes. The javadoc states:
The number of bytes read is, at most, equal to the length of b