I want to get sha2 hash of particular java object. I don't want it to be int, I want byte[] or at least String. I've got the following code to create sha2:
static byte[] sha2(byte[] message) {
if (message == null || message.length == 0) {
throw new IllegalArgumentException("message is null");
}
try {
MessageDigest sha256 = MessageDigest.getInstance(SHA_256);
return sha256.digest(message);
} catch (NoSuchAlgorithmException e) {
throw new IllegalArgumentException(e);
}
}
I can just convert my object to byte[], but I don't think it's a good idea to store big array in memory just to create 32 byte array. So how can I compute sha2(or maybe another crypto hash function) of object?
You do not have to load the whole object into memory, you can load parts of it into temporary buffer.
Dump object into a temporary file using FileOutputStream/BufferedOutputStream, this will make sure serialized object does not pollute JVM memory.
The load serialize object from temporary file using FileInputStream/BufferedInputStream and feed it to MessageDigest#update(buf) method in a loop.
Finally call MessageDigest#digest() to finish work:
int[] buf = new int[1024];
while (/* has more data */) {
int readBytes = readIntoBuf(buf);
sha256.update(buf, 0, readBytes);
}
return sha256.digest();
If you can afford to store entire serialized object in memory, use ByteArrayOutputStream and pass result byte[] to MessageDigest#digest(buf):
try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOuputStream(baos)) {
oos.writeObject(obj);
MessageDigest sha256 = MessageDigest.getInstance(SHA_256);
return sha256.digest(baos.toByteArray());
}
Related
I'm trying to use the java DeflaterOutputStream and InflaterOutputStream classes to compress a byte array, but both appear to not be working correctly. I assume I'm incorrectly implementing them.
public static byte[] compress(byte[] in) {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
DeflaterOutputStream defl = new DeflaterOutputStream(out);
defl.write(in);
defl.flush();
defl.close();
return out.toByteArray();
} catch (Exception e) {
e.printStackTrace();
System.exit(150);
return null;
}
}
public static byte[] decompress(byte[] in) {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
InflaterOutputStream infl = new InflaterOutputStream(out);
infl.write(in);
infl.flush();
infl.close();
return out.toByteArray();
} catch (Exception e) {
e.printStackTrace();
System.exit(150);
return null;
}
}
Here's the two methods I'm using to compress and decompress the byte array. Most implementations I've seen online use a fixed size buffer array for the decompression portion, but I'd prefer to avoid that if possible, because I'd need to make that buffer array have a size of one if I want to have any significant compression.
If anyone can explain to me what I'm doing wrong it would be appreciated. Also, to explain why I know these methods aren't working correctly: The "compressed" byte array that it outputs is always larger than the uncompressed one, no matter what size byte array I attempt to provide it.
This will depend on the data you are compressing. For example if we take an array of 0 bytes it compresses well:
byte[] plain = new byte[10000];
byte[] compressed = compress(plain);
System.out.println(compressed.length); // 33
byte[] result = decompress(compressed);
System.out.println(result.length); // 10000
Compression always has overhead to allow for future decompression. If the compression produced no reduction in length (the data was unique or nearly unique) then the output file could be longer than the input file
if I send a List Object to the Server via ObjectOutputStream which is null, then I got an exception.
So, I wanna check if the ObjectInputStream at server-side's is empty.
But I dont know how to do this.
neither ois.available() nor ois.readObject != null works.
Here is an example-code:
Client:
String str= null;//"Hello \n";
//Nachricht mit AES verschlüsseln und an Server senden
cipherAES.init(Cipher.ENCRYPT_MODE, encryptionKey, iv);
byte[] input = str.getBytes();
byte[] ctLength = cipherAES.doFinal(input);
List<Byte> messageToServer = new ArrayList<>();
for(int i = 0; i < ctLength.length; i++){
messageToServer.add(ctLength[i]);
}
//3. OUTPUTSTREAM#############################################################################
oos.writeObject(messageToServer);
os.flush();
Server
//3.INPUTSTREAM#####################################################################
//ois.availale();At this place, it doesnt work. if List<Byte> isn't null result = 0;
//Empfange Clients verschlüsselte Nachricht
List<Byte> encryptedMessageFromClient = (List<Byte>) ois.readObject();
// At this place ois.available retruns the value of the Object, if the object isnt null;
//#####################################################################
If you serialize List which contains null value with the help of writeObject() and try to deserialize it at server end you will get null again.
We don't have method as objectOutputStream.available() but we have objectInputStream.available() which returns the number of bytes that can be read without blocking.
Note that
ObjectOutputStream in Java can be used to convert an object to OutputStream. The process of converting object to stream is called serialization in java.
and
The Java ObjectInputStream class (java.io.ObjectInputStream) enables you to read Java objects from an InputStream instead of just raw bytes. You wrap an InputStream in a ObjectInputStream and then you can read objects from it. Of course the bytes read must represent a valid, serialized Java object. Otherwise reading objects will fail.
Resolved. There is no patent remedy.
So, if you have several ObjectInputStreams, then work with "instanceof" or "!instanceof".
Please note: instanceof doesnt accept generic-objects.
Edit: Code-Example: Client:
if (str != null){
//Nachricht mit AES verschlüsseln und an Server senden
cipherAES.init(Cipher.ENCRYPT_MODE, encryptionKey, iv);
byte[] input = str.getBytes();
byte[] ctLength = cipherAES.doFinal(input);
List<Byte> messageToServer = new ArrayList<>();
for(int i = 0; i < ctLength.length; i++){
messageToServer.add(ctLength[i]);
}
//3.
OUTPUTSTREAM##############################################
oos.writeObject(messageToServer);
os.flush();
}
Code example Server:
//3.INPUTSTREAM######################################
Object readed = ois.readObject();
//##################################################
if(!(readed instanceof requiredObject)){
objectYouNeed = (cast_it)Object;
}else
{
--> Go on with another Object which is in the stream
}
I have a ByteArrayOutputStream that has large amounts of data written into, which is ultimately converted into a byte array and written to a cache:
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (JsonGenerator jg = mapper.getFactory().createGenerator(baos)) {
for (Object result : results) {
jg.writeObject(result);
}
}
localCache.put(cacheKey, baos.toByteArray());
}
catch (IOException e) {
throw Throwables.propagate(e);
}
Here baos.toByteArray() creates a whole new copy of the data in memory, which I'm trying to avoid. Is there a way to convert the stream to a byte array without using the extra memory?
The internal buffer and current count are protected fields documented in the Javadoc. This means you should be OK to subclass ByteArrayOutputStream and provide a byte[] getBuffer() method to access the buffer directly. Use the existing size() method to determine how much data is present.
public class MyBAOS extends ByteArrayOutputStream
{
public MyBAOS() { super(); }
public MyBAOS(int size) { super(size); }
public byte[] getBuffer() { return buf; }
}
I have a little problem: I decompress byte array and everything is ok with following code but sometimes with some data it throws DataFormatException with incorrect data check. Any ideas?
private byte[] decompress(byte[] compressed) throws DecoderException {
Inflater decompressor = new Inflater();
decompressor.setInput(compressed);
ByteArrayOutputStream outPutStream = new ByteArrayOutputStream(compressed.length);
byte temp [] = new byte[8196];
while (!decompressor.finished()) {
try {
int count = decompressor.inflate(temp);
logger.info("count = " + count);
outPutStream.write(temp, 0, count);
}
catch (DataFormatException e) {
logger.info(e.getMessage());
throw new DecoderException("Wrong format", e);
}
}
try {
outPutStream.close();
} catch (IOException e) {
throw new DecoderException("Cant close outPutStream ", e);
}
return outPutStream.toByteArray();
}
Try with a different compression level or using the nowrap options
1 Some warning: do you use the same algorithm in both sides ?
do you use bytes ? (not String)
your arrays have the good sizes ?
2
I suggest you check step by step, catching exceptions, checking sizes, null, and comparing bytes.
like this: Using Java Deflater/Inflater with custom dictionary causes IllegalArgumentException
Take your input
Compress it
copy your bytes
decompress them
compare output with input
3 if you cant find, take another example which works, and modify it step by step
hope it helps
I found out why its happening
byte temp [] = new byte[8196];
its too big, it must be exactly size of decompressed array cause it was earlier Base64 encoded, how i can get this size before decompressing it?
I want to compress/decompress and serialize/deserialize String content. I'm using the following two static functions.
/**
* Compress data based on the {#link Deflater}.
*
* #param pToCompress
* input byte-array
* #return compressed byte-array
* #throws NullPointerException
* if {#code pToCompress} is {#code null}
*/
public static byte[] compress(#Nonnull final byte[] pToCompress) {
checkNotNull(pToCompress);
// Compressed result.
byte[] compressed = new byte[] {};
// Create the compressor.
final Deflater compressor = new Deflater();
compressor.setLevel(Deflater.BEST_SPEED);
// Give the compressor the data to compress.
compressor.setInput(pToCompress);
compressor.finish();
/*
* Create an expandable byte array to hold the compressed data.
* You cannot use an array that's the same size as the orginal because
* there is no guarantee that the compressed data will be smaller than
* the uncompressed data.
*/
try (ByteArrayOutputStream bos = new ByteArrayOutputStream(pToCompress.length)) {
// Compress the data.
final byte[] buf = new byte[1024];
while (!compressor.finished()) {
final int count = compressor.deflate(buf);
bos.write(buf, 0, count);
}
// Get the compressed data.
compressed = bos.toByteArray();
} catch (final IOException e) {
LOGWRAPPER.error(e.getMessage(), e);
throw new RuntimeException(e);
}
return compressed;
}
/**
* Decompress data based on the {#link Inflater}.
*
* #param pCompressed
* input string
* #return compressed byte-array
* #throws NullPointerException
* if {#code pCompressed} is {#code null}
*/
public static byte[] decompress(#Nonnull final byte[] pCompressed) {
checkNotNull(pCompressed);
// Create the decompressor and give it the data to compress.
final Inflater decompressor = new Inflater();
decompressor.setInput(pCompressed);
byte[] decompressed = new byte[] {};
// Create an expandable byte array to hold the decompressed data.
try (final ByteArrayOutputStream bos = new ByteArrayOutputStream(pCompressed.length)) {
// Decompress the data.
final byte[] buf = new byte[1024];
while (!decompressor.finished()) {
try {
final int count = decompressor.inflate(buf);
bos.write(buf, 0, count);
} catch (final DataFormatException e) {
LOGWRAPPER.error(e.getMessage(), e);
throw new RuntimeException(e);
}
}
// Get the decompressed data.
decompressed = bos.toByteArray();
} catch (final IOException e) {
LOGWRAPPER.error(e.getMessage(), e);
}
return decompressed;
}
Yet, compared to non-compressed values it's orders of magnitudes slower even if I'm caching the decompressed-result and the values are only decompressed if the content is really needed.
That is, it's used for a DOM-like persistable tree-structure and XPath-queries which force the decompression of the String-values are about 50 times if not even more slower (not really benchmarked, just executed unit tests). My laptop even freezes after some unit tests (everytime, checked it about 5-times), because Eclipse isn't responding anymore due to heavy disk I/O and what not. I've even set the compression level to Deflater.BEST_SPEED, whereas other compression levels might be better, maybe I'm providing a configuration option parameter which can be set for resources. Maybe I've messed something up as I haven't used the deflater before. I'm even only compressing content where the String lenght is > 10.
Edit: After considering to extract the Deflater instantiation to a static field it seems creating an instance of deflater and inflater is very costly as the performance bottleneck is gone and perhaps without microbenchmarks or the like I can't see any performance loss :-) I'm just resetting the deflater/inflater before using a new input.
How you considered using the higher level api like Gzip.
Here is an example for compressing:
public static byte[] compressToByte(final String data, final String encoding)
throws IOException
{
if (data == null || data.length == 0)
{
return null;
}
else
{
byte[] bytes = data.getBytes(encoding);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream os = new GZIPOutputStream(baos);
os.write(bytes, 0, bytes.length);
os.close();
byte[] result = baos.toByteArray();
return result;
}
}
Here is an example for uncompressing:
public static String unCompressString(final byte[] data, final String encoding)
throws IOException
{
if (data == null || data.length == 0)
{
return null;
}
else
{
ByteArrayInputStream bais = new ByteArrayInputStream(data);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
GZIPInputStream is = new GZIPInputStream(bais);
byte[] tmp = new byte[256];
while (true)
{
int r = is.read(tmp);
if (r < 0)
{
break;
}
buffer.write(tmp, 0, r);
}
is.close();
byte[] content = buffer.toByteArray();
return new String(content, 0, content.length, encoding);
}
}
We get very good performance and compression ratio with this.
The zip api is also an option.
Your comments are the correct answer.
In general, if a method is going to be used frequently, you want to eliminate any allocations and copying of data. This often means removing instance initialization and other setup to either static variables or to the constructor.
Using statics is easier, but you may run into lifetime issues (as in how do you know when to clean up the statics - do they exist forever?).
Doing the setup and initialization in the constructor allows the user of the class to determine the lifetime of the object and clean up appropriately. You could instantiate it once before going into a processing loop and GC it after exiting.