Develop Reference

How do I create a dictionary object for passing to Zstd.compress?

I am using Zstd compression in Java for compressing a large JSON payload. I am using methods from the zstd-jni library for Java. I create a byte array out of the JSON string and use this method.
public static byte[] compress(byte[] var0, int var1)
I read that ZSTD will give more optimal results when a dictionary is passed during compression and decompression. How do I create a ZstdDictCompress object? What byte array and integer should I pass to the constructor?
public static long compress(byte[] var0, byte[] var1, ZstdDictCompress var2)

This example is for https://github.com/luben/zstd-jni.
First of all you need to get many samples of your jsons. You shouldn't use just one or couple samples. After that you can train your dictionary:
List<String> jsons = ...; // List of your jsons samples
ZstdDictTrainer trainer = new ZstdDictTrainer(1024 * 1024, 16 * 1024); // 16 KB dictionary
for(String json : jsons) {
trainer.addSample(json.getBytes(StandardCharsets.UTF_8));
}
byte[] dictionary = trainer.trainSamples();
Now you have you dictionary in byte array.
Next step is using SAME dictionary to compress and decompress.
// Compress
byte[] json = jsonString.getBytes(StandardCharsets.UTF_8);
ZstdDictCompress zstdDictCompress = new ZstdDictCompress(dictionary, Zstd.defaultCompressionLevel());
byte[] compressed = Zstd.compress(json, zstdDictCompress);
// Tricky moment, you have to pass json full length to decompress method
int jsonFullLength = json.length;
// Decompress
ZstdDictDecompress zstdDictDecompress = new ZstdDictDecompress(dictionary);
byte[] decompressed = Zstd.decompress(compressed, zstdDictDecompress, jsonFullLength);
String jsonString2 = new String(decompressed, StandardCharsets.UTF_8);
That's all!

How to convert binary payload (file) to byte[] in java?

I receive json data that contains binary data like that ,and I would like to convert that data to byte[] in java but I don't know how ?.
"payload": "7V1bcxs3ln6frfdcfvfbghfdX8HSw9Zu1QzzartyhblfdcvberCObjvJpkiJUpmhRI1pKXYeXHRsZLSrCy
5dElN5tfvQaO72TdSoiOS3TH8Yxdffgtg754679513qdfrgvlslsqdeqaepdccngrdzedrtghBD+d++e7v//p80/v96v7h+u72
+z1gfK/39x/+9t391cPTzeP88aE/++Fvvd53n+8+Xd1c/fBm/unqAf+7
N7v65en++vGP3vx2fvPHw/XDdwfpHf5mevhq/vQDcnAAwD+gEPwDF+bDxTv+3UF61d/4eesrfP356uFx"

Based on the observation that the "binary" string consists of ASCII letters, digits and "+" and "/", I am fairly confident that it is actually Base64 encoded data.
To decode Base64 to a byte[] you can do something like this:
String s = "7V1bcxs3ln6...";
byte [] bytes = java.util.Base64.getDecoder().decode(s);
The decode call will throw IllegalArgumentException if the input string is not properly Base64 encoded.
When I decoded that particular string using an online Base64 decoder, the result is unintelligible. But that is what I would expect for an arbitrary "blob" of binary data.

In general if you have a String in some object that denotes the json payload you can :
String s = "7V1bcxs3ln6...";
byte [] bytes = s.getBytes();
Other than that if this payload should be decoded somehow then additional code will be required.

In my case I had to convert payload that I knew it was a text something like:
{"payload":"eyJ1c2VyX2lkIjo0LCJ1c2VybmFtZSI6IngiLCJjaXR5IjoiaGVyZSJ9"}
This is the difference between java.util.Base64.getDecoder() and getBytes():
String s = "eyJ1c2VyX2lkIjo0LCJ1c2VybmFtZSI6IngiLCJjaXR5IjoiaGVyZSJ9";
byte [] bytes = s.getBytes();
byte [] bytes_base64 = java.util.Base64.getDecoder().decode(s);
String bytesToStr = new String(bytes, StandardCharsets.UTF_8);
String bytesBase64Tostr = new String(bytes_base64, StandardCharsets.UTF_8);
System.out.println("bytesToStr="+bytesToStr);
System.out.println("bytesBase64Tostr="+bytesBase64Tostr);
Output:
bytesToStr=eyJ1c2VyX2lkIjo0LCJ1c2VybmFtZSI6IngiLCJjaXR5IjoiaGVyZSJ9
bytesBase64Tostr={"user_id":4,"username":"x","city":"here"}
java.util.Base64.getDecoder() worked for in my case

Serializing/Deserializing a standalone integer using protocol buffers

Up to now, I've been using Protocol Buffers to serialize and deserialize objects using the code-generated classes.
Now I am attempting to serialize and deserialize a single 64-bit integer. The problem is, I'm getting different results in Java and C#.
Here's how I'm doing it in Java....
private static byte[] convertLongToByteArray(long value) throws IOException {
int size = CodedOutputStream.computeInt64SizeNoTag(value);
byte[] buffer = new byte[size];
CodedOutputStream codedOutputStream = CodedOutputStream.newInstance(buffer);
codedOutputStream.writeInt64NoTag(value);
codedOutputStream.flush();
codedOutputStream.checkNoSpaceLeft();
return buffer;
}
And here's how I'm doing this in C#, using protobuf.net:
public void SerializeLongValue()
{
long n = 9876;
byte[] memoryBuffer = null;
using (MemoryStream destination = new MemoryStream())
{
ProtoBuf.Serializer.Serialize(destination, n);
destination.Flush();
memoryBuffer = destination.ToArray();
}
using (MemoryStream source = new MemoryStream(memoryBuffer))
{
long result = ProtoBuf.Serializer.Deserialize<long>(source);
Assert.AreEqual(n, result);
}
}
The java code converted the number 9876 to [0x94, 0x4D]
The C# code converted the number 9876 to [0x08, 0x94, 0x4D]
How do I do this so that both com.google.protobuf and protobuf.net produce identical outputs?

The protobuf.net method ProtoBuf.Serializer.Serialize forces a field header (field number=1) into the stream. That's the only way you can perform a serialization; this method invokes a number of internal methods which are not publicly available.
The solution I'm using is to change the Java code to also include a field header.
Here is my new Java code.
private static byte[] convertLongToByteArray(long value) throws IOException {
int size = CodedOutputStream.computeTagSize(1) + CodedOutputStream.computeInt64SizeNoTag(value);
byte[] buffer = new byte[size];
CodedOutputStream codedOutputStream = CodedOutputStream.newInstance(buffer);
codedOutputStream.writeInt64(1, value);
codedOutputStream.flush();
codedOutputStream.checkNoSpaceLeft();
return buffer;
}
public static long convertByteArrayToLong(byte[] byteArray) throws IOException {
CodedInputStream codedInputStream = CodedInputStream.newInstance(byteArray);
codedInputStream.readTag();
return codedInputStream.readInt64();
}
The changes I've made are:
When computing the required buffer size, include a tag size
Instead of CodedOutputStream.WriteInt64NoTag, call CodedOutputStream.WriteInt64
When reading back, call CodedOutputStream.ReadTag before calling CodedOutputStream.ReadInt64

How to use ByteBuffer properly?

I am trying to use ByteBuffer properly with BigEndian byte order format..
I have couple of fields which I am trying to put together into a single ByteBuffer before storing it in Cassandra database.
That Byte Array which I will be writing into Cassandra is made up of three Byte Arrays as described below-
short employeeId = 32767;
long lastModifiedDate = "1379811105109L";
byte[] attributeValue = os.toByteArray();
Now, I will write employeeId , lastModifiedDate and attributeValue together into a single Byte Array and that resulting Byte Array I will write into Cassandra and then I will be having my C++ program which will retrieve that Byte Array data from Cassandra and then deserialize it to extract employeeId , lastModifiedDate and attributeValue from it.
So to do this, I am using ByteBuffer with BigEndian byte order format.
I have put up this code together -
public static void main(String[] args) throws Exception {
String text = "Byte Buffer Test";
byte[] attributeValue = text.getBytes();
long lastModifiedDate = 1289811105109L;
short employeeId = 32767;
int size = 2 + 8 + 4 + attributeValue.length; // short is 2 bytes, long 8 and int 4
ByteBuffer bbuf = ByteBuffer.allocate(size);
bbuf.order(ByteOrder.BIG_ENDIAN);
bbuf.putShort(employeeId);
bbuf.putLong(lastModifiedDate);
bbuf.putInt(attributeValue.length);
bbuf.put(attributeValue);
bbuf.rewind();
// best approach is copy the internal buffer
byte[] bytesToStore = new byte[size];
bbuf.get(bytesToStore);
// write bytesToStore in Cassandra...
// Now retrieve the Byte Array data from Cassandra and deserialize it...
byte[] allWrittenBytesTest = bytesToStore;//magicFunctionToRetrieveDataFromCassandra();
ByteBuffer bb = ByteBuffer.wrap(allWrittenBytesTest);
bb.order(ByteOrder.BIG_ENDIAN);
bb.rewind();
short extractEmployeeId = bb.getShort();
long extractLastModifiedDate = bb.getLong();
int extractAttributeValueLength = bb.getInt();
byte[] extractAttributeValue = new byte[extractAttributeValueLength];
bb.get(extractAttributeValue); // read attributeValue from the remaining buffer
System.out.println(extractEmployeeId);
System.out.println(extractLastModifiedDate);
System.out.println(new String(extractAttributeValue));
}
Is there any better way of doing this, the way I am doing it currently? Or some minor improvements that we can do it here??
This is the first time I am using ByteBuffer so having little bit problem...
Can anyone take a look and let me know whether this is the right way to use ByteBuffer?

The default order is always BIG_ENDIAN, so you don't meed to set it. Also when you wrap() is is already rewind()ed.
Instead of copying the underlying array, I would use the underlying array.
Replace
bbuf.rewind();
// best approach is copy the internal buffer
byte[] bytesToStore = new byte[size];
bbuf.get(bytesToStore);
with
byte[] bytesToStore = bbuf.array();

Convert a String to a byte array and then back to the original String

Is it possible to convert a string to a byte array and then convert it back to the original string in Java or Android?
My objective is to send some strings to a microcontroller (Arduino) and store it into EEPROM (which is the only 1  KB). I tried to use an MD5 hash, but it seems it's only one-way encryption. What can I do to deal with this issue?

I would suggest using the members of string, but with an explicit encoding:
byte[] bytes = text.getBytes("UTF-8");
String text = new String(bytes, "UTF-8");
By using an explicit encoding (and one which supports all of Unicode) you avoid the problems of just calling text.getBytes() etc:
You're explicitly using a specific encoding, so you know which encoding to use later, rather than relying on the platform default.
You know it will support all of Unicode (as opposed to, say, ISO-Latin-1).
EDIT: Even though UTF-8 is the default encoding on Android, I'd definitely be explicit about this. For example, this question only says "in Java or Android" - so it's entirely possible that the code will end up being used on other platforms.
Basically given that the normal Java platform can have different default encodings, I think it's best to be absolutely explicit. I've seen way too many people using the default encoding and losing data to take that risk.
EDIT: In my haste I forgot to mention that you don't have to use the encoding's name - you can use a Charset instead. Using Guava I'd really use:
byte[] bytes = text.getBytes(Charsets.UTF_8);
String text = new String(bytes, Charsets.UTF_8);

You can do it like this.
String to byte array
String stringToConvert = "This String is 76 characters long and will be converted to an array of bytes";
byte[] theByteArray = stringToConvert.getBytes();
http://www.javadb.com/convert-string-to-byte-array
Byte array to String
byte[] byteArray = new byte[] {87, 79, 87, 46, 46, 46};
String value = new String(byteArray);
http://www.javadb.com/convert-byte-array-to-string

Use [String.getBytes()][1] to convert to bytes and use [String(byte[] data)][2] constructor to convert back to string.

byte[] pdfBytes = Base64.decode(myPdfBase64String, Base64.DEFAULT)

import java.io.FileInputStream;
import java.io.ByteArrayOutputStream;
public class FileHashStream
{
// write a new method that will provide a new Byte array, and where this generally reads from an input stream
public static byte[] read(InputStream is) throws Exception
{
String path = /* type in the absolute path for the 'commons-codec-1.10-bin.zip' */;
// must need a Byte buffer
byte[] buf = new byte[1024 * 16]
// we will use 16 kilobytes
int len = 0;
// we need a new input stream
FileInputStream is = new FileInputStream(path);
// use the buffer to update our "MessageDigest" instance
while(true)
{
len = is.read(buf);
if(len < 0) break;
md.update(buf, 0, len);
}
// close the input stream
is.close();
// call the "digest" method for obtaining the final hash-result
byte[] ret = md.digest();
System.out.println("Length of Hash: " + ret.length);
for(byte b : ret)
{
System.out.println(b + ", ");
}
String compare = "49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d";
String verification = Hex.encodeHexString(ret);
System.out.println();
System.out.println("===")
System.out.println(verification);
System.out.println("Equals? " + verification.equals(compare));
}
}