I have a legacy system that uses hibernate interceptor to encrypt (and encode) and decrypt (and decode) some fields on some database tables. It makes use of the OnSave, OnLoad and OnFlushDirty methods. This code turns out to be buggy as data read from this system, when transferred to another application still has some of the records encrypted and encoded (some encrypted multiple times). The challenge for me here is that I could perform the decryption and decoding (as many times as necessary) when the receiving application is on a Windows machine. I get a BadPaddingException when I try to repeat the same thing when the receiving application is a linux VM.
Any help/suggestions will be greatly appreciated
here is a snippet of the hibernate interceptor
public boolean onLoad(Object entity, Serializable arg1, Object[] state, String[] propertyNames, Type[] arg4) throws CallbackException {
if (key != null){
try {
if (entity instanceof BasicData) {
for (int i = 0; i < state.length; i++) {
if (state[i] instanceof String){
String cipherText = (String)state[i];
byte[] cipherTextBytes = Base64Coder.decode(cipherText);
byte[] plainTextBytes = dCipher.doFinal(cipherTextBytes);
state[i] = new String(plainTextBytes, "UTF8");
}
}
return true;
}
} catch (Exception e) {
e.printStackTrace();
}}return false;}
I'd have to guess here but if you mean this Base64Coder the problem might be the following:
It is unclear how the base64 string has been created, i.e. which encoding had been used.
If you use UTF-8 to get the bytes of a string and create a base64 from those bytes you'll get a different result than if you'd use ISO Latin-1, for example.
Afterwards you create a string from those bytes using UTF-8, but if the base64 string had not been created using UTF-8, you'll get wrong results.
Just a quote from the linked source (if this is the correct one):
public static String encodeString (String s) {
return new String(encode(s.getBytes())); }
Here, s.getBytes() will use the system's/jvm's default encoding, so you should really ensure it is UTF-8!
If you control both sides, encode and decode, better way to use DatatypeConverter:
String buffer = DatatypeConverter.printBase64Binary( symKey );
byte[] supposedSymKey = DatatypeConverter.parseBase64Binary( buffer );
Related
I need some consulting here.
I'm writing a client/server software.
One requirement is to use an XML file to transfer information between the client and the server. This decision was made to allow clients to be created on any language. I know I could use json for simpler communication, but I needed a better Object to Text Mapping, so I ended up with XML. So, to diminish the XMl size problem, I decided to compress it before sending over socket. On top of it, I want to encrypt the data so it will be harder to be broken by some interceptor. I have read a lot about encrypting and encryption types, but for testing purposes I'll just use Base64 and generic compression just to join up all the technologies, than I'll specialize the specifics. I found a lot of code and good advice here on stack exchange, and now I'm having some trouble.
What I have:
In memory Objects;
Static methods to convert Objects to XML in memory;
No use for Serialization because the clients can be written on C, .net, perl, etc;
Avoid disk write to reduce disk I/O, keeping everything in-memory;
The need to work with punctuation chars (UTF-8);
The need to work with stronger encrypt on the future;
The process I idealized is:
Convert object to XML - OK
Establish Client/server communication - OK
Encode XML String to Base64 - OK (no compression)
3.1. Compress XML String and than encode to Base64 - OK
Transfer between client/server - OK
Decode Base64 String to XML String - OK
5.1. Uncompress transferred byte array and decode Base64 - NOT OK
So, I can't find where I'm getting wrong on the "Compress String -> Encode it to Base64 -> transmit -> decode Base64 to String -> uncompress it" process... Here is the snippet from my code "inspired" on stack overflow:
I'm using this answer (with .net compatibility) as a reference to compress/uncompress:
How can I Zip and Unzip a string using GZIPOutputStream that is compatible with .Net?
public class Compressor2 {
public static byte[] compress(String string) throws IOException {
byte[] blockcopy = ByteBuffer
.allocate(4)
.order(java.nio.ByteOrder.LITTLE_ENDIAN)
.putInt(string.length())
.array();
ByteArrayOutputStream os = new ByteArrayOutputStream(string.length());
GZIPOutputStream gos = new GZIPOutputStream(os);
gos.write(string.getBytes());
gos.close();
os.close();
byte[] compressed = new byte[4 + os.toByteArray().length];
System.arraycopy(blockcopy, 0, compressed, 0, 4);
System.arraycopy(os.toByteArray(), 0, compressed, 4, os.toByteArray().length);
compressed = MyBase64.encode(new String(compressed)).getBytes();
return compressed;
}
public static String decompress(byte[] compressed) throws IOException {
compressed = MyBase64.decode(new String(compressed)).getBytes();
System.out.println(compressed);
System.out.println(new String(compressed));
final int BUFFER_SIZE = 32;
//ByteArrayInputStream is = new ByteArrayInputStream(compressed, 4, compressed.length - 4);
ByteArrayInputStream is = new ByteArrayInputStream(compressed);
GZIPInputStream gis = new GZIPInputStream(is, BUFFER_SIZE);
StringBuilder string = new StringBuilder();
byte[] data = new byte[BUFFER_SIZE];
int bytesRead;
while ((bytesRead = gis.read(data)) != -1) {
string.append(new String(data, 0, bytesRead));
}
gis.close();
is.close();
return string.toString();
}
from MyBase 64:
public static String encode(String text) throws UnsupportedEncodingException {
byte[] encodedBytes = Base64.encodeBase64(text.getBytes());
return new String(encodedBytes, "UTF-8");
}
public static String decode(String text) throws UnsupportedEncodingException {
byte[] decodedBytes = Base64.decodeBase64(text);
return new String(decodedBytes, "UTF-8");
}
and the test case:
/**
* Test of compress method, of class Compressor2.
*/
#Test
public void testCompress() throws Exception {
System.out.println("compress");
String string = "Hello all, how can I transfer this with punctuation? like á é í ó ú";
byte[] expResult = Compressor2.compress(string);
byte[] result = Compressor2.compress(string);
System.out.write(result);
System.out.print("\n");
System.out.println(new String(result));
assertArrayEquals(expResult, result);
for (byte character : result) {
System.out.print(String.valueOf(character));
}
System.out.print("\n");
for (byte character : result) {
System.out.print((character));
}
System.out.print("\n");
}
/**
* Test of decompress method, of class Compressor2.
*/
#Test
public void testDecompress() throws Exception {
System.out.println("decompress");
String expResult = "Hello all, how can I transfer this with punctuation? like á é í ó ú";
//String expResult = "The lazy dog ate an apple";
byte[] compressed = Compressor2.compress(expResult);
System.out.println(expResult);
System.out.write(compressed);
System.out.print("\n");
System.out.println("going to decompress");
String result = Compressor2.decompress(compressed);
System.out.println(result);
assertEquals(expResult, result);
// TODO punctuation errors?
}
System Out for checking the values:
compress
QwAAAB/vv70IAAAAAAAAAAXvv73vv70N77+9MAwE77+9VX4Adu+/ve+/vTHvv73vv70oFu+/vRMl77+9Mg8lBRUd77+9F++/vW5j77+9AlJd77+977+9RCLvv70Ob2Tvv73vv70G77+977+9Me+/vTPvv73vv71c77+977+9UDkZcSFuxIN4Ee+/vQ/nn5ENSAAAAA==
QwAAAB/vv70IAAAAAAAAAAXvv73vv70N77+9MAwE77+9VX4Adu+/ve+/vTHvv73vv70oFu+/vRMl77+9Mg8lBRUd77+9F++/vW5j77+9AlJd77+977+9RCLvv70Ob2Tvv73vv70G77+977+9Me+/vTPvv73vv71c77+977+9UDkZcSFuxIN4Ee+/vQ/nn5ENSAAAAA==
811196565656647118118554873656565656565656565658811811855511181185548785555435777651196955554357868852651001174347118101434711884721181185551118118554811170117434711882771085555435777103561086682851005555435770434347118875310655554357651087410055554357555543578267761181185548799850841181185551118118554871555543575555435777101434711884801181185551118118554999555543575555435785681079099837011712073785269101434711881471101105369788365656565656161
811196565656647118118554873656565656565656565658811811855511181185548785555435777651196955554357868852651001174347118101434711884721181185551118118554811170117434711882771085555435777103561086682851005555435770434347118875310655554357651087410055554357555543578267761181185548799850841181185551118118554871555543575555435777101434711884801181185551118118554999555543575555435785681079099837011712073785269101434711881471101105369788365656565656161
decompress
Hello all, how can I transfer this with punctuation? like á é í ó ú
QwAAAB/vv70IAAAAAAAAAAXvv73vv70N77+9MAwE77+9VX4Adu+/ve+/vTHvv73vv70oFu+/vRMl77+9Mg8lBRUd77+9F++/vW5j77+9AlJd77+977+9RCLvv70Ob2Tvv73vv70G77+977+9Me+/vTPvv73vv71c77+977+9UDkZcSFuxIN4Ee+/vQ/nn5ENSAAAAA==
going to decompress
[B#19f9bdc4
C
Sorry for the long post, I'm trying to give you something to work on to check where I'm getting this wrong. I Appreciate any help...
Edit.: Forgot to say, if you take the "MyBase64.decode or encode" line from the Compressor2 Classe, it works like a charm...
I'm trying to decompress a json object in Java that was initially compressed in PHP. Here's how it gets compressed into PHP:
function zip_json_encode(&$arr) {
$uncompressed = json_encode($arr);
return pack('L', strlen($uncompressed)).gzcompress($uncompressed);
}
and decoded (again in PHP):
function unzip_json_decode(&$data) {
$uncompressed = #gzuncompress(substr($data,4));
return json_decode($uncompressed, $array_instead_of_object);
}
That gets put into MySQL and now it must be pulled out of the db by Java. We pull it out from the ResultSet like this:
String field = rs.getString("field");
I then pass that string to a method to decompress it. This is where it falls apart.
private String decompressHistory(String historyString) throws SQLException {
StringBuffer buffer = new StringBuffer();
try {
byte[] historyBytes = historyString.substring(4).getBytes();
ByteArrayInputStream bin = new ByteArrayInputStream(historyBytes);
InflaterInputStream in = new InflaterInputStream(bin, new Inflater(true));
int len;
byte[] buf = new byte[1024];
while ((len = in.read(buf)) != -1) {
// buf should be decoded, right?
}
} catch (IOException e) {
e.getStackTrace();
}
return buffer.toString();
}
Not quite sure what's going wrong here, but any pointers would be appreciated!
You need to get rid of the true in Inflater(true). Use just Inflater(). The true makes it expect raw deflate data. Without the true, it is expecting zlib-wrapped deflate data. PHP's gzcompress() produces zlib-wrapped deflate data.
Gzipped data is binary, byte[]. Using String, Unicode text, not only needs conversion, but is faulty.
For instance this involves a conversion:
byte[] historyBytes = historyString.substring(4).getBytes();
byte[] historyBytes = historyString.substring(4).getBytes("ISO-8859-1");
The first version uses the default platform encoding, making the application non-portable.
The first to-do is to use binary data in the database as VARBINARY or BLOB.
ImputStream field = rs.getBinaryStream("field");
try (InputStream in = new GZIPInputStream(field)) {
...
}
Or so. Mind the other answer.
In the end, neither of the above solutions worked, but both have merits. When we pulled the data out of mysql and cast it to bytes we have a number of missing character bytes (67). This made it impossible to decompress on the java side. As for the answers above. Mark is correct that gzcompress() uses zlib and therefore you should use the Inflater() class in Java.
Joop is correct that the data conversion is faulty. Our table was too large to convert it to varbinary or blob. That may have solved the problem, but didn't work for us. We ended up having java make a request to our PHP app, then simply unpacked the compressed data on the PHP side. This worked well. Hopefully this is helpful to anyone else that stumbles across it.
I am writing an App which does the following with a given String str:
encode(encrypt(encode(stringToBytearray(str))));
The Server receives the Base64 encoded String, which is then decoded -> decrypted -> decoded, to get the sent String str from the App.
Unfortunately it doesnt work for all Strings, long Strings lead to a long Base64 String and my Server throws the following Exception:
Exception in thread "main" java.lang.IllegalArgumentException: Illegal base64 character 5b
at java.util.Base64$Decoder.decode0(Base64.java:714)
at java.util.Base64$Decoder.decode(Base64.java:526)
at Main.decode(Main.java:113)
at Main.main(ain.java:33)
The String has the format "[string, string, ..., string]" without "s.
Like I mentioned above, for Strings which are not too long (sorry I cant quantify the length yet), this works. So I think I implemented it right.
Whats weird is, that if I dont send it, but decode(decrypt(decode(stringToBytearray(str)))); the String on the Device itself, it all works perfectly.
My Setup:
JDK 7, eclipse (adt-bundle, android development) (Windows 7)
JDK 8, eclipse ('normal' java) (Linux (xubuntu))
Is it because both Classes (Base64) are implemented diffrently? If so, how can I make it work?
Here are the Encoding/Decoding Methods:
Encoding (Device: on Windows 7, adt-bundle, Android-Dev, jdk 7):
import android.util.Base64
public byte[] encode(byte[] bytearrayToEncode){
return Base64.encode(bytearrayToEncode, Base64.NO_WRAP|Base64.URL_SAFE);
}
Decoding (Server: on Linux, jdk 8):
import java.util.Base64
public byte[] decode(byte[] bytearrayToEncode){
return Base64.getUrlDecoder().decode(bytearrayToDecode);
}
Strings are all De-/Encoded with the same charset (utf-8)!
Encoding/Decoding: Base64
Crypto: AES
If you need more Information, just ask, but I think I provided all neccessary informations.
Edit:
public String bytearrayToString(byte[] bytearray){
String str = null;
try {
str = new String(bytearray, "UTF-8");
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return str;
}
public byte[] stringToBytearray(String str){
byte[] bytearray = null;
try {
bytearray = str.getBytes("UTF-8");
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return bytearray;
}
You must be using JSONObjects to send the Base64 String to the server. If so then the JSONObject will add " \ " escape characters to the string which are illegal Base64 characters. All you need to do is
String rectifiedString = Base64String.replace("\\","");
on the server end before using the acquired string.
Optional
Also on the server side use
import javax.xml.bind.DatatypeConverter;
and
String result = DatatypeConverter.printBase64Binary(bytearrayToDecode);
byte[] result = DatatypeConverter.parseBase64Binary(str);
instead.
It will not only get the job done but is much faster at encoding/decoding (as benchmarked here).
If you have a space in your encoded string, which is an illegal character in base64. If you remove it, it still generates the same image/data in the converter you linked to, and can be now decoded by your code as well.
UPDATE:
Some decoders (like the one you linked, or Base64.getMimeDecoder() in Java) ignore illegal characters, others (like Base64.getDecoder()) don't allow them.
Use for encode:
Base64.getEncoder().encodeToString(yourString.getBytes("UTF-8"));
and decode:
byte[] decodedBytes = Base64.getDecoder().decode(yourString);
String stringDecode = new String(decodedBytes, "UTF-8");
according to the specification: http://wiki.theory.org/BitTorrentSpecification
info_hash: urlencoded 20-byte SHA1 hash of the value of the info key from the Metainfo file. Note that the value will be a bencoded dictionary, given the definition of the info key above.
torrentMap is my dictionary, I get the info key which is another dictionary, I calculate the hash and I URLencode it.
But I always get an invalid info_hash message when I try to send it to the tracker.
This is my code:
public String GetInfo_hash() {
String info_hash = "";
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = null;
try {
out = new ObjectOutputStream(bos);
out.writeObject(torrentMap.get("info"));
byte[] bytes = bos.toByteArray(); //Map => byte[]
MessageDigest md = MessageDigest.getInstance("SHA1");
info_hash = urlencode(md.digest(bytes)); //Hashing and URLEncoding
out.close();
bos.close();
} catch (Exception ex) { }
return info_hash;
}
private String urlencode(byte[] bs) {
StringBuffer sb = new StringBuffer(bs.length * 3);
for (int i = 0; i < bs.length; i++) {
int c = bs[i] & 0xFF;
sb.append('%');
if (c < 16) {
sb.append('0');
}
sb.append(Integer.toHexString(c));
}
return sb.toString();
}
This is almost certainly the problem:
out = new ObjectOutputStream(bos);
out.writeObject(torrentMap.get("info"));
What you're going to be hashing is the Java binary serialization format of the value of torrentMap.get("info"). I find it very hard to believe that all BitTorrent programs are meant to know about that.
It's not immediately clear to me from the specification what the value of the "info" key is meant to be, but you need to work out some other way of turning it into a byte array. If it's a string, I'd expect some well-specified encoding (e.g. UTF-8). If it's already binary data, then use that byte array directly.
EDIT: Actually, it's sounds like the value will be a "bencoded dictionary" as per your quote, which looks like it will be a string. Quite how you're meant to encode that string (which sounds like it may include values which aren't in ASCII, for example) before hashing it is up for grabs. If your sample strings are all ASCII, then using "ASCII" and "UTF-8" as the encoding names for String.getBytes(...) will give the same result anyway, of course...
What I am trying to do is read from a text file where each line has the path to a file and then space for a separator and a hash that accompanies it. So I call checkVersion() and loadStrings(File f_) returns a String[], one place for each line. When I try to check the hashes however I end up with something that isn't even hex and is twice as long as it should be, it's probably something obvious that my eyes are just overlooking. The idea behind this is an auto-update for my game to save bandwidth, thanks for your time.
The code is fixed, here is the final version if anyone else has this issue, thanks a lot everyone.
void checkVersion() {
String[] v = loadStrings("version.txt");
for(int i=0; i<v.length; i++) {
String[] piece = split(v[i], " "); //BREAKS INTO FILENAME, HASH
println("Checking "+piece[0]+"..."+piece[1]);
if(checkHash(piece[0], piece[1])) {
println("ok!");
} else {
println("NOT OKAY!");
//CONTINUE TO DOWNLOAD FILE AND THEN CALL CHECKVERSION AGAIN
}
}
}
boolean checkHash(String path_, String hash_) {
return createHash(path_).equals(hash_);
}
byte[] messageDigest(String message, String algorithm) {
try {
java.security.MessageDigest md = java.security.MessageDigest.getInstance(algorithm);
md.update(message.getBytes());
return md.digest();
} catch(java.security.NoSuchAlgorithmException e) {
println(e.getMessage());
return null;
}
}
String createHash(String path_) {
byte[] md5hash = messageDigest(new String(loadBytes(path_)),"MD5");
BigInteger bigInt = new BigInteger(1, md5hash);
return bigInt.toString(16);
}
The String.getBytes() method returns the bytes that represent the character encodings for the string. It doesn't parse it into bytes that represent a number in some arbitrary radix. For example "AA".getBytes() would yield you 0x41 0x41 on windows, not 10101010b, which is what it appears you were expecting? To get that you could, for example Byte.parseByte("AA", 16)
The library you're using to create hashes probably has a method for taking back in its own string representation. How to convert back depends on the representation, which you didn't give us.
use following code to convert hash bytes to string
//byte[] md5sum = digest.digest();
BigInteger bigInt = new BigInteger(1, md5sum);
String output = bigInt.toString(16);
System.out.println("MD5: " + output);
for more information