I need some consulting here.
I'm writing a client/server software.
One requirement is to use an XML file to transfer information between the client and the server. This decision was made to allow clients to be created on any language. I know I could use json for simpler communication, but I needed a better Object to Text Mapping, so I ended up with XML. So, to diminish the XMl size problem, I decided to compress it before sending over socket. On top of it, I want to encrypt the data so it will be harder to be broken by some interceptor. I have read a lot about encrypting and encryption types, but for testing purposes I'll just use Base64 and generic compression just to join up all the technologies, than I'll specialize the specifics. I found a lot of code and good advice here on stack exchange, and now I'm having some trouble.
What I have:
In memory Objects;
Static methods to convert Objects to XML in memory;
No use for Serialization because the clients can be written on C, .net, perl, etc;
Avoid disk write to reduce disk I/O, keeping everything in-memory;
The need to work with punctuation chars (UTF-8);
The need to work with stronger encrypt on the future;
The process I idealized is:
Convert object to XML - OK
Establish Client/server communication - OK
Encode XML String to Base64 - OK (no compression)
3.1. Compress XML String and than encode to Base64 - OK
Transfer between client/server - OK
Decode Base64 String to XML String - OK
5.1. Uncompress transferred byte array and decode Base64 - NOT OK
So, I can't find where I'm getting wrong on the "Compress String -> Encode it to Base64 -> transmit -> decode Base64 to String -> uncompress it" process... Here is the snippet from my code "inspired" on stack overflow:
I'm using this answer (with .net compatibility) as a reference to compress/uncompress:
How can I Zip and Unzip a string using GZIPOutputStream that is compatible with .Net?
public class Compressor2 {
public static byte[] compress(String string) throws IOException {
byte[] blockcopy = ByteBuffer
.allocate(4)
.order(java.nio.ByteOrder.LITTLE_ENDIAN)
.putInt(string.length())
.array();
ByteArrayOutputStream os = new ByteArrayOutputStream(string.length());
GZIPOutputStream gos = new GZIPOutputStream(os);
gos.write(string.getBytes());
gos.close();
os.close();
byte[] compressed = new byte[4 + os.toByteArray().length];
System.arraycopy(blockcopy, 0, compressed, 0, 4);
System.arraycopy(os.toByteArray(), 0, compressed, 4, os.toByteArray().length);
compressed = MyBase64.encode(new String(compressed)).getBytes();
return compressed;
}
public static String decompress(byte[] compressed) throws IOException {
compressed = MyBase64.decode(new String(compressed)).getBytes();
System.out.println(compressed);
System.out.println(new String(compressed));
final int BUFFER_SIZE = 32;
//ByteArrayInputStream is = new ByteArrayInputStream(compressed, 4, compressed.length - 4);
ByteArrayInputStream is = new ByteArrayInputStream(compressed);
GZIPInputStream gis = new GZIPInputStream(is, BUFFER_SIZE);
StringBuilder string = new StringBuilder();
byte[] data = new byte[BUFFER_SIZE];
int bytesRead;
while ((bytesRead = gis.read(data)) != -1) {
string.append(new String(data, 0, bytesRead));
}
gis.close();
is.close();
return string.toString();
}
from MyBase 64:
public static String encode(String text) throws UnsupportedEncodingException {
byte[] encodedBytes = Base64.encodeBase64(text.getBytes());
return new String(encodedBytes, "UTF-8");
}
public static String decode(String text) throws UnsupportedEncodingException {
byte[] decodedBytes = Base64.decodeBase64(text);
return new String(decodedBytes, "UTF-8");
}
and the test case:
/**
* Test of compress method, of class Compressor2.
*/
#Test
public void testCompress() throws Exception {
System.out.println("compress");
String string = "Hello all, how can I transfer this with punctuation? like á é í ó ú";
byte[] expResult = Compressor2.compress(string);
byte[] result = Compressor2.compress(string);
System.out.write(result);
System.out.print("\n");
System.out.println(new String(result));
assertArrayEquals(expResult, result);
for (byte character : result) {
System.out.print(String.valueOf(character));
}
System.out.print("\n");
for (byte character : result) {
System.out.print((character));
}
System.out.print("\n");
}
/**
* Test of decompress method, of class Compressor2.
*/
#Test
public void testDecompress() throws Exception {
System.out.println("decompress");
String expResult = "Hello all, how can I transfer this with punctuation? like á é í ó ú";
//String expResult = "The lazy dog ate an apple";
byte[] compressed = Compressor2.compress(expResult);
System.out.println(expResult);
System.out.write(compressed);
System.out.print("\n");
System.out.println("going to decompress");
String result = Compressor2.decompress(compressed);
System.out.println(result);
assertEquals(expResult, result);
// TODO punctuation errors?
}
System Out for checking the values:
compress
QwAAAB/vv70IAAAAAAAAAAXvv73vv70N77+9MAwE77+9VX4Adu+/ve+/vTHvv73vv70oFu+/vRMl77+9Mg8lBRUd77+9F++/vW5j77+9AlJd77+977+9RCLvv70Ob2Tvv73vv70G77+977+9Me+/vTPvv73vv71c77+977+9UDkZcSFuxIN4Ee+/vQ/nn5ENSAAAAA==
QwAAAB/vv70IAAAAAAAAAAXvv73vv70N77+9MAwE77+9VX4Adu+/ve+/vTHvv73vv70oFu+/vRMl77+9Mg8lBRUd77+9F++/vW5j77+9AlJd77+977+9RCLvv70Ob2Tvv73vv70G77+977+9Me+/vTPvv73vv71c77+977+9UDkZcSFuxIN4Ee+/vQ/nn5ENSAAAAA==
811196565656647118118554873656565656565656565658811811855511181185548785555435777651196955554357868852651001174347118101434711884721181185551118118554811170117434711882771085555435777103561086682851005555435770434347118875310655554357651087410055554357555543578267761181185548799850841181185551118118554871555543575555435777101434711884801181185551118118554999555543575555435785681079099837011712073785269101434711881471101105369788365656565656161
811196565656647118118554873656565656565656565658811811855511181185548785555435777651196955554357868852651001174347118101434711884721181185551118118554811170117434711882771085555435777103561086682851005555435770434347118875310655554357651087410055554357555543578267761181185548799850841181185551118118554871555543575555435777101434711884801181185551118118554999555543575555435785681079099837011712073785269101434711881471101105369788365656565656161
decompress
Hello all, how can I transfer this with punctuation? like á é í ó ú
QwAAAB/vv70IAAAAAAAAAAXvv73vv70N77+9MAwE77+9VX4Adu+/ve+/vTHvv73vv70oFu+/vRMl77+9Mg8lBRUd77+9F++/vW5j77+9AlJd77+977+9RCLvv70Ob2Tvv73vv70G77+977+9Me+/vTPvv73vv71c77+977+9UDkZcSFuxIN4Ee+/vQ/nn5ENSAAAAA==
going to decompress
[B#19f9bdc4
C
Sorry for the long post, I'm trying to give you something to work on to check where I'm getting this wrong. I Appreciate any help...
Edit.: Forgot to say, if you take the "MyBase64.decode or encode" line from the Compressor2 Classe, it works like a charm...
Related
I have a byte array file with me which I am trying to convert into human readable. I tried below ways :
public static void main(String args[]) throws IOException
{
//System.out.println("Platform Encoding : " + System.getProperty("file.encoding"));
FileInputStream fis = new FileInputStream("<Path>");
// Using Apache Commons IOUtils to read file into byte array
byte[] filedata = IOUtils.toByteArray(fis);
String str = new String(filedata, "UTF-8");
System.out.println(str);
}
Another approach :
public static void main(String[] args) {
File file = new File("<Path>");
readContentIntoByteArray(file);
}
private static byte[] readContentIntoByteArray(File file) {
FileInputStream fileInputStream = null;
byte[] bFile = new byte[(int) file.length()];
try {
FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
for (int i = 0; i < bFile.length; i++) {
System.out.print((char) bFile[i]);
}
} catch (Exception e) {
e.printStackTrace();
}
return bFile;
}
These codes are compiling but its not yielding output file in a human readable fashion. Excuse me if this is a repeated or basic question.
Could someone please correct me where I am going wrong here?
Your code (from the first snippet) for decoding a byte file into a UTF-8 text file looks correct to me (assuming FileInputStream fis = new FileInputStream("Path") is yielding the correct fileInputStream) .
If you're expecting a text file format but are not sure which encoding the file format is in (perhaps it's not UTF-8) , you can use a library like the below to find out.
https://code.google.com/archive/p/juniversalchardet/
or just explore some of the different Charsets in the Charset library and see what they produce in your String initialization line and what you produce:
new String(byteArray, Charset.defaultCharset()) // try other Charsets here.
The second method you show has associated catches with byte to char conversion , depending on the characters, as discussed here (Byte and char conversion in Java).
Chances are, if you cannot find a valid encoding for this file, it is not human readable to begin with, before byte conversion, or the byte array file being passed to you lost something that makes it decodeable along the way.
I have extracted base64 string from an image, then perform compression on the base64 string.Then generate QR code using this compressed string.
But when I scan(QR CODE) the result using android phone I am getting value like 17740296 which is not value I want. My purpose is that after getting the scanned value I will decompress it and display the image using bitmap from base64. What is the wrong in my code. I am using java code to generate QR CODE.(I have tried UTF-8 also but not working). This code is working for String but not for the image.
Compressing Code is -
public static String compressBD(String str) throws IOException {
if (str == null || str.length() == 0) {
return str;
}
ByteArrayOutputStream out = new ByteArrayOutputStream();
GZIPOutputStream gzip = new GZIPOutputStream(out);
gzip.write(str.getBytes());
gzip.close();
return out.toString("ISO-8859-1");
}
Decompress Code is -
public static String decompressBD(String str) throws Exception {
if (str == null || str.length() == 0) {
return str.toString();
}
// System.out.println("Input String length : " + str.toString().length());
GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(str.getBytes("ISO-8859-1")));
BufferedReader bf = new BufferedReader(new InputStreamReader(gis, "ISO-8859-1"));
String outStr = "";
String line;
while ((line=bf.readLine())!=null) {
outStr += line;
}
//System.out.println("Output String lenght : " + outStr.length());
return outStr;
}
this wont work.
Base64 ist an encoding that can be used entirely in strings. You can print it on a piece of paper, type it into your machine and decode an image fromt it.
However, if you use a gzip compression and transform this to a string, the compression generates bytes that are outside the string encoding and can not be printed or presented as string.
Base64 is meant to be the "compressed" encoding for strings. I would really encourage you not to use string as storage, but directly store the binary data, or transmitt it. This would also be considerably faster, since base64 encoding is very slow.
Its purpose is entirely to store binary content, that contains non printable bytes, in text messages. For what ever reason.
I hope it is understandable, but it basically means, you cant store a base64 zip content in a string. You have to store the binary representation if you want to compress.
So I have a program that is supposed to encode UTF-8 Base64, but is not. My Encoder code is as follows:
public class Encoder {
public static void Encode() throws IOException{
byte[] bytes = Base64.encodeBase64(readFile("C:\\Users\\Dragon\\Desktop\\Binary\\Diamond.png", StandardCharsets.UTF_8).getBytes(StandardCharsets.UTF_8));
String binary = new String(bytes);
PrintWriter out = new PrintWriter("C:\\Users\\Dragon\\Desktop\\Binary\\Base64.txt");
out.println(binary);
out.close();
System.out.println("Your File has been saved at C:\\Users\\Dragon\\Desktop\\Binary\\Base64.txt");
}
static String readFile(String path, Charset encoding)
throws IOException
{
byte[] encoded = Files.readAllBytes(Paths.get(path));
return new String(encoded, encoding);
}
}
And when I decode it with this website (That has UTF-8 Functionality): http://www.base64decode.org/ I get:
Input (My code made this):
77+9UE5HDQoaCgAAAA1JSERSAAAAEAAAABAIBgAAAB/vv73vv71hAAAAGHRFWHRBdXRob3IAbWluZWNyYWZ0aW5mby5jb23vv71mJ0sAAADvv71JREFUOO+/vWPvv73vv73vv70/AyUY77+9QAM877+9Bu+/ve+/vTHvv708XgNAGu+/vQHvv70hOA3vv71pXu+/ve+/vTkcX++/ve+/ve+/ve+/ve+/vSjvv71gNQDvv70Z77+9Ae+/vRkE77+977+9XgRj77+9De+/vWkEGQTvv70gA++/vV3vv71hAO+/ve+/vS/vv71Q77+977+977+9RhRX77+9NQBkIy4DYO+/ve+/vWkASO+/ve+/vdmNWA3vv73vv73vv70GY++/vQbvv70CDO+/vQEZ77+9DAVhEBvvv70B77+9QARpBu+/vQLvv70A77+9MDHvv71hADMA77+9R1wG77+9bCfvv73vv73vv73vv73vv73vv71077+977+9Mjbvv703KVPvv70b77+977+9AB0QYe+/vV4VTO+/vQAAAABJRU5E77+9QmDvv70=
Expected output from website:
‰PNG
IHDR óÿa tEXtAuthor minecraftinfo.comßf'K ²IDAT8Ëcøÿÿ?%Á#<æÿ‘1º<^#к!8
€i^ôí9_üõù¿Áí½(†`5 ¦¤¦¼Þ^c¢
€ià ]a ºæž/P°ÂÔFWà5 d#.`®Ài H³ÎÙX
®Îc¼À¤ƒa«°#iÙÓ Ã01œa 3 äG\Àl'˜Œît¢“26Œ7)SœÉÅ aÍ^L‹ IEND®B`‚
Actual output From website:
�PNG
IHDR��atEXtAuthorminecraftinfo.com�f'K�IDAT8�c���?%�#<���1�<^#��!8
�i^��9_�����(�`5�����^c�
�i� �]�a��/�P���FW�5d#.`��iH��ٍX
���c����a��#i���01�a3�G\�l'������t��26�7)S���a�^L�IEND�B`�
However, when I encode the same exact thing using this site and not my code, It DOES have UTF-8 Formatting and decodes correctly. So what am I doing wrong?
You Base64.encodeBase64 take byte[] as input. You should pass result of Files.readAllBytes(Paths.get(path)) to Base64.encodeBase64 directly.
according to the specification: http://wiki.theory.org/BitTorrentSpecification
info_hash: urlencoded 20-byte SHA1 hash of the value of the info key from the Metainfo file. Note that the value will be a bencoded dictionary, given the definition of the info key above.
torrentMap is my dictionary, I get the info key which is another dictionary, I calculate the hash and I URLencode it.
But I always get an invalid info_hash message when I try to send it to the tracker.
This is my code:
public String GetInfo_hash() {
String info_hash = "";
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = null;
try {
out = new ObjectOutputStream(bos);
out.writeObject(torrentMap.get("info"));
byte[] bytes = bos.toByteArray(); //Map => byte[]
MessageDigest md = MessageDigest.getInstance("SHA1");
info_hash = urlencode(md.digest(bytes)); //Hashing and URLEncoding
out.close();
bos.close();
} catch (Exception ex) { }
return info_hash;
}
private String urlencode(byte[] bs) {
StringBuffer sb = new StringBuffer(bs.length * 3);
for (int i = 0; i < bs.length; i++) {
int c = bs[i] & 0xFF;
sb.append('%');
if (c < 16) {
sb.append('0');
}
sb.append(Integer.toHexString(c));
}
return sb.toString();
}
This is almost certainly the problem:
out = new ObjectOutputStream(bos);
out.writeObject(torrentMap.get("info"));
What you're going to be hashing is the Java binary serialization format of the value of torrentMap.get("info"). I find it very hard to believe that all BitTorrent programs are meant to know about that.
It's not immediately clear to me from the specification what the value of the "info" key is meant to be, but you need to work out some other way of turning it into a byte array. If it's a string, I'd expect some well-specified encoding (e.g. UTF-8). If it's already binary data, then use that byte array directly.
EDIT: Actually, it's sounds like the value will be a "bencoded dictionary" as per your quote, which looks like it will be a string. Quite how you're meant to encode that string (which sounds like it may include values which aren't in ASCII, for example) before hashing it is up for grabs. If your sample strings are all ASCII, then using "ASCII" and "UTF-8" as the encoding names for String.getBytes(...) will give the same result anyway, of course...
Is it possible to convert a string to a byte array and then convert it back to the original string in Java or Android?
My objective is to send some strings to a microcontroller (Arduino) and store it into EEPROM (which is the only 1 KB). I tried to use an MD5 hash, but it seems it's only one-way encryption. What can I do to deal with this issue?
I would suggest using the members of string, but with an explicit encoding:
byte[] bytes = text.getBytes("UTF-8");
String text = new String(bytes, "UTF-8");
By using an explicit encoding (and one which supports all of Unicode) you avoid the problems of just calling text.getBytes() etc:
You're explicitly using a specific encoding, so you know which encoding to use later, rather than relying on the platform default.
You know it will support all of Unicode (as opposed to, say, ISO-Latin-1).
EDIT: Even though UTF-8 is the default encoding on Android, I'd definitely be explicit about this. For example, this question only says "in Java or Android" - so it's entirely possible that the code will end up being used on other platforms.
Basically given that the normal Java platform can have different default encodings, I think it's best to be absolutely explicit. I've seen way too many people using the default encoding and losing data to take that risk.
EDIT: In my haste I forgot to mention that you don't have to use the encoding's name - you can use a Charset instead. Using Guava I'd really use:
byte[] bytes = text.getBytes(Charsets.UTF_8);
String text = new String(bytes, Charsets.UTF_8);
You can do it like this.
String to byte array
String stringToConvert = "This String is 76 characters long and will be converted to an array of bytes";
byte[] theByteArray = stringToConvert.getBytes();
http://www.javadb.com/convert-string-to-byte-array
Byte array to String
byte[] byteArray = new byte[] {87, 79, 87, 46, 46, 46};
String value = new String(byteArray);
http://www.javadb.com/convert-byte-array-to-string
Use [String.getBytes()][1] to convert to bytes and use [String(byte[] data)][2] constructor to convert back to string.
byte[] pdfBytes = Base64.decode(myPdfBase64String, Base64.DEFAULT)
import java.io.FileInputStream;
import java.io.ByteArrayOutputStream;
public class FileHashStream
{
// write a new method that will provide a new Byte array, and where this generally reads from an input stream
public static byte[] read(InputStream is) throws Exception
{
String path = /* type in the absolute path for the 'commons-codec-1.10-bin.zip' */;
// must need a Byte buffer
byte[] buf = new byte[1024 * 16]
// we will use 16 kilobytes
int len = 0;
// we need a new input stream
FileInputStream is = new FileInputStream(path);
// use the buffer to update our "MessageDigest" instance
while(true)
{
len = is.read(buf);
if(len < 0) break;
md.update(buf, 0, len);
}
// close the input stream
is.close();
// call the "digest" method for obtaining the final hash-result
byte[] ret = md.digest();
System.out.println("Length of Hash: " + ret.length);
for(byte b : ret)
{
System.out.println(b + ", ");
}
String compare = "49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d";
String verification = Hex.encodeHexString(ret);
System.out.println();
System.out.println("===")
System.out.println(verification);
System.out.println("Equals? " + verification.equals(compare));
}
}