Develop Reference

ZLib decompression fails on large byte array

When experimenting with ZLib compression, I have run across a strange problem. Decompressing a zlib-compressed byte array with random data fails reproducibly if the source array is at least 32752 bytes long. Here's a little program that reproduces the problem, you can see it in action on IDEOne. The compression and decompression methods are standard code picked off tutorials.
public class ZlibMain {
private static byte[] compress(final byte[] data) {
final Deflater deflater = new Deflater();
deflater.setInput(data);
deflater.finish();
final byte[] bytesCompressed = new byte[Short.MAX_VALUE];
final int numberOfBytesAfterCompression = deflater.deflate(bytesCompressed);
final byte[] returnValues = new byte[numberOfBytesAfterCompression];
System.arraycopy(bytesCompressed, 0, returnValues, 0, numberOfBytesAfterCompression);
return returnValues;
}
private static byte[] decompress(final byte[] data) {
final Inflater inflater = new Inflater();
inflater.setInput(data);
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length)) {
final byte[] buffer = new byte[Math.max(1024, data.length / 10)];
while (!inflater.finished()) {
final int count = inflater.inflate(buffer);
outputStream.write(buffer, 0, count);
}
outputStream.close();
final byte[] output = outputStream.toByteArray();
return output;
} catch (DataFormatException | IOException e) {
throw new RuntimeException(e);
}
}
public static void main(final String[] args) {
roundTrip(100);
roundTrip(1000);
roundTrip(10000);
roundTrip(20000);
roundTrip(30000);
roundTrip(32000);
for (int i = 32700; i < 33000; i++) {
if(!roundTrip(i))break;
}
}
private static boolean roundTrip(final int i) {
System.out.printf("Starting round trip with size %d: ", i);
final byte[] data = new byte[i];
for (int j = 0; j < data.length; j++) {
data[j]= (byte) j;
}
shuffleArray(data);
final byte[] compressed = compress(data);
try {
final byte[] decompressed = CompletableFuture.supplyAsync(() -> decompress(compressed))
.get(2, TimeUnit.SECONDS);
System.out.printf("Success (%s)%n", Arrays.equals(data, decompressed) ? "matching" : "non-matching");
return true;
} catch (InterruptedException | ExecutionException | TimeoutException e) {
System.out.println("Failure!");
return false;
}
}
// Implementing Fisher–Yates shuffle
// source: https://stackoverflow.com/a/1520212/342852
static void shuffleArray(byte[] ar) {
Random rnd = ThreadLocalRandom.current();
for (int i = ar.length - 1; i > 0; i--) {
int index = rnd.nextInt(i + 1);
// Simple swap
byte a = ar[index];
ar[index] = ar[i];
ar[i] = a;
}
}
}
Is this a known bug in ZLib? Or do I have an error in my compress / decompress routines?

It is an error in the logic of the compress / decompress methods; I am not this deep in the implementations but with debugging I found the following:
When the buffer of 32752 bytes is compressed, the deflater.deflate() method returns a value of 32767, this is the size to which you initialized the buffer in the line:
final byte[] bytesCompressed = new byte[Short.MAX_VALUE];
If you increase the buffer size for example to
final byte[] bytesCompressed = new byte[4 * Short.MAX_VALUE];
the you will see, that the input of 32752 bytes actually is deflated to 32768 bytes. So in your code, the compressed data does not contain all the data which should be in there.
When you then try to decompress, the inflater.inflate()method returns zero which indicates that more input data is needed. But as you only check for inflater.finished() you end in an endless loop.
So you can either increase the buffer size on compressing, but that probably just means haveing the problem with bigger files, or you better need to rewrite to compress/decompress logic to process your data in chunks.

Apparently the compress() method was faulty.
This one works:
public static byte[] compress(final byte[] data) {
try (final ByteArrayOutputStream outputStream =
new ByteArrayOutputStream(data.length);) {
final Deflater deflater = new Deflater();
deflater.setInput(data);
deflater.finish();
final byte[] buffer = new byte[1024];
while (!deflater.finished()) {
final int count = deflater.deflate(buffer);
outputStream.write(buffer, 0, count);
}
final byte[] output = outputStream.toByteArray();
return output;
} catch (IOException e) {
throw new IllegalStateException(e);
}
}

inputStream read method constantly reading 0 [duplicate]

This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 6 years ago.
I am first transferring a file from a client to my master, the stores the byte array and then sends to the slave. Where the slave stores the byte array. But when The file is sent properly from client to master but when I send the byte array to the slave it to the slave the read method in input stream constantly reads 0.
// This method writes the file to the master
public void writeFile(File file) {
try {
this.write(String.valueOf(file.length()));
byte[] bytearray = new byte[(int) file.length()];
FileInputStream fin = new FileInputStream(file);
BufferedInputStream bin = new BufferedInputStream(fin);
bin.read(bytearray, 0, bytearray.length);
BufferedOutputStream bos;
OutputStream os = socket.getOutputStream();
bos= new BufferedOutputStream(os);
bos.write(bytearray, 0, bytearray.length);
bos.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
//This method reads the file into the master as a byte array and the byte array from the master into slave
public byte[] readFile() {
byte[] bytearray = null;
try {
int currentTot = 0;
int filesize = Integer.parseInt(this.read());
System.out.println(filesize);
bytearray = new byte[filesize];
InputStream is = socket.getInputStream();
int bytesRead;
bytesRead = is.read(bytearray, 0, bytearray.length);
currentTot = bytesRead;
int count = 0;
do {
bytesRead = is.read(bytearray, currentTot, (bytearray.length - currentTot));
if (bytesRead > 0) {
currentTot += bytesRead;
count = 0;
} else {
count++;
System.out.println("count " + count);
}
} while (bytesRead > -1);
System.out.println(currentTot);
// bos.write(bytearray, 0, currentTot);
// bos.flush();
// bos.close();
} catch (IOException e) {
e.printStackTrace();
}
return bytearray;
}
//This method writes from the master to the slave
public void writeByte(byte[] m) {
this.write(String.valueOf(m.length));
System.out.println("File side inside sender" + m.length);
// byte[] bytearray = m;
OutputStream os;
try {
os = socket.getOutputStream();
os.write(m, 0, m.length);
os.flush();
//os.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Interestingly if I close my output stream after I send my byte array from my master it works well. But I cannot close stream because the slave needs to communicate with the master further. Thanks in advance.
public void write(String output) {
if (pw == null)
this.openWriter();
pw.println(output);
}
public String read() {
try {
if (br == null) {
if (this.socket != null)
br = new BufferedReader(new InputStreamReader(this.socket.getInputStream()));
}
return br.readLine();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}

You're misreading the file length in the receiver. You are getting zero, so you're constructing a zero length byte array, so read() returns zero.
You need to send the length via DataOutputStream.writeLong() and read it via DataInputStream.readLong(). And then your sending and receiving code is all wrong as well. See my answer here for complete code.

Getting game ID of psx game

I was wondering which is the best way to get the title from a disk image in .iso or .cue+.bin format,
Is there any java library that can do this or should I read from the file header?
UPDATE:
I managed to do it, i was particularly interested in PSX ISOs title. It's 10 bytes long and this is a sample code to read it:
File f = new File("cdimage2.bin");
FileInputStream fin = new FileInputStream(f);
fin.skip(37696);
int i = 0;
while (i < 10) {
System.out.print((char) fin.read());
i++;
}
System.out.println();
UPDATE2: This method is better:
private String getPSXId(File f) {
FileInputStream fin;
try {
fin = new FileInputStream(f);
fin.skip(32768);
byte[] buffer = new byte[4096];
long start = System.currentTimeMillis();
while (fin.read(buffer) != -1) {
String buffered = new String(buffer);
if (buffered.contains("BOOT = cdrom:\\")) {
String tmp = "";
int lidx = buffered.lastIndexOf("BOOT = cdrom:\\") + 14;
for (int i = 0; i < 11; i++) {
tmp += buffered.charAt(lidx + i);
}
long elapsed = System.currentTimeMillis() - start;
// System.out.println("BOOT = cdrom:\\" + tmp);
tmp = tmp.toUpperCase().replace(".", "").replace("_", "-");
fin.close();
return tmp;
}
}
fin.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}

Just start reading after 32768 bytes (unused by ISO9660) in 2048 byte chunks (Volume Descriptor). The first byte determines the type of the descriptor, and 1 means Primary Volume Descriptor, which contain the title after the first 7 bytes (which are always \x01CD001\x01). The next byte is a NUL (\x00) and it is followed by 32 bytes of system and 32 bytes of volume identifier, the latter usually known and displayed as title. See http://alumnus.caltech.edu/~pje/iso9660.html for a more detailed description.

Assign a String to byte and specify string length at the start on Java

I would like to assign a String data to the byte array and also put a 4-byte String data length at the start. What is the best way to accomplish? I need this for transmitting the byte data over the socket connection. Server side reads as many bytes mentioned at the start.
Is there a better way of doing this?
private byte[] getDataSendBytes(String data) {
int numberOfDataBytes = data.getBytes().length;
ByteBuffer bb = ByteBuffer.allocate(HEADER_LENGTH_BYTES);
bb.putInt(numberOfDataBytes);
byte[] headerBytes = bb.array();
byte[] dataBytes = data.getBytes();
// create a Datagram packet
byte[] sendDataBytes = new byte[HEADER_LENGTH_BYTES + dataBytes.length];
System.arraycopy(headerBytes, 0, sendDataBytes, 0, headerBytes.length);
System.arraycopy(dataBytes, 0, sendDataBytes, headerBytes.length,
dataBytes.length);
return sendDataBytes;
}

I would use either DataOutputStream
public byte[] getDataSendBytes(String text) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
new DataOutputStream(baos).writeUTF(text);
} catch (IOException e) {
throw new AssertionError(e);
}
return baos.toByteArray();
}
or ByteBuffer for control of the length type and endianess.
public byte[] getDataSendBytes(String text) {
try {
byte[] bytes = text.getBytes("UTF-8");
ByteBuffer bb = ByteBuffer.allocate(4 + bytes.length).order(ByteOrder.LITTLE_ENDIAN);
bb.putInt(bytes.length);
bb.put(bytes);
return bb.array();
} catch (UnsupportedEncodingException e) {
throw new AssertionError(e);
}
}
or for performance, reuse the ByteBuffer and assume a ISO-8859-1 character encoding
// GC-less method.
public void writeAsciiText(ByteBuffer bb, String text) {
assert text.length() < (1 << 16);
bb.putShort((short) text.length());
for(int i=0;i<text.length();i++)
bb.put((byte) text.charAt(i));
}

Bluetooth-connection; can't send strings properly

I have troubles with my program when i need to send Strings from my server bluetooth-socket to my client bluetooth-socket.
Everything works fine as long as I am only sending one String at a time (for example chatting) but if I need to write more Strings at a short period of time (to interchange informations), the Strings will not get seperated from the client code. For example if I'm sending "FirstUser" and right after that "SecondUser" the client does not read "FirstUser" and then "SecondUser". It will read "FirstUserSecondUser". How can I avoid this behaviour?
Edit: If I let the Thread sleep before it is able to send a new message, it reads the right strings but this solution is not working fine for my need.
Server-Code: sending to all clients(edited)
public synchronized void sendToAll(String message)
{
try {
Thread.sleep(100);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
publishProgress(message);
for(OutputStream writer:outputList) {
try {
writer.write(message.getBytes());
writer.flush();
} catch (IOException e) {
System.out.println("Some-Error-Code");
}
}
}
Server-Code: reading from a client:
public void run() {
String nachricht;
int numRead;
byte[] buffer = new byte[1024];
while (runningFlag)
{
try {
if((numRead = inputStream.read(buffer)) >= 0) {
nachricht = new String(buffer, 0, numRead);
serverThread.handleMessage(nachricht);
}
}
catch (IOException e) {
this.cancel();
e.printStackTrace();
}
}
}
Client-Code: reading from server(edited)
#Override
protected Void doInBackground(Integer... ints) {
String nachricht = new String();
byte[] buffer = new byte[1024];
int numRead;
while (runningFlag)
{
try {
if(((numRead = inputStream.read(buffer)) >= 0)) {
nachricht = new String(buffer, 0, numRead);
publishProgress(nachricht);
}
}
catch (IOException e) {
clientGame.finish();
e.printStackTrace();
}
}
return null;
}
Client-Code: writing to server
public synchronized void write(String nachricht)
{
try {
Thread.sleep(100);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
try {
outputStream.write(nachricht.getBytes());
outputStream.flush();
} catch (IOException e) {
this.cancel();
e.printStackTrace();
}
}
I appreciate every little help :) .

You need to encapsulate your data item to avoid concatenation.
It means that you have to write and read a whole data item before continuing.
You should have some utility methods to do that instead of directly using methods of OutputStream and InputStream :
public static void writeItem(OutputStream out, String s) throws IOException
{
// Get the array of bytes for the string item:
byte[] bs = s.getBytes(); // as bytes
// Encapsulate by sending first the total length on 4 bytes :
// - bits 7..0 of length
out.write(bs.length); // modulo 256 done by write method
// - bits 15..8 of length
out.write(bs.length>>>8); // modulo 256 done by write method
// - bits 23..16 of length
out.write(bs.length>>>16); // modulo 256 done by write method
// - bits 31..24 of length
out.write(bs.length>>>24); // modulo 256 done by write method
// Write the array content now:
out.write(bs); // Send the bytes
out.flush();
}
public static String readItem(InputStream in) throws IOException
{
// first, read the total length on 4 bytes
// - if first byte is missing, end of stream reached
int len = in.read(); // 1 byte
if (len<0) throw new IOException("end of stream");
// - the other 3 bytes of length are mandatory
for(int i=1;i<4;i++) // need 3 more bytes:
{
int n = in.read();
if (n<0) throw new IOException("partial data");
len |= n << (i<<3); // shift by 8,16,24
}
// Create the array to receive len bytes:
byte[] bs = new byte[len];
// Read the len bytes into the created array
int ofs = 0;
while (len>0) // while there is some byte to read
{
int n = in.read(bs, ofs, len); // number of bytes actually read
if (n<0) throw new IOException("partial data");
ofs += n; // update offset
len -= n; // update remaining number of bytes to read
}
// Transform bytes into String item:
return new String(bs);
}
Then you use these methods both for server & client to read and write your String items.