I have an Android app that uses ByteBuffer to send an array of ints (converted to a string) to my server, which is written in Ruby. The server takes the string and unpacks it. My code to build the ByteBuffer looks like this:
ByteBuffer bytes = ByteBuffer.allocate(16);
bytes.order(ByteOrder.LITTLE_ENDIAN);
bytes.putInt(int1);
bytes.putInt(int2);
bytes.putInt(int3);
bytes.putInt(int4);
String byteString = new String(bytes.array());
This works great when the ints are all positive values. When it has a negative int, things go awry. For example, in iOS when I submit an array of ints like [1,1,-1,0], the byte string on the server is:
"\x01\x00\x00\x00\x01\x00\x00\x00\xFF\xFF\xFF\xFF\x00\x00\x00\x00"
That gets correctly unpacked to [1,1,-1,0].
In Android, however, when I try to submit the same array of [1,1,-1,0], my string is:
"\x01\x00\x00\x00\x01\x00\x00\x00\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00\x00\x00"
Which gets unpacked to [1, 1, -272777233, -1074807361].
If I convert the negative int to an unsigned int:
byte intByte = (byte) -1;
int unsignedInt = intByte & 0xff;
I get the following:
"\x01\x00\x00\x00\x01\x00\x00\x00\xEF\xBF\xBD\x00\x00\x00\x00\x00\x00\x00"
Which gets unpacked to [1, 1, 12435439, 0]. I'm hoping someone can help me figure out how to properly handle this so I can send negative values properly.
Your problem is here:
String byteString = new String(bytes.array());
Why do you do that? You want to send a stream of bytes, so why convert it to a stream of chars?
If you want to send bytes, send bytes. Use an OutputStream, not a Writer; use an InputStream, not a Reader. The fact that integers are "negative" or "positive" does not matter.
Related
I am trying to send data from PHP TCP server to JAVA TCP client.
I am comparing my results by comparing hex values of the data.
PHP script reads STDIN, sends it through socket one byte at a time and java reads it using DataInputStream.read(), converts to hex and displays.
If I manually type data into script - it works ok.
If I use file with data - it works OK
But when I assign /dev/urandom(even few bytes) - the data on the java side is coming corrupted. There is always a hex of value efbfbd in random places instead of correct data.
Please help me with this issue.
PHP code:
$f = fopen( 'php://stdin', 'rb' );
while($line = fread($f, 1)){
$length = 1;
echo bin2hex($line)."\n";
echo socket_write($client, $line, 1)."\n";
$sent = socket_write($client, $line, $length);
if ($sent === false) {
break;
}
// Check if the entire message has been sented
if ($sent < $length) {
// If not sent the entire message.
// Get the part of the message that has not yet been sented as message
$line = substr($line, $sent);
// Get the length of the not sented part
$length -= $sent;
}
Java code:
in = new DataInputStream(clientSocket.getInputStream());
byte[] data = new byte[1];
int count = 0;
while(in.available() > 0){
//System.out.println(in.available());
in.read(data);
String message = new String(data);
System.out.println(message);
//System.out.flush();
System.out.println( toHex(message) );
//in.flush();
message = "";
}
You're stumbling upon encoding. By calling new String(data) the byte array is converted using your default encoding to a string, whatever this encoding may is (you can set the encoding by java -Dfile.encoding=UTF-8 to UTF-8 for example).
The Java code you want would most likely look the following:
in = new DataInputStream(clientSocket.getInputStream());
byte[] data = new byte[1];
int count = 0;
while (in.available() > 0) {
// System.out.println(in.available());
in.read(data);
String hexMessage = Integer.toHexString(data[0] & 0xFF);
String stringMessage = new String(data, "UTF-8"); // US-ASCII, ISO-8859-1, ...
System.out.println(hexMessage);
}
Update: I missed the 32bit issue. The 8-bit byte, which is signed in Java, is sign-extended to a 32-bit int. To effectively undo this sign extension, one can mask the byte with 0xFF.
There are two main issues with your Java program.
First - the use of in.available(). It does not tell you how many bytes there are still in the message. It merely says how many bytes are ready in the stream and for available reading without being blocked. For example, if the server sends two packets of data over the socket, one has arrived, but one is still being sent over the Internet, and each packet has 200 bytes (this is just an example), then in the first call you'll get the answer 200. If you read 200 bytes, you're sure not to be blocked. But if the second packet has not arrived yet, your next check of in.available() will return 0. If you stop at this point, you only have half the data. Not what you wanted.
Typically you either have to read until you reach end-of-stream (InputStream.read() returns -1), and then you can't use the same stream anymore and you close the socket, or you have a specific protocol that tells you how many bytes to expect and you read that number of bytes.
But that's not the reason for the strange values you see in output from your program. The reason is that Java and PHP represent strings completely differently. In PHP, a string can contain any bytes at all, and the interpretation of them as characters is up to the prorgrammer.
This basically means that a PHP string is the equivalent of a byte[] in Java.
But Java Strings are completely different. It consists internally of an array of char, and char is always two bytes in UTF-16 encoding. When you convert bytes you read into a Java String, it's always done by encoding the bytes using some character encoding so that the appropriate characters are stored in the string.
For example, if your bytes are 44 4F 4C 4C, and the character encoding is ISO-8859-1, this will be interpreted as the characters \u0044, \u004F, \u004C, \u004C. It will be a string of four characters - "DOLL". But if your character encoding is UTF-16, the bytes will be interpreted as \u444F and \u4C4C. A string of only two characters, "䑏䱌".
When you were reading from the console or from a file, the data was probably in the encoding that Java expects by default. This is usually the case when the file is written in pure English, with just English letters, spaces and punctuation. These are all 7-bit characters which are the same in ISO-8859-1 and UTF-8, which are the common defaults. But in /dev/urandom you'd have some bytes in the range 80 through FF, which may be treated differently when interpreted into a UTF-16 Java string.
Furthermore, you didn't show your toHex() method in Java. It probably reads bytes back from the string again, but using which encoding? If you read the bytes into the String using ISO-8859-1, and got them out in UTF-8, you'd get completely different bytes.
If you want to see exactly what PHP sent you, don't put the bytes in a String. Write a toHex method that works on byte arrays, and use the byte[] you read directly.
Also, always remember to check the number of bytes returned by read() and only interpret that number of bytes! read() does not always fill the entire array. So in your new toHex() method, you need to also pass the number of bytes read as a parameter, so that it doesn't display the parts of the array after them. In your case you just have a one-byte array - which is not recommended - but even in this case, read() can return 0, and it's a perfectly legal value indicating that in this particular call to read() there were no bytes available although there may be some available in the next read().
As the comment above says you might be having troubles with the string representation of the bytes String message = new String(data); To be certain, you should get the data bytes and encode them in Base64 for example. You can use a library such as Apache Commons or Java 8 to do that. You should be able to do something similar in PHP to compare.
I'm new to bit manipulation and I'm receiving data via a socket connection.
Data received is populated in a byte buffer, size is 8.
The number of data fields is 5.
The data/fields are variable length. So, it appears two of data/fields are combined as one byte.
Sent dataset1: 1, 0, 0, 2, 22
Sent dataset2: 1, 0, 0, 2, 500
Example data received: [1, 0, -128, 22] or [1,0,-127, -12]
From the example byte array elements total 4 when 5 data items was sent over a socket.
How can I parse/unpack/extract byte array element (e.g., -128, -12, etc.) in order to capture the dataset values that were sent?
the question sounds very strange, if you have a byte array then there is not any problem to get the needed byte for its index, it you need get access to parts of bytes then as the first one you need find the specification of the protocol because without that you have not any idea about field lengths
p.s.
for bit parsing you can try JBBP
It is even more impossible, 500 does not fit into a byte.
To see regularities, you might puzzle with hex; that shows the bit packing (if that is the case);
int[] dataset = new int[5];
byte[] received = new byte[5];
dataset[0] = received[0];
dataset[1] = received[1];
dataset[2] = received[2] - (received[4] < 0 ? 0x7f : 0x80);
dataset[3] = 2;
dataset[4] = received[3];
if (received[4] < 0) {
dataset[
dataset[4] += 0x200; // 512
}
You really need more information. The above is an attempt to match both datasets to those received bytes, by minimal bits & operations. I would not bet for 1:100 that that is the correct decoding though.
I modified my implementation to use DataInputStream to read data from socket as outline here:Java TCP How do you read a sent stream size (header) and keep reading based upon that size
This approach provide flexibility to manipulate the various values such as individual header fields by using ByteBuffer
Thanks
I have a byte array in Java. The values are both positive and negative (since they are greater than 127 in the original unsigned array). Now I want to send this array with Quickserver (http://www.quickserver.org/) to my TCP client in an iOS application I am writing as well. I pass the byte array to sendClientBinary() method which accepts the byte array as its input. However, when I receive the array in the iOS client app, all the negative values seem to have been converted to some kind of complement form and mainly into two-byte values: -71 (0xB9) in Netbeans looks in Xcode memory view as 0xC2 0xB9 and -67 (0xBD) in Netbeans appears as 0xC2 0xBD in Xcode.
Can anyone please provide explanation for this?
I am also able to convert my byte array to char array and masking out all the upper bytes, so now the char array holds the correct values in the full 0-255 range, however, there is no way how to pass a char array to sendClientBinary() method that only accepts byte array as input.
Should I try to be casting or converting char array to byte array somehow again?
//Some code in Java:
//reading my byte array from a method and converting it to char array (sorry if it's not the most efficient way, just need something simple right now
byte byteArray[] = (byte[])functionReturningByteArray();
char charArray[] = new char[byteArray.length];
for (int ij = 0; ij < byteArray.length; ij++)
{
charArray[ij] = (char) byteArray[ij];
if (charArray[ij] > 255)
charArray[ij] &= 0xFF;
}
//and the code sending the data over TCP socket (via Quickserver):
clientH.setDataMode(DataMode.BINARY, DataType.OUT);
clientH.sendClientBinary(byteArray);
//--this is received in iOS as 16-bit values with some prefix such as 0xC2 or 0xC3 for negative values, if not for the prefix the value would be correct
//or an attempt to send the charArray:
clientH.setDataMode(DataMode.byte, DataType.OUT);
clientH.sendClientBytes(charArray.toString());
//--this doesn't resemble my bytes once received in iOS at all
//iOS reception code:
case NSStreamEventHasBytesAvailable:
{
if(stream == inputStream)
{
int len = 0;
len = [inputStream read:receptionBuf maxLength:2048*2048*2];
packetBytesReceived += len;
[packetData appendBytes:receptionBuf length:len];
NSString* fullData = [[NSString alloc] initWithData:packetData encoding:NSASCIIStringEncoding];
...
...
I think the problem might be in NSASCIIStringEncoding since I am receiving characters in the main part of my data packet, but some content is just byte data values and this probably could be the cause...? Will start working on it.
0xc2 is the prepend for byte in UTF-8 encoding. It denotes that you are sending a special character in UTF-8 in the 0xc2 sequence. So 0xC2 0xB9 would translate to a superscript character; in particular ^1. My guess (since I assume this is not what you are actually sending) is that your encoding is set incorrectly some place.
Problem solved. I am reading my binary portion of data payload directly from packetData variable in iOS application (instead of fullData which is NSString) without converting it first to string and then decoding to byte with UTF8 encoding again.
I've no coding experience in PHP at all. But while looking for a solution for my Java project, i found an example of the problem in PHP, which incidentally is alien to me.
Can anyone please explain the working and the result of the unpack('N*',"string") function of PHP and how to implement it in Java?
An example would help me a lot!
Thanks!
In PHP (and in Perl, where PHP copied it from), unpack("N*", ...) takes a string (actually representing a sequence of bytes) and parses each 4-byte segment of it as a signed 32-bit big-endian ("Network byte order") integer, returning them in an array.
There are several ways to do the same in Java, but one way would be to wrap the input byte array in a java.nio.ByteBuffer, convert it to an IntBuffer and then read the integers from that:
public static int[] unpackNStar ( byte[] bytes ) {
// first, wrap the input array in a ByteBuffer:
ByteBuffer byteBuf = ByteBuffer.wrap( bytes );
// then turn it into an IntBuffer, using big-endian ("Network") byte order:
byteBuf.order( ByteOrder.BIG_ENDIAN );
IntBuffer intBuf = byteBuf.asIntBuffer();
// finally, dump the contents of the IntBuffer into an array
int[] integers = new int[ intBuf.remaining() ];
intBuf.get( integers );
return integers;
}
Of course, if you just want to iterate over the integers, you don't really need the IntBuffer or the array:
ByteBuffer buf = ButeBuffer.wrap( bytes );
buf.order( ByteOrder.BIG_ENDIAN );
while ( buf.hasRemaining() ) {
int num = buf.getInt();
// do something with num...
}
In fact, iterating over a ByteBuffer like this is a convenient way to emulate the behavior of even more complicated examples of unpack() in Perl or PHP.
(Disclaimer: I have not tested this code. I believe it should work, but it's always possible that I may have mistyped or misunderstood something. Please test before using.)
Ps. If you're reading the bytes from an input stream, you could also wrap it in a DataInputStream and use its readInt() method. Of course, it's also possible to use a ByteArrayInputStream to read the input from a byte array, achieving the same results as the ByteBuffer examples above.
Previously I posed a question about converting a byte[] to short[] and a new problem I encountered is converting/ not converting the data from byte[] to BigEndian.
Here is what is going on:
I am using TargetDataLine to read data into a byte[10000].
The AudioFormat object has BigEndian set to true, arbitrarily.
This byte[] needs to be converted to short[] so that it can be encoded using Xuggler
I don't know whether the AudioFormat BigEndian should be set to true or false.
I have tried both the cases and I get an Exception in both the cases.
To convert byte[] to short[], I do this:
fromMic.read(tempBufferByte, 0, tempBufferByte.length);
for(int i=0;i<tempBufferShort.length;i++){
tempBufferShort[i] = (short) tempBufferByte[i];
}
where:
fromMic is TargetDataLine
tempBufferbyte is byte[10000]
tempBufferShort is short[10000]
I get the Exception:
java.lang.RuntimeException: failed to write packet: com.xuggle.xuggler.IPacket#90098448[complete:true;dts:12;pts:12;size:72;key:true;flags:1;stream index:1;duration:1;position:-1;time base:9/125;]
Miscellaneous information that may be needed:
How I set the stream for adding audio in Xuggler:
writer.addAudioStream(0,1,fmt.getChannels(),(int)fmt.getSampleRate());
How I perform the encoding
writer.encodeAudio(1,tempBufferShort,timeStamp,TimeUnit.NANOSECONDS);
Java Doc on AudioFormat
...In addition to the encoding, the audio format includes other
properties that further specify the exact arrangement of the data.
These include the number of channels, sample rate, sample size, byte
order, frame rate, and frame size...
and
For 16-bit samples (or any other sample size larger than a byte), byte
order is important; the bytes in each sample are arranged in either
the "little-endian" or "big-endian" style.
Questions:
Do I need to keep the BigEndian as true in javax.sound.sampled.AudioFormat object?
What is causing the error? Is it the format?
I guess I get BigEndian data preformatted by the AudioFormat object.
If your data is indeed big endian, you can directly convert it to a (big endian) short array like this:
ByteBuffer buf = ByteBuffer.wrap(originalByteArray);
short[] shortArray = buf.asShortBuffer().array();
The resulting short array will have all the original byte array directly, and correctly, mapped, given that your data is big endian. So, an original array, such as:
// bytes
[00], [ae], [00], [7f]
will be converted to:
// shorts
[00ae], [007f]
You need to convert two bytes into one short, so this line is wrong:
tempBufferShort[i] = (short) tempBufferByte[i];
You need something along the lines of
tempBufferShort[i] = (short)
(tempBufferByte[i*2] & 0xFF)*256 + (tempBufferByte[i*2+1] & 0xFF);
This would be in line with a big-endian byte array.
What others here have said about the byte-to-short conversion is correct, but it cannot cause the problem you see, it would just cause the output audio to be mostly noise. You can call writeAudio with a buffer of all zeros (or anything, really) so, everything else being equal, the values in the buffer don't matter to whether the call succeeds (they do matter to what you hear in the output, of course :)
Does the exception happen at the beginning of the stream (first audio chunk)? Can you write an audio-only stream successfully?
Set the audio codec when you call addAudioStream. Try ICodec.ID.CODEC_ID_MP3 or ICodec.ID.CODEC_ID_AAC.
Check that fmt.getChannels() and fmt.getSampleRate() are correct. Not all possible values are supported by any particular codec. (2 ch, 44100 Hz should be supported by just about anything).
Are you writing your audio and video such that the timestamps are strictly non-decreasing?
Do you have enough audio samples for the duration your timestamps are indicating? Does tempBufferShort.length == ((timeStamp - lastTimeStamp) / 1e+9) * sampleRate * channels ? (This may be only approximately equal, but it should be very close, slight rounding errors probably ok).