I've run into mind twisting bafflement after putting my hands into an old legacy project. The project consists of a Java application and a c++ application, which communicate with sockets. Both applications are designed to work on cross platform environments, so I'd be happy to keep the code as universal as possible.
I ended up rewriting parts of the communication logic, since the previous implementation had some issues with foreign characters. Now I ran into a problem with endianness, which I hope someone could spell out for me.
The Java software writes messages to socket with OutputStreamWriter, using UTF-16LE encoding, as follows.
OutputStream out = _socket.getOutputStream();
outputWriter = new OutputStreamWriter(new BufferedOutputStream(out), "UTF-16LE");
// ... create msg
outputWriter.write(msg, 0, msg.length());
outputWriter.flush();
The c++ program receives the message character by character as follows:
char buf[1];
std::queue<char> q;
std::u16string recUtf16Msg;
do {
int iResult = recv(socket, buf, 1, 0);
if (iResult <= 0)
break; // Error or EOS
for (int i = 0; i < iResult; i++) {
q.push(buf[i]);
}
while (q.size() >= 2) {
char firstByte = q.front();
q.pop();
char secondByte = q.front();
q.pop();
char16_t utf16char = (firstByte << (sizeof(char) * CHAR_BIT)) ^
(0x00ff & secondByte);
// Change endianness, if necessary
utf16char = ntohs(utf16char);
recUtf16Msg.push_back(utf16char);
}
// ... end of message check removed for clarity
} while (true);
Now the issue which I'm really facing is that the code above actually works, but I'm not really sure why. The c++ side is written to receive messages which use network byte order (big endian) but it seems that java is sending the data using little endian encoding.
On c++ side we even use ntons-function to change endianness to the one desired by host machine. According to specification I understand that hton is supposed to do swap endianness if host platform uses little endian byte order. However ntonhs actually swaps the endianness of the received small endian characters, which ends up as big endian and the software works flawlessly.
Maybe someone can point out what exactly is happening? Do I accidentally switch bytes already to when creating utf16char? Why htons makes everything work, while it seems to act exactly opposite to the documentation? To compile I'm using Clang with libc++.
I left out parts of the code for clarity, but you should get the general idea. Also, I'm aware that using queue and dynamic array may not be the most effective way of handling data, but it's clean and performs well enough for this purpose.
I am trying to implement RTSP Server with Java for fun.
Since I do not have any pre-knowledge about RTSP.
I am starting with analysis already made source code.
I found those code in internet.
Link :
http://nsl.cs.sfu.ca/teaching/09/820/streaming/Client.html
http://nsl.cs.sfu.ca/teaching/09/820/streaming/Server.html
http://nsl.cs.sfu.ca/teaching/09/820/streaming/VideoStream.html
http://nsl.cs.sfu.ca/teaching/09/820/streaming/RTPpacket.html
For this posting, I got question about VideoStream.java.
it has a method like below :
public int getnextframe(byte[] frame) throws Exception
{
int length = 0;
String length_string;
byte[] frame_length = new byte[5];
//read current frame length
fis.read(frame_length,0,5);
//transform frame_length to integer
length_string = new String(frame_length);
length = Integer.parseInt(length_string);
return(fis.read(frame,0,length));
}
As you can see, it casts byte[] to String than Integer. However, in my experience, the String turns out a hexa String. So I changed like... below.
Integer.parseInt(length_string.trim(), 16);
It looks OK sometimes, but sometimes gets Number Format Exception.
When I print length_string variables, it show in console like iso2a, vc1mp, 41��� ....
I can not know what I am missing here. Can you explain what is the purpose of codes here?
length_string = new String(frame_length);
length = Integer.parseInt(length_string);
P.S
Is there anyone knows full implementation of these code or other samples which does not uses extra third party libs, it could be much help for me.
I answer to myself to close this question.
Each video types has its own data structure.
In this case, I set data type for 'MJPEG', but I load 'MPEG' file.
So bytes array appears weird way.
FYI, MJPEG is just pack of jpeg. So every frame is key frame.
Other type I handled before, for instance, TS file, has i-frames and b-frames. Former one has clear image since it is key frame. However second one looks broken since it is not a key frame.
Can someone explain to me why this works just fine with in.available()>0 commented out, but when I put it back in it breaks?
mySocket = new Socket("blahblah", 12345);
BufferedInputStream in = new BufferedInputStream(mySocket.getInputStream());
....
char result[] = new char[length];
for (int i = 0; i < length && !mySocket.isClosed() /*&& in.available()>0*/; i++){
result[i] = (char)in.read();
}
More specifically: I'm making an Android app where a user can search for a term, that search is sent to some thingy in interspace, I get back results in xml form, and do stuff with them. When the xml I get back is small enough (see "length" in code above), the code works just fine with in.available()>0 left in. But if the length is large, in.available() returns 0. But with that commented out, everything continues to run smoothly.
Why is that? And is it something I need to worry about and fix?
in.available() lets you know if you can read data at that moment without blocking. As Sockets have a stream of data, it may not available immediately but in a short time. e.g. if you have a 1 Gbit connection, full packets will be no closer than 15 micro-seconds which is long time for a computer.
I think the reason in.available() == 0 when the data is large is because it hasn't had a chance to write it to your socket yet. You shouldn't need to use in.available(). Also, I wouldn't suggest reading a single char at a time, that will be really slow with a lot of data and VERY chatty over the network. Consider reading in a byte array of size "length".
I am trying to write Long using Java command for Random File IO as follows:
fstreamOut = new FileOutputStream(new File("C:\\Basmah","dataOutput.7"),true);
DataOutputStream out=new DataOutputStream(fstreamOut);
Long p= Long.parseLong(longNumberInString ); // Number of digits for this long key are 7-15
out.writeLong(p);
The problem is that when I write 7-15 digit number using writeLong ; it writes 8 bytes in file.
Then I am trying to read the same record into my program and decode it
Long l=in.readLong();
but I dont get the same number as I wrote ; Instead Iget EOF exception.
A long id 64-bit long. That makes 8 bytes. The DataOutputStream's writeLong method writes the binary representation of the long, not the textual one.
Without knowing the code used to read the long value, it's impossible to tell why it doesn't work.
The code given in your example and comment should work. The fact that it doesn't suggests that something else is going on here:
Maybe the writing and reading is happening on different files.
Maybe the file being written is not flushed / closed before you attempt to read it.
Maybe something else is overwriting the file.
Maybe the snippets of code you have provided are different enough to the real code to make a difference.
In the code that attempts to read the file, print what you get when you call f.length().
I have to read a binary file in a legacy format with Java.
In a nutshell the file has a header consisting of several integers, bytes and fixed-length char arrays, followed by a list of records which also consist of integers and chars.
In any other language I would create structs (C/C++) or records (Pascal/Delphi) which are byte-by-byte representations of the header and the record. Then I'd read sizeof(header) bytes into a header variable and do the same for the records.
Something like this: (Delphi)
type
THeader = record
Version: Integer;
Type: Byte;
BeginOfData: Integer;
ID: array[0..15] of Char;
end;
...
procedure ReadData(S: TStream);
var
Header: THeader;
begin
S.ReadBuffer(Header, SizeOf(THeader));
...
end;
What is the best way to do something similar with Java? Do I have to read every single value on its own or is there any other way to do this kind of "block-read"?
To my knowledge, Java forces you to read a file as bytes rather than being able to block read. If you were serializing Java objects, it'd be a different story.
The other examples shown use the DataInputStream class with a File, but you can also use a shortcut: The RandomAccessFile class:
RandomAccessFile in = new RandomAccessFile("filename", "r");
int version = in.readInt();
byte type = in.readByte();
int beginOfData = in.readInt();
byte[] tempId;
in.read(tempId, 0, 16);
String id = new String(tempId);
Note that you could turn the responce objects into a class, if that would make it easier.
If you would be using Preon, then all you would have to do is this:
public class Header {
#BoundNumber int version;
#BoundNumber byte type;
#BoundNumber int beginOfData;
#BoundString(size="15") String id;
}
Once you have this, you create Codec using a single line:
Codec<Header> codec = Codecs.create(Header.class);
And you use the Codec like this:
Header header = Codecs.decode(codec, file);
You could use the DataInputStream class as follows:
DataInputStream in = new DataInputStream(new BufferedInputStream(
new FileInputStream("filename")));
int x = in.readInt();
double y = in.readDouble();
etc.
Once you get these values you can do with them as you please. Look up the java.io.DataInputStream class in the API for more info.
I may have misunderstood you, but it seems to me you're creating in-memory structures you hope will be a byte-per-byte accurate representation of what you want to read from hard-disk, then copy the whole stuff onto memory and manipulate thence?
If that's indeed the case, you're playing a very dangerous game. At least in C, the standard doesn't enforce things like padding or aligning of members of a struct. Not to mention things like big/small endianness or parity bits... So even if your code happens to run it's very non-portable and risky - you depend on the compiler's creator not changing its mind on future versions.
Better to create an automaton to both validate the structure being read (byte per byte) from HD is valid, and filling an in-memory structure if it's indeed OK. You may loose some milliseconds (not so much as it may seem for modern OSes do a lot of disk read caching) though you gain platform and compiler independence. Plus, your code will be easily ported to another language.
Post Edit: In a way I sympathize with you. In the good-ol' days of DOS/Win3.11, I once created a C program to read BMP files. And used exactly the same technique. Everything was nice until I tried to compile it for Windows - oops!! Int was now 32 bits long, rather than 16! When I tried to compile on Linux, discovered gcc had very different rules for bit fields allocation than Microsoft C (6.0!). I had to resort to macro tricks to make it portable...
I used Javolution and javastruct, both handles the conversion between bytes and objects.
Javolution provides classes that represent C types. All you need to do is to write a class that describes the C structure. For example, from the C header file,
struct Date {
unsigned short year;
unsigned byte month;
unsigned byte day;
};
should be translated into:
public static class Date extends Struct {
public final Unsigned16 year = new Unsigned16();
public final Unsigned8 month = new Unsigned8();
public final Unsigned8 day = new Unsigned8();
}
Then call setByteBuffer to initialize the object:
Date date = new Date();
date.setByteBuffer(ByteBuffer.wrap(bytes), 0);
javastruct uses annotation to define fields in a C structure.
#StructClass
public class Foo{
#StructField(order = 0)
public byte b;
#StructField(order = 1)
public int i;
}
To initialize an object:
Foo f2 = new Foo();
JavaStruct.unpack(f2, b);
I guess FileInputStream lets you read in bytes. So, opening the file with FileInputStream and read in the sizeof(header). I am assuming that the header has a fixed format and size. I don't see that mentioned in the initial post, but assuming that is the case as it would get much more complex if the header has optional args and different sizes.
Once you have the info, there can be a header class in which you assign the contents of the buffer that you've already read. And then parse the records in a similar fashion.
Here is a link to read byte using a ByteBuffer (Java NIO)
http://exampledepot.com/egs/java.nio/ReadChannel.html
As other people mention DataInputStream and Buffers are probably the low-level API's you are after for dealing with binary data in java.
However you probably want something like Construct (wiki page has good examples too: http://en.wikipedia.org/wiki/Construct_(python_library), but for Java.
I don't know of any (Java versions) off hand, but taking that approach (declaratively specifying the struct in code) would probably be the right way to go. With a suitable fluent interface in Java it would probably be quite similar to a DSL.
EDIT: bit of googling reveals this:
http://javolution.org/api/javolution/io/Struct.html
Which might be the kind of thing you are looking for. I have no idea whether it works or is any good, but it looks like a sensible place to start.
I would create an object that wraps around a ByteBuffer representation of the data and provide getters to read directly from the buffer. In this way, you avoid copying data from the buffer to primitive types. Furthermore, you could use a MappedByteBuffer to get the byte buffer. If your binary data is complex, you can model it using classes and give each class a sliced version of your buffer.
class SomeHeader {
private final ByteBuffer buf;
SomeHeader( ByteBuffer fileBuffer){
// you may need to set limits accordingly before
// fileBuffer.limit(...)
this.buf = fileBuffer.slice();
// you may need to skip the sliced region
// fileBuffer.position(endPos)
}
public short getVersion(){
return buf.getShort(POSITION_OF_VERSION_IN_BUFFER);
}
}
Also useful are the methods for reading unsigned values from byte buffers.
HTH
I've written up a technique to do this sort of thing in java - similar to the old C-like idiom of reading bit-fields. Note it is just a start but could be expanded upon.
here
In the past I used DataInputStream to read data of arbitrary types in a specified order. This will not allow you to easily account for big-endian/little-endian issues.
As of 1.4 the java.nio.Buffer family might be the way to go, but it seems that the your code might actually be more complicated. These classes do have support for handling endian issues.
A while ago I found this article on using reflection and parsing to read binary data. In this case, the author is using reflection to read the java binary .class files. But if you are reading the data into a class file, it may be of some help.