So by my understanding for thrift, Java is the only language supported that does not have binary-safe Strings, hence the thrift binary type. My problem is it doesn't seem to work.
My definition File is:
service myService {
int myMethod(1:binary input)
}
My Java client builds a ByteBuffer from binary data that is observed to have positive length, and printable bytes prior to calling myMethod.
Immediately inside the C++ implementation of myMethod (from the thrift generated server skeleton), attempts to print input show it as always being empty of size 0.
Any ideas what I'm missing here? Changing binary to string makes everything work like a charm, minus the fact that I don't want the unsafe java-converted string to deal with later...
Most likely you're having problem because ByteBuffer in Java has mutable state. So, any read operation actually modifies ByteBuffer, since it modifies read position.
The simpliest (whereas not the most effective) way to work with thrift binaries in java is creating binaries as byte arrays and using wrapping them info buffers immidiately before invocation, i.e.:
byte[] input = ....;
myService.myMethod(ByteBuffer.wrap(input));
Another possible solution is to use ByteBuffer.duplicate to keep original buffer safe, i.e.:
ByteBuffer input = ....;
dump(input.duplicate());// dump function may change buffer position
myService.myMethod(input);
Related
I need to represent a DirectX D3DCOLOR (which is a 32-bit ARGB) in Java.
I don't need to perform any operations on it, other than string encoding it to transport over XML-RPC to a DirectX application. It's very likely to enter my Java system in a similar way, i.e. as an encoded string.
What string form would be considered "canonical", or rather, what string representation will be the friendliest for my DirectX client to parse?
Therefore, what type should I use within Java to capture a D3DCOLOR? The immediately obvious answers are:
Keep it as a java.lang.String, and don't bother validating it
Wrap a java.lang.Long and implement some toString() method
Neither seems wholly satisfactory, but I suppose it depends on how the client would prefer me to serialize it.
Considering that ARGB channels are restricted to a single byte they are usually stored as a byte[] or some equivalent-in-size structure. An obvious equivalent would be a 4-byte int/uint. This is doubly-useful when you consider that D3DXCOLOR can be constructed from a DWORD.
EDIT: Although Java doesn't support unsigned, this doesn't really matter as a C++ cast will turn it back around.
In C/C++, you can do the following:
struct DataStructure
{
char member1;
char member2;
};
DataStructure ds;
char bytes[] = {0xFF, 0xFE};
memcpy(&ds, bytes, sizeof(ds));
and you would essentially get the following:
ds.member1 = 0xFF;
ds.member2 = 0xFE;
What is the Java equivalent?
What is the Java equivalent?
There is no Java equivalent.
Java does not allow you to create or modify objects by accessing them at that level. You should be using new or setter methods, depending on what you are trying to achieve.
(There are a couple of ways to do this kind of thing, but they are unsafe, non-portable and "not Java" ... and they are not warranted in this situation.)
The memcpy you wrote depends on the internal implementation of the struct and would not necessarily work. In java, you need to define a constructor that accepts a byte array and set the fields. No shortcuts like this, as the memory structure of the class is not defined.
In Java you cannot work with the memory directly (no memcpy) it is the advantage (disadvantage?) of Java. There are some java library methods to copy arrays: System.arraycopy().
In general, to copy some object you need to ship it with clone method.
You might be able to do that in C. But you'd be wandering into aliasing problems and a hunka hunka burning undefined behavior.
And because struct padding is up to a compiler, what you might get with your memcpy is just ds.member1 = 0xFF, ds.member2 = whatever junk happened to be on the stack at the time, because member1 was padded to occupy 4 bytes rather than just 1. Or maybe you get junk for both, because you set the top 2 bytes of a 4-byte and they're in the bottom 2 bytes.
What you're wandering into is compiler/runtime-specific memory layouts. The same is true in Java. Java itself won't let you do something so horrendously un-Java, but if you write your own JVM or debug an existing JVM written in C or C++, you could do something like that. And who knows what would happen; I'm not Java god enough to know exactly how much the JVM spec pins down JVM implementation, but my guess is, not to the degree necessary to enable interoperability of the in-memory, runtime representations of objects.
So you get undefined behavior in every language flavor. Tastes just as good in each language, too - like mystery meat.
I want to try abusing Java classes as structures and for that I'm wondering if it is possible to serialize a byte array to a class and other way around.
So if I have a class like this:
public class Handshake
{
byte command;
byte error;
short size;
int major;
int ts;
char[] secret; // aligned size = 32 bytes
}
Is there an easy way (without having to manually read bytes and fill out the class which requires 3 times as much code) to deserialize a set of bytes into this class? I know that Java doesn't have structs but I'm wondering if it is possible to simplify the serialization process so it does it automatically. The bytes are not from Java's serializer, they are just aligned bytes derived from C structs.
The bytes are not from Java's serializer, they are just aligned bytes
derived from C structs.
Bad idea. It can break as soon as someone compiles that code on a different platform, using a different compiler or settings, etc.
Much better: use a standardized binary interface with implementations in Java and C++ like ASN.1 or Google's Protocol Buffers.
You can write a library to do the deserializtion using reflection. This may result in more code being required, but may suit your needs. It worth nothing that char in Java 16-bit rather than 8 bit and a char[] is a separate Object, unlike in C.
In short you can write a library which reads this data without touching the Handshake class. Only you can decide if this is actually easier than adding a method or two to the handshake class..
Do not do that! I will break sooner or later. Use some binary serialization format, like [Hessian][1], which supports both java and C++ (I'm not aware of anything that works on plain C)
Also remember C does not force size for int's or long's, they are platform dependent.
So if you must use C, and you are forced to write your own library, be very careful.
My problem is to serialize protobuf data in C++ and deserialize the data in Java probably.
Here is the code I use to the hints given by dcn:
With this I create the protobuf data in C++ and write it to an ostream which is send via socket.
Name name;
name.set_name("platzhirsch");
boost::asio::streambuf b;
std::ostream os(&b);
ZeroCopyOutputStream *raw_output = new OstreamOutputStream(&os);
CodedOutputStream *coded_output = new CodedOutputStream(raw_output);
coded_output->WriteLittleEndian32(name.ByteSize());
name.SerializeToCodedStream(coded_output);
socket.send(b);
This is the Java side where I try to parse it:
NameProtos.Name name = NameProtos.Name.parseDelimitedFrom(socket.getInputStream());
System.out.println(name.newBuilder().build().toString());
However by this I get this Exception:
com.google.protobuf.UninitializedMessageException: Message missing required fields: name
What am I missing?
The flawed code line is: name.newBuilder().build().toString()
This would have never worked, a new instance is created with uninitialized name field. Anyway the answer here solved the rest of my problem.
One last thing, which I was told in the protobuf mailinglist: In order to flush the CodedOutputStreams, the objects have to be deleted!
delete coded_output;
delete raw_output;
I don't know what received is in your Java code, but your problem may be due to some charset conversion. Note also that protobuf does not delimit the messages when serializing.
Therefore you should use raw data to transmit the messages (byte array or directly (de)serialize from/to streams).
If you intent to send many message you should also send the size before you send the actual messages.
In Java you can do it directly via parseDelimitedFrom(InputStream) and writeDelimitedTo(OutputStream). You can do the same in C++ a litte more complex using CodedOutputStream like
codedOutput.WriteVarint32(protoMessage.ByteSize());
protoMessage.SerializeToCodedStream(&codedOutput);
See also this ealier thread.
You're writing two things to the stream, a size and the Name object, but only trying to read one.
As a general question: why do you feel the need to use CodedInputStream? To quote the docs:
Typically these classes will only be
used internally by the protocol buffer
library in order to encode and decode
protocol buffers. Clients of the
library only need to know about this
class if they wish to write custom
message parsing or serialization
procedures
And to emphasize jtahlborn's comment: why little-endian? Java deals with big-endian values, so will have to convert on reading.
What is the 'correct' way to store a native pointer inside a Java object?
I could treat the pointer as a Java int, if I happen to know that native pointers are <= 32 bits in size, or a Java long if I happen to know that native pointers are <= 64 bits in size. But is there a better or cleaner way to do this?
Edit: Returning a native pointer from a JNI function is exactly what I don't want to do. I would rather return a Java object that represents the native resource. However, the Java object that I return must presumably have a field containing a pointer, which brings me back to the original question.
Or, alternatively, is there some better way for a JNI function to return a reference to a native resource?
IIRC, both java.util.zip and java.nio just use long.
java.nio.DirectByteBuffer does what you want.
Internally it uses a private long address to store pointer value. Dah !
Use JNI function env->NewDirectByteBuffer((void*) data, sizeof(MyNativeStruct)) to create a DirectByteBuffer on C/C++ side and return it to Java side as a ByteBuffer. Note: It's your job to free this data at native side! It miss the automatic Cleaner available on standard DirectBuffer.
At Java side, you can create a DirectByteBuffer this way :
ByteBuffer directBuff = ByteBuffer.allocateDirect(sizeInBytes);
Think it as sort of C's malloc(sizeInBytes). Note: It has as automatic Cleaner, which deallocates the memory previously requested.
But there are some points to consider about using DirectByteBuffer:
It can be Garbage Collected (GC) if you miss your direct ByteBuffer reference.
You can read/write values to pointed structure, but beware with both offset and data size. Compiler may add extra spaces for padding and break your assumed internal offsets in structure. Structure with pointers (stride is 4 or 8 bytes ?) also puzzle your data.
Direct ByteBuffers are very easy to pass as a parameter for native methods, as well to get it back as return.
You must cast to correct pointer type at JNI side. Default type returned by env->GetDirectBufferAddress(buffer) is void*.
You are unable to change pointer value once created.
Its your job to free memory previously allocated for buffers at native side. That ones you used with env->NewDirectByteBuffer().
There is no good way. In SWT, this code is used:
int /*long*/ hModule = OS.GetLibraryHandle ();
and there is a tool which converts the code between 32bit and 64bit by moving the comment. Ugly but it works. Things would have been much easier if Sun had added an object "NativePointer" or something like that but they didn't.
A better way might by to store it in a byte array, since native pointers aren't very Java-ish in the first place. ints and longs are better reserved for storing numeric values.
I assume that this is a pointer returned from some JNI code and my advice would be just dont do it :)
Ideally the JNI code should pass you back some sort of logical reference to the resource and not an actual pointer ?
As to your question there is nothing that comes to mind about a cleaner way to store the pointer - if you know what you have then use either the int or long or byte[] as required.
You could look to the way C# handles this with the IntPtr type. By creating your own type for holding pointers, the same type can be used as a 32-bit or 64-bit depending on the system you're on.