I'm new to Google Protobuf. I want to solve an issue and i can't find the answer on the internet, maybe i'm asking the wrong questions..
I want to build a tool which uses a .proto schema (and the generated classes) and convert input strings from readable to base64 string and the reverse (with java)
the purpose is to debug the messages. I don't know exactly which message is encoded (so which class to use..)
how can i proceed please? could you give me some pointers, links to projects or anything that may help me..
edit: for the moment, i will try to use ParseFromString function, with every class until i find the protobuf class that matches the base64 blob ..
Thank you
Protobuf can be used to serialize data on 1 machine and send it to the 2nd machine where it is de-serialized to the original form.
Protobuf uses its own logic to serialize things like Strings, Signed/Unsigned number values, etc
So knowing this, protobuf cannot/shouldnot be used as a converter to convert things into other things. It is used to transport data accross machine boundaries.
Maybe you just want to convert a String into BASE64 representation without protobuf? Then the hint of Thomas Timbul might be useful for you.
Related
Currently I've a job to rewrite some library which communicate with COBOL Mainframe via ViewCall (Not sure if this is a universal term or not). The response will be a fixed length string according to the copybook. Are there any solutions or approach for Java to map this fixed length string to model class using copybook as a mapper instead of manually cut the strings and set it to model class?
Some possible Solutions:
For small Text Copybooks hand coding is feasible.
next step up is to use cb2xml and generate the code. Cb2xml will calculate position and length for you. This answer shows what can be done with cb2xml.
Use JRecord ~ CodeGen to generate java classes. See Generating Java Code for details on generating Java code. Java Jrecord is orientated to Files but should be usable.
Remember there is a fair overhead in passing the Cobol copybook.
For a one off request with a small record, hand coding is viable option but you run the risk of subsequent requests coming along.
I created the following Thrift Object:
struct Student{
1: string id;
2: string firstName;
3: string lastName
}
Now I would like to read this object from JSON. According to this post this is possible
So I wrote the following code:
String json = "{\"id\":\"aaa\",\"firstName\":\"Danny\",\"lastName\":\"Lesnik\"}";
StudentThriftObject s = new StudentThriftObject();
byte[] jsonAsByte = json.getBytes("UTF-8");
TMemoryBuffer memBuffer = new TMemoryBuffer(jsonAsByte.length);
memBuffer.write(jsonAsByte);
TProtocol proto = new TJSONProtocol(memBuffer);
s.read(proto);
What I'm getting is the following exception:
Exception in thread "main" org.apache.thrift.protocol.TProtocolException: Unexpected character:i
at org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:322)
at org.apache.thrift.protocol.TJSONProtocol.readJSONInteger(TJSONProtocol.java:698)
at org.apache.thrift.protocol.TJSONProtocol.readFieldBegin(TJSONProtocol.java:837)
at com.vanilla.thrift.example.entities.StudentThriftObject$StudentThriftObjectStandardScheme.read(StudentThriftObject.java:486)
at com.vanilla.thrift.example.entities.StudentThriftObject$StudentThriftObjectStandardScheme.read(StudentThriftObject.java:479)
at com.vanilla.thrift.example.entities.StudentThriftObject.read(StudentThriftObject.java:413)
at com.vanilla.thrift.controller.Main.main(Main.java:24)
Am I missing something?
You are missing the fact, that Thrift's JSON is different from yours. The field names are not written, instead the assigned field ID numbers are written (and expected). Here's an example for Thrift's JSON protocol:
[1,"MyService",2,1,{"1":{"rec":{"1":{"str":"Error: Process() failed"}}}}]
In other words, Thrift is not intended to parse any kind of JSON. It supports a very specific JSON format as one of the possible transports.
However, depending on what the origin of your JSON data is, Thrift can possibly still help you out, if you are able to use it on both sides. In that case, write an IDL to describe the data structures, feed it to the Thrift compiler and integrate both the generated code and the neccessary parts of the library with your projects.
If the origin of the JSON lies outside of your reach, or if the JSON format cannot be changed for some reason, you need to find another way.
Format and semantics are different beasts
To some extent, the whole issue can be compared with XML: There is one general XML syntax, which tells us how we have to fomat things so any standard conformant XML processor can read them.
But knowing the rules of XML is only half the answer, if we get a certain XML file from someone. Even if our XML parser can read the file successfully, because it is well-formed XML, we need to know the semantics of the data to really make use of what's within that file: Is it a customer data record? Or is it a SOAP envelope? Maybe a configuration file?
That is where DTDs or XML Schema come into play, they exist to describe the contents of the XML data. Without knowing the logical structure you are lost, because there are myriads of possible ways to express things in XML. And exactly the same is true with JSON, except that JSON schema descriptions are less commonly used.
"So you mean, we need just a way to tell Thrift how the JSON is organized?"
No, because the purpose and idea behind Thrift is to have a framework to de/serialize things and/or implement RPC servers and clients as efficiently as possible. It is not intended to have a general purpose file parser. Instead, Thrift reads and speaks only its own set of formats, which are plugged into the architecture as protocols: Thrift Binary, Thrift JSON, Thrift Compact, and a few more.
What you could do: In addition to what I said at in the first section of my answer, you may consider writing your own custom Thrift protocol implementation to support your particular JSON format of choice. It is not that hard, and worth a try.
For my project, I need to store info about protocols (the data sent (most likely integers) and in the order it's sent) and info that might be formatted something like this:
'ID' 'STRING' 'ADDITIONAL INTEGER DATA'
This info will be read by a Java program and stored in memory for processing, but I don't know what would be the most sensible format to store this data in?
EDIT: Here's some extra information:
1)I will be using this data in a game server.
2)Since it is a game server, speed is not the primary concern, since this data will primary be read and utilized during startup, which shouldn't occur very often.
3)Memory consumption I would like to keep at a minimum, however.
4)The second data "example" will be used as a "dictionary" to look up names of specific in-game items, their stats and other integer data (and therefore might become very large, unlike the first data containing the protocol information, where each file will only note small protocol bites, like a login protocol for instance).
5)And yes, I would like the data to be "human-editable".
EDIT 2: Here's the choices that I've made:
JSON - For the protocol descriptions
CSV - For the dictionaries
There are many factors that could come to weigh--here are things that might help you figure this out:
1) Speed/memory usage: If the data needs to load very quickly or is very large, you'll probably want to consider rolling your own binary format.
2) Portability/compatibility: Balanced against #1 is the consideration that you might want to use the data elsewhere, with programs that won't read a custom binary format. In this case, your heavy hitters are probably going to be CSV, dBase, XML, and my personal favorite, JSON.
3) Simplicity: Delimited formats like CSV are easy to read, write, and edit by hand. Either use double-quoting with proper escaping or choose a delimiter that will not appear in the data.
If you could post more info about your situation and how important these factors are, we might be able to guide you further.
How about XML, JSON or CSV ?
I've written a similar protocol-specification using XML. (Available here.)
I think it is a good match, since it captures the hierarchal nature of specifying messages / network packages / fields etc. Order of fields are well defined and so on.
I even wrote a code-generator that generated the message sending / receiving classes with methods for each message type in XSLT.
The only drawback as I see it is the verbosity. If you have a really simple structure of the specification, I would suggest you use some simple home-brewed format and write a parser for it using a parser-generator of your choice.
In addition to the formats suggested by others here (CSV, XML, JSON, etc.) you might consider storing the info in a Java properties file. (See the java.util.Properties class.) The code is already there for you, so all you have to figure out is the properties names (or name prefixes) you want to use.
The Properties class also provides for storing/loading properties in a simple XML format.
I have a Java module that is receiving a compressed string from a remote Python script. The Python script compresses the string using zlib.compress(). I simply want to uncompress it in Java and display it to the user.
The man page for Java's built-in zip.Deflater object describes pretty explicitly how to uncompress something that has been compressed using zlib.compress(). However, this method does not work for me. Depending on which encoding I use, I either get "Incorrect Header Check" errors or the uncompression returns an empty string.
So, how am I supposed to uncompress this? The data are not getting corrupted in transmission, and the compressed string begins with "x\x9c", which is apparently appropriate for zlib-compressed stuff.
I've never dealt with compression/uncompression on this level before and am getting confused. For extra credit, I'd appreciate an explanation between compressed/uncompressed and inflated/deflated. According to this they are different, but most of the internet seems to use them interchangeably for zlib. This just makes trying to find a solution even more difficult, as I couldn't tell you whether I'm actually trying to "uncompress" or "inflate" these data.
The confusion has arisen because some bright spark started describing the zlib protocol as "deflate". It might help you to read the RFCs mentioned in these Java docs.
Also this SO topic is quite relevant.
I suggest that you do
print repr(zlib.compress("The quick brown dog etc etc")
in Python (A) and compare the result from using the equivalent Java code using Deflater (B). Also ensure that you can Inflate B to recover your test input. Check that you are not suffering from unicode <-> bytes complications in Python or Java or both.
Have you tried doing a Python "deflate" as per the answer by #patthoyts in the SO topic that you quoted?
It seems Python's zlib.compress() uses gzip, are you sure to create Inflater with nowrap parameter for gzip compatible uncompression?
Inflate/deflate is used only regarding DEFLATE algorithm I believe, whereas compress/uncompress is more general term.
Is there any way to deserialize in PHP an object serialized in Java? IE If I have a Java class that implements Serialization and I use an ObjectOutputStream to write the object, and convert the result to a string, is there a way in PHP to take that string and create a similar object representation from it?
What does the Java Serialized data look like?
Response:
���sr�com.site.entity.SessionV3Data���������xpsr�java.util.HashMap���`��F�
loadFactorI� thresholdxp?#�����w������t� sessionIdt�0NmViMzUxYWItZDRmZC00MWY4LWFlMmUtZjg2YmZjZGUxNjg5xx
:)
I would heavily recommend you don't do this. Java serialization is meant for a Java instance to both save and load the data (for either transmission to another Java application or persistence between invocations of the same application). It was not at all meant to be a cross-platform protocol.
I would advise you to make an API adapter layer between the two. Output the contents of your Java object to a format you can work with in PHP, be it XML, YAML, or even a binary format (where you could use DataOutputStream).
What is the easiest way to eat soup with chopsticks when the soup was put in a bowl with a ladle? Put the soup in a cup and discard your chopsticks, because chopsticks are a poor choice for aiding in the consumption of soup. A cup (ubiquitous) eliminates external dependencies except for "mouth" and "opposable thumbs", both of which come with the standard library of humans.
A more elegant solution would be to encode that Java object with a JSON Serializer or XML serializer. Protocol Buffers or any other intentionally cross-language serialization technique would work fine plus Protocol Buffers can efficiently encode binary data.
Some time ago i did something simillar. However i didn't make PHP read "Java serialize" format. I did the oposite, that is, made Java serialize itself to a "PHP serialize" format. This is actually quite easy. Have look at PHPSerializedResponseWriter class that is a part of Solr package:
https://github.com/terrancesnyder/solr-analytics/blob/master/solr/core/src/java/org/apache/solr/response/PHPSerializedResponseWriter.java
...then all you have to do is just read the string and call:
$result = unserialize($string);
From comments in the online PHP manual, there is a Java class that serializes to the PHP serialization format that you can look into. Then you can unserialize the data using the standard PHP functionality.
Is it possible to use one of the more common cross platform data formats like JSON to communicate between your Java app and PHP? PHP has plenty of parsers for those formats. Check out json_decode for an example.
Is there any way to deserialize in PHP
an object serialized in Java?
Yes. The question is, should you? Exporting the Java object as XML or JSON probably makes more sense.
The following SO question might also help.
Dynamically create PHP object based on string