Develop Reference

Protobuf JsonFormater.printer convert long to String in Json

Currently I encounter a behavior of JsonFormater.printer printing the long(fixed64) value as String in JSON.
Is there any way/option to set the JsonFormater.printer not to do this conversion (Long(fixed64) -> String in Json)?
The Json is consumed by Java app, representing fixed64 as integer in JSON should not be a problem for Java.
Here is the code:
In data.proto
syntax = "proto2";
message data{
required fixed64 user_id = 1;
required int32 member_id = 2
}
Here is the java code, the file format is *.pb.gz
import com.google.protobuf.util.JsonFormat;
.......
//print data in JSON format
final InputStream input = new GZIPInputStream(new FileInputStream(pathToFile));
Message m;
m = defaultMsg.getParserForType().parseDelimitedFrom(input));
String jsonString = JsonFormat.printer().preservingProtoFieldNames().omittingInsignificantWhitespace().print(m);
Generated Java class: Data.java (Generated with protoc 2.6.1)
...
private long userId_;
...
private int memberId_;
...
expected result:
{"user_id":6546585813946021349,member_id":7521}
actual result:
{"user_id":"6546585813946021349",member_id":7521}
The user_id is String in json, but I want it as integer
Thanks
David

It seems this is by design, according to the source code. UINT64 and FIXED64 types are always printed out with surrounding double quotes, no questions asked:
https://github.com/protocolbuffers/protobuf/blob/f9d8138376765d229a32635c9209061e4e4aed8c/java/util/src/main/java/com/google/protobuf/util/JsonFormat.java#L1081-L1084
case INT64:
case SINT64:
case SFIXED64:
generator.print("\"" + ((Long) value).toString() + "\"");
In that same file, a few lines above, you can see that INT32 types are only double quoted if they're keys in a map (which your proto obviously doesn't have).
So, I'd ask for more information on the protobuf mailing list, or maybe report it as a bug/feature-request.

Convert json to dynamically generated protobuf in Java

Given the following json response:
{
"id" : "123456",
"name" : "John Doe",
"email" : "john.doe#example.com"
}
And the following user.proto file:
message User {
string id = 1;
string name = 2;
string email = 3;
}
I would like to have the possibility to dynamically create the protobuf message class (compile a .proto at runtime), so that if the json response gets enhanced with a field "phone" : "+1234567890" I could just upload a new version of the protobuf file to contain string phone = 4 and get that field exposed in the protobuf response, without a service restart.
If I were to pull these classes from a hat, I would like to be able to write something along the following code.
import com.googlecode.protobuf.format.JsonFormat;
import com.googlecode.protobuf.Message;
import org.apache.commons.io.FileUtils;
...
public Message convertToProto(InputStream jsonInputStream){
// get the latest user.proto file
String userProtoFile = FileUtils.readFileToString("user.proto");
Message userProtoMessage = com.acme.ProtobufUtils.compile(userProtoFile);
Message.Builder builder = userProtoMessage.newBuilderForType();
new JsonFormat().merge(jsonInputStream, Charset.forName("UTF-8"), builder);
return builder.build();
}
Is there an existing com.acme.ProtobufUtils.compile(...) method? Or how to implement one? Running a protoc + load class seems overkill, but I'm willing to use it if no other option...

You cannot compile the .proto file (at least not in Java), however you can pre-compile the .proto into a descriptor .desc
protoc --descriptor_set_out=user.desc user.proto
and then use the DynamicMessage's parser:
DynamicMessage.parseFrom(Descriptors.Descriptor type, byte[] data)
Source: google groups thread

Java classes generated by proto compiler fail to build

I have a simple .proto file, from which I generate java classes. The proto file look like this.
message Address {
string city = 1;
string country = 2;
}
message PersonalInfo {
string name = 1;
repeated Address adresses = 2;
}
The error is:
error: incompatible types: com.google.protobuf.GeneratedMessageV3.BuilderParent cannot be converted to com.google.protobuf.AbstractMessage.BuilderParent
getParentForChildren(),
^
I am using 3.1.0 to generate the classes and build the java source. Do I have something misconfigured, Is the proto file incorrect or is it a bug in proto?

How to convert a String FIX message to FIX FIX50SP2 format using QuickFixJ

Need a quick help. I am a newbie in QuickFixJ. I have a FIX message in a txt file. I need to convert that into FIX50SP2 format. I am enclosing the code snippet.
String fixMsg = "1128=99=25535=X49=CME34=47134052=20100318-03:21:11.36475=20120904268=2279=122=848=336683=607400107=ESU2269=1270=140575271=152273=121014000336=2346=521023=1279=122=848=336683=607401107=ESU2269=1270=140600271=206273=121014000336=2346=681023=210=159";
System.out.println("FixMsg String:"+fixMsg);
Message FIXMessage = new Message();
DataDictionary dd = new DataDictionary("FIX50SP2.xml");
FIXMessage.fromString(fixMsg, dd, false);
System.out.println("FIXMessage Output:" + FIXMessage.toString()); // Print message after parsing
MsgType msgType = new MsgType();
System.out.println(FIXMessage.getField(msgType));
Here is the output:
FixMsg String:1128=99=15835=X49=CME34=47164052=2012090312102051175=20120904268=1279=122=848=336683=607745107=ESU2269=1270=140575271=123273=121020000336=2346=501023=110=205
FIXMessage Output:9=6135=X34=47164049=CME52=2012090312102051175=20120904268=110=117
quickfix.FieldNotFound: Field [35] was not found in message.
at quickfix.FieldMap.getField(FieldMap.java:216)
at quickfix.FieldMap.getFieldInternal(FieldMap.java:353)
at quickfix.FieldMap.getField(FieldMap.java:349)
at MainApp.main(MainApp.java:52)
I want to extract MsgType field (field 35). Could you please tell me where I am wrong? The thing I have observed is that after parsing to FIX50SP2 format, the convert FIX message is missing many data element (for details see the output)
Thanks

Like others mentioned the MsgType is an header field and you get it by using the following
String msgType = null;
if(FIXMessage.getHeader().isSetField(MsgType.FIELD)) {
msgType = FIXMessage.getHeader().getString(MsgType.FIELD);
}
System.out.println("MsgType is " + msgType);`
The reason you are missing many data element after parsing is, probably your message have some custom tags(like tag 2346), which is not defined in your data dictionary(FIXSP02.xml). hence the parsing of those tags failed and missing in the output.
To fix this, get the data dictionary from the party that is sending you the message and use it to parse the message

I'm not familiar with FIX messages and QuickFixJ, but glancing at the Javadoc, it seems like you should use the identifyType method :
String fixMsg = "1128=99=25535=X49=CME34=47134052=20100318-03:21:11.36475=20120904268=2279=122=848=336683=607400107=ESU2269=1270=140575271=152273=121014000336=2346=521023=1279=122=848=336683=607401107=ESU2269=1270=140600271=206273=121014000336=2346=681023=210=159";
MsgType msgType = Message.identifyType(fixMsg);

You may find FixB framework useful as it deals well with non-standard use cases of FIX.
As in your case, to extract only data you are interested in, you need to define a class that will represent this data and to bind it to FIX using annotations. E.g.:
#FixBlock
public class MDEntry {
#FixField(tag=269) public int entryType; // you could define an enum type for it as well
#FixField(tag=278) public String entryId;
#FixField(tag=55) public String symbol;
}
...
FixFieldExtractor fixExtractor = new NativeFixFieldExtractor();
List<MDEntry> mdEntries = fixExtractor.getGroups(fixMsg, List.class, 268, FixMetaScanner.scanClass(MDEntry.class))
In more common cases, FixSerializer interface should be used, but it requires a message with MsgType(35) tag and a class annotated with #FixMessage(type="...") accordingly. E.g.:
#FixMessage(type="X")
public class MarketData {
#FixGroup(tag=268) public List<MDEntry> entries;
}
...
FixMetaDictionary fixMetaDictionary = FixMetaScanner.scanClassesIn("my.fix.classes.package");
FixSerializer fixSerializer = new NativeFixSerializer("FIX.5.0.SP2", fixMetaDictionary);
MarketData marketData = fixSerializer.deserialize(fixMsg);
I hope you will find it useful.

If you need just a MsgTyp, you're sure the message is correct and you do not need any other field from the message, then I would recommend extracting MsgType from string using regexp.
e.g.: \u000135=(\w+)\u0001
It is MUCH FASTER than parsing (and validating) a string via QuickFix.

error with serialization with protobuf

I'm trying to serialize a structure with protobuf. after many hours trying to figure out what I'm doing wrong I decided to test the google's example and it didn't worked as well
I have the following protocol from google (https://developers.google.com/protocol-buffers/docs/javatutorial):
package tutorial;
option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
repeated PhoneNumber phone = 4;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
}
message AddressBook {
repeated Person person = 1;
}
and I'm trying to serialize it with:
Person john = Person.newBuilder()
.setId(1234)
.setName("John Doe")
.setEmail("jdoe#example.com")
.addPhone(
Person.PhoneNumber.newBuilder()
.setNumber("555-4321")
.setType(Person.PhoneType.HOME))
.build();
byte[] serialized = john.toByteArray();
and I get "java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses."
Thanks;

As Marc said, A mismatch in Protocol Buffer versions will give you this exact message. In particular if
The .proto definition is converted to java using the 2.4.3 (or earlier) protoc.exe
You use the 2.5.0 protobuffers library
you will get this message in many methods (e.g. getParserForType, getUnknownFields) of class GeneratedMessage. There are no doubt other potential mismatch's that will cause this error
With protocol buffers 2.5.0 it is essential you regenerate all java classes with the 2.5.0 version of protoc (or on windows protoc.exe).
If you do the reverse - run code generated by protoc version 2.5 with the libraries for protocol buffers version 2.4. You will get the following message
java.lang.VerifyError: class xxx.xxx.xx..
overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;