Develop Reference

Protobuf JsonFormater.printer convert long to String in Json

Currently I encounter a behavior of JsonFormater.printer printing the long(fixed64) value as String in JSON.
Is there any way/option to set the JsonFormater.printer not to do this conversion (Long(fixed64) -> String in Json)?
The Json is consumed by Java app, representing fixed64 as integer in JSON should not be a problem for Java.
Here is the code:
In data.proto
syntax = "proto2";
message data{
required fixed64 user_id = 1;
required int32 member_id = 2
}
Here is the java code, the file format is *.pb.gz
import com.google.protobuf.util.JsonFormat;
.......
//print data in JSON format
final InputStream input = new GZIPInputStream(new FileInputStream(pathToFile));
Message m;
m = defaultMsg.getParserForType().parseDelimitedFrom(input));
String jsonString = JsonFormat.printer().preservingProtoFieldNames().omittingInsignificantWhitespace().print(m);
Generated Java class: Data.java (Generated with protoc 2.6.1)
...
private long userId_;
...
private int memberId_;
...
expected result:
{"user_id":6546585813946021349,member_id":7521}
actual result:
{"user_id":"6546585813946021349",member_id":7521}
The user_id is String in json, but I want it as integer
Thanks
David

It seems this is by design, according to the source code. UINT64 and FIXED64 types are always printed out with surrounding double quotes, no questions asked:
https://github.com/protocolbuffers/protobuf/blob/f9d8138376765d229a32635c9209061e4e4aed8c/java/util/src/main/java/com/google/protobuf/util/JsonFormat.java#L1081-L1084
case INT64:
case SINT64:
case SFIXED64:
generator.print("\"" + ((Long) value).toString() + "\"");
In that same file, a few lines above, you can see that INT32 types are only double quoted if they're keys in a map (which your proto obviously doesn't have).
So, I'd ask for more information on the protobuf mailing list, or maybe report it as a bug/feature-request.

Convert json to dynamically generated protobuf in Java

Given the following json response:
{
"id" : "123456",
"name" : "John Doe",
"email" : "john.doe#example.com"
}
And the following user.proto file:
message User {
string id = 1;
string name = 2;
string email = 3;
}
I would like to have the possibility to dynamically create the protobuf message class (compile a .proto at runtime), so that if the json response gets enhanced with a field "phone" : "+1234567890" I could just upload a new version of the protobuf file to contain string phone = 4 and get that field exposed in the protobuf response, without a service restart.
If I were to pull these classes from a hat, I would like to be able to write something along the following code.
import com.googlecode.protobuf.format.JsonFormat;
import com.googlecode.protobuf.Message;
import org.apache.commons.io.FileUtils;
...
public Message convertToProto(InputStream jsonInputStream){
// get the latest user.proto file
String userProtoFile = FileUtils.readFileToString("user.proto");
Message userProtoMessage = com.acme.ProtobufUtils.compile(userProtoFile);
Message.Builder builder = userProtoMessage.newBuilderForType();
new JsonFormat().merge(jsonInputStream, Charset.forName("UTF-8"), builder);
return builder.build();
}
Is there an existing com.acme.ProtobufUtils.compile(...) method? Or how to implement one? Running a protoc + load class seems overkill, but I'm willing to use it if no other option...

You cannot compile the .proto file (at least not in Java), however you can pre-compile the .proto into a descriptor .desc
protoc --descriptor_set_out=user.desc user.proto
and then use the DynamicMessage's parser:
DynamicMessage.parseFrom(Descriptors.Descriptor type, byte[] data)
Source: google groups thread

Get token string from tokenID using Stanford Parser in GATE

I am trying to use some Java RHS to get the string value of dependent tokens using Stanford dependency parser in GATE, and add them as features of a new annotation.
I am having problems targeting just the 'dependencies' feature of the token, and getting the string value from the tokenID.
Using below specifying only 'depdencies' also throws a java null pointer error:
for(Annotation lookupAnn : tokens.inDocumentOrder())
{
FeatureMap lookupFeatures = lookupAnn.getFeatures();
token = lookupFeatures.get("dependencies").toString();
}
I can use below to get all the features of a token,
gate.Utils.inDocumentOrder
but it returns all features, including the dependent tokenID's; i.e:
dependencies = [nsubj(8390), dobj(8394)]
I would like to get just the dependent token's string value from these tokenID's.
Is there any way to access dependent token string value and add them as a feature to the annotation?
Many thanks for your help

Here is a working JAPE example. It only printns to the GATE's message window (std out), It doesn't create any new annotations with features you asked for. Please finish it yourself...
Stanford_CoreNLP plugin has to be loaded in GATE to make this JAPE file loadable. Otherwise you will get class not found exception for DependencyRelation class.
Imports: {
import gate.stanford.DependencyRelation;
}
Phase: GetTokenDepsPhase
Input: Token
Options: control = all
Rule: GetTokenDepsRule
(
{Token}
): token
-->
:token {
//note that tokenAnnots contains only a single annotation so the loop could be avoided...
for (Annotation token : tokenAnnots) {
Object deps = token.getFeatures().get("dependencies");
//sometimes the dependencies feature is missing - skip it
if (deps == null) continue;
//token.getFeatures().get("string") could be used instead of gate.Utils.stringFor(doc,token)...
System.out.println("Dependencies for token " + gate.Utils.stringFor(doc, token));
//the dependencies feature has to be typed to List<DependencyRelation>
List<DependencyRelation> typedDeps = (List<DependencyRelation>) deps;
for (DependencyRelation r : typedDeps) {
//use DependencyRelation.getTargetId() to get the id of the target token
//use inputAS.get(id) to get the annotation for its id
Annotation targetToken = inputAS.get(r.getTargetId());
//use DependencyRelation.getType() to get the dependency type
System.out.println(" " +r.getType()+ ": " +gate.Utils.stringFor(doc, targetToken));
}
}
}

Java classes generated by proto compiler fail to build

I have a simple .proto file, from which I generate java classes. The proto file look like this.
message Address {
string city = 1;
string country = 2;
}
message PersonalInfo {
string name = 1;
repeated Address adresses = 2;
}
The error is:
error: incompatible types: com.google.protobuf.GeneratedMessageV3.BuilderParent cannot be converted to com.google.protobuf.AbstractMessage.BuilderParent
getParentForChildren(),
^
I am using 3.1.0 to generate the classes and build the java source. Do I have something misconfigured, Is the proto file incorrect or is it a bug in proto?

Compatibility between Protobuf and Protostuff

are Classes generated by the Protostuff code generator compatible with those created by Protobuf?
I tried to (de)serialize some simple messages and got several exceptions:
Proto-File (WrapperClass.proto)
package tutorial;
option java_package = "com.example.tutorial";
message ProjectId {
required int32 id = 1;
}
message UserId {
required ProjectId project = 1;
required int32 projectUserId = 2;
}
message ChannelId {
required ProjectId project = 1;
required string name = 2;
}
Protostuff to Protobuf Test (example)
ProjectId projectId = new ProjectId(1);
byte[] projectarray = ProtostuffIOUtil.toByteArray(projectId, ProjectId.getSchema(), buffer);
com.example.tutorial.WrapperClass.ProjectId returnBufProject = com.example.tutorial.WrapperClass.ProjectId.parseFrom(projectarray);
Problem:
Everything works for ProjectId, but for UserId and ChannelId (everything a little bit more complex), i get:
com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: project
at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81)
at com.example.tutorial.WrapperClass$ChannelId$Builder.buildParsed(Test.java:1278)
at com.example.tutorial.WrapperClass$ChannelId$Builder.access$17(Test.java:1273)
at com.example.tutorial.WrapperClass$ChannelId.parseFrom(Test.java:1142)
...
And the other way around:
Protobuf to Protostuff Test (example)
com.example.tutorial.WrapperClass.ProjectId projectId2 = com.example.tutorial.WrapperClass.ProjectId.newBuilder().setId(1).build();
byte[] project2array = projectId2.toByteArray();
ProjectId returnStufProject = new ProjectId();
ProtostuffIOUtil.mergeFrom(project2array, returnStufProject, ProjectId.getSchema());
Problem
again, for everything other than the ProjectId, there is an exception:
java.lang.RuntimeException: Reading from a byte array threw an IOException (should never happen).
at com.dyuproject.protostuff.IOUtil.mergeFrom(IOUtil.java:53)
at com.dyuproject.protostuff.ProtostuffIOUtil.mergeFrom(ProtostuffIOUtil.java:96)
at JacksonTest.main(JacksonTest.java:92)
Caused by: com.dyuproject.protostuff.ProtobufException: Protocol message contained an invalid tag (zero).
at com.dyuproject.protostuff.ProtobufException.invalidTag(ProtobufException.java:98)
at com.dyuproject.protostuff.ByteArrayInput.readFieldNumber(ByteArrayInput.java:220)
at com.example.tutorial.ProjectId$1.mergeFrom(ProjectId.java:115)
at com.example.tutorial.ProjectId$1.mergeFrom(ProjectId.java:1)
at com.dyuproject.protostuff.ByteArrayInput.mergeObjectEncodedAsGroup(ByteArrayInput.java:390)
at com.dyuproject.protostuff.ByteArrayInput.mergeObject(ByteArrayInput.java:362)
at com.example.tutorial.UserId$1.mergeFrom(UserId.java:138)
at com.example.tutorial.UserId$1.mergeFrom(UserId.java:1)
at com.dyuproject.protostuff.IOUtil.mergeFrom(IOUtil.java:43)
... 2 more
Am i trying something impossible or do i only do something wrong?

The problem was simple:
Instead of using ProtostuffIOUtil to (de)serialize my messages i need to use ProtobufIOUtil