Given the following json response:
{
"id" : "123456",
"name" : "John Doe",
"email" : "john.doe#example.com"
}
And the following user.proto file:
message User {
string id = 1;
string name = 2;
string email = 3;
}
I would like to have the possibility to dynamically create the protobuf message class (compile a .proto at runtime), so that if the json response gets enhanced with a field "phone" : "+1234567890" I could just upload a new version of the protobuf file to contain string phone = 4 and get that field exposed in the protobuf response, without a service restart.
If I were to pull these classes from a hat, I would like to be able to write something along the following code.
import com.googlecode.protobuf.format.JsonFormat;
import com.googlecode.protobuf.Message;
import org.apache.commons.io.FileUtils;
...
public Message convertToProto(InputStream jsonInputStream){
// get the latest user.proto file
String userProtoFile = FileUtils.readFileToString("user.proto");
Message userProtoMessage = com.acme.ProtobufUtils.compile(userProtoFile);
Message.Builder builder = userProtoMessage.newBuilderForType();
new JsonFormat().merge(jsonInputStream, Charset.forName("UTF-8"), builder);
return builder.build();
}
Is there an existing com.acme.ProtobufUtils.compile(...) method? Or how to implement one? Running a protoc + load class seems overkill, but I'm willing to use it if no other option...
You cannot compile the .proto file (at least not in Java), however you can pre-compile the .proto into a descriptor .desc
protoc --descriptor_set_out=user.desc user.proto
and then use the DynamicMessage's parser:
DynamicMessage.parseFrom(Descriptors.Descriptor type, byte[] data)
Source: google groups thread
Related
The task:
I need to perform a post request to an endpoint.
Request body type is JSON:
{
"id": "${__UUID()}"
}
I want to simulate 10 users that each will send a payload with unique "id" field generated.
From JMeter GUI it works as expected, but from Java Code it seems not to be recognized and reads it like a simple string or so.
Here is the Java Code:
public JMeterClient httpSamplerProxy(String name, String endpoint, String payload, String httpMethod) {
Arguments arguments = new Arguments();
HTTPArgument httpArgument = new HTTPArgument();
httpArgument.setMetaData("=");
httpArgument.setValue(payload);
List<Argument> args = new ArrayList<>();
args.add(httpArgument);
arguments.setArguments(args);
HTTPSamplerProxy httpSamplerProxy = new HTTPSamplerProxy();
httpSamplerProxy.setProperty(TestElement.TEST_CLASS, HTTPSamplerProxy.class.getName());
httpSamplerProxy.setProperty(TestElement.GUI_CLASS, HttpTestSampleGui.class.getName());
httpSamplerProxy.setName(name);
httpSamplerProxy.setEnabled(true);
httpSamplerProxy.setPostBodyRaw(true);
httpSamplerProxy.setFollowRedirects(true);
httpSamplerProxy.setAutoRedirects(false);
httpSamplerProxy.setUseKeepAlive(true);
httpSamplerProxy.setDoMultipart(false);
httpSamplerProxy.setPath(endpoint);
httpSamplerProxy.setMethod(httpMethod);
httpSamplerProxy.setArguments(arguments);
httpSamplerProxies.add(httpSamplerProxy);
return this;
}
where payload is a json in a string representative.
I Use JMeter 5.4.1
Besides that I need this to work, how can I enable logging of Post Body in Java to see it in console?
I was missing dependency of functions:
<dependency>
<groupId>org.apache.jmeter</groupId>
<artifactId>ApacheJMeter_functions</artifactId>
<version>5.4.1</version>
</dependency>
After adding this to class path the function got recognized
Currently I have a working code implementation of submitting an application to Yarn using spark.deploy.yarn.Client. It's complex to aggregate all the arguments this client needs, but the submission of the application is simple:
ClientArguments cArgs = new ClientArguments(args.toArray(new String[0]));
client = new Client(cArgs, sparkConf);
applicationID = client.submitApplication();
Most of the code before this point was accumulating the sparkConf and args. Now I wish to retire the Client and work with Rest only. Spark offers a full REST api including submitting applications - according to the Spark documentation it's a matter of this easy json/xml post:
POST http://<rm http address:port>/ws/v1/cluster/apps
Accept: application/json
Content-Type: application/json
{
"application-id":"application_1404203615263_0001",
"application-name":"test",
"am-container-spec":
{
"local-resources":
{
"entry":
[
{
"key":"AppMaster.jar",
"value":
{
"resource":"hdfs://hdfs-namenode:9000/user/testuser/DistributedShell/demo-app/AppMaster.jar",
"type":"FILE",
"visibility":"APPLICATION",
"size": 43004,
"timestamp": 1405452071209
}
}
]
},
"commands":
{
"command":"{{JAVA_HOME}}/bin/java -Xmx10m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --container_vcores 1 --num_containers 1 --priority 0 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr"
},
"environment":
{
"entry":
[
{
"key": "DISTRIBUTEDSHELLSCRIPTTIMESTAMP",
"value": "1405459400754"
},
{
"key": "CLASSPATH",
"value": "{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties"
},
{
"key": "DISTRIBUTEDSHELLSCRIPTLEN",
"value": "6"
},
{
"key": "DISTRIBUTEDSHELLSCRIPTLOCATION",
"value": "hdfs://hdfs-namenode:9000/user/testuser/demo-app/shellCommands"
}
]
}
},
"unmanaged-AM":false,
"max-app-attempts":2,
"resource":
{
"memory":1024,
"vCores":1
},
"application-type":"YARN",
"keep-containers-across-application-attempts":false,
"log-aggregation-context":
{
"log-include-pattern":"file1",
"log-exclude-pattern":"file2",
"rolled-log-include-pattern":"file3",
"rolled-log-exclude-pattern":"file4",
"log-aggregation-policy-class-name":"org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AllContainerLogAggregationPolicy",
"log-aggregation-policy-parameters":""
},
"attempt-failures-validity-interval":3600000,
"reservation-id":"reservation_1454114874_1",
"am-black-listing-requests":
{
"am-black-listing-enabled":true,
"disable-failure-threshold":0.01
}
}
I tried to translate my arguments into this JSON body of the POST request, but it seems impossible. Does anyone know if I can reverse-engineer from a running application I submitted the JSON payload to send via REST? Or what mapping I could use to take the Client arguments and place them in the JSON?
After a little searching, I managed to submit an application from the REST API only. It's not a well-documented process, so I'm posting it here.
NOTE: if at any time you wish to compare the content of the request to the request sent by the client, use debug breakpoints to inspect the application context used by the Client.
Open the class org.apache.hadoop.yarn.client.api.impl.YarnClientImpl and go to the method submitApplication(ApplicationSubmissionContext appContext).
Firstly, to replace the spark.deploy.yarn.Client with a REST API request, the solution must make sure all the files mentioned in the configuration are available on the HDFS.
Later, it needs to compose and upload one extra file called __spark_conf__.zip.
Step 1
Go over the files from the SparkConf (the Client's 2nd argument): files mentioned in "AllJars" tag, file mentioned in "mainJarPath", and files mentioned in "FilesList".
For each file, check if it exists in the HDFS and if not - upload it from the local machine. For each file get its FileStatus from the HDFS.
aggregate the resources list, which is an attribute map for each file containing these 6 attributes:
size = getLen()
timestamp = getModificationTime()
type=FILE
visibility=PUBLIC
Two other attributes: key and resource.
Files from the allJars list: Key is spark_libs/{{filename}}, and the resource is the filename.
Files from the FilesList: key is "localEntry" tag, resource is "hdfsPath" tag.
the File in mainJarPath: key is "app.jar", resource is the filename.
Step 2
Creating the __spark_conf__.zip file. You can create it directly in the hdfs, in the staging path which is usually {{HDFS_base_folder}}/user/{{username}}/.sparkStaging/{{application_id}}/__spark_conf__.zip.
This archive file contains two files and one empty directory: one file __spark_hadoop_conf__.xml (a rename to core-site.xml), and the other file is called __spark_conf__.properties which is a slightly modified version
of the sparkConf section from the configuration.
To create __spark_conf__.properties you will need to read the JSON map from "sparkConf"->"org$apache$spark$SparkConf$$settings", and convert each line from the JSON format "spark.safemine.addcontrol.driverMemory": "5120M"
to spark.safemine.addcontrol.driverMemory=5120M
To the bottom of the file add 6 new lines:
spark.yarn.cache.confArchive={{the location to which you will upload __spark_conf__.zip in the sparkStaging}}
spark.yarn.cache.visibilities={{all the visibilities of the files, comma delimited - basically "PUBLIC,PUBLIC, ... ,PUBLIC"}}
spark.yarn.cache.timestamps={{All the timestamps for the files, comma delimited}}
spark.yarn.cache.types={{all the types of the files, comma delimited - basically "FILE,FILE, ... ,FILE"}}
spark.yarn.cache.filenames={{All the filenames and keys, recorded as resource#key and comma delimited}}
spark.yarn.cache.sizes={{All the sizes for the files, comma delimited}}
Make sure you compile the 5 aggregated lines in respective order. I used this code:
String confArchive = "spark.yarn.cache.confArchive="+hdfs+"/user/"+userName+"/.sparkStaging/"+applicationId+"/__spark_conf__.zip";
String filenames = "spark.yarn.cache.filenames=";
String sizes = "spark.yarn.cache.sizes=";
String timestamps = "spark.yarn.cache.timestamps=";
String types = "spark.yarn.cache.types=";
String visibilities = "spark.yarn.cache.visibilities=";
for (Map<String,String> localResource:localResources) {
filenames+=localResource.get("resource")+"#"+localResource.get("key")+",";
sizes+=localResource.get("size")+",";
timestamps+=localResource.get("timestamp")+",";
types+=localResource.get("type")+",";
visibilities+=localResource.get("visibility")+",";
}
properties+=confArchive+"\n";
properties+=filenames.substring(0,filenames.length()-1)+"\n";
properties+=sizes.substring(0,sizes.length()-1)+"\n";
properties+=timestamps.substring(0,timestamps.length()-1)+"\n";
properties+=types.substring(0,types.length()-1)+"\n";
properties+=visibilities.substring(0,visibilities.length()-1)+"\n";
The __spark_hadoop_conf__.xml file is a simple rename of core-site.xml, and the folder created with them is named __hadoop_conf__ and is left empty.
you can save the files to the hdfs directly like so:
private void generateSparkConfInHdfs(String applicationId, String userName, String sparkConfProperties, String sparkHadoopConf) throws IOException {
String path = hdfs+"/user/"+userName+"/.sparkStaging/"+applicationId+"/__spark_conf__.zip";
Path hdfsPath = new Path(path);
ZipOutputStream os = new ZipOutputStream(getHdfs().create(hdfsPath));
os.putNextEntry(new ZipEntry("__hadoop_conf__/"));
os.putNextEntry(new ZipEntry("__spark_conf__.properties"));
os.write(sparkConfProperties.getBytes(),0,sparkConfProperties.getBytes().length);
os.putNextEntry(new ZipEntry("__spark_hadoop_conf__.xml"));
os.write(sparkHadoopConf.getBytes(),0,sparkHadoopConf.getBytes().length);
os.close();
}
After you finish creating the file, add it to the resources list with these specifications:
size = getLen()
timestamp = getModificationTime()
type = ARCHIVE
visibility = PRIVATE
key = __spark_conf__
resource is the staging directory (usually {{HDFS_base_folder}}/user/{{username}}/.sparkStaging/{{application_id}}/__spark_conf__.zip).
Go over the full resources list and create an XML/JSON from them with this structure for each one, using the values we collected in the {{}} placeholders:
<entry>
<key>{{key}}</key>
<value>
<resource>{{resource}}</resource>
<size>{{size}}</size>
<timestamp>{{timestamp}}</timestamp>
<type>{{type}}</type>
<visibility>{{visibility}}</visibility>
</value>
</entry>
The accumulated string will be your localResources XML segment shown below.
Step 3
Generating the Java command. You will need to extract a few elements from the SparkConfig:
driverMemory - from the same attribute in the sparkConf
extraJavaOptions = from spark.driver.extraJavaOptions within the attribute collection
mainClass - from the same attribute in the sparkConf
argstr - collect all the ClientArgs except the --class one.
The result command with the elements included is:
String command = "$JAVA_HOME/bin/java -server -Xmx"+driverMemory+" -Djava.io.tmpdir=$PWD/tmp "+extraJavaOptions+" -Dspark.yarn.app.container.log.dir=<LOG_DIR> "
+ "org.apache.spark.deploy.yarn.ApplicationMaster --class "+mainClass+" "+argstr+" "
+ "--properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr";
Step 4
Compiling the request XML.
NOTE: my implementation requires a label on the AM container, so am-container-node-label-expression is added. This will not be applicable in all cases.
The mapping from the sparkConf to the REST request is (shown here in XML, JSON implementation is also supported):
<application-submission-context>
<application-id>"+applicationId+"</application-id>
<application-name>"+appName+"</application-name>
<queue>default</queue>
<priority>0</priority>
<am-container-spec>
<local-resources>+localResources+</local-resources>
<environment>
<entry>
<key>SPARK_YARN_STAGING_DIR</key>
<value>"+hdfs+"/user/"+userName+"/.sparkStaging/"+applicationId+"</value>
</entry>
<entry>
<key>CLASSPATH</key>
<value>$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/spark-non-hdfs-storage/spark-assembly-2.3.0-hadoop2.7/*:%HADOOP_CONF_DIR%:%HADOOP_COMMON_HOME%/share/hadoop/common/*:%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*:%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*:%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*:%HADOOP_YARN_HOME%/share/hadoop/yarn/*:%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*:%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*:%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*:$PWD/__spark_conf__/__hadoop_conf__</value>
</entry>
<entry>
<key>SPARK_USER</key>
<value>"+userName+"</value>
</entry>
</environment>
<commands>
<command>"+command+"</command>
</commands>
</am-container-spec>
<unmanaged-AM>false</unmanaged-AM>
<max-app-attempts>1</max-app-attempts>
<resource>
<memory>5632</memory>
<vCores>1</vCores>
</resource>
<application-type>SPARK</application-type>
<keep-containers-across-application-attempts>false</keep-containers-across-application-attempts>
<application-tags>
<tag>"+sparkYarnTag+"</tag>
</application-tags>
<am-container-node-label-expression>appMngr</am-container-node-label-expression>
<log-aggregation-context/>
<attempt-failures-validity-interval>1</attempt-failures-validity-interval>
<reservation-id/>
</application-submission-context>
Step 5:
Submitting the application via REST http PUT:
private void submitApplication (String body, String userName) throws SMSparkManagerException {
HttpClient client = HttpClientBuilder.create().build();
HttpPost request = new HttpPost(uri+"?user.name="+userName);
try {
request.setEntity(new StringEntity(body, ContentType.APPLICATION_XML));
HttpResponse response = client.execute(request);
if (response.getStatusLine().getStatusCode()!=202) {
throw new SMSparkManagerException("The application could not be submitted to Yarn, response http code "+response.getStatusLine().getStatusCode());
}
} catch (UnsupportedEncodingException e) {
logger.error("The application Could not be submitted due to UnsupportedEncodingException in the provided body: "+body, e );
throw new SMSparkManagerException("Error in submitting application to yarn");
} catch (ClientProtocolException e) {
logger.error("The application Could not be submitted due to ClientProtocolException", e);
throw new SMSparkManagerException("Error in submitting application to yarn");
} catch (IOException e) {
logger.error("The application Could not be submitted due to IOException", e);
throw new SMSparkManagerException("Error in submitting application to yarn");
}
}
I am trying to use some Java RHS to get the string value of dependent tokens using Stanford dependency parser in GATE, and add them as features of a new annotation.
I am having problems targeting just the 'dependencies' feature of the token, and getting the string value from the tokenID.
Using below specifying only 'depdencies' also throws a java null pointer error:
for(Annotation lookupAnn : tokens.inDocumentOrder())
{
FeatureMap lookupFeatures = lookupAnn.getFeatures();
token = lookupFeatures.get("dependencies").toString();
}
I can use below to get all the features of a token,
gate.Utils.inDocumentOrder
but it returns all features, including the dependent tokenID's; i.e:
dependencies = [nsubj(8390), dobj(8394)]
I would like to get just the dependent token's string value from these tokenID's.
Is there any way to access dependent token string value and add them as a feature to the annotation?
Many thanks for your help
Here is a working JAPE example. It only printns to the GATE's message window (std out), It doesn't create any new annotations with features you asked for. Please finish it yourself...
Stanford_CoreNLP plugin has to be loaded in GATE to make this JAPE file loadable. Otherwise you will get class not found exception for DependencyRelation class.
Imports: {
import gate.stanford.DependencyRelation;
}
Phase: GetTokenDepsPhase
Input: Token
Options: control = all
Rule: GetTokenDepsRule
(
{Token}
): token
-->
:token {
//note that tokenAnnots contains only a single annotation so the loop could be avoided...
for (Annotation token : tokenAnnots) {
Object deps = token.getFeatures().get("dependencies");
//sometimes the dependencies feature is missing - skip it
if (deps == null) continue;
//token.getFeatures().get("string") could be used instead of gate.Utils.stringFor(doc,token)...
System.out.println("Dependencies for token " + gate.Utils.stringFor(doc, token));
//the dependencies feature has to be typed to List<DependencyRelation>
List<DependencyRelation> typedDeps = (List<DependencyRelation>) deps;
for (DependencyRelation r : typedDeps) {
//use DependencyRelation.getTargetId() to get the id of the target token
//use inputAS.get(id) to get the annotation for its id
Annotation targetToken = inputAS.get(r.getTargetId());
//use DependencyRelation.getType() to get the dependency type
System.out.println(" " +r.getType()+ ": " +gate.Utils.stringFor(doc, targetToken));
}
}
}
I want to access Domino data via the Domino Access Services (DAS) as REST provider in java e.g.
String url = "http://malin1/fakenames.nsf/api/data/collections/name/groups";
ObjectMapper mapper = new ObjectMapper();
JsonFactory factory = new JsonFactory();
JsonParser parser = factory.createParser(new URL(url));
JsonNode rootNode = mapper.readTree(parser);
however, I notice DAS binds the JSON in square brackets:
[
{
"#entryid":"1-D68BB54DEA77AC8085256B700078923E",
"#unid":"D68BB54DEA77AC8085256B700078923E",
"#noteid":"1182",
"#position":"1",
"#read":true,
"#siblings":3,
"#form":"Group",
"name":"LocalDomainAdmins",
"description":"This group should contain all Domino administrators in your domain. Most system databases and templates give people in this group Manager access."
},
{
"#entryid":"3-9E6EABBF405A1A9985256B020060E64E",
"#unid":"9E6EABBF405A1A9985256B020060E64E",
"#noteid":"F46",
"#position":"3",
"#read":true,
"#siblings":3,
"#form":"Group",
"name":"OtherDomainServers",
"description":"You should add all Domino servers in other domains with which you commonly replicate to this group."
}
]
How can I easily get rid of these brackets?
As already mentioned you should leave them intact. You can parse theJSON array for example with Jackson.
find an example snippet below
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.JsonProcessingException;
import org.codehaus.jackson.map.ObjectMapper;
...
String response = ... your posted string
ObjectMapper mapper = new ObjectMapper();
try {
JsonNode taskIdsjsonNode = mapper.readTree(response);
for (JsonNode next : taskIdsjsonNode) {
System.out.printf("%s: %s%n", "#entryid", next.get("#entryid"));
System.out.printf("%s: %s%n", "name", next.get("name"));
}
} catch (.... ) {
// your exception handling goes here
}
output
#entryid: "1-D68BB54DEA77AC8085256B700078923E"
name: "LocalDomainAdmins"
#entryid: "3-9E6EABBF405A1A9985256B020060E64E"
name: "OtherDomainServers"
The brackets are not nasty but a correct notation. To access the contens just use [0] in your client side script or with your JSON parser in Java you like.
Perhaps the explanation here can help:
https://quintessens.wordpress.com/2015/05/08/processing-json-data-from-domino-access-services-with-jackson/
Basically you establish a call to DAS via the Jersey client and then you parse the json via Jackson library to a map in java.
During the parsing process you can define which values you want to parse and transform them.
Take a look at the Person class...
I am developing an android application using JAVA. All I want is to
record a song and generate its hash(CODE), then query the echoprint server for a match.
If a match is not found, then upload it to the server (ingest) for future references.
I have been able to achieve the first part. Can someone suggest me about the second part in JAVA? (P.S. : I've seen how to do it using python codes - but that won't be helpful in my case.)
Another question, may I achieve the second objective with the global echoprint server? Or, do I need to set up one of my own?
The references I've used are:
http://masl.cis.gvsu.edu/2012/01/25/android-echoprint/
https://github.com/gvsumasl/EchoprintForAndroid
To insert a song into the echoprint server database, all you need to do is call the ingest method. Basically, it is only a HTTP POST request with correct json body. Here is a Scala code (Java would be very similar) that I am using for that:
import EasyJSON.JSON
import EasyJSON.ScalaJSON
import dispatch.Defaults.executor
import dispatch._
class EchoprintAPI {
val API_URL = "http://your.api.server"
def queryURL(code: String) = url(s"$API_URL/query?fp_code=$code")
def query(code: String): scala.concurrent.Future[ScalaJSON] = {
jsonResponse(queryURL(code))
}
def ingest(json: ScalaJSON, trackId: String): scala.concurrent.Future[ScalaJSON] = {
val metadata = json("metadata")
val request = url(s"$API_URL/ingest").POST
.addParameter("fp_code", json("code").toString)
.addParameter("artist", metadata("artist").toString)
.addParameter("release", metadata("release").toString)
.addParameter("track", metadata("title").toString)
.addParameter("codever", metadata("version").toString)
.addParameter("length", metadata("duration").toString)
.addParameter("genre", metadata("genre").toString)
.addParameter("bitrate", metadata("bitrate").toString)
.addParameter("source", metadata("filename").toString)
.addParameter("track_id", trackId)
.addParameter("sample_rate", metadata("sample_rate").toString)
jsonResponse(request)
}
def delete(trackId: String): scala.concurrent.Future[ScalaJSON] = {
jsonResponse(url(s"$API_URL/query?track_id=$trackId").DELETE)
}
protected def jsonResponse(request: dispatch.Req): scala.concurrent.Future[EasyJSON.ScalaJSON] = {
val response = Http(request OK as.String)
for (c <- response) yield JSON.parseJSON(c)
}
}
To generate the fingerprint code, you can use echoprint-codegen command line call or use the Java JNI integration with C lib