JSONata is an expression language designed to query and transform JSON data structures.
I find that current implementations of JSONata are in Javascript only. (https://github.com/jsonata-js/jsonata)
I want to use JSONata in my Java code. It'll make life much easier to manipulate JSON documents in Java.
A possible way could be to use the standard Java classes under javax.script package to interact with the Javascript-based JSONata implementation.
Has anyone already done this? Is there any sample code to demonstrate how this can be achieved?
Has anyone implemented other mechanisms of using JSONata in Java?
The following snippet shows how you could invoke the JSONata processor from Java using the embedded JavaScript engine...
ScriptEngineManager factory = new ScriptEngineManager();
ScriptEngine engine = factory.getEngineByName("JavaScript");
Invocable inv = (Invocable) engine;
FileReader jsonata = new FileReader("jsonata.js");
// load the JSONata processor
engine.eval(jsonata);
// read and JSON.parse the input data
byte[] sample = Files.readAllBytes(Paths.get("sample.json"));
engine.put("input", new String(sample));
Object inputjson = engine.eval("JSON.parse(input);");
// query the data
String expression = "$sum(Account.Order.Product.(Price * Quantity))"; // JSONata expression
Object expr = inv.invokeFunction("jsonata", expression);
Object resultjson = inv.invokeMethod(expr, "evaluate", inputjson);
// JSON.stringify the result
engine.put("resultjson", resultjson);
Object result = engine.eval("JSON.stringify(resultjson);");
System.out.println(result);
In this example, the jsonata.js file has been pulled down from the JSONata GitHub repo as well as the 'Invoice' sample code from try.jsonata.org.
Extra code would be needed to handle errors, but this gives the general idea.
I have just posted a Java implementation of JSONata called JSONata4Java.
maven central:
JSONata4Java jar files are located here in Maven Central: https://search.maven.org/search?q=g:com.ibm.jsonata4java
pom.xml dependency
<dependency>
<groupId>com.ibm.jsonata4java</groupId>
<artifactId>JSONata4Java</artifactId>
<version>1.0.0</version>
</dependency>
github:
The Java port of jsonata project named JSONata4Java has been posted here: https://github.com/IBM/JSONata4Java
If you would like to contribute, please print and sign the appropriate JSONata4Java cla document and mail it to me:
IBM Corporation
c/o Nathaniel Mills
16 Deer Hill Ln.
Coventry, CT, 06238, US
Attn: OSS CLA Processing
Thanks in advance.
You can use the JSONata-Java project:
https://github.com/cow-co/jsonata-java
A Java port of the original (JavaScript) interpreter for the JSONata
JSON query and transformation language.
Related
I am trying python and java integration using Jep. I have loaded randomforest model from pickle file (rf.pkl) as sklearn.ensemble.forest.RandomForestClassifier object from java program using Jep.
I want this loading to be one time so that I wanted to execute a python function defined in python script prediction.py (to predict using rf model) by sending "rfmodel" argument from java to call python function.
But the argument sent to python from java is read as string in python. How can I retain the datatype of argument in python as sklearn.ensemble.forest.RandomForestClassifier?
Jep jep = new Jep();
jep.eval("import pickle");
jep.eval("clf = pickle.load(open('C:/Downloads/DSRFmodel.pkl', 'rb'))");
jep.eval("print(type(clf))");
Object randomForest = jep.getValue("clf");
jep.eval("import integration");
jep.set("arg1", requestId);
jep.set("arg2", randomForest);
jep.eval("result = integration.trainmodel(arg1, arg2)");
------------
python.py
import pickle
def trainmodel(requestid, rf):
//when rf is printed it is 'str' format.
When Jep converts a Python object into a Java object if it does not recognize the Python type it will return the String representation of the Python object, see this bug for discussion on that behavior. If you are running the latest version of Jep(3.8) you can override this behavior by passing a Java class to the getValue function. The PyObject class was created to serve as a generic wrapper around arbitrary python objects. The following code should do what you want:
Jep jep = new Jep();
jep.eval("import pickle");
jep.eval("clf = pickle.load(open('C:/Downloads/DSRFmodel.pkl', 'rb'))");
jep.eval("print(type(clf))");
Object randomForest = jep.getValue("clf", PyObject.class);
jep.eval("import integration");
jep.set("arg1", requestId);
jep.set("arg2", randomForest);
jep.eval("result = integration.trainmodel(arg1, arg2)");
I'm trying to check velocity scripting engine 2.0 which Provide JSR 223 implementation and support of Compilable
the Compilable interface has been implemented in the process.
I use jars: velocity-engine-scripting-2.0.jar, velocity-1.7.jar, commons-collections-3.2.2.jar
from previous answer I use the following code
//class org.apache.velocity.script.VelocityScriptEngine
final ScriptEngine engine = engineFactory.getScriptEngine();
if (engine instanceof Compilable) {
try {
((Compilable) engine).compile("");
...
For velocity I get the following:
javax.script.ScriptException: org.apache.velocity.exception.ResourceNotFoundException: No template name provided
at org.apache.velocity.script.VelocityScriptEngine.compile(VelocityScriptEngine.java:311)
at org.apache.velocity.script.VelocityScriptEngine.compile(VelocityScriptEngine.java:288)
at com.Workers.LevelCheck.main(LevelCheck.java:69)
Caused by: org.apache.velocity.exception.ResourceNotFoundException: No template name provided
at org.apache.velocity.runtime.resource.loader.StringResourceLoader.getResourceStream(StringResourceLoader.java:353)
at org.apache.velocity.Template.process(Template.java:108)
at org.apache.velocity.script.VelocityScriptEngine.compile(VelocityScriptEngine.java:306)
... 2 more
Also when I tried to give template name ((Compilable) engine).compile("v.vm"); it failed with same exception
You cannot use velocity-engine-scripting-2.0.jar with velocity-1.7.jar, you need to use velocity-engine-core-2.0.jar otherwise you'll get unpredictable results.
Am using Nashorn script engine in java for evaluating expressions,
the below code for log works,
ScriptEngineManager mgr = new ScriptEngineManager();
ScriptEngine engine = mgr.getEngineByName("noshorn");
engine.eval("Math.log(99);");
whereas for log10 fails,
ScriptEngineManager mgr = new ScriptEngineManager();
ScriptEngine engine = mgr.getEngineByName("noshorn");
engine.eval("Math.log10(99);");
it throws an exception saying,
Caused by: <eval>:1 TypeError: Cannot call undefined
How can i resolve this.Please help.
The Math that is initialized by default not the java system Math. If you want the java's Math use:
var JavaMath = Java.type("java.lang.Math");
Then you can use all of the math methods. I am surprised Nashorn doesn't provide the javascript Math Object, which would have log10.
Nashorns current Math implementation is based on this version of the standard which does not yet contain Math.log10.
#matt has already given a workaround (+1), here is another one
engine.eval("java.lang.Math.log10(99);");
Math.log10 is from ECMAScript6 -> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/log10 , http://www.ecma-international.org/ecma-262/6.0/#sec-math.log10
Nashorn as of jdk8u implements ECMAScript 5.1. (few) Elements of ES6 being added to jdk9.
Background
I'm using wsimport to create what is essentially a Java webservice client, connecting to a .Net webservice that is returning datasets (unfortunately). To be more specific I'm working on a project (inbound transport) for the GeoEvent Processor suite of ESRI ArcGIS Server 10.2, but I think this might be answered on more general terms in relation to JAXB and WSDL bindings. Bear with me as I haven't touched Java since college (10+ years).
For purposes of the WSDL, the .Net DataSet is a polymorphic type whose actual layout isn't determined until run time, after the DataSet has been filled with data. This causes problems when you want to use that webservice with anything but .Net.
After some research I've managed to use wsimport to generate from the webservice wsdl. I was then able to put together a basic proof of concept program that gets results from the webservice as a DOM, then walks that DOM as a nodelist.
Reference:
JAX-WS error on WSDL file: "Error resolving component 's:schema'"
https://weblogs.java.net/blog/vivekp/archive/2007/05/how_to_deal_wit_1.html
The section on Toolkit Bindings and figure 6 in http://msdn.microsoft.com/en-us/magazine/cc188755.aspx
My wsimport looks like this (domain names have been changed to protect the innocent):
C:\Development\ArcGIS\WSDL>wsimport -b http://www.w3.org/2001/XMLSchema.xsd -b xsd.xjb -keep -p com.somecompany.services -XadditionalHeaders http://services.somecompany.com/DataRetrieval.asmx?wsdl
The Problem
Unfortunately, the same codebase that worked in my proof of concept, getting results from the webservice, fails once I implement in the ArcGIS GeoEvent Processor. My project is part of an OSGI bundle that the ArcGIS GeoEvent Processor will control. The error below is as shown in the Apache Karaf log for the GeoEvent Processor.
Based on the error, my understanding is there is a problem with how I did the binding in wsimport, referencing the generic schema per those links I have listed above. Looks like the generic schema lacks definitions for some of the elements that exist as classes generated by wsimport. Those classes appear to be properly generated when I check the output from wsimport.
I've not included the WSDL due to posting limitations, but will include in later responses if needed.
What I'm trying to figure out
How should this error be interpreted?
Why does the same wsimport generated code used to access the webservice in my basic proof of concept fail when run in the ArcGIS GeoEvent Processor?
The error mentions JAXB and SAX, I'm not consciously referencing either of those libraries in the proof of concept or the project for the ArcGIS GeoEvent Processor. Could it be that the binding/unmarshalling of the webservice is handled differently, with ArcGIS GeoEvent Processor wrapping in JAXB/SAX and the proof of concept not?
What can I do to resolve this?
Use a different, custom, xsd and xjb that spells out the expected schema for the webservice? I'm not sure exactly how that would be done.
Use something other than wsimport to generate the webservice reference classes?
Tweak something in the java environment for the ArcGIS GeoEvent Processor?
Other options?
Commit seppuku, then it's not my problem?
The Error
2014-09-23 16:10:14,365 | ERROR | ansport Listener | SomeInboundTransport | 367 - com.somecompany.arcgis.geoevent.transport.inbound.somecompanyInboundTransport - 1.0.0 | Unable to call Webservice
javax.xml.ws.soap.SOAPFaultException: Unmarshalling Error: unexpected element (uri:"http://www.w3.org/2001/XMLSchema", local:"element"). Expected elements are <{http://services.somecompany.com/}complexType>,<{http://services.somecompany.com/}annotation>,<{http://services.somecompany.com/}redefine>,<{http://services.somecompany.com/}element>,<{http://services.somecompany.com/}include>,<{http://services.somecompany.com/}attributeGroup>,<{http://services.somecompany.com/}group>,<{http://services.somecompany.com/}notation>,<{http://services.somecompany.com/}import>,<{http://services.somecompany.com/}simpleType>,<{http://services.somecompany.com/}attribute>
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:156)[120:org.apache.cxf.cxf-rt-frontend-jaxws:2.6.1]
at com.sun.proxy.$Proxy198.getCompanyArcgisData(Unknown Source)[367:com.somecompany.arcgis.geoevent.transport.inbound.somecompanyInboundTransport:1.0.0]
at com.somecompany.arcgis.geoevent.transport.inbound.SomeInboundTransport.callWebService(SomeInboundTransport.java:184)[367:com.somecompany.arcgis.geoevent.transport.inbound.somecompanyInboundTransport:1.0.0]
at com.somecompany.arcgis.geoevent.transport.inbound.SomeInboundTransport.run(SomeInboundTransport.java:257)[367:com.somecompany.arcgis.geoevent.transport.inbound.somecompanyInboundTransport:1.0.0]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
Caused by: javax.xml.bind.UnmarshalException
- with linked exception:
[com.sun.istack.SAXParseException2; lineNumber: 1; columnNumber: 651; unexpected element (uri:"http://www.w3.org/2001/XMLSchema", local:"element"). Expected elements are <{http://services.somecompany.com/}complexType>,<{http://services.somecompany.com/}annotation>,<{http://services.somecompany.com/}redefine>,<{http://services.somecompany.com/}element>,<{http://services.somecompany.com/}include>,<{http://services.somecompany.com/}attributeGroup>,<{http://services.somecompany.com/}group>,<{http://services.somecompany.com/}notation>,<{http://services.somecompany.com/}import>,<{http://services.somecompany.com/}simpleType>,<{http://services.somecompany.com/}attribute>]
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.handleStreamException(UnmarshallerImpl.java:425)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:362)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:339)
at org.apache.cxf.jaxb.JAXBEncoderDecoder.doUnmarshal(JAXBEncoderDecoder.java:784)[91:org.apache.cxf.cxf-rt-databinding-jaxb:2.6.1]
at org.apache.cxf.jaxb.JAXBEncoderDecoder.access$100(JAXBEncoderDecoder.java:97)[91:org.apache.cxf.cxf-rt-databinding-jaxb:2.6.1]
at org.apache.cxf.jaxb.JAXBEncoderDecoder$1.run(JAXBEncoderDecoder.java:812)
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_17]
at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:810)[91:org.apache.cxf.cxf-rt-databinding-jaxb:2.6.1]
at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:644)[91:org.apache.cxf.cxf-rt-databinding-jaxb:2.6.1]
at org.apache.cxf.jaxb.io.DataReaderImpl.read(DataReaderImpl.java:157)[91:org.apache.cxf.cxf-rt-databinding-jaxb:2.6.1]
at org.apache.cxf.interceptor.DocLiteralInInterceptor.handleMessage(DocLiteralInInterceptor.java:108)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:798)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1667)[118:org.apache.cxf.cxf-rt-transports-http:2.6.1]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1520)[118:org.apache.cxf.cxf-rt-transports-http:2.6.1]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1428)[118:org.apache.cxf.cxf-rt-transports-http:2.6.1]
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:658)[118:org.apache.cxf.cxf-rt-transports-http:2.6.1]
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:532)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:464)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:367)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:320)[87:org.apache.cxf.cxf-api:2.6.1]
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:89)[119:org.apache.cxf.cxf-rt-frontend-simple:2.6.1]
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:134)[120:org.apache.cxf.cxf-rt-frontend-jaxws:2.6.1]
... 4 more
Caused by: com.sun.istack.SAXParseException2; lineNumber: 1; columnNumber: 651; unexpected element (uri:"http://www.w3.org/2001/XMLSchema", local:"element"). Expected elements are <{http://services.somecompany.com/}complexType>,<{http://services.somecompany.com/}annotation>,<{http://services.somecompany.com/}redefine>,<{http://services.somecompany.com/}element>,<{http://services.somecompany.com/}include>,<{http://services.somecompany.com/}attributeGroup>,<{http://services.somecompany.com/}group>,<{http://services.somecompany.com/}notation>,<{http://services.somecompany.com/}import>,<{http://services.somecompany.com/}simpleType>,<{http://services.somecompany.com/}attribute>
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.handleEvent(UnmarshallingContext.java:642)
at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportError(Loader.java:254)
at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportError(Loader.java:249)
at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportUnexpectedChildElement(Loader.java:116)
at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.childElement(Loader.java:101)
at com.sun.xml.bind.v2.runtime.unmarshaller.StructureLoader.childElement(StructureLoader.java:243)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext._startElement(UnmarshallingContext.java:478)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:459)
at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:242)
at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:176)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:360)
... 28 more
Caused by: javax.xml.bind.UnmarshalException: unexpected element (uri:"http://www.w3.org/2001/XMLSchema", local:"element"). Expected elements are <{http://services.somecompany.com/}complexType>,<{http://services.somecompany.com/}annotation>,<{http://services.somecompany.com/}redefine>,<{http://services.somecompany.com/}element>,<{http://services.somecompany.com/}include>,<{http://services.somecompany.com/}attributeGroup>,<{http://services.somecompany.com/}group>,<{http://services.somecompany.com/}notation>,<{http://services.somecompany.com/}import>,<{http://services.somecompany.com/}simpleType>,<{http://services.somecompany.com/}attribute>
... 39 more
The Code (snippet)
import com.somecompany.services.*; //generated by wsimport
import javax.xml.ws.*;
//...
private com.somecompany.services.DataRetrieval myWS;
private com.somecompany.services.DataRetrievalSoap port;
private byte[] callWebService(String userName, String pwd, long dataTimeFrame)
{
try
{
myWS = new com.somecompany.services.DataRetrieval();
port = myWS.getDataRetrievalSoap();
com.somecompany.services.AuthSoapHeader mySoapHeader = new com.somecompany.services.AuthSoapHeader();
mySoapHeader.setUserName(userName);
//Hash the password then set it for the SOAP header
String pwdHash = hashMD5(pwd);
mySoapHeader.setPassword(pwdHash);
Holder holder = new Holder<AuthSoapHeader>(mySoapHeader);
Date endTime = new Date();
Date startTime = new Date(endTime.getTime() - dataTimeFrame);
XMLGregorianCalendar gcEndTime = dateToGregorianTime(endTime);
XMLGregorianCalendar gcStartTime = dateToGregorianTime(startTime);
GetCompanyArcgisDataResponse.GetCompanyArcgisDataResult companyData = port.getCompanyArcgisData(gcStartTime, gcEndTime, holder);
if( ((AuthSoapHeader)holder.value).getError() != null)
{
log.error("Authentication to web services failed!");
//OSGI stop service
this.stop();
return null;
}else
log.info("Authentication to web services successful.");
//Convert the results to a java object and then to a byte array to send to the adapter
Object companyDataAny = companyData.getAny();
byte[] companyDataBytes = objectToBytes(companyDataAny);
return companyDataBytes;
}
catch(Exception ex)
{
log.error("Unable to call Webservice", ex);
//OSGI stop service
this.stop();
return null;
}
}
Environment Specifics
JDK 7u17 (1.7.0_17) 64 bit. The ArcGIS GeoEvent Processor is using this version of the JRE, so I'm locked into that version for execution. Though I've done some development in 1.7.0_51 before I realized that.
wsimport - JAX-WS RI 2.2.4-b01
ArcGIS Server 10.2
ArcGIS GeoEvent Processor Extension
Karaf (used by ArcGIS Geovent Processor to run OSGI bundles)
This is probably not the best answer on this, but it's what I came up with.
The ArcGIS GeoEvent Processor that wrapped my OSGI project appeared to be doing some additional binding/unbinding of the web service that I referenced in my application. The work-around that I employed to get that .Net (DataSet return values) web service to function in Java just wasn't acceptable to that wrapper from the GeoEvent Processor.
My Solution
Ultimately what I did was create a secondary .Net web service which took the DataSet values and converted them to JSON, and returned JSON strings. This removed the problems encountered when attempting to reference DataSet return values from the web service, now I was dealing with a simple JSON string. The wsimport of that JSON web service went smooth, no work-around required. I tucked the newly imported web service files into my java project and now have no problems.
For Reference on C# DataSet to JSON:
Using Newtonsoft.Json (http://james.newtonking.com/json). After playing with a few other libraries for JSON serialization that is what I found worked best for me.
Newtonsoft.Json is available via NuGet package
Rick Strahl's site was a big help http://weblog.west-wind.com/posts/2008/Sep/03/DataTable-JSON-Serialization-in-JSONNET-and-JavaScriptSerializer
I am trying to figure out how to pass a codec to the saveAsSequenceFile method in Apache Spark. Below is the code I am trying to run. I am running Scala 2.10.4, Spark 1.0.0, Java 1.7.60, and Apache Hadoop 2.4.0.
val rdd:RDD[(String, String)] = sc.sequenceFile(secPath,
classOf[Text],
classOf[Text]
).map { case (k,v) => (k.toString, v.toString)}
val sortedOutput = rdd.sortByKey(true, 1)
sortedOutput.saveAsSequenceFile(secPathOut)
My issue is that I am new to Spark and Scala. I do not understand what the javadoc means for the codec variable passed to the saveAsSequenceFile method.
def saveAsSequenceFile(path: String, codec: Option[Class[_ <: CompressionCodec]] = None): Unit
What does the <: mean? I get that the codec is optional, because when I run the above code it works. Could someone please show an example of a properly formatted codec call to this method?
Thanks!
The <: indicates that the class you pass in should extend org.apache.hadoop.io.compress.CompressionCodec (read this), spark uses a lot of HDFS features and is pretty heavily integrated with it at this point. This means you can pass the class of any of the following as the codec, BZip2Codec, DefaultCodec, GzipCodec, there are likely also other extensions of CompressionCodec not built into hadoop. Here is an example of calling the method
sc.parallelize(List((1,2))).saveAsSequenceFile("path",Some(classOf[GzipCodec]))
The Option[...] is used in scala in favor of java's null even though null exists in scala. Option can be Some(...) or None