Develop Reference

Validate JSON before further processing

I have the following csv file (In production the number of records can range from 20k-100k and has many fields )
id,firstname,lastname,email,profession
100,Betta,Wandie,Betta.Wandie#gmail.com,developer
101,Janey,Firmin,Janey.Firmin#gmail.com,doctor
I need to convert this to json and do further processing.
CSV->JSON->PROCESS FURTHER
.I am able to convert it to JSON directly using the code given here
directly convert CSV file to JSON file using the Jackson library
But i want do validations for json like if lastname has null value then ignore that record or id is missing then ignore that record.
How can i handle the validation?I am using Java 8 and spring boot latest version

I have done something similar by using JavaScript (Nashorn). Yes, that is nasty, but it works, and it is astonishingly fast!
Unfortunately, I do not have the source code at hand …
Why I did it that way had the same reasons as #chrylis-on strike implies in their comment: the validation is much easier if you have an object for each JSON record. But as I was lazy, and there was definitely no need for the Java object I had to create for this, I had the idea with the JavaScript script inside my Java program.
Basically, a JSON String is the source for a JavaScript program; you can assign it directly to a JavaScript variable and then you can access the records as array elements and the fields by their name. So your JavaScript code walks through the records and drops those that do not match your validation rules.
Now the Java part: the keyword here is JSR223; it is an API that allows you to execute scripts inside your Java environment. In your case, you have to provide the converted JSON in the context, then you start the script that writes the modified JSON back to the context.
If the converted JSON is too large, you can even use the same technique to check record by record; if you compile the script, it is nearly as fast as native Java.
You can even omit Jackson and let JavaScript do the conversion …
Sorry that I cannot provide code samples; I will try to get hold on them and add them if I get them.

Parsing non-fixed format binary payload with a custom javascript conversion in Vorto

We are using Vorto now mainly as a normalized format and are starting to look into using the mapping engine for mapping different payload formats to Vorto model as well. I more or less understand how to map functionblock properties from JSON or binary payload using xpath and the conversion functions. However, I'm not clear how to support parsing of non-fixed format binary payload using this method.
For instance we have an off the shelf LoRaWAN sensor which transmits in the following format:
<length><frame type>[<sensor-id><sensor-value>] where length is the total frame length and sensor-id (for eg temperature, humidity, battery, ...) describes how to parse the sensor-value (ie length, datatype). In one frame multiple of these readings may be present in random order.
Parsing this can be done easily in for instance loraserver.io using a small javascript function which iterates over all the bytes en returns the parsed properties. The same way will work in the Ditto payload mapping engine afaik.
However, currently I don't see how to do something similar in Vorto mapping. This is just one specific sensor example of course, but more examples exist on the market using similar dynamic payload format. I know there is already an open issue (#1535) to improve the documentation, but it would already be helpful to know if such flexible parsing would be possible using the mapping DSL.
I tried passing the raw payload as bytearray to the javascript function. In order to test this I duplicated the org.eclipse.vorto.mapping.engine.converter.binary.BinaryMappingTest#testMappingBinaryContaining2DataPoints and adapted the model to use a custom javascript function like this
evaluator.addScriptFunction(new ScriptClassFunction("extractTemperature",
"function extractTemperature(value) { " +
" print(\"parameter of type \" + typeof value + \", value = \" + value);" +
" print(value[1]);" +
"}"));
The output of this function is
parameter of type number, value = 1
undefined
Where the value 1 is the first element of the bytearray used.
So the function does not seem to receive the parameter as bytarray.
The model is configured with .withXPathStereotype("custom:extractTemperature(data)", "demo") so the payload is passed (as BinaryData) in the same way as in the testMappingBinaryContaining2DataPoints test (.withXPathStereotype("custom:convert(vorto_conversion1:byteArrayToInt(data,0,0,0,2))", "demo")). The only difference I see now is that in the testMappingBinaryContaining2DataPoints test is that the byetarray parameter is passed to a Java function instead of a javascript function. Or am I missing something?
Also, I noticed that loop keywords like for and while are not allowed in the javascript code. So even if I can access the bytearray parameter in the javascript function I see no way for now how to iterate over this.
On gitter I received following reply (together with the suggestion to move discussion to SO)
You are right. We restricted the Javascript function usage to very rudimentary set of language keywords excluding for loops as nasty stuff can be implemented there. What you could do Instead is to register a java function In your own namespace to the mapping engine. That function can hold a byte array. Later this function can be contributed to the mapping engine as a standard function to extract a certain value out for other developers to reuse.
I don't think this is solution to the problem however. As mentioned above this is just one example of an off the shelf sensor payload format, and I don't see how this can be generalized enough to include as a generic function in the mapping engine. And I don't think it should be required to implement a sensor specific conversion in Java, since (as an end-user of an IoT platform wanting to deploy a new sensor type) this is more complex to develop and deploy than a little javascript function which can be altered at runtime in the mapping spec. I see a lot of value in being able to do simple mappings in javascript, just like this can be done in for example loraserver.io and Eclipse Ditto.
I think being able to pass a byte array to javascript is a first step. Also I wonder where exactly the risk is in allowing loops in the javascript? For example Ditto also has some restrictions in the javascript sandbox (see here) but this allows loops and only prevents endless looping and recursion.
They state the following:
Using Rhino instead of Nashorn, the newer JavaScript engine shipped with Java, has the benefit that sandboxing can be applied in a better way.
Sandboxing of different payload scripts is required as Ditto is intended to be run as cloud service where multiple connections to different endpoints are managed for different tenants at the same time. This requires the isolation of each single script to avoid interference with other scripts and to protect the JVM executing the script against harmful code execution.
Would using Rhino in Vorto as well allow to control the risks you see and allow loop construct in Vorto mapping?
PS: can someone with enough SO reputation points add the tag eclipse-vorto please?

I created an issue for you request to support this in the Javascript converters: https://github.com/eclipse/vorto/issues/2029
As stated in the issue, as a current workaround, you can register your own custom converter function with Java and re-use this function across your mappings. In these java converter functions, you have all the power of the java language to convert to extract the right property from the arbitrary list.
In order to find out how to implement your own custom converter function with Java, take a look here: https://github.com/eclipse/vorto/tree/master/mapping-engine#Advanced-Usage

Since Eclipse Vorto 0.12.3 release, a fix for your request is available. With this it is possible to pass array object to javascript Converter as well as use for loops inside javascript functions. You might wanna give it a try.
See release notes https://github.com/eclipse/vorto/blob/master/docs/release-notes.md

Efficient java library for text templating?

I've got a simple string coming in from a UI component as The device id is %{test}. Assume %{test} is a dynamic variable and the values for it are being assigned from the backend code. The final string should look like:
The device id is some text
----------------------------^ should be replaced with %{test} and appended to the whole string
I've read a bit and tried out some of the libraries which were pointed out here, such as Velocity and FreeMarker. But I'm quite unaware in terms of efficiency and performance on using those libraries.
Hope I could get some insights on this since I'm pretty new to this. Any help could be appreciated.

I suggest you to take a look at Arco Template Engine: It compiles the template in compile-time, producing a .java (or .class) file. And so, at run-time, the expansion is done very fast.
The templates should be coded in JSP format. Thus, all variables references must be written ${variable} (not %{variable}).
The only thing to take in account is that templates must be staticly generated (in order to be processed at compile-time).
(Read the FAQ and the examples).

Manipulating HTML nodes with java javascript scripting API

I'm using the Java Scripting API which is working quite well. Now I have a function where I want to get all <a> tags from a String and then add/remove attributes before returning the manipulated String. The problem of course is, that I can't just use document.getElementsByTagName. Is there any easy option that comes to your mind without going through regex-hell?
Please note that I'm currently running on Java 7 (with Rhino), planning to update to Java 8 (with Nashorn), so I don't want to use any Rhino specific APIs.

In the book "Learning JavaScript Design Patterns" by Addi Osmani, author mentions 3 alternatives to a similar problem, obviously being getElementById() the fastest.
Excerpt from book:
Imagine that we have a script where for each DOM element found on page
with class "foo," we wish to increment a counter. What's the most
efficient way to query for this collection of elements? Well, there
are a few different ways this problem could be tackled:
Select all of the elements in the page and then store references to them. Next, filter this collection and use regular expressions (or
another means) to store only those with the class "foo."
Use a modern native browser feature such as querySelectorForAll() to select all of the elements with the class "foo."
Use a netive feature such as getElementsByClassName() to similarly...
Another way is, since you're using Nashorn/Rhino, you could use the Java implementation of the Xerces library to manipulate the DOM.
Hope this helps you find out the solution.

java: using a variable's value as an object name (not the eval() way)

Ok, so coming from a background of mostly perl, and mostly writing dirty little apps to automate my tasks, I've read the pages about the evils of eval(), and I always use a hash (in perl). I'm currently working on a little project (mostly for me and a couple of other technical people at work), for creating "canned response" e-mails. To allow for additions, subtractions, edits, etc., I'd like to essentially describe the response form(s) in XML, and have my app parse the XML and create the response forms at runtime. I want to use Java (to integrate it into an existing Java tool that I created), and boiled down to a trivial example, what I'm trying to do is take some XML like:
<Form Name="first" Title="Title!">
<Label Name="before">Your Request:</Label>
<Textbox Name="input"/>
<Label Name="after">has been completed.</Label>
<Output>%before%%input%%after%</Output>
</Form>
<Form Name="second">
...
and from parsing that, I want to create a JFrame named first, which contains a JLabel named before with the obvious text, then a textbox, then another JLabel... you get the idea (I eventually want to use the output tag to control exactly how the response is formatted).
I can parse the XML, and get the element name and such, but I don't know how to instantiate the Objects with a name that is the value of a variable, effectively:
JFrame $(thisNode.getAttributes().getNamedItem("Name").getNodeValue()) = new JFrame(thisNode.getAttributes().getNamedItem("Title").getNodeValue());
I've read basically the whole first page of google results on java reflection, but I haven't come across anyone doing quite what I'm looking for (at least not that I could tell). Having basically zero experience with reflection, I'm curious if this is something that can be accomplished using it, or if I should take the same approach as I would in Perl, and create a HashMap or HashTable of Objects, and tie them to a entry in a Hash of JFrames. Or, I'm open to ideas that don't fall into those two categories. The Hash is sort of my stand-by answer, because I've done it in Perl plenty of times, and I'm sure I can make it work in Java, but if there's a feature (like reflection) that's made to do this task, then why not do it the way it was intended to be done?

What you're asking isn't possible in Java. It doesn't work that way and these sorts of tricks, which are common in dynamic languages, aren't the Java way. You can certainly do:
JFrame frame = JFrameBuilder.buildFromTemplate("frame.xml");
where you create a JFrameBuilder class that reads the XML and creates an object from it but the variable name can't be dynamic. You have to remember that there are two steps in Java.
Java source files are compiled into bytecode;
The bytecode is read by a Java interpreter (JVM) and executed.
What you want is essentially asking to execute code in step (1). Now annotations can do things in a compile step (like adding interfaces, implementing methods and so on) but local variable naming is not one of those things.

You could (not necessarily that you should) generate Java source based on your XML, compile the generated code, and finally, execute the compiled code. This could be more efficient if you saved the generated .class files and reused them instead of parsing the XML every time the program is run (it can check the timestamp on the XML and only generate and compile if it's been modified since the last code generation).