Develop Reference

Parsing non-fixed format binary payload with a custom javascript conversion in Vorto

We are using Vorto now mainly as a normalized format and are starting to look into using the mapping engine for mapping different payload formats to Vorto model as well. I more or less understand how to map functionblock properties from JSON or binary payload using xpath and the conversion functions. However, I'm not clear how to support parsing of non-fixed format binary payload using this method.
For instance we have an off the shelf LoRaWAN sensor which transmits in the following format:
<length><frame type>[<sensor-id><sensor-value>] where length is the total frame length and sensor-id (for eg temperature, humidity, battery, ...) describes how to parse the sensor-value (ie length, datatype). In one frame multiple of these readings may be present in random order.
Parsing this can be done easily in for instance loraserver.io using a small javascript function which iterates over all the bytes en returns the parsed properties. The same way will work in the Ditto payload mapping engine afaik.
However, currently I don't see how to do something similar in Vorto mapping. This is just one specific sensor example of course, but more examples exist on the market using similar dynamic payload format. I know there is already an open issue (#1535) to improve the documentation, but it would already be helpful to know if such flexible parsing would be possible using the mapping DSL.
I tried passing the raw payload as bytearray to the javascript function. In order to test this I duplicated the org.eclipse.vorto.mapping.engine.converter.binary.BinaryMappingTest#testMappingBinaryContaining2DataPoints and adapted the model to use a custom javascript function like this
evaluator.addScriptFunction(new ScriptClassFunction("extractTemperature",
"function extractTemperature(value) { " +
" print(\"parameter of type \" + typeof value + \", value = \" + value);" +
" print(value[1]);" +
"}"));
The output of this function is
parameter of type number, value = 1
undefined
Where the value 1 is the first element of the bytearray used.
So the function does not seem to receive the parameter as bytarray.
The model is configured with .withXPathStereotype("custom:extractTemperature(data)", "demo") so the payload is passed (as BinaryData) in the same way as in the testMappingBinaryContaining2DataPoints test (.withXPathStereotype("custom:convert(vorto_conversion1:byteArrayToInt(data,0,0,0,2))", "demo")). The only difference I see now is that in the testMappingBinaryContaining2DataPoints test is that the byetarray parameter is passed to a Java function instead of a javascript function. Or am I missing something?
Also, I noticed that loop keywords like for and while are not allowed in the javascript code. So even if I can access the bytearray parameter in the javascript function I see no way for now how to iterate over this.
On gitter I received following reply (together with the suggestion to move discussion to SO)
You are right. We restricted the Javascript function usage to very rudimentary set of language keywords excluding for loops as nasty stuff can be implemented there. What you could do Instead is to register a java function In your own namespace to the mapping engine. That function can hold a byte array. Later this function can be contributed to the mapping engine as a standard function to extract a certain value out for other developers to reuse.
I don't think this is solution to the problem however. As mentioned above this is just one example of an off the shelf sensor payload format, and I don't see how this can be generalized enough to include as a generic function in the mapping engine. And I don't think it should be required to implement a sensor specific conversion in Java, since (as an end-user of an IoT platform wanting to deploy a new sensor type) this is more complex to develop and deploy than a little javascript function which can be altered at runtime in the mapping spec. I see a lot of value in being able to do simple mappings in javascript, just like this can be done in for example loraserver.io and Eclipse Ditto.
I think being able to pass a byte array to javascript is a first step. Also I wonder where exactly the risk is in allowing loops in the javascript? For example Ditto also has some restrictions in the javascript sandbox (see here) but this allows loops and only prevents endless looping and recursion.
They state the following:
Using Rhino instead of Nashorn, the newer JavaScript engine shipped with Java, has the benefit that sandboxing can be applied in a better way.
Sandboxing of different payload scripts is required as Ditto is intended to be run as cloud service where multiple connections to different endpoints are managed for different tenants at the same time. This requires the isolation of each single script to avoid interference with other scripts and to protect the JVM executing the script against harmful code execution.
Would using Rhino in Vorto as well allow to control the risks you see and allow loop construct in Vorto mapping?
PS: can someone with enough SO reputation points add the tag eclipse-vorto please?

I created an issue for you request to support this in the Javascript converters: https://github.com/eclipse/vorto/issues/2029
As stated in the issue, as a current workaround, you can register your own custom converter function with Java and re-use this function across your mappings. In these java converter functions, you have all the power of the java language to convert to extract the right property from the arbitrary list.
In order to find out how to implement your own custom converter function with Java, take a look here: https://github.com/eclipse/vorto/tree/master/mapping-engine#Advanced-Usage

Since Eclipse Vorto 0.12.3 release, a fix for your request is available. With this it is possible to pass array object to javascript Converter as well as use for loops inside javascript functions. You might wanna give it a try.
See release notes https://github.com/eclipse/vorto/blob/master/docs/release-notes.md

Java JSON editor / visualizer

I wrote an application that takes a JSON file as configuration. Up to this point, I wrote the JSON config by hand. However, now I want to allow other users who are not familiar with JSON format to make their own configurations. There's only 3 types of objects that the configuration needs to store, but the user should be allowed to add multiple copies of these objects. I want to write a configuration application where the user can press a button such as "Add Type A" and an object of type A is populated with default values and visualized so that the user can select properties and edit them. I know how to write an application that does this, but I feel like I'm re-inventing the wheel. Does anyone know of an open-source Java library that I can use inside my config application to handle visualization and editing a JSON file? If I'm approaching this from completely the wrong direction, please let me know.

I'm tempted to say "don't fear to reinvent the wheel".
But the fact is that there a bunch solutions available :
How do I convert an object to JSON representation (and vice-versa)
Java representation of JSON Object
Javascript to Java using JSON
among them, GSON looks fine.
For my personal experience, Yaml turned out to be an appropriate solution.

How to use Lucene for next token recommendation?

I am trying to build a recommender that suggests the next token based on some context. An example would be recommending method calls where the context would be the method calls that were seen before.
I would need Lucene to build a language model (e.g. the n-gram model) from a prebuilt list of strings. It should then support queries that contain a list of tokens (the context) and return the token that has the highest probability to come next.
What would be the best way to implement this with the Lucene API?
edit:
I tried the suggestion from femtoRgon to try the Suggesters from the suggest package. Unfortunately, they don't completely solve my use case. AnalyzingSuggester and AnalyzingInfixSuggester need to be built with query data,
while I want to build the model with a list of strings or text.
The FreeTextSuggester is pretty close to what I need but it does not support context. While it builds an n-gram model from the text and the autocompletion works, I would also like to input a context. The context would be a list of strings, for example when using it with method calls it would be the tokens seen before the triggering the code completion.
Would there be a way to use such context in these suggesters or is there another method in Lucene?

Compare 2 objects in eclipse in runtime (while debugging)?

Can I compare 2 Objects using the Eclipse - Expressions tool, am trying to debug a Java program. The max I can do is, add both the objects to Expressions and manually expand the object to compare, I wish there was like, select 2 Object (of same kind - Class, Of course) and say "Compare" and Eclipse parses both the objects and highlights all the differences..
PS: I'm not lazy to do this manually, its just that the Object I'm dealing with is very complicated, like its got 10 levels or arraylist of Objects :)

Not a elegant solution but it has worked well for me in the past is to include something like gson or other json parser in your classpath. Then in the Display view type:
new Gson().gson.toJson(yourObject1)
and
new Gson().gson.toJson(yourObject2)
Then use the json ouput save it to files and use a file comparison tool.
Longwinded but with complex objects its often worth it.

They can be as detailed or simple as you want them to be.
Using a equals expression in the Display view may tel you if two objects are the same, but if their different, it won't tell you what those differences are.
Consider using Eclipse detail formatters. They are awesome.
Eclipse's Inspect view can often be littered with unwanted data that just clutters your mind and what your really interested in. The ouput you decide to display is your choice with detail formatters.
See the link below for an idea of how to use them.
http://www.howardism.org/Technical/Eclipse/Eclipse_Detail_Formatter.html

Find "Display" view under debug category. It is not open by default in the java debug perspective. You will be able to execute pretty much arbitrary java statements during the debug session.

java: using a variable's value as an object name (not the eval() way)

Ok, so coming from a background of mostly perl, and mostly writing dirty little apps to automate my tasks, I've read the pages about the evils of eval(), and I always use a hash (in perl). I'm currently working on a little project (mostly for me and a couple of other technical people at work), for creating "canned response" e-mails. To allow for additions, subtractions, edits, etc., I'd like to essentially describe the response form(s) in XML, and have my app parse the XML and create the response forms at runtime. I want to use Java (to integrate it into an existing Java tool that I created), and boiled down to a trivial example, what I'm trying to do is take some XML like:
<Form Name="first" Title="Title!">
<Label Name="before">Your Request:</Label>
<Textbox Name="input"/>
<Label Name="after">has been completed.</Label>
<Output>%before%%input%%after%</Output>
</Form>
<Form Name="second">
...
and from parsing that, I want to create a JFrame named first, which contains a JLabel named before with the obvious text, then a textbox, then another JLabel... you get the idea (I eventually want to use the output tag to control exactly how the response is formatted).
I can parse the XML, and get the element name and such, but I don't know how to instantiate the Objects with a name that is the value of a variable, effectively:
JFrame $(thisNode.getAttributes().getNamedItem("Name").getNodeValue()) = new JFrame(thisNode.getAttributes().getNamedItem("Title").getNodeValue());
I've read basically the whole first page of google results on java reflection, but I haven't come across anyone doing quite what I'm looking for (at least not that I could tell). Having basically zero experience with reflection, I'm curious if this is something that can be accomplished using it, or if I should take the same approach as I would in Perl, and create a HashMap or HashTable of Objects, and tie them to a entry in a Hash of JFrames. Or, I'm open to ideas that don't fall into those two categories. The Hash is sort of my stand-by answer, because I've done it in Perl plenty of times, and I'm sure I can make it work in Java, but if there's a feature (like reflection) that's made to do this task, then why not do it the way it was intended to be done?

What you're asking isn't possible in Java. It doesn't work that way and these sorts of tricks, which are common in dynamic languages, aren't the Java way. You can certainly do:
JFrame frame = JFrameBuilder.buildFromTemplate("frame.xml");
where you create a JFrameBuilder class that reads the XML and creates an object from it but the variable name can't be dynamic. You have to remember that there are two steps in Java.
Java source files are compiled into bytecode;
The bytecode is read by a Java interpreter (JVM) and executed.
What you want is essentially asking to execute code in step (1). Now annotations can do things in a compile step (like adding interfaces, implementing methods and so on) but local variable naming is not one of those things.

You could (not necessarily that you should) generate Java source based on your XML, compile the generated code, and finally, execute the compiled code. This could be more efficient if you saved the generated .class files and reused them instead of parsing the XML every time the program is run (it can check the timestamp on the XML and only generate and compile if it's been modified since the last code generation).