I'm trying to figure out how I can pass multiple side inputs to my DoFn and refer them separately inside ProcessContext.
I wasn't able to find anything for this in the beam documentation and wanted to gain some idea on how I can achieve this in JAVA
Though the example of using side inputs only has a single side input, the same pattern holds for multiple side inputs.
Specifically, the withSideInputs method of ParDo takes any number of PCollectionViews, each of which can be used as its own key in ProcessContext.sideInput.
Related
I am new to Corda and am designing a Cordap. I want to display the states onto the terminal based on some arguments passed during initiating the flow.
My approach was I created a List of type StateAndRef storing all the states. Now I loop through the list and display the state based on if else condition.
But the issue I am facing is somethimes all the states the printed and sometimes only few of them are printed. Please help me to solve this problem. My main motive is I should be able to display on the screen those states which have name Car.
PS: I am using Java for designing the Cordapp.
You don't really need a flow to display states. You con using vaultQuery to do so:
run vaultQuery contractStateType: <fully-qualified name of the state type>
Refer documentation here for: https://docs.corda.net/docs/corda-os/4.4/shell.html#examples
In case you want to do complex querying, it's not supported in the CRaSH shell. However, you could do so in the RPC Client using the vault Query API. The VaultQuery API is very flexible and provide a number of mechanisms to access the vault.
Check the documentation here: https://docs.corda.net/docs/corda-os/4.4/api-vault-query.html#api-vault-query
As Ashutosh has already mentioned, complex querying doesn't seem to be supported yet. However a way to accomplish querying a subset of all states would be to create another class for those states.
For Ex - CARState for the subset you want to query on and the other states can be CAR2State, and then just use the
run vaultQuery contractStateType: com.template.CARState
Im not sure if this helps since I don't know for what reasons you want to use arguments passed in the flow. But the RPC client is the way to achieve complex queries, unfortunately that is not covered in my course on Udemy.
We are using Vorto now mainly as a normalized format and are starting to look into using the mapping engine for mapping different payload formats to Vorto model as well. I more or less understand how to map functionblock properties from JSON or binary payload using xpath and the conversion functions. However, I'm not clear how to support parsing of non-fixed format binary payload using this method.
For instance we have an off the shelf LoRaWAN sensor which transmits in the following format:
<length><frame type>[<sensor-id><sensor-value>] where length is the total frame length and sensor-id (for eg temperature, humidity, battery, ...) describes how to parse the sensor-value (ie length, datatype). In one frame multiple of these readings may be present in random order.
Parsing this can be done easily in for instance loraserver.io using a small javascript function which iterates over all the bytes en returns the parsed properties. The same way will work in the Ditto payload mapping engine afaik.
However, currently I don't see how to do something similar in Vorto mapping. This is just one specific sensor example of course, but more examples exist on the market using similar dynamic payload format. I know there is already an open issue (#1535) to improve the documentation, but it would already be helpful to know if such flexible parsing would be possible using the mapping DSL.
I tried passing the raw payload as bytearray to the javascript function. In order to test this I duplicated the org.eclipse.vorto.mapping.engine.converter.binary.BinaryMappingTest#testMappingBinaryContaining2DataPoints and adapted the model to use a custom javascript function like this
evaluator.addScriptFunction(new ScriptClassFunction("extractTemperature",
"function extractTemperature(value) { " +
" print(\"parameter of type \" + typeof value + \", value = \" + value);" +
" print(value[1]);" +
"}"));
The output of this function is
parameter of type number, value = 1
undefined
Where the value 1 is the first element of the bytearray used.
So the function does not seem to receive the parameter as bytarray.
The model is configured with .withXPathStereotype("custom:extractTemperature(data)", "demo") so the payload is passed (as BinaryData) in the same way as in the testMappingBinaryContaining2DataPoints test (.withXPathStereotype("custom:convert(vorto_conversion1:byteArrayToInt(data,0,0,0,2))", "demo")). The only difference I see now is that in the testMappingBinaryContaining2DataPoints test is that the byetarray parameter is passed to a Java function instead of a javascript function. Or am I missing something?
Also, I noticed that loop keywords like for and while are not allowed in the javascript code. So even if I can access the bytearray parameter in the javascript function I see no way for now how to iterate over this.
On gitter I received following reply (together with the suggestion to move discussion to SO)
You are right. We restricted the Javascript function usage to very rudimentary set of language keywords excluding for loops as nasty stuff can be implemented there. What you could do Instead is to register a java function In your own namespace to the mapping engine. That function can hold a byte array. Later this function can be contributed to the mapping engine as a standard function to extract a certain value out for other developers to reuse.
I don't think this is solution to the problem however. As mentioned above this is just one example of an off the shelf sensor payload format, and I don't see how this can be generalized enough to include as a generic function in the mapping engine. And I don't think it should be required to implement a sensor specific conversion in Java, since (as an end-user of an IoT platform wanting to deploy a new sensor type) this is more complex to develop and deploy than a little javascript function which can be altered at runtime in the mapping spec. I see a lot of value in being able to do simple mappings in javascript, just like this can be done in for example loraserver.io and Eclipse Ditto.
I think being able to pass a byte array to javascript is a first step. Also I wonder where exactly the risk is in allowing loops in the javascript? For example Ditto also has some restrictions in the javascript sandbox (see here) but this allows loops and only prevents endless looping and recursion.
They state the following:
Using Rhino instead of Nashorn, the newer JavaScript engine shipped with Java, has the benefit that sandboxing can be applied in a better way.
Sandboxing of different payload scripts is required as Ditto is intended to be run as cloud service where multiple connections to different endpoints are managed for different tenants at the same time. This requires the isolation of each single script to avoid interference with other scripts and to protect the JVM executing the script against harmful code execution.
Would using Rhino in Vorto as well allow to control the risks you see and allow loop construct in Vorto mapping?
PS: can someone with enough SO reputation points add the tag eclipse-vorto please?
I created an issue for you request to support this in the Javascript converters: https://github.com/eclipse/vorto/issues/2029
As stated in the issue, as a current workaround, you can register your own custom converter function with Java and re-use this function across your mappings. In these java converter functions, you have all the power of the java language to convert to extract the right property from the arbitrary list.
In order to find out how to implement your own custom converter function with Java, take a look here: https://github.com/eclipse/vorto/tree/master/mapping-engine#Advanced-Usage
Since Eclipse Vorto 0.12.3 release, a fix for your request is available. With this it is possible to pass array object to javascript Converter as well as use for loops inside javascript functions. You might wanna give it a try.
See release notes https://github.com/eclipse/vorto/blob/master/docs/release-notes.md
I am attempting to implement a solution where I need to write data (json) messages from pubsub into GCS using dataflow. My question is exactly similar to this one
I need to write either based on windowing or element count.
Here is the code sample for writes from the the above question:
windowedValues.apply(FileIO.<String, String>writeDynamic()
.by(Event::getKey)
.via(TextIO.sink())
.to("gs://data_pipeline_events_test/events/")
.withDestinationCoder(StringUtf8Coder.of())
.withNumShards(1)
.withNaming(key -> FileIO.Write.defaultNaming(key, ".json")));
The solution suggests using FileIO.WriteDynamic function. But i am not able to understand what .by(Event::getKey) does and where it comes from.
Any help on this is greatly appreciated.
It's partitioning elements into groups according to events' keys.
From my understanding, the events come from a PCollection using the KV class since it has the getKey method.
Note that :: is a new operator included in Java 8 that is used to refer a method of a class.
Hey I'm using NetBeans and I am using it's Test feature.
It works fine when input is string but how can I test it with when input is java.util.list?
All I see is an open textbox, I tried doing something like new List or comma seperated value, but none worked.
The same with custom objects, how can I test those?
I've been trying that for a while. And believe me I tried with a lot of possibilities.
But then I thought about it, and realized that the NetBeans Tester does not make any conversions to the data that you enter. This means, it is not intelligent enough to convert your String input to a Java specific object (java.util.List).
And if you think about it, it is alright. Because Web Services Definition Language (WSDL) uses XSD as its type system and therefore, there is a limited group of types we can use (without making it too complicated).
So my recommendation is to keep your Web Service parameters as simple as possible (int, string, another string, another int, and so on).
I have two javascript files:
http://www.dublinbikes.ie/design/marseille/javascript/gmaps_common.js
http://www.dublinbikes.ie/design/marseille/javascript/gmaps_search_station.js
Which for http://www.dublinbikes.ie/All-Stations/Station-map# website
What I want to do is to simulate the search a station status action that on the left hand side by set variable 'value' a station number and pass it to gmaps_common.js for execute, then read the result from gmaps_search_station.js that executed variable called html from LeftBlock(id) function.
Because I need run whole process automatically several times, so please suggest me could I use Java to do this and how.
Thank you very much for the help.
You should never try to emulate a JavaScript interpreter just to get js variables out of it - just do the same thing the script would do in your language. Java is mighty enough.
The script just gets a list of stations from http://www.dublinbikes.ie/service/carto, and then queries details from http://www.dublinbikes.ie/service/stationdetails/dublin/{number} (example). Java has lots of HTTP and XML libraries, it will be no problem for you to do that, too.