I am interfacing with Shopify and they use RESTful API. When I request a resource that returns an array of items, they use RFC8288 pagination format.
For example, https://example.com/api/inventory_levels.json?limit=10 returns 10 entities along with the following response header:
Link: <https://example.com/api/inventory_levels.json?limit=10&page_info=eyJs9pZHMiO>;
rel="previous", <https://example.com/api/inventory_levels.json?limit=10&page_info=MiZHeyJs9pO>; rel="next"
Appearantly if I want to retrieve all entites from that resource I need to iterate through the 'next' URL until there's no more 'next' returning. But how am I going to parse these info using JAVA or C# code? I could use a regular expression like <(?<next_url>.*)>; rel="next" to retrieve the 'next_url' from it. But it feels like re-inventing the wheel and not robust.
If this is a well-defined feature, shouldn't there be a readily available library/infrastructure that could be used? I just don't want to be caught by surprise if one day the formatting shows up different (like having an extra space and such) and, whilst abiding to the RFC defination, breaks my hastily scrambled up RegEx solution.
Suggestion welcome for Java or C#.
Related
We are using Vorto now mainly as a normalized format and are starting to look into using the mapping engine for mapping different payload formats to Vorto model as well. I more or less understand how to map functionblock properties from JSON or binary payload using xpath and the conversion functions. However, I'm not clear how to support parsing of non-fixed format binary payload using this method.
For instance we have an off the shelf LoRaWAN sensor which transmits in the following format:
<length><frame type>[<sensor-id><sensor-value>] where length is the total frame length and sensor-id (for eg temperature, humidity, battery, ...) describes how to parse the sensor-value (ie length, datatype). In one frame multiple of these readings may be present in random order.
Parsing this can be done easily in for instance loraserver.io using a small javascript function which iterates over all the bytes en returns the parsed properties. The same way will work in the Ditto payload mapping engine afaik.
However, currently I don't see how to do something similar in Vorto mapping. This is just one specific sensor example of course, but more examples exist on the market using similar dynamic payload format. I know there is already an open issue (#1535) to improve the documentation, but it would already be helpful to know if such flexible parsing would be possible using the mapping DSL.
I tried passing the raw payload as bytearray to the javascript function. In order to test this I duplicated the org.eclipse.vorto.mapping.engine.converter.binary.BinaryMappingTest#testMappingBinaryContaining2DataPoints and adapted the model to use a custom javascript function like this
evaluator.addScriptFunction(new ScriptClassFunction("extractTemperature",
"function extractTemperature(value) { " +
" print(\"parameter of type \" + typeof value + \", value = \" + value);" +
" print(value[1]);" +
"}"));
The output of this function is
parameter of type number, value = 1
undefined
Where the value 1 is the first element of the bytearray used.
So the function does not seem to receive the parameter as bytarray.
The model is configured with .withXPathStereotype("custom:extractTemperature(data)", "demo") so the payload is passed (as BinaryData) in the same way as in the testMappingBinaryContaining2DataPoints test (.withXPathStereotype("custom:convert(vorto_conversion1:byteArrayToInt(data,0,0,0,2))", "demo")). The only difference I see now is that in the testMappingBinaryContaining2DataPoints test is that the byetarray parameter is passed to a Java function instead of a javascript function. Or am I missing something?
Also, I noticed that loop keywords like for and while are not allowed in the javascript code. So even if I can access the bytearray parameter in the javascript function I see no way for now how to iterate over this.
On gitter I received following reply (together with the suggestion to move discussion to SO)
You are right. We restricted the Javascript function usage to very rudimentary set of language keywords excluding for loops as nasty stuff can be implemented there. What you could do Instead is to register a java function In your own namespace to the mapping engine. That function can hold a byte array. Later this function can be contributed to the mapping engine as a standard function to extract a certain value out for other developers to reuse.
I don't think this is solution to the problem however. As mentioned above this is just one example of an off the shelf sensor payload format, and I don't see how this can be generalized enough to include as a generic function in the mapping engine. And I don't think it should be required to implement a sensor specific conversion in Java, since (as an end-user of an IoT platform wanting to deploy a new sensor type) this is more complex to develop and deploy than a little javascript function which can be altered at runtime in the mapping spec. I see a lot of value in being able to do simple mappings in javascript, just like this can be done in for example loraserver.io and Eclipse Ditto.
I think being able to pass a byte array to javascript is a first step. Also I wonder where exactly the risk is in allowing loops in the javascript? For example Ditto also has some restrictions in the javascript sandbox (see here) but this allows loops and only prevents endless looping and recursion.
They state the following:
Using Rhino instead of Nashorn, the newer JavaScript engine shipped with Java, has the benefit that sandboxing can be applied in a better way.
Sandboxing of different payload scripts is required as Ditto is intended to be run as cloud service where multiple connections to different endpoints are managed for different tenants at the same time. This requires the isolation of each single script to avoid interference with other scripts and to protect the JVM executing the script against harmful code execution.
Would using Rhino in Vorto as well allow to control the risks you see and allow loop construct in Vorto mapping?
PS: can someone with enough SO reputation points add the tag eclipse-vorto please?
I created an issue for you request to support this in the Javascript converters: https://github.com/eclipse/vorto/issues/2029
As stated in the issue, as a current workaround, you can register your own custom converter function with Java and re-use this function across your mappings. In these java converter functions, you have all the power of the java language to convert to extract the right property from the arbitrary list.
In order to find out how to implement your own custom converter function with Java, take a look here: https://github.com/eclipse/vorto/tree/master/mapping-engine#Advanced-Usage
Since Eclipse Vorto 0.12.3 release, a fix for your request is available. With this it is possible to pass array object to javascript Converter as well as use for loops inside javascript functions. You might wanna give it a try.
See release notes https://github.com/eclipse/vorto/blob/master/docs/release-notes.md
I have a found a free API to get some data I need for my app. The thing is the values that I get are in English.
I wanted to know if there is some way to translate this strings in my language before showing them to the users.
You can translate text but it will require another API call. Not only that, but you will have to create an appropriate request object and parse a response object from your chosen API.
You have various API providers to choose from, the top Google hits being:
Google: https://cloud.google.com/translate/
Yandex: https://tech.yandex.com/translate/
But beware! Machine translation is patchy at best. The likelihood of getting odd sounding or outright wrong results, particularly for anything other than simple text, is very high.
I want to get the notifications about any change in any issues in my jira server.
I have basic code for connecting jira from java code using jira-rest-java-client library that they have provided.
I searched their javadocs and also went through some classes in that API library but I could not find any methods/classes which would be helpful to me.
Does anyone know if it is possible to get notification events from changes in jira to my java code (may be via polling or something like that).
What do you want to achieve?
You want to have push notifications? There isn't any, IMHO.
UPDATE: However, there is this WebHook thingy: https://confluence.atlassian.com/display/JIRA/Managing+Webhooks.
I have no expertise with it, but it is promising, please read this short introduction also: http://blogs.atlassian.com/2012/10/jira-5-2-remote-integration-webhooks/.
You are looking for something that gives you back what changed in the last N minutes, something like the Activity Stream? You can get the RSS feed of Activity Streams for Projects and for Users.
How
The base URL is https://jira.contoso.com/activity. Then you can append querystring parameters, like maxResults for paginating.
Selecting the data source is through the filters you provide in the streams parameter. It looks like it is JQL, but it's not.
Examples:
List a project's activites: ?streams=key+IS+SOMEPROJ.
List a user's activites: ?streams=user+IS+foobar.
List event between two dates: ?streams=update-date+BETWEEN+1425300236000+1425300264999. (Note: the epoch is the millisecond precision epoch.)
List user activities in one project: ?streams=user+IS+JohnDoe&streams=key+IS+PROJECTKEY.
More complex ones: ?streams=user+IS+JohnDoe&streams=key+IS+PROJECTKEY&streams=activity+IS+issue:close
Watch out, it is case sensitive, on my JIRA 6.1.9, if I write Is instead of IS, I get an error page (but not if AFTER is not all uppercase o.O).
Also note, that spaces should be encoded as plus signs (+), not URL encoded (%20 for spaces).
If you go to your JIRA, and fetch the following URL: https://jira.yourserver.com/rest/activity-stream/1.0/config, it will list all the combinations it accepts.
What
The call returns a standard Atom feed. You can then process it with XML query tools, or with other Java-based RSS/ATOM reader libraries.
Noteworthy document about this topic: https://developer.atlassian.com/docs/atlassian-platform-common-components/activity-streams/consuming-an-activity-streams-feed
I am using PlayFramework2 and I can't find a way to properly handle HTML escaping.
In the template system, HTML entities are filtered by default.
But when I use REST requests with Backbone.js, my JSON objects are not filtered.
I use play.libs.Json.toJson(myModel) to transform an Object into a String.
So, in my controller, I use return ok(Json.toJson(myModel)); to send the response ... but here, the attributes of my model are not secured.
I can't find a way to handle it ...
Second question :
The template engine filters HTML entities by default, this means that we have to store into our database the raw user inputs.
Is it a save behaviour ?
Third questdion :
Is there in the PlayFramework a function to manualy escape strings ? All those I can find require to add new dependencies.
Thanks !
Edit : I found a way at the Backbone.js templating level :
- Use myBackboneModel.escape('attr'); instead of myBackboneModel.get('attr');
Underscore.js templating system also includes that options : <%= attr %> renders without escaping but <%- attr %> renders with escaping !
Just be careful to the efficiency, strings are re-escaped at each rendering. That's why the Backbone .create() should be prefered.
The best practices on XSS-attacks prevention usually recommend you to reason about your output rather than your input. There's a number of reasons behind that. In my opinion the most important are:
It doesn't make any sense to reason about escaping something unless you exactly know how you are going to output/render your data. Because different ways of rendering will require different escaping strategies, e.g. properly escaped HTML string is not good enough to use it in Javascript block. Requirements and technologies change constantly, today you render your data one way - tomorrow you might be using another (let's say you will be working on a mobile client which doesn't require HTML-escaping, because it doesn't use HTML at all to render data) You can only be sure about proper escaping strategy while rendering your data. This is why modern frameworks delegate escaping to templating engines. I'd recommend reviewing the following article: XSS (Cross Site Scripting) Prevention Cheat Sheet
Escaping user's input is actually a destructive/lossy operation – if you escape user's input before persisting it to a storage you will never find out what was his original input. There's no deterministic way to 'unescape' HTML-escaped string, consider my mobile client example above.
That is why I believe that the right way to go would be to delegate escaping to your templating engines (i.e. Play and JS-templating engine you're using for Backbone). There's no need to HTML-escape string you serialize to JSON. Notice that behind the scenes JSON-serializer will JSON-escape your strings, e.g. if you have a quote in your string it will be properly escaped to ensure resulting JSON is correct, because it's a JSON serializer after all that's why it only cares about proper JSON rendering, it knows nothing about HTML (and it shouldn't). However when you rendering your JSON data in the client side you should properly HTML-escape it using the functionality provided by the JS-templating engine you're using for Backbone.
Answering another question: you can use play.api.templates.HtmlFormat to escape raw HTML-string manually:
import play.api.templates.HtmlFormat
...
HtmlFormat.escape("<b>hello</b>").toString()
// -> <b>hello</b>
If you really need to make JSON-encoder escape certain HTML strings, a good idea might be to create a wrapper for them, let's say RawString and provide custom Format[RawString] which will also HTML-escape a string in its writes method. For details see: play.api.libs.json API documentation
I'm looking for a good web framework for compositing multiple JSON sources fetched with HTTP requests in to static HTML. I am not looking to do this on the client-side (browser, javascript), I am looking for the best server-side solution.
So, what I need to do is:
Fetch several different JSON documents over HTTP
Format that JSON as HTML content, mostly with templates but some dynamic custom HTML
Basic login/logout/preferences customization, nothing major
Mostly stateless pages; what state there is, comes already in the JSON
User / search engine friendly / bookmarkable URLs; should be customizable accurately
How I'd like to do it:
A lean solution, perhaps just a template engine
HTML templates that have no custom syntax over HTML/XML, like Wicket and almost like Tapestry
Application server that is scalable and utilizes multiple CPUs properly (for example, a single Python process does not)
Preferably Java, but if Java doesn't have anything that fits, willing to consider others
As for the template part, if this were to be in JavaScript in the browser, something like PURE would be my tool of choice.
You might want to check out RESTx. That's an open source platform for the easy creation of RESTful resources. It allows you to write custom data access and integration logic in either Java or Python. Getting data from multiple sources and combining them is what it's made for, so this should be a pretty close fit. Data output is rendered according to what the user requested. For example, a further JSON data source, or the same data rendered in HTML.
The HTML rendering is currently according to a built-in template. However, that should be easy enough to modify. I'm one of the developers on that project, so if you need some special template functionality, let me know and I will see what I can do.
To give you an example: Assume you have two JSON resources, in your code you would write this (I'm giving a Python example here, but the Java example would look very similar):
status, data_1 = accessResource("/resource/some_resource")
status, data_2 = accessResource("/resource/some_other_resource")
# data_1 and data_2 now hold a dictionary, list, etc., depending on the JSON
# that was returned.
# ... some code that combines and processes the data and produces a dict or list
# with the result. The data object you return here is then automatically rendered
# in either HTML or JSON, depending on the client request.
return Result.ok(data)
Also take a look at the example for some simple data integration here.
I think that the only framework you need is a library that reads json. The templates can very well be standard jsp pages.