Develop Reference

Jolt Transform JSON Spec

I need to transform below Input JSON to output JSON and not sure about how to write spec for that. Need to re-position one field ("homePage") as a root element. Any help or suggestion would be appreciated.
Input JSON :
[{
"uuid": "cac40601-ffc9-4fd0-c5a1-772ac65f0587",
"pageId": 123456,
"page": {
"indexable": true,
"rootLevel": false,
"homePage": false
}
}]
Output JSON :
[{
"uuid": "cac40601-ffc9-4fd0-c5a1-772ac65f0587",
"pageId": 123456,
"homePage": false,
"page": {
"indexable": true,
"rootLevel": false
}
}]

This Jolt Spec should work for you. Tested with https://jolt-demo.appspot.com/
[
{
"operation": "shift",
"spec": {
"*": {
"uuid": "[&1].uuid",
"pageId": "[&1].pageId",
"page": {
"indexable": "[&2].page.indexable",
"rootLevel": "[&2].page.rootLevel",
"homePage": "[&2].homePage"
}
}
}
}
]
input:
{
"uuid" : "cac40601-ffc9-4fd0-c5a1-772ac65f0587",
"pageId" : 123456,
"page" : {
"indexable" : true,
"rootLevel" : false
},
"homePage" : false
}
output:
[ {
"uuid" : "cac40601-ffc9-4fd0-c5a1-772ac65f0587",
"pageId" : 123456,
"page" : {
"indexable" : true,
"rootLevel" : false
},
"homePage" : false
} ]
Explanation:
From the javadoc
& Path lookup
As Shiftr processes data and walks down the spec, it maintains a data structure describing the path it has walked.
The & wildcard can access data from that path in a 0 major, upward oriented way.
Example:
{
"foo" : {
"bar": {
"baz": // &0 = baz, &1 = bar, &2 = foo
}
}
}
Next thing: How to wrap the output object into the array?
A good example can be found in this post.
So, in our case:
"[&1].uuid" says:
Place the uuid value in the object inside the array. The index of the array is indicated by the &1 wildcard. For uuid it will be the index of the array, where the object with uuid key is placed in the original json.
Next, [&2] is similar to [&1]. However, looking at the "indexable" key, it is one level deeper in the input json. Thats why instead of [&1] we used [&2] (have a look again at the foo-bar example from the docs).

What point of same properites at that json?

When I get this json at POST method I need create a object. But I can't see what I need do exactly with this code. Why there is a lot of same properties? Are they a object of different class in fields class?or what? If they are why not like that host:{ "value":"120.515.151.124"}. Could someone help or show some documents about that?
{
"fields": [
{
"name": "host",
"value": "102.164.152.128"
},
{
"name": "port",
"value": "8564"
},
{
"name": "accessKey",
"value":
"(here is a 64 bit a key like(531b8e6c...)"
},
{
"name": "secretKey",
"value":
"(also here is a 64 bit a key like(531b8e6c...)"
}
]
}

fields is an array of key value pairs.. So, this object will contain a fields object, which is a map. Can you try like this..
public class MyClass {
Map<String, String> fields;
}

Dynamic JSON key/value pairs generation in ESQL

How to transform JSON response retrieved from external system to meaningful data (key/value pairs) in ESQL?
Retrieved JSON:
{
"data": [
{
"name": "application.info.header",
"value": "headerValue"
},
{
"name": "entity.statistics.name.fullName",
"value": "fullNameValue"
},
{
"name": "application.info.matter",
"value": "matterValue"
},
{
"name": "entity.statistics.skill",
"value": "skillValue"
}
]
}
where,
name ~ hierarchy of JSON (last attribute being the key)
value ~ value against the key
Expected JSON:
{
"data": {
"application": {
"info": {
"header": "headerValue",
"matter": "matterValue"
}
},
"entity": {
"statistics": {
"name": {
"fullName": "fullNameValue"
},
"skill": "skillValue"
}
}
}
}
Needless to say this can be easily achieved in Java through Split method - I'm looking for a suitable method in ESQL.
Current ESQL Module:
CREATE COMPUTE MODULE getDetails_prepareResponse
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
DECLARE data REFERENCE TO InputRoot.JSON.Data.data.Item[1];
SET OutputRoot.JSON.Data = InputRoot.JSON.Data;
SET OutputRoot.JSON.Data.data = NULL;
WHILE LASTMOVE(data) DO
DECLARE keyA CHARACTER SUBSTRING(data.name BEFORE '.');
DECLARE name CHARACTER SUBSTRING(data.name AFTER '.');
DECLARE keyB CHARACTER SUBSTRING(name BEFORE '.');
DECLARE key CHARACTER SUBSTRING(name AFTER '.');
CREATE LASTCHILD OF OutputRoot.JSON.Data.data.{EVAL('keyA')}.{EVAL('keyB')}
NAME key VALUE data.value;
MOVE data NEXTSIBLING;
END WHILE;
RETURN TRUE;
END;
END MODULE;
This is currently handled through SUBSTRING method in ESQL (for 3 levels only), but now the JSON levels are dynamic (no limit to key/value pairs) as per requirements.

You could implement your own procedure to split a string. Take a look at this answer for an example.
ESQL for splitting a string into mulitple values
The method splits S on Delim into an array in Env (Environment.Split.Array[]) and removes Environment.Split before refilling it.

Jackson map JSON containing different items for same object

I've got the follwoing JSON structure with the corresponding DTOs in Java:
{
"kind": "object 1",
"selfLink": "some_link",
"items": [
{
"kind": "subkind 1",
"name": "server 1",
"anotherObject": {
"link": "some_link",
"isSubcollection": true,
"items": [
{
"att 1": "value",
"att2": "value",
"att3": "value"
},
{
"att5": "value" ,
"att6": "value" ,
"att7": "value" ,
"att8": "value"
}
]
}
}
]
}
Now I want to map this into corresponding DTOs using Jackson. Using the #JsonIgnoreUnknown annotation, this works fine. The problem is within the items array: How can I map different classes from the same JSON list in Jackson? Of course I could create a huge class containing both's attributes, but that would not be my way of choice.
I hope you can help me.

ElasticSearch mapping for dynamic keys for indexing a map

I have a sample json which I want to index into elasticsearch.
Sample Json Indexed:
put test/names/1
{
"1" : {
"name":"abc"
},
"2" : {
"name":"def"
},
"3" : {
"name":"xyz"
}
}
where ,
index name : test,
type name : names,
id :1
Now the default mapping generated by elasticsearch is :
{
"test": {
"mappings": {
"names": {
"properties": {
"1": {
"properties": {
"name": {
"type": "string"
}
}
},
"2": {
"properties": {
"name": {
"type": "string"
}
}
},
"3": {
"properties": {
"name": {
"type": "string"
}
}
},
"metadataFieldDefinition": {
"properties": {
"name": {
"type": "string"
}
}
}
}
}
}
}
}
If the map size increases from 3 ( currently) to suppose thousand or million, then ElasticSearch will create a mapping for each which may cause a performance issue as the mapping collection will be huge .
I tried creating a mapping by setting :
"dynamic":false,
"type":object
but it was overriden by ES. since it didnt match the indexed data.
Please let me know how can I define a mapping so that ES. doesnot creates one like the above .

I think there might be a little confusion here in terms of how we index documents.
put test/names/1
{...
document
...}
This says: the following document belongs to index test and is of type name with id 1. The entire document is treated as type name. Using the PUT API as you currently are, you cannot index multiple documents at once. ES immediately interprets 1, 2, and 3 as a properties of type object, each containing a property name of type string.
Effectively, ES thinks you are trying to index ONE document, instead of three
To get many documents into index test with a type of name, you could do this, using the CURL syntax:
curl -XPUT"http://your-es-server:9200/test/names/1" -d'
{
"name": "abc"
}'
curl -XPUT"http://your-es-server:9200/test/names/2" -d'
{
"name": "ghi"
}'
curl -XPUT"http://your-es-server:9200/test/names/3" -d'
{
"name": "xyz"
}'
This will specify the document ID in the endpoint you are index to. Your mapping will then look like this:
"test": {
"mappings": {
"names": {
"properties": {
"name": {
"type": "string"
}
}
}
}
}
Final Word: Split your indexing up into discrete operations, or check out the Bulk API to see the syntax on how to POST multiple operations in a single request.