I'am new using jolt tool and i wonder if there a way to take several json objects and put them to an array with a new field named as follows:
Input :
{
"userId": 1,
"age": 20,
"desc1": "value desc1",
"desc2": "value desc2",
"desc3": "value desc3"
}
JSON spec :
[
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"*": "additionalInformation"
}
}
]
Expected result :
{
"ID": 1,
"age": 20,
"additionalInformation": [
{
"code": "desc1",
"value": "value desc1"
},
{
"code": "desc2",
"value": "value desc2"
},
{
"code": "desc3",
"value": "value desc3"
}
]
}
Using the spec above i can only obtain this result :
{
"ID": 1,
"test": 20,
"additionalInformation": [
"value desc1",
"value desc2",
"value desc3"
]
}
Any suggestion what i have missed ?
You can use $ and # wildcards for code and value, respectively while combining the elements under common factor by using &. as prefixes for them in the first step. Then combine elements other than ID and age within the same array in the last step such as
[
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"*": {
"$": "&.code",
"#": "&.value"
}
}
},
{
"operation": "shift",
"spec": {
"ID": "&",
"age": "&",
"*": "additionalInformation[]"
}
}
]
Edit(due to the comment) : If you need to pick some individual elements such as desc1 and desc3 rather than all("*"), then replace the first spec with following
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"desc1|desc3": {
"$": "&.code",
"#": "&.value"
}
}
}
Related
I'm struggling to trans my input response to a specific output using jolt, below is my example
The input that describe my message
{
"firstAttribute": true,
"secondAttribute": "12",
"data": {
"propertyKey1": "1",
"propertyKey2": [
"a"
],
"propertyKey3": "2",
"propertyKey4": "3",
"propertyKey_test": [
"option1",
"option2",
"option3"
],
"propertyKey5": "4",
"propertyKey6": "87.0"
},
"Keytest1": "value1",
"KeyTest2": "value2"
}
The used jolt Spec
[
{
"operation": "shift",
"spec": {
"data": {
"propertyKey2": {
"0": "propertyKey2"
},
"*": {
"#": "data.&"
}
},
"*": {
"#": "&"
}
}
}
]
My actual output after a jolt transfomration
{
"firstAttribute" : true,
"secondAttribute" : "12",
"data" : {
"propertyKey1" : "1",
"propertyKey3" : "2",
"propertyKey4" : "3",
"propertyKey_test" : [ "option1", "option2", "option3" ],
"propertyKey5" : "4",
"propertyKey6" : "87.0"
},
"propertyKey2" : "a",
"Keytest1" : "value1",
"KeyTest2" : "value2"
}
The desired output is to convert every array fields to separated fields such as propertyKey_test as below
{
"firstAttribute" : true,
"secondAttribute" : "12",
"data" : {
"propertyKey1" : "1",
"propertyKey3" : "2",
"propertyKey4" : "3",
"propertyKey_test": "option1",
"propertyKey5" : "4",
"propertyKey6" : "87.0"
},
"propertyKey2" : "a",
"Keytest1" : "value1",
"KeyTest2" : "value2"
}
Any help would be appreciated. Thanks in advance
PS: we can receive a dynamic array fields (ex: today we have propertyKey_test:["option1","option2","option3"] tomorrow we could receive another array field for example new_field:["a","b","c"].
My goal is to check everytime if the field is an array and take just the first element as described above
The current desired output is not a valid JSON value, since there may not exist more than one attribute with identical key within a common object. But, if you need to differentiate the components of the "propertyKey_test" array, then using the indexes of the array as below might be preferred such as
[
{
"operation": "shift",
"spec": {
"data": {
"propertyKey2": {
"0": "&1"
},
"*": {
"#": "&2.&"
},
"propertyKey_*": {
"*": {
"#": "&3.&2&1"
}
}
},
"*": {
"#": "&"
}
}
}
]
in order to get
{
"firstAttribute" : true,
"secondAttribute" : "12",
"data" : {
"propertyKey1" : "1",
"propertyKey3" : "2",
"propertyKey4" : "3",
"propertyKey_test0" : "option1",
"propertyKey_test1" : "option2",
"propertyKey_test2" : "option3",
"propertyKey5" : "4",
"propertyKey6" : "87.0"
},
"propertyKey2" : "a",
"Keytest1" : "value1",
"KeyTest2" : "value2"
}
as output.
I'm struggling with that Jolt transformation:
Here is my input JSON
{
"bloc1-1": {
"bloc1-2": [
{
"key": "key1",
"value": "value1-1"
},
{
"key": "key2",
"value": "value1-2"
}
]
},
"bloc2-1": {
"bloc2-2": [
{
"key": "key1",
"value": "value2-1"
},
{
"key": "key2",
"value": "value2-2"
},
{
"key": "key3",
"value": "value2-3"
}
]
}
}
Here is what I'm expecting
{
"bloc1-key1" : "value1-1",
"bloc1-key2" : "value1-2",
"bloc2-key1" : "value2-1",
"bloc2-key2" : "value2-2",
"bloc2-key3" : "value2-3"
}
I have tried the following spec, but I cannot figure out how to prefix the key in the RHS (the # should be the first character)
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"value": "#(1,key)"
}
}
}
}
}
]
and got that
{
"key1" : [ "value1-1", "value2-1" ],
"key2" : [ "value1-2", "value2-2" ],
"key3" : "value2-3"
}
Any help would be appreciated
The spec below should do the trick:
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"$2": "keyParts1",
"key": "keyParts2",
"value": "values"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"keyParts1": {
"*": "[#1].part1"
},
"keyParts2": {
"*": "[#1].part2"
},
"values": {
"*": "[#1].value"
}
}
},
{
"operation": "modify-default-beta",
"spec": {
"*": {
"newKey": "=split('-', #(1,part1))",
"newKeyFirst": "#(1,newKey[0])",
"newKeyComplete": "=concat(#(1,newKeyFirst),'-',#(1,part2))"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,newKeyComplete)"
}
}
}
]
Try to apply each of the operations separately (for example in the demo app).
Check out the links to the jolt docs and the examples from this answer
Step by step explanation:
First "shift" transformation places the components of the final result in the separate arrays:
{
"keyParts1" : [ "bloc1-1", "bloc1-1", "bloc2-1", "bloc2-1", "bloc2-1" ],
"keyParts2" : [ "key1", "key2", "key1", "key2", "key3" ],
"values" : [ "value1-1", "value1-2", "value2-1", "value2-2", "value2-3" ]
}
The above operation let us have each of the components of the end result positioned in the right index of the desired result.
Note that each array has the same size.
In the second operation we group each of the array elements by its index using the # operator. Looking at the docs [#1] can be translated to:
"Go up one level and ask that node how many matches it has had before finding the current element".
The node above is an array, so for the 3rd array element it will be the array index 2.
[
...
{
"part1" : "bloc2-1",
"part2" : "key1",
"value" : "value2-1"
}
...
]
In the 3rd operation I have assumed that the key of the desired result is the concatenation of the "blocX" substring of the "blocX-Y" key concatenated with the "key" value. That's why I've used the "modify-default-beta". Have a look at the example from the demo app
Each of five elements of the array is transformed similarly to the 3rd array element:
[
...
{
"part1" : "bloc2-1",
"part2" : "key1",
"value" : "value2-1",
"newKey" : [ "bloc2", "1" ],
"newKeyFirst" : "bloc2",
"newKeyComplete" : "bloc2-key1"
}
...
]
The last operation produces the desired result from the 3rd operation's output.
You may consider another library Josson.
https://github.com/octomix/josson
Josson josson = Josson.fromJsonString(inputJSON);
JsonNode node = josson.getNode(
"entries()" + // Field "key" generated by entries() is the same as the inner one.
".field(k:key)" + // So, rename it to "k".
".unwind(value.*)" + // Unwind the "value" generated by entries(), not the inner "value".
".map(concat(k.keepBefore('-'),'-',key)::value)" +
".mergeObjects()");
System.out.println(node.toPrettyString());
Output
{
"bloc1-key1" : "value1-1",
"bloc1-key2" : "value1-2",
"bloc2-key1" : "value2-1",
"bloc2-key2" : "value2-2",
"bloc2-key3" : "value2-3"
}
When Calling an external API I am receiving a complex json structure, the same I need to transform into a simple json as per our need. I found jolt has the capability to transform a json but not able to come up with the jolt spec.
My input Json array --
{
"attribute": [
"teamalloc",
"prodAlloc"
],
"item": {
"id": "abcde",
"name": "Champak Kumar",
"allocDetails": {
"updatedAt": "2020-08-10T14:26:48-07:00",
"token": 1134,
"items": [
{
"allocation": 0.2,
"team": {
"id": 90,
"name": "Some Team Name 1",
"createdAt": "2010-01-19T10:52:52-07:00"
}
},
{
"allocation": 0.9,
"team": {
"id": 80,
"name": "Some Team Name 2",
"createdAt": "2010-01-19T10:52:52-07:00",
"product": {
"id": 20,
"name": "Some Product Name 1",
"otherDetails": {
"key": "Id",
"value": "GEC"
}
}
}
},
{
"allocation": 0.1,
"team": {
"id": 10,
"name": "Some Team Name 3",
"createdAt": "2010-01-19T10:52:52-07:00",
"product": {
"id": 22,
"name": "Some Product Name 2",
"otherDetails": {
"key": "Id1",
"value": "GEC1"
}
}
}
}
]
}
}
}
My output Json structure should look like --
{
"name": "Champak Kumar",
"allocDetails": [
{
"allocation": 0.2,
"team": {
"id": 90,
"name": "Some Team Name 1"
}
},
{
"allocation": 0.9,
"team": {
"id": 80,
"name": "Some Team Name 2",
"product": {
"id": 20,
"name": "Some Product Name 1"
}
}
},
{
"allocation": 0.1,
"team": {
"id": 10,
"name": "Some Team Name 3",
"product": {
"id": 22,
"name": "Some Product Name 2"
}
}
}
]
}
I tried multiple jolt spec but couldn't come up with the desired output. What should be the ideal jolt spec for this?
This spec should using shift operation should work:
[
{
"operation": "shift",
"spec": {
"item": {
"name": "name",
"allocDetails": {
"items": {
"*": {
"allocation": "allocDetails[&1].allocation",
"team": {
"id": "allocDetails[&2].team.id",
"name": "allocDetails[&2].team.name",
"product": "allocDetails[&2].team.product"
}
}
}
}
}
}
}
]
Edit #1:
The explanation:
The shift operation spec defines where do we want to put the values from the input json in the result json.
Each key - value pair in the spec defines the source - target relation.
For example (let's start from the simplest one): value from the ["item"]["name"] will land under the ["name"] key in the output JSON.
The
"items": {
"*": {
...
}
}
part says: "For each element of the array under the 'items' key do ..."
The ampersand operator "&X" lets us reference the key that is X levels above the LHS (the key in question). For example:
Let's imagine we have the 57th element of the items array (counting from 0):
...
"name": "allocDetails[&2].team.name"
...
says: "Place the value found under the "name" key (that is in item["items"][56]["team"]["name"] in the 57th element of the array placed under the "allocDetails" key.
'*' matches the 57th element. '&2' lets us find out that the element of the array we are working on right now is the 57th element.
Have a look at the shift operation javadocs, especially at the "&" and "*" wildcards
Edit #2:
considering the otherDetails comment:
You could also approach it like this:
...
"team": {
"id": "allocDetails[&2].team.id",
"name": "allocDetails[&2].team.name",
"product": {
"otherDetails": null,
"*": "allocDetails[&3].team.product.&"
}
}
...
the above: puts all of the products' part keys (matched by the "*") to the key with the same name ("&") placed in the "allocDetails[&3].team.product" output json's key... with the exception of the "otherDetails" key.
Working jolt spec --
{
"operation": "shift",
"spec": {
"item": {
"name": "name",
"allocDetails": {
"items": {
"*": {
"allocation": "allocDetails[&1].allocation",
"team": {
"id": "allocDetails[&2].team.id",
"name": "allocDetails[&2].team.name",
"product": {
"id": "allocDetails[&3].team.product.id",
"name": "allocDetails[&3].team.product.name"
}
}
}
}
}
}
}
}
]```
I am trying to do JOLT shift operation with below spec which is not working. Not sure what mistake I have done. Need help in this case. Output JSON is coming as an object instead of Array and shift also not working as expected.
Input : [
{
"Header": {
"Number": 1,
"Id": "JO"
},
"Name": "John"
},
{
"Header": {
"Number": 2,
"Id": "JS"
},
"Name": "Justin"
}
]
Spec : [
{
"operation": "shift",
"spec": {
"*": {
"Header": "Header",
"Name": "Header.Name"
}
}
}
]
Expected Output : [
{
"Header": {
"Number": 1,
"Id": "JO",
"Name": "John"
}
},
{
"Header": {
"Number": 2,
"Id": "JS",
"Name": "Justin"
}
}
]
Actual Output : {
"Header" : [ {
"Number" : 1,
"Id" : "JO",
"Name" : "John"
}, {
"Number" : 2,
"Id" : "JS"
} ]
}
You have to also specify that the "Header" object is inside the array.
Moreover, the index of the array where you place the "Header" object for each of the element of the array.
That's what the spec below does (using the [&1] - apmersand wildcard combined with array):
[
{
"operation": "shift",
"spec": {
"*": {
"Header": "[&1].Header",
"Name": "[&1].Header.Name"
}
}
}
]
Sources:
Shiftr.java javadocs:
The arrays
The ampersand wildcard
Other answer: "How do I transform an array using Jolt?"
Demo application linked in the jolt repo to test the spec
I have a bit of a tricky question to ask, as far as I know, I haven't seen anyone with the same problem, at least not all of them at once!
I have the following test JSON to transform:
INPUT
[
{
"test_id": 1212101011,
"someDate": "2020-03-06",
"keyToCheck": "true",
"someData": "123"
},
{
"test_id": 8787909099,
"someDate": "2020-03-09",
"keyToCheck": "false",
"someData": "456"
}
]
And I am trying to get the following output.
DESIRED
[{
"test_id": 1212101011,
"someDate": "2020-03-06",
"keyToCheck": "true",
"someData": "123",
"objToAdd": {
"id": "1",
"label": "dummy1"
}
},{
"test_id": 8787909099,
"someDate": "2020-03-09",
"keyToCheck": "false",
"someData": "456",
"objToAdd": {
"id": "2",
"label": "dummy2"
}]
I already managed to propagate all the values and change the fields' names if needed, but adding a condition is just ruining everything. Also, with the current spec I have, I am not getting an array after the transform which I can already get with my former spec.
The spec I have so far is:
SPEC USED
[
{
"operation": "shift",
"spec": {
"*": {
"keyToCheck": {
"true": {
"#1": "objToAdd.id",
"#dummy1": "objToAdd.label"
},
"false": {
"#2": "objToAdd.id",
"#dummy2": "objToAdd.label"
}
}
}
}
}
]
But as said before, it is only giving me this:
RESULT SO FAR
{
"objToAdd" : {
"id" : [ "1", "2" ],
"label" : [ "dummy1", "dummy2" ]
}
}
Which seems to be a problem of indentation. Adding [&1] before the values in the condition didn't help, and when I propagate the values with "*": "[&1].&", I am losing the item generated by the if-else condition.
Is there any way to do this?
Any help would be appreciated! Thanks in advance.
Check this spec,
[
{
"operation": "shift",
"spec": {
"*": {
"test_id": "[&1].test_id",
"someDate": "[&1].someDate",
"someData": "[&1].someData",
"keyToCheck": {
"true": {
"$": "[&3].keyToCheck",
"#1": "[&3].objToAdd.id",
"#dummy1": "[&3].objToAdd.label"
},
"false": {
"$": "[&3].keyToCheck",
"#2": "[&3].objToAdd.id",
"#dummy2": "[&3].objToAdd.label"
}
}
}
}
}
]