I'm struggling with that Jolt transformation:
Here is my input JSON
{
"bloc1-1": {
"bloc1-2": [
{
"key": "key1",
"value": "value1-1"
},
{
"key": "key2",
"value": "value1-2"
}
]
},
"bloc2-1": {
"bloc2-2": [
{
"key": "key1",
"value": "value2-1"
},
{
"key": "key2",
"value": "value2-2"
},
{
"key": "key3",
"value": "value2-3"
}
]
}
}
Here is what I'm expecting
{
"bloc1-key1" : "value1-1",
"bloc1-key2" : "value1-2",
"bloc2-key1" : "value2-1",
"bloc2-key2" : "value2-2",
"bloc2-key3" : "value2-3"
}
I have tried the following spec, but I cannot figure out how to prefix the key in the RHS (the # should be the first character)
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"value": "#(1,key)"
}
}
}
}
}
]
and got that
{
"key1" : [ "value1-1", "value2-1" ],
"key2" : [ "value1-2", "value2-2" ],
"key3" : "value2-3"
}
Any help would be appreciated
The spec below should do the trick:
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"$2": "keyParts1",
"key": "keyParts2",
"value": "values"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"keyParts1": {
"*": "[#1].part1"
},
"keyParts2": {
"*": "[#1].part2"
},
"values": {
"*": "[#1].value"
}
}
},
{
"operation": "modify-default-beta",
"spec": {
"*": {
"newKey": "=split('-', #(1,part1))",
"newKeyFirst": "#(1,newKey[0])",
"newKeyComplete": "=concat(#(1,newKeyFirst),'-',#(1,part2))"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,newKeyComplete)"
}
}
}
]
Try to apply each of the operations separately (for example in the demo app).
Check out the links to the jolt docs and the examples from this answer
Step by step explanation:
First "shift" transformation places the components of the final result in the separate arrays:
{
"keyParts1" : [ "bloc1-1", "bloc1-1", "bloc2-1", "bloc2-1", "bloc2-1" ],
"keyParts2" : [ "key1", "key2", "key1", "key2", "key3" ],
"values" : [ "value1-1", "value1-2", "value2-1", "value2-2", "value2-3" ]
}
The above operation let us have each of the components of the end result positioned in the right index of the desired result.
Note that each array has the same size.
In the second operation we group each of the array elements by its index using the # operator. Looking at the docs [#1] can be translated to:
"Go up one level and ask that node how many matches it has had before finding the current element".
The node above is an array, so for the 3rd array element it will be the array index 2.
[
...
{
"part1" : "bloc2-1",
"part2" : "key1",
"value" : "value2-1"
}
...
]
In the 3rd operation I have assumed that the key of the desired result is the concatenation of the "blocX" substring of the "blocX-Y" key concatenated with the "key" value. That's why I've used the "modify-default-beta". Have a look at the example from the demo app
Each of five elements of the array is transformed similarly to the 3rd array element:
[
...
{
"part1" : "bloc2-1",
"part2" : "key1",
"value" : "value2-1",
"newKey" : [ "bloc2", "1" ],
"newKeyFirst" : "bloc2",
"newKeyComplete" : "bloc2-key1"
}
...
]
The last operation produces the desired result from the 3rd operation's output.
You may consider another library Josson.
https://github.com/octomix/josson
Josson josson = Josson.fromJsonString(inputJSON);
JsonNode node = josson.getNode(
"entries()" + // Field "key" generated by entries() is the same as the inner one.
".field(k:key)" + // So, rename it to "k".
".unwind(value.*)" + // Unwind the "value" generated by entries(), not the inner "value".
".map(concat(k.keepBefore('-'),'-',key)::value)" +
".mergeObjects()");
System.out.println(node.toPrettyString());
Output
{
"bloc1-key1" : "value1-1",
"bloc1-key2" : "value1-2",
"bloc2-key1" : "value2-1",
"bloc2-key2" : "value2-2",
"bloc2-key3" : "value2-3"
}
Related
I'm struggling to trans my input response to a specific output using jolt, below is my example
The input that describe my message
{
"firstAttribute": true,
"secondAttribute": "12",
"data": {
"propertyKey1": "1",
"propertyKey2": [
"a"
],
"propertyKey3": "2",
"propertyKey4": "3",
"propertyKey_test": [
"option1",
"option2",
"option3"
],
"propertyKey5": "4",
"propertyKey6": "87.0"
},
"Keytest1": "value1",
"KeyTest2": "value2"
}
The used jolt Spec
[
{
"operation": "shift",
"spec": {
"data": {
"propertyKey2": {
"0": "propertyKey2"
},
"*": {
"#": "data.&"
}
},
"*": {
"#": "&"
}
}
}
]
My actual output after a jolt transfomration
{
"firstAttribute" : true,
"secondAttribute" : "12",
"data" : {
"propertyKey1" : "1",
"propertyKey3" : "2",
"propertyKey4" : "3",
"propertyKey_test" : [ "option1", "option2", "option3" ],
"propertyKey5" : "4",
"propertyKey6" : "87.0"
},
"propertyKey2" : "a",
"Keytest1" : "value1",
"KeyTest2" : "value2"
}
The desired output is to convert every array fields to separated fields such as propertyKey_test as below
{
"firstAttribute" : true,
"secondAttribute" : "12",
"data" : {
"propertyKey1" : "1",
"propertyKey3" : "2",
"propertyKey4" : "3",
"propertyKey_test": "option1",
"propertyKey5" : "4",
"propertyKey6" : "87.0"
},
"propertyKey2" : "a",
"Keytest1" : "value1",
"KeyTest2" : "value2"
}
Any help would be appreciated. Thanks in advance
PS: we can receive a dynamic array fields (ex: today we have propertyKey_test:["option1","option2","option3"] tomorrow we could receive another array field for example new_field:["a","b","c"].
My goal is to check everytime if the field is an array and take just the first element as described above
The current desired output is not a valid JSON value, since there may not exist more than one attribute with identical key within a common object. But, if you need to differentiate the components of the "propertyKey_test" array, then using the indexes of the array as below might be preferred such as
[
{
"operation": "shift",
"spec": {
"data": {
"propertyKey2": {
"0": "&1"
},
"*": {
"#": "&2.&"
},
"propertyKey_*": {
"*": {
"#": "&3.&2&1"
}
}
},
"*": {
"#": "&"
}
}
}
]
in order to get
{
"firstAttribute" : true,
"secondAttribute" : "12",
"data" : {
"propertyKey1" : "1",
"propertyKey3" : "2",
"propertyKey4" : "3",
"propertyKey_test0" : "option1",
"propertyKey_test1" : "option2",
"propertyKey_test2" : "option3",
"propertyKey5" : "4",
"propertyKey6" : "87.0"
},
"propertyKey2" : "a",
"Keytest1" : "value1",
"KeyTest2" : "value2"
}
as output.
I'm trying to construct an array of objects named catalogs with different combinations of input values, but can't set some attributes in the same object. This is the current progress:
[
{
"operation": "shift",
"spec": {
"new": {
"bc_sku_channel": {
"*": {
"Partner": null, //ignore if value is Partner
"*": {
"#BC": "catalogs[#3].catalog",
"#1": "catalogs[#3].channel"
}
}
},
"bc_sku_partner": {
"*": {
"#BC": "catalogs[].catalog", // don't know which index should be here to group these 3 attributes into the same object
"#Partner": "catalogs[].channel", // tried #3,#2,#1,#4 but doesn't work
"#": "catalogs[].partner"
}
},
"cc_sku_channel": {
"*": {
"Partner": null, //ignore if value is Partner
"*": {
"#CC": "catalogs[#3].catalog",
"#1": "catalogs[#3].channel"
}
}
}
}
}
}
]
Input:
{
"new": {
"bc_sku_partner": [
"Amazon",
"Ebay"
],
"bc_sku_channel": [
"Partner",
"Online",
"Store"
],
"cc_sku_channel": [
"Store"
]
}
}
Expected Output:
{
"catalogs": [
{
"catalog": "BC",
"channel": "Partner",
"partner": "Amazon"
},
{
"catalog": "BC",
"channel": "Partner",
"partner": "Ebay"
},
{
"catalog": "BC",
"channel": "Online"
},
{
"catalog": "BC",
"channel": "Store"
},
{
"catalog": "CC",
"channel": "Store"
}
]
}
Now I have only been able to build the last 3 objects.
Notes:
If channel is Online or Store the object shouldn't have the partner attribute.
For each partner, the channel attribute should be always Partner
This spec produces the expected output
Version which assumes that bc_ and cc_ prefixes are dynamic:
Simpler version can be found at the end of this answer.
[
{
"operation": "shift",
"spec": {
"new": {
"*_sku_partner": { // 1
"*": { // 2
"*": { // 3
"#1": { // 4
"$1": "partner__&1__&(4,1).partner", // 5
"#Partner": "partner__&1__&(4,1).channel", // 6
"$(3,1)": "partner__&1__&(4,1).catalogLowerCase" // 7
}
}
}
},
"*_sku_channel": {
"*": {
"Partner": null, // 8
"*": {
"#1": {
"$1": "channel__&1__&(4,1).channel",
"$(3,1)": "channel__&1__&(4,1).catalogLowerCase"
}
}
}
}
}
}
},
{
"operation": "shift", // 9
"spec": {
"partner__*": "catalogs[#1]",
"channel__*": "catalogs[#1]"
}
},
{
"operation": "modify-default-beta", // 10
"spec": {
"catalogs": {
"*": {
"catalog": "=toUpper(#(1,catalogLowerCase))"
}
}
}
},
{
"operation": "shift", // 11
"spec": {
"catalogs": {
"*": {
"catalogLowerCase": null,
"*": "&2[&1].&"
}
}
}
}
]
The first operation is the trickiest, I guess:
It creates a dictionary with unique entries, like:
{
"channel__Online__bc" : {
"catalogLowerCase" : "bc",
"channel" : "Online"
}
// ...
"partner__Amazon__bc": {
"catalogLowerCase": "bc",
"channer": "Partner",
"partner": "Amazon"
}
}
from:
// ...
"bc_sku_partner": [
"Amazon",
"Ebay"
]
// ...
Let's focus on the first entry generated for "Amazon":
Ad. 1. *_sku_partner matches "bc_sku_partner". You could also replace the asterisk with the explicit bc value and instead of $(3,1) in ad. 7 place #BC
Ad. 2. Matches the elements of the ["Amazon", ...] list.
Ad. 3. We are now inside "Amazon" :D
Ad. 4. We go into "Amazon" and reach for it (#1). Related docs: # shift wildcard
Ad. 5. Looking at the ($ wildcard docs)
'$' has the same syntax as the '&' wildcard, and can be read as, dereference to get a value, and then use that value as the data to be output.
$1 goes up one level, grabs the value ("Amazon") and places it under they "partner__&1__&(4,1).partner: key.
"partner__&1__&(4,1)" for the "Amazon" element generates: "partner__Amazon__bc". &(4,1) means here:
Go up 4 levels (we are at "bc_sku_partner") and take the value matched by the asterisk. See: & wildcard docs
Ad. 6. Places "Partner" string under the "channel" key (next to the "partner": "Amazon" key-value).
Ad. 7. $(3,1) again but here we reach for the "bc" matched by the asterisk and place it under the "catalogLowerCase" key.
The "*_sku_channel" part is quite similar, so I am not going to explain it.
At ad. 8. is the removal of the "Partner" entries.
Ad. 9. The second shift operation. It's purpose is to transform the dictionary produced by the 1st shift operation into the list placed under the "catalogs" key. # wildcard docs may be useful here.
Ad. 10 and 11 replace the "catalogLowerCase": "bc" with "catalog": "BC". Same for the similar "catalogLowerCase" fields related to the "Ebay", "Online" and "Store" entries.
Try to apply the operations incrementally, to see the intermediate result. I recommend this website for testing: https://jolt-demo.appspot.com/. It also has a lot of examples.
Edit:
Other way to do it, if the bc_ and cc_ prefixes are not dynamic.
[
{
"operation": "shift",
"spec": {
"new": {
"bc_sku_partner": { // 1
"*": { // 2
"*": { // 3
"#1": { // 4
"$1": "partner__&1__bc.partner", // 5
"#Partner": "partner__&1__bc.channel", // 6
"#BC": "partner__&1__bc.catalog" // 7
}
}
}
},
"bc_sku_channel": {
"*": {
"Partner": null,
"*": {
"#1": {
"$1": "channel__&1__bc.channel",
"#BC": "channel__&1__bc.catalog"
}
}
}
},
"cc_sku_channel": {
"*": {
"Partner": null,
"*": {
"#1": {
"$1": "channel__&1__cc.channel",
"#CC": "channel__&1__cc.catalog"
}
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"partner__*": "catalogs[#1]",
"channel__*": "catalogs[#1]"
}
}
]
You may consider another library Josson.
https://github.com/octomix/josson
Deserialization
Josson josson = Josson.fromJsonString(
"{" +
" \"new\": {" +
" \"bc_sku_partner\": [" +
" \"Amazon\"," +
" \"Ebay\"" +
" ]," +
" \"bc_sku_channel\": [" +
" \"Partner\"," +
" \"Online\"," +
" \"Store\"" +
" ]," +
" \"cc_sku_channel\": [" +
" \"Store\"" +
" ]" +
" }" +
"}");
Transformation
JsonNode node = josson.getNode(
"new" +
".entries()" +
".unwind(value)" +
".[key='bc_sku_partner' | value!='Partner']*" +
".map(catalog:key.substr(0,2).upperCase()," +
" channel:if([key='bc_sku_partner'],'Partner',value)," +
" partner:if([key='bc_sku_partner'],value))" +
".toObject('catalogs')");
System.out.println(node.toPrettyString());
Output
{
"catalogs" : [ {
"catalog" : "BC",
"channel" : "Partner",
"partner" : "Amazon"
}, {
"catalog" : "BC",
"channel" : "Partner",
"partner" : "Ebay"
}, {
"catalog" : "BC",
"channel" : "Online"
}, {
"catalog" : "BC",
"channel" : "Store"
}, {
"catalog" : "CC",
"channel" : "Store"
} ]
}
I'am new using jolt tool and i wonder if there a way to take several json objects and put them to an array with a new field named as follows:
Input :
{
"userId": 1,
"age": 20,
"desc1": "value desc1",
"desc2": "value desc2",
"desc3": "value desc3"
}
JSON spec :
[
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"*": "additionalInformation"
}
}
]
Expected result :
{
"ID": 1,
"age": 20,
"additionalInformation": [
{
"code": "desc1",
"value": "value desc1"
},
{
"code": "desc2",
"value": "value desc2"
},
{
"code": "desc3",
"value": "value desc3"
}
]
}
Using the spec above i can only obtain this result :
{
"ID": 1,
"test": 20,
"additionalInformation": [
"value desc1",
"value desc2",
"value desc3"
]
}
Any suggestion what i have missed ?
You can use $ and # wildcards for code and value, respectively while combining the elements under common factor by using &. as prefixes for them in the first step. Then combine elements other than ID and age within the same array in the last step such as
[
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"*": {
"$": "&.code",
"#": "&.value"
}
}
},
{
"operation": "shift",
"spec": {
"ID": "&",
"age": "&",
"*": "additionalInformation[]"
}
}
]
Edit(due to the comment) : If you need to pick some individual elements such as desc1 and desc3 rather than all("*"), then replace the first spec with following
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"desc1|desc3": {
"$": "&.code",
"#": "&.value"
}
}
}
I am trying to do JOLT shift operation with below spec which is not working. Not sure what mistake I have done. Need help in this case. Output JSON is coming as an object instead of Array and shift also not working as expected.
Input : [
{
"Header": {
"Number": 1,
"Id": "JO"
},
"Name": "John"
},
{
"Header": {
"Number": 2,
"Id": "JS"
},
"Name": "Justin"
}
]
Spec : [
{
"operation": "shift",
"spec": {
"*": {
"Header": "Header",
"Name": "Header.Name"
}
}
}
]
Expected Output : [
{
"Header": {
"Number": 1,
"Id": "JO",
"Name": "John"
}
},
{
"Header": {
"Number": 2,
"Id": "JS",
"Name": "Justin"
}
}
]
Actual Output : {
"Header" : [ {
"Number" : 1,
"Id" : "JO",
"Name" : "John"
}, {
"Number" : 2,
"Id" : "JS"
} ]
}
You have to also specify that the "Header" object is inside the array.
Moreover, the index of the array where you place the "Header" object for each of the element of the array.
That's what the spec below does (using the [&1] - apmersand wildcard combined with array):
[
{
"operation": "shift",
"spec": {
"*": {
"Header": "[&1].Header",
"Name": "[&1].Header.Name"
}
}
}
]
Sources:
Shiftr.java javadocs:
The arrays
The ampersand wildcard
Other answer: "How do I transform an array using Jolt?"
Demo application linked in the jolt repo to test the spec
I have a bit of a tricky question to ask, as far as I know, I haven't seen anyone with the same problem, at least not all of them at once!
I have the following test JSON to transform:
INPUT
[
{
"test_id": 1212101011,
"someDate": "2020-03-06",
"keyToCheck": "true",
"someData": "123"
},
{
"test_id": 8787909099,
"someDate": "2020-03-09",
"keyToCheck": "false",
"someData": "456"
}
]
And I am trying to get the following output.
DESIRED
[{
"test_id": 1212101011,
"someDate": "2020-03-06",
"keyToCheck": "true",
"someData": "123",
"objToAdd": {
"id": "1",
"label": "dummy1"
}
},{
"test_id": 8787909099,
"someDate": "2020-03-09",
"keyToCheck": "false",
"someData": "456",
"objToAdd": {
"id": "2",
"label": "dummy2"
}]
I already managed to propagate all the values and change the fields' names if needed, but adding a condition is just ruining everything. Also, with the current spec I have, I am not getting an array after the transform which I can already get with my former spec.
The spec I have so far is:
SPEC USED
[
{
"operation": "shift",
"spec": {
"*": {
"keyToCheck": {
"true": {
"#1": "objToAdd.id",
"#dummy1": "objToAdd.label"
},
"false": {
"#2": "objToAdd.id",
"#dummy2": "objToAdd.label"
}
}
}
}
}
]
But as said before, it is only giving me this:
RESULT SO FAR
{
"objToAdd" : {
"id" : [ "1", "2" ],
"label" : [ "dummy1", "dummy2" ]
}
}
Which seems to be a problem of indentation. Adding [&1] before the values in the condition didn't help, and when I propagate the values with "*": "[&1].&", I am losing the item generated by the if-else condition.
Is there any way to do this?
Any help would be appreciated! Thanks in advance.
Check this spec,
[
{
"operation": "shift",
"spec": {
"*": {
"test_id": "[&1].test_id",
"someDate": "[&1].someDate",
"someData": "[&1].someData",
"keyToCheck": {
"true": {
"$": "[&3].keyToCheck",
"#1": "[&3].objToAdd.id",
"#dummy1": "[&3].objToAdd.label"
},
"false": {
"$": "[&3].keyToCheck",
"#2": "[&3].objToAdd.id",
"#dummy2": "[&3].objToAdd.label"
}
}
}
}
}
]