Develop Reference

Java Jolt: can't find proper spec for my transformation

I'm struggling with that Jolt transformation:
Here is my input JSON
{
"bloc1-1": {
"bloc1-2": [
{
"key": "key1",
"value": "value1-1"
},
{
"key": "key2",
"value": "value1-2"
}
]
},
"bloc2-1": {
"bloc2-2": [
{
"key": "key1",
"value": "value2-1"
},
{
"key": "key2",
"value": "value2-2"
},
{
"key": "key3",
"value": "value2-3"
}
]
}
}
Here is what I'm expecting
{
"bloc1-key1" : "value1-1",
"bloc1-key2" : "value1-2",
"bloc2-key1" : "value2-1",
"bloc2-key2" : "value2-2",
"bloc2-key3" : "value2-3"
}
I have tried the following spec, but I cannot figure out how to prefix the key in the RHS (the # should be the first character)
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"value": "#(1,key)"
}
}
}
}
}
]
and got that
{
"key1" : [ "value1-1", "value2-1" ],
"key2" : [ "value1-2", "value2-2" ],
"key3" : "value2-3"
}
Any help would be appreciated

The spec below should do the trick:
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"$2": "keyParts1",
"key": "keyParts2",
"value": "values"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"keyParts1": {
"*": "[#1].part1"
},
"keyParts2": {
"*": "[#1].part2"
},
"values": {
"*": "[#1].value"
}
}
},
{
"operation": "modify-default-beta",
"spec": {
"*": {
"newKey": "=split('-', #(1,part1))",
"newKeyFirst": "#(1,newKey[0])",
"newKeyComplete": "=concat(#(1,newKeyFirst),'-',#(1,part2))"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,newKeyComplete)"
}
}
}
]
Try to apply each of the operations separately (for example in the demo app).
Check out the links to the jolt docs and the examples from this answer
Step by step explanation:
First "shift" transformation places the components of the final result in the separate arrays:
{
"keyParts1" : [ "bloc1-1", "bloc1-1", "bloc2-1", "bloc2-1", "bloc2-1" ],
"keyParts2" : [ "key1", "key2", "key1", "key2", "key3" ],
"values" : [ "value1-1", "value1-2", "value2-1", "value2-2", "value2-3" ]
}
The above operation let us have each of the components of the end result positioned in the right index of the desired result.
Note that each array has the same size.
In the second operation we group each of the array elements by its index using the # operator. Looking at the docs [#1] can be translated to:
"Go up one level and ask that node how many matches it has had before finding the current element".
The node above is an array, so for the 3rd array element it will be the array index 2.
[
...
{
"part1" : "bloc2-1",
"part2" : "key1",
"value" : "value2-1"
}
...
]
In the 3rd operation I have assumed that the key of the desired result is the concatenation of the "blocX" substring of the "blocX-Y" key concatenated with the "key" value. That's why I've used the "modify-default-beta". Have a look at the example from the demo app
Each of five elements of the array is transformed similarly to the 3rd array element:
[
...
{
"part1" : "bloc2-1",
"part2" : "key1",
"value" : "value2-1",
"newKey" : [ "bloc2", "1" ],
"newKeyFirst" : "bloc2",
"newKeyComplete" : "bloc2-key1"
}
...
]
The last operation produces the desired result from the 3rd operation's output.

You may consider another library Josson.
https://github.com/octomix/josson
Josson josson = Josson.fromJsonString(inputJSON);
JsonNode node = josson.getNode(
"entries()" + // Field "key" generated by entries() is the same as the inner one.
".field(k:key)" + // So, rename it to "k".
".unwind(value.*)" + // Unwind the "value" generated by entries(), not the inner "value".
".map(concat(k.keepBefore('-'),'-',key)::value)" +
".mergeObjects()");
System.out.println(node.toPrettyString());
Output
{
"bloc1-key1" : "value1-1",
"bloc1-key2" : "value1-2",
"bloc2-key1" : "value2-1",
"bloc2-key2" : "value2-2",
"bloc2-key3" : "value2-3"
}

Jolt transform several object to array with new field name

I'am new using jolt tool and i wonder if there a way to take several json objects and put them to an array with a new field named as follows:
Input :
{
"userId": 1,
"age": 20,
"desc1": "value desc1",
"desc2": "value desc2",
"desc3": "value desc3"
}
JSON spec :
[
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"*": "additionalInformation"
}
}
]
Expected result :
{
"ID": 1,
"age": 20,
"additionalInformation": [
{
"code": "desc1",
"value": "value desc1"
},
{
"code": "desc2",
"value": "value desc2"
},
{
"code": "desc3",
"value": "value desc3"
}
]
}
Using the spec above i can only obtain this result :
{
"ID": 1,
"test": 20,
"additionalInformation": [
"value desc1",
"value desc2",
"value desc3"
]
}
Any suggestion what i have missed ?

You can use $ and # wildcards for code and value, respectively while combining the elements under common factor by using &. as prefixes for them in the first step. Then combine elements other than ID and age within the same array in the last step such as
[
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"*": {
"$": "&.code",
"#": "&.value"
}
}
},
{
"operation": "shift",
"spec": {
"ID": "&",
"age": "&",
"*": "additionalInformation[]"
}
}
]
Edit(due to the comment) : If you need to pick some individual elements such as desc1 and desc3 rather than all("*"), then replace the first spec with following
{
"operation": "shift",
"spec": {
"userId": "ID",
"age": "age",
"desc1|desc3": {
"$": "&.code",
"#": "&.value"
}
}
}

Jolt Transform JSON Spec for Array Input

I am trying to do JOLT shift operation with below spec which is not working. Not sure what mistake I have done. Need help in this case. Output JSON is coming as an object instead of Array and shift also not working as expected.
Input : [
{
"Header": {
"Number": 1,
"Id": "JO"
},
"Name": "John"
},
{
"Header": {
"Number": 2,
"Id": "JS"
},
"Name": "Justin"
}
]
Spec : [
{
"operation": "shift",
"spec": {
"*": {
"Header": "Header",
"Name": "Header.Name"
}
}
}
]
Expected Output : [
{
"Header": {
"Number": 1,
"Id": "JO",
"Name": "John"
}
},
{
"Header": {
"Number": 2,
"Id": "JS",
"Name": "Justin"
}
}
]
Actual Output : {
"Header" : [ {
"Number" : 1,
"Id" : "JO",
"Name" : "John"
}, {
"Number" : 2,
"Id" : "JS"
} ]
}

You have to also specify that the "Header" object is inside the array.
Moreover, the index of the array where you place the "Header" object for each of the element of the array.
That's what the spec below does (using the [&1] - apmersand wildcard combined with array):
[
{
"operation": "shift",
"spec": {
"*": {
"Header": "[&1].Header",
"Name": "[&1].Header.Name"
}
}
}
]
Sources:
Shiftr.java javadocs:
The arrays
The ampersand wildcard
Other answer: "How do I transform an array using Jolt?"
Demo application linked in the jolt repo to test the spec

How to return all fields instead of just id and count after sortByCount operation in Mongodb/Java?

I need to do sortByCount and return all the fields instead of just _id and count.
sortByCount returns:
{ "_id" : "1", "count" : 4 }
{ "_id" : "2", "count" : 3 }
{ "_id" : "3", "count" : 2 }
{ "_id" : "4", "count" : 2 }
{ "_id" : "5", "count" : 1 }
But, I need a complete document like below:
{
"_id": 1,
"title": "The Pillars of Society",
"artist": "Grosz",
"year": 1926,
"tags": ["painting", "satire", "Expressionism", "caricature"]
} {
"_id": 2,
"title": "Melancholy III",
"artist": "Munch",
"year": 1902,
"tags": ["woodcut", "Expressionism"]
} {
"_id": 3,
"title": "Dancer",
"artist": "Miro",
"year": 1925,
"tags": ["oil", "Surrealism", "painting"]
} {
"_id": 4,
"title": "The Great Wave off Kanagawa",
"artist": "Hokusai",
"tags": ["woodblock", "ukiyo-e"]
} {
"_id": 5,
"title": "The Persistence of Memory",
"artist": "Dali",
"year": 1931,
"tags": ["Surrealism", "painting", "oil"]
}
Is there any way to replace root after sortByCount? In Java, I don't see any push method after sortByCount

$sortByCount is essentially a combination of $group followed by $sort on count field of group stage. If you really want the entire documents, you can try this:
db.collection.aggregate([
{
"$group": {
"_id": {
"id": "$_id"
},
"count": {
$sum: 1
},
"reqItems": {
$push: {
"title": "$title",
"artist": "$artist"
}
}
}
},
{
$sort: {
count: -1
}
}
])
Playground link

JOLT - Adding fields in JSONArray based on if condition

I have a bit of a tricky question to ask, as far as I know, I haven't seen anyone with the same problem, at least not all of them at once!
I have the following test JSON to transform:
INPUT
[
{
"test_id": 1212101011,
"someDate": "2020-03-06",
"keyToCheck": "true",
"someData": "123"
},
{
"test_id": 8787909099,
"someDate": "2020-03-09",
"keyToCheck": "false",
"someData": "456"
}
]
And I am trying to get the following output.
DESIRED
[{
"test_id": 1212101011,
"someDate": "2020-03-06",
"keyToCheck": "true",
"someData": "123",
"objToAdd": {
"id": "1",
"label": "dummy1"
}
},{
"test_id": 8787909099,
"someDate": "2020-03-09",
"keyToCheck": "false",
"someData": "456",
"objToAdd": {
"id": "2",
"label": "dummy2"
}]
I already managed to propagate all the values and change the fields' names if needed, but adding a condition is just ruining everything. Also, with the current spec I have, I am not getting an array after the transform which I can already get with my former spec.
The spec I have so far is:
SPEC USED
[
{
"operation": "shift",
"spec": {
"*": {
"keyToCheck": {
"true": {
"#1": "objToAdd.id",
"#dummy1": "objToAdd.label"
},
"false": {
"#2": "objToAdd.id",
"#dummy2": "objToAdd.label"
}
}
}
}
}
]
But as said before, it is only giving me this:
RESULT SO FAR
{
"objToAdd" : {
"id" : [ "1", "2" ],
"label" : [ "dummy1", "dummy2" ]
}
}
Which seems to be a problem of indentation. Adding [&1] before the values in the condition didn't help, and when I propagate the values with "*": "[&1].&", I am losing the item generated by the if-else condition.
Is there any way to do this?
Any help would be appreciated! Thanks in advance.

Check this spec,
[
{
"operation": "shift",
"spec": {
"*": {
"test_id": "[&1].test_id",
"someDate": "[&1].someDate",
"someData": "[&1].someData",
"keyToCheck": {
"true": {
"$": "[&3].keyToCheck",
"#1": "[&3].objToAdd.id",
"#dummy1": "[&3].objToAdd.label"
},
"false": {
"$": "[&3].keyToCheck",
"#2": "[&3].objToAdd.id",
"#dummy2": "[&3].objToAdd.label"
}
}
}
}
}
]