How to keep other fields in the Jolt transform JSON array, I am trying to use wildcard but fields are not added in the final output?
Here is the example input I am using
[
{
"foundduring": "D-DC",
"user_type": "type1",
"location": "location1"
},
{
"foundduring": "D-DG",
"user_type": "type2",
"location": "location2"
},
{
"foundduring": "D-DI",
"user_type": "type3",
"location": "location3"
}
]
I am using the following Jolt transformation and also trying wildcard:
[
{
"operation": "shift",
"spec": {
"*": {
"foundduring": {
"D-DC": {
"#CycleCount": "[&3].foundduring"
},
"D-DG": {
"#Pick": "[&3].foundduring"
},
"D-DI": {
"#Issue": "[&3].foundduring"
}
},
"#": "&"
}
}
}
]
Following is my expected output where shift operation happened and then need to keep all other fields as it it
[
{
"foundduring" : "CycleCount",
"user_type" : "type1",
"location" : "location1"
},
{
"foundduring" : "Pick",
"user_type" : "type2",
"location" : "location2"
},
{
"foundduring" : "Issue",
"user_type" : "type3",
"location" : "location3"
}
]
Actual Output coming:
[
{
"foundduring": "CycleCount"
},
{
"foundduring": "Pick"
},
{
"foundduring": "Issue"
}
]
Consider using "*" wildcard as else case instead of "#" such as
[
{
"operation": "shift",
"spec": {
"*": {
"foundduring": {
"D-DC": {
"#CycleCount": "[&3].&2"
},
"D-DG": {
"#Pick": "[&3].&2"
},
"D-DI": {
"#Issue": "[&3].&2"
}
},
"*": "[&1].&"
}
}
}
]
Btw, no need to get the key name "foundduring", just use &2 substitution to go 2 level up from the current branch and grab that value.
The demo on the site http://jolt-demo.appspot.com/ is
You may consider another library Josson.
https://github.com/octomix/josson
Deserialization
Josson josson = Josson.fromJsonString(
"[" +
" {" +
" \"foundduring\": \"D-DC\"," +
" \"user_type\": \"type1\"," +
" \"location\": \"location1\"" +
" }," +
" {" +
" \"foundduring\": \"D-DG\"," +
" \"user_type\": \"type2\"," +
" \"location\": \"location2\"" +
" }," +
" {" +
" \"foundduring\": \"D-DI\"," +
" \"user_type\": \"type3\"," +
" \"location\": \"location3\"" +
" }" +
"]");
Transformation
JsonNode node = josson.getNode(
"field(foundduring.caseValue('D-DC','CycleCount','D-DG','Pick','D-DI','Issue'))");
System.out.println(node.toPrettyString());
Output
[ {
"foundduring" : "CycleCount",
"user_type" : "type1",
"location" : "location1"
}, {
"foundduring" : "Pick",
"user_type" : "type2",
"location" : "location2"
}, {
"foundduring" : "Issue",
"user_type" : "type3",
"location" : "location3"
} ]
Related
I am trying to transform a JSON using Jolt transformation looking for some input here.
I am trying to get all the items from all the levels into an array.
My goal is to get an array that contains all the items without knowing how many levels I have in the json.
Here is my input and expected output:
if I've 3 levels:
Input:
{
"id": 11,
"item": [
{
"id": "11_1",
"item": [
{
"id": "11_1_1",
"item": [
{
"id": "11_1_1_1"
}
]
},
{
"id": "11_1_2",
"item": [
{
"id": "11_1_2_1"
}
]
}
]
}
]
}
Expected output:
[
{
"id": "11_1"
},
{
"id": "11_1_1"
},
{
"id": "11_1_1_1"
},
{
"id": "11_1_2"
},
{
"id": "11_1_2_1"
}
]
if I've 2 levels:
Input:
{
"id": 11,
"item": [
{
"id": "11_1",
"item": [
{
"id": "11_1_1"
},
{
"id": "11_1_2"
}
]
}
]
}
Expected output:
[
{
"id": "11_1"
},
{
"id": "11_1_1"
},
{
"id": "11_1_2"
}
]
I tried to write something like:
[
{
"operation": "shift",
"spec": {
"item": { //to cover the second level
"*": "item"
}
}
},
{
"operation": "shift",
"spec": {
"item": {
"*": { //to cover the 3td level
"item": {
"*": "item"
}
}
}
}
}
]
The result was null, if I run each transformation separately, I get results when applicable
Can you please help me to write a simple spec that will do this?
If the inputs have at most 3 levels as in your case, then use this spec
[
{
"operation": "shift",
"spec": {
"item": {
"*": {
"id": "&",
"item": {
"*": {
"id": "&",
"item": {
"*": {
"id": "&"
}
}
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"id": {
"*": "[#1].&1"
}
}
}
]
in order to handle the both cases using only this one. If one more level is needed as well, then add
,
"item": {
"*": {
"id": "&"
}
}
just after the inner most
"id": "&"
Edit : If you have some other attributes than id as lately commented like in the below input sample
{
"id": 11,
"item": [
{
"id": "11_1",
"quantity": 1,
"action": "add",
"state": "x",
"item": [
{
"id": "11_1_1",
"quantity": 2,
"action": "drop",
"state": "y"
},
{
"id": "11_1_2",
"quantity": 3,
"action": "modify",
"state": "z"
}
]
}
]
}
than just a little modification would handle your new case :
[
{
"operation": "shift",
"spec": {
"item": {
"*": {
"*": "&",
"item": {
"*": {
"*": "&",
"item": {
"*": {
"*": "&"
}
}
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": "[#1].&1"
}
}
}
]
which would generate :
[
{
"id": "11_1",
"quantity": 1,
"action": "add",
"state": "x"
},
{
"id": "11_1_1",
"quantity": 2,
"action": "drop",
"state": "y"
},
{
"id": "11_1_2",
"quantity": 3,
"action": "modify",
"state": "z"
}
]
Solution for unlimited levels using library Josson.
https://github.com/octomix/josson
Deserialization
Josson josson = Josson.fromJsonString(
"{" +
" \"id\": 11," +
" \"item\": [" +
" {" +
" \"id\": \"11_1\"," +
" \"quantity\": 1," +
" \"action\": \"add\"," +
" \"state\": \"x\"," +
" \"item\": [" +
" {" +
" \"id\": \"11_1_1\"," +
" \"quantity\": 2," +
" \"action\": \"drop\"," +
" \"state\": \"y\"" +
" }," +
" {" +
" \"id\": \"11_1_2\"," +
" \"quantity\": 3," +
" \"action\": \"modify\"," +
" \"state\": \"z\"" +
" }" +
" ]" +
" }" +
" ]" +
"}");
Transformation
JsonNode node = josson.getNode("item.cumulateCollect(field(item:), item)");
System.out.println(node.toPrettyString());
Statement field(item:) removes field item from the current object.
The 2nd argument of cumulateCollect() means the next level. In this case is item.
Output
[ {
"id" : "11_1",
"quantity" : 1,
"action" : "add",
"state" : "x"
}, {
"id" : "11_1_1",
"quantity" : 2,
"action" : "drop",
"state" : "y"
}, {
"id" : "11_1_2",
"quantity" : 3,
"action" : "modify",
"state" : "z"
} ]
Hi working in Jolt transformation tool and getting know some in-depth details
Input
{
"characteristic": [
{
"name": "usageCharacteristicName",
"value": "availableBalance",
"#type": "usageCharacteristic",
"arrayIndex": "2"
},
{
"name": "usageCharacteristicValue",
"value": "2999.25",
"#type": "usageCharacteristic",
"arrayIndex": "2"
},
{
"name": "usageCharacteristicName",
"value": "Name",
"#type": "usageCharacteristic",
"arrayIndex": "0"
},
{
"name": "usageCharacteristicValue",
"value": "Jack",
"#type": "usageCharacteristic",
"arrayIndex": "0"
},
{
"name": "usageCharacteristicName",
"value": "Likes",
"#type": "usageCharacteristic",
"arrayIndex": "1"
},
{
"name": "usageCharacteristicValue",
"value": "Code",
"#type": "usageCharacteristic",
"arrayIndex": "1"
}
]
}
Jolt Spec :
[
{
"operation": "shift",
"spec": {
"characteristic": {
"*": {
"arrayIndex": {
"2": {
"#(2,value)": "data.queryBalance.accountBalance"
},
"0": {
"#(2,value)": "data.queryBalance.Name"
},
"1": {
"#(2,value)": "data.queryBalance.Likes"
},
}
}
}
}
}
]
Output :
{
"data" : {
"queryBalance" : {
"accountBalance" : [ "availableBalance", "2999.25" ],
"Name" : [ "Name", "Jack" ],
"Likes" : [ "Likes", "Code" ]
}
}
}
the Output i was getting is not good enough for me, want to do like String like bellow Expected.
Expected Output :
"availableBalance" : "2999.25",
"Name" : "Jack",
"Likes" : "Code"
how do i get like a String ?
I think this solve the problem
spec
[
{
//group keys and valeus for arrayIndex
"operation": "shift",
"spec": {
"characteristic": {
"*": {
"value": "newArray[#(1,arrayIndex)].#(1,name)"
}
}
}
},
{
//create a new object use a key and valeus matchs previous
"operation": "shift",
"spec": {
"newArray": {
"*": {
"usageCharacteristicValue": "#(1,usageCharacteristicName)"
}
}
}
}
]
output
{
"Name" : "Jack",
"Likes" : "Code",
"availableBalance" : "2999.25"
}
You need a modify transformation spec as well such as
[
{
// reduce to simple array "value" : [ "availableBalance", "2999.25" ] nested within an object
"operation": "shift",
"spec": {
"*": {
"*": {
"value": "&"
}
}
}
},
{
// combine components of the array so as to reform as a string
"operation": "modify-overwrite-beta",
"spec": {
"*": "=join(' : ',#(1,&))"
}
}
]
Edit: You can alternatively try the following for the new case :
[
{
// dissipate each array into different object with respect to arrayIndex values
"operation": "shift",
"spec": {
"*": {
"*": {
"value": "#(1,arrayIndex).&"
}
}
}
},
{
// combine components of each array colon-separatedly
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"*": "=join(' : ',#(1,&))"
}
}
},
{
// convert to a unique array
"operation": "shift",
"spec": {
"*": {
"*": "&"
}
}
},
{
// combine components of each array comma-separatedly
"operation": "modify-overwrite-beta",
"spec": {
"*": "=join(' , ',#(1,value))"
}
}
]
I know how to do it by another library Josson. You may consider to use it.
https://github.com/octomix/josson
Deserialization
Josson josson = Josson.fromJsonString(inputJSON);
Transform to an array of string
JsonNode node = josson.getNode(
"characteristic" +
".group(arrayIndex)" +
".concat(elements[name='usageCharacteristicName'].value" +
" ,' : '" +
" ,elements[name='usageCharacteristicValue'].value)");
System.out.println(node.toPrettyString());
Output
[ "availableBalance : 2999.25", "Name : Jack", "Likes : Code" ]
Transform to a single string
String str = josson.getString(
"characteristic" +
".group(arrayIndex)" +
".concat(elements[name='usageCharacteristicName'].value" +
" ,' : '" +
" ,elements[name='usageCharacteristicValue'].value)" +
".join(', ')");
System.out.println(str);
Output
availableBalance : 2999.25, Name : Jack, Likes : Code
Transform to a map
JsonNode node = josson.getNode(
"characteristic" +
".group(arrayIndex)" +
".map(elements[name='usageCharacteristicName'].value::elements[name='usageCharacteristicValue'].value)" +
".mergeObjects()");
System.out.println(node.toPrettyString());
Output
{
"availableBalance" : "2999.25",
"Name" : "Jack",
"Likes" : "Code"
}
Transform to a map with sorted "arrayIndex" as key
JsonNode node = josson.getNode(
"characteristic" +
".group(arrayIndex)" +
".sort(arrayIndex)" +
".map(arrayIndex.prepend('arrayIndex')::" +
" concat(elements[name='usageCharacteristicName'].value" +
" ,' : '" +
" ,elements[name='usageCharacteristicValue'].value))" +
".mergeObjects()");
System.out.println(node.toPrettyString());
Output
{
"arrayIndex0" : "Name : Jack",
"arrayIndex1" : "Likes : Code",
"arrayIndex2" : "availableBalance : 2999.25"
}
I have an array of Jsonnodes. something like that
[
{
"coupon": "VAR",
"currency": "USD",
"sip": "94989WAX5",
"lastModifiedDate": "2022-09-23T08:16:25Z"
},
{
"coupon": "VAR1",
"currency": "USD",
"sip": "94989WAX5",
"lastModifiedDate": "2022-09-21T08:16:25Z"
},
{
"coupon": "VAR3",
"currency": "USD",
"sip": "XHBRYWEB1",
"lastModifiedDate": "2022-09-20T08:16:25Z"
}
]
I have a requirement, if the sip value of two nodes are same then I need to pick only that sip which lastModifiedDate is latest. In above example the final output should be remaining two nodes.
[
{
"coupon": "VAR",
"currency": "USD",
"sip": "94989WAX5",
"lastModifiedDate": "2022-09-23T08:16:25Z"
},
{
"coupon": "VAR3",
"currency": "USD",
"sip": "XHBRYWEB1",
"lastModifiedDate": "2022-09-20T08:16:25Z"
}
]
I was try to solve it by creating HashMap<String,JsonNode> where Sip is the key and the JsonNode is complete node. It doesn't seems to be a cleaner way. is there any other way to achieve it. I am using fasterxml.jackson.databind.JsonNode
Map<String, JsonNode> map =
jsonNodeList.stream()
.collect(
toMap(
jsonNode -> jsonNode.get("sip").asText(),
jsonNode -> jsonNode,
(jsonNode1, jsonNode2) -> {
boolean after =
LocalDateTime.parse(String.valueOf(jsonNode1.get("lastModifiedDate")))
.isAfter(
LocalDateTime.parse(
String.valueOf(jsonNode2.get("lastModifiedDate"))));
return after ? jsonNode1 : jsonNode2;
},
HashMap::new));
https://github.com/octomix/josson
Deserialization
Josson josson = Josson.fromJsonString(
"[" +
" {" +
" \"coupon\": \"VAR\"," +
" \"currency\": \"USD\"," +
" \"sip\": \"94989WAX5\"," +
" \"lastModifiedDate\": \"2022-09-23T08:16:25Z\"" +
" }," +
" {" +
" \"coupon\": \"VAR1\"," +
" \"currency\": \"USD\"," +
" \"sip\": \"94989WAX5\"," +
" \"lastModifiedDate\": \"2022-09-21T08:16:25Z\"" +
" }," +
" {" +
" \"coupon\": \"VAR3\"," +
" \"currency\": \"USD\"," +
" \"sip\": \"XHBRYWEB1\"," +
" \"lastModifiedDate\": \"2022-09-20T08:16:25Z\"" +
" }" +
" ]");
Transformation
JsonNode node = josson.getNode("group(sip)#.elements.findByMax(lastModifiedDate)");
System.out.println(node.toPrettyString());
Output
[ {
"coupon" : "VAR",
"currency" : "USD",
"sip" : "94989WAX5",
"lastModifiedDate" : "2022-09-23T08:16:25Z"
}, {
"coupon" : "VAR3",
"currency" : "USD",
"sip" : "XHBRYWEB1",
"lastModifiedDate" : "2022-09-20T08:16:25Z"
} ]
i want to get the maximun id form a subfield of an aptitude object,
{
"mappings": {
"aptitude": {
"dynamic": "strict",
"properties": {
"id": {
"type": "long"
},
"es": {
"type": "text"
},
"en": {
"type": "text"
},
"behaviors": {
"properties": {
"id": {
"type": "long"
},
"es": {
"type": "text"
},
"en": {
"type": "text"
}
}
}
}
}
}
as you can see the aptitude have an array of behaviors who in turn have an id, afaik i should use the maxAggregation from Jest, but cant find a decent example of how to do it in java, can someone help?
i found the way like this:
String query = "{\n"
+" \"query\" : {\n"
+" \"match\" : {\"id\":" + aptitudeId + "}\n"
+" },\n"
+" \"aggs\" : {\n"
+" \"max1\" : {\n"
+" \"max\" : {\n"
+" \"field\" : \"behaviors.id\"\n"
+" }\n"
+" }\n"
+" }\n"
+"}";
i was looking into the aggregation builders from jest but doing it via query was much easier.
the return looks like this:
Search search = new Search.Builder(query)
.addIndex(aptitudeIndexName)
.addType(aptitudeTypeName)
.build();
try {
SearchResult result = client.execute(search);
MaxAggregation max1 = result.getAggregations().getMaxAggregation("max1");
Double max = max1.getMax();
return max.longValue() + 1;//so it would add 1 to the current maximum id
I have this query I made using sense, I'm breaking my head how to transform it into Java.
I can manage the aggs part, the real pain is the "constant_score"
GET /xxxx/yyyy/_search
{
"size": 0,
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : [
{ "term" : {"userId" : 275}},
{ "range" :{"logDate" : { "gte" : "2016-04-30", "lte" : "now/d" }}}
]
}
}
}
},
"aggs" : {
"datebucket" : {
"date_histogram" : {
"field" : "logDate",
"interval": "day",
"format": "yyyy-MM-dd",
"min_doc_count": 0
},
"aggs": {
"info": {
"filters": {
"filters" : [
{"term": { "logAction": "sleep" }},
{"term": { "logAction": "stop" }}
]
}
}
}
}
}
}
I had a more simple query I managed to do it like this
SearchResponse res = client.prepareSearch("xxxx").setTypes("yyyy")
.setSize(0)
.setQuery(QueryBuilders.termQuery("userId", 95))
.addAggregation(
AggregationBuilders.dateHistogram("date_histogram")
.field("logDate")
.interval(DateHistogramInterval.DAY)
.format("yyyy-MM-dd")
.minDocCount(0)
).execute().get();
Well, I found the solution. I hope this will help someone
String query = "{\"constant_score\" : "
+ "{ \"filter\" : "
+ "{\"bool\" : "
+ "{\"must\" : "
+ "[{ \"term\" : {\"userId\" : " + userID + "}}, "
+ "{ \"range\" :{\"logDate\" : { \"gte\" : \"" + startdate + "\", \"lte\" : \"" + enddate + "\" }}}]"
+ "}"
+ "}"
+ "}"
+ "}";
SearchResponse res = client.prepareSearch(xxxx).setTypes(yyyy)
.setQuery(query).addAggregation(
AggregationBuilders.dateHistogram("date_histogram")
.field("logDate")
.interval(DateHistogramInterval.DAY)
.format("dd-MM-yyyy")
.minDocCount(0)
.subAggregation(AggregationBuilders.filters("info")
.filter(QueryBuilders.termQuery("logAction", "click"))
.filter(QueryBuilders.termQuery("logAction", "view")))
).setSize(0).execute().get();