Nested Jolt Transformation not working - java

{
"items": [
{
"itemId": 1,
"itemName": "OKRA",
"stock": [
{
"stock": 50,
"salePrice": 150
},
{
"stock": 100,
"salePrice": 75
}
]
},
{
"itemId": 2,
"itemName": "CUCUMBER",
"stock": [
{
"stock": 100,
"salePrice": 10
},
{
"stock": 200,
"salePrice": 5
}
]
}
],
"count": 2,
"total_records": 428,
"current_page": 1,
"per_page": 2,
"total_pages": 215
}
Excepted Output should be
{
"items": [
{
"itemId": 1,
"itemName": "OKRA",
"salePrice": 150,
"stock": 50
},
{
"itemId": 1,
"itemName": "OKRA",
"salePrice": 75,
"stock": 100
},
{
"itemId": 2,
"itemName": "CUCUMBER",
"salePrice": 10,
"stock": 100
},
{
"itemId": 2,
"itemName": "CUCUMBER",
"salePrice": 5,
"stock": 200
}
]
}
My spec file
[
{
"operation": "shift",
"spec": {
"items": {
"*": {
"stock": {
"*": {
"#2": "itemsArray[]",
"#": "stockArray[]"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"itemsArray": {
"*": { // bizArray index
"itemId": "items[&1].itemId",
"itemName": "items[&1].itemName",
"stock": "items[&1].stock[&1].stock",
"salePrice": "[&1].stock[&1].salePrice"
}
}
}
}
]
Anyone help me to get the Excepted output.I am new to JOLT.


[
{
"operation": "shift",
"spec": {
"items": {
"*": {
"stock": {
"*": {
"#2": "itemsArray[]",
"#": "stockArray[]"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"itemsArray": {
"*": {
"itemId": "items[&1].itemId",
"itemName": "items[&1].itemName"
}
},
"stockArray": {
"*": {
"stock": "items[&1].stock",
"salePrice": "items[&1].salePrice"
}
}
}
}
]

Related

jolt transformation of this complex scenario?

I want the jolt transform for the given input . Your help in this is highly appreciated . thanks i am providing the input and expected output. in input json Photos array is dynamic in nature. Here it is 3 , it can be 3 or 4 or5 any .
Input JSON :
{
"Entity": {
"card": {
"cardNo":"123456789",
"cardStatus":"10",
"cardAddress":"UK",
"cardAddress1":"US",
"cardCity":"mk" ,
"name": "RAM",
"lastName": "ABU",
"name1": "RAM1",
"lastName1": "ABU1"
},
"Photos": [
{
"Id": 327703,
"Caption": "TEST>> photo 1",
"Url": "http://bob.com/0001/327703/photo.jpg"
},
{
"Id": 327704,
"Caption": "TEST>> photo 2",
"Url": "http://bob.com/0001/327704/photo.jpg"
},
{
"Id": 327704,
"Caption": "TEST>> photo 2",
"Url": "http://bob.com/0001/327704/photo.jpg"
}
]
}
}
Used Jolt Spec :
[
{
"operation": "shift",
"spec": {
"Entity": {
"card": {
"cardNo": "tab.text",
"cardAddress": "address[0].add",
"cardAddress1": "address[0].add2",
"cardC*": "address[0].mk",
"Id1": "Photos.no",
"#http.1.com": "Photos.caption2",
"Id2": "Photos.no",
"#http.2.com": "Photos.caption2"
},
"Photos": {
"*": {
"Id": "Photos.no",
"Caption": "Photos.caption2"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"tab": "&",
"address": "&",
"Photos": {
"*": {
"*": {
"#": "&3[&1].&2"
}
}
}
}
}
]
Current Output :
{
"tab": {
"text": "123456789"
},
"address": [
{
"add": "UK",
"add2": "US",
"mk": "mk"
}
],
"Photos": [
{
"caption2": "http.1.com",
"no": 222444
},
{
"caption2": "http.2.com",
"no": 222444
},
{
"caption2": "TEST>> photo 1",
"no": 327703
},
{
"caption2": "TEST>> photo 2",
"no": 327704
},
{
"caption2": "TEST>> photo 2",
"no": 327704
}
]
}
Expected Output ( i need "no" in string like below ) :
{
"tab": {
"text": "123456789"
},
"address": [
{
"add": "UK",
"add2": "US",
"mk": "mk"
}
],
"Photos": [
{
"caption2": "http.1.com",
"no": "222444"
},
{
"caption2": "http.2.com",
"no": "222444"
},
{
"caption2": "TEST>> photo 1",
"no": "327703"
},
{
"caption2": "TEST>> photo 2",
"no": "327704"
},
{
"caption2": "TEST>> photo 2",
"no": "327704"
}
]
}

You can add a modify transformation to the current spec along with toString function such as
,
{
"operation": "modify-overwrite-beta",
"spec": {
"Photos": {
"*": {
"no": "=toString"
}
}
}
}

How can I map inner fields to the same array object in jolt?

Having this input json:
{
"orderItems": [
{
"itemName": "Mozzz",
"quantity": 1
},
{
"itemName": "zer",
"quantity": 0,
"bar": {
"arr": [
{
"meta_itemName": "Small Barqs2",
"meta_quantity": 22
}
]
}
}
]
}
I'm trying to shift the "meta_" prefixed fields up inside the orderItems object array, eliminating the bar and arr keys.
I tried this spec:
[
{
"operation": "shift",
"spec": {
"orderItems": {
"*": {
"quantity": "basket_item[#2].quantity",
"itemName": "basket_item[#2].itemName",
"bar": {
"arr": {
"*": {
"meta_itemName": "basket_item[#2].m2",
"meta_quantity": "basket_item[#2].m3"
}
}
}
}
}
}
}
]
but the fields renamed to m2 and m3 don't stay inside the "zer" item.

This spec should work for you (tested with https://jolt-demo.appspot.com/)
[
{
"operation": "shift",
"spec": {
"orderItems": {
"*": {
"bar": {
"arr": {
"*": { "meta_*": "orderItems[&4].&" }
}
},
"*": "orderItems[&1].&"
}
}
}
}
]
output for your input:
{
"orderItems" : [ {
"itemName" : "Mozzz",
"quantity" : 1
}, {
"itemName" : "zer",
"quantity" : 0,
"meta_itemName" : "Small Barqs2",
"meta_quantity" : 22
} ]
}

Elasticsearch Rest High Level Client aggregate fields dynamically

I am trying to generate query dynamically based on the inputs but in the generated query i can see there are only two aggregations are getting generated how can i make each fields to have the separate aggregations below is the code what i have tried and the response what i'm getting.
From main() i'm calling
buildSearchCriteria("1");
Here i am setting the aggregation type and respective values:
public static void buildSearchCriteria(String... exceptionId) {
SearchCriteria searchCriteria = new SearchCriteria();
Map<String, List<FieldNameAndPath>> stringListMap = new HashMap<>();
stringListMap.put("nested", asList(new FieldNameAndPath("nested", "recommendations",
"recommendations", null, emptyList(), 1)));
stringListMap.put("filter", asList(new FieldNameAndPath("filter", "exceptionIds", "recommendations.exceptionId.keyword",
asList(exceptionId),
asList(new NestedAggsFields("terms", "exceptionIdsMatch")), 2)));
stringListMap.put("terms", asList(new FieldNameAndPath("terms", "by_exceptionId", "recommendations.exceptionId.keyword", null, emptyList(), 3),
new FieldNameAndPath("terms", "by_item", "recommendations.item.keyword", null, emptyList(), 4),
new FieldNameAndPath("terms", "by_destination", "recommendations.location.keyword", null, emptyList(), 5),
new FieldNameAndPath("terms", "by_trans", "recommendations.transportMode.keyword", null, emptyList(), 6),
new FieldNameAndPath("terms", "by_sourcelocation", "recommendations.sourceLocation.keyword", null, emptyList(), 7),
new FieldNameAndPath("terms", "by_shipdate", "recommendations.shipDate", null, emptyList(), 8),
new FieldNameAndPath("terms", "by_arrival", "recommendations.arrivalDate", null, emptyList(), 9)));
stringListMap.put("sum", asList(new FieldNameAndPath("sum", "quantity", "recommendations.transferQuantity", null, emptyList(), 10),
new FieldNameAndPath("sum", "transfercost", "recommendations.transferCost", null, emptyList(), 11),
new FieldNameAndPath("sum", "revenueRecovered", "recommendations.revenueRecovered", null, emptyList(), 12)));
System.out.println(stringListMap);
searchCriteria.setStringListMap(stringListMap);
aggregate(searchCriteria);
}
Below is the aggregate function which will get the the above information and builds query:
public static void aggregate(SearchCriteria searchCriteria) throws IOException {
Map<String, List<FieldNameAndPath>> map = searchCriteria.getStringListMap();
List<FieldNameAndPath> nesteds = map.get("nested");
List<FieldNameAndPath> filter = map.get("filter");
List<FieldNameAndPath> terms = map.get("terms");
List<FieldNameAndPath> sums = map.get("sum");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
AggregationBuilder aggregationBuilder = new SamplerAggregationBuilder("parent");
nesteds.stream().forEach(l -> buildAggregations(l, aggregationBuilder));
filter.stream().forEach(l -> buildAggregations(l, aggregationBuilder));
terms.stream().forEach(l -> buildAggregations(l, aggregationBuilder));
sums.stream().forEach(l -> buildAggregations(l, aggregationBuilder));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("index");
searchRequest.types("type");
sourceBuilder.aggregation(aggregationBuilder);
searchRequest.source(sourceBuilder);
System.out.println(searchRequest.source().toString());
}
buildAggregations method:
private static AggregationBuilder buildAggregations(FieldNameAndPath fieldNameAndPath , AggregationBuilder parentAggregationBuilder) {
if(fieldNameAndPath.getAggType().equals("nested")){
parentAggregationBuilder = AggregationBuilders.nested(fieldNameAndPath.getFieldName(), fieldNameAndPath.getFieldPath());
}
if(fieldNameAndPath.getAggType().equals("filter")){
parentAggregationBuilder.subAggregation(AggregationBuilders
.filter(fieldNameAndPath.getFieldName(),
QueryBuilders.termsQuery(fieldNameAndPath.getNestedAggs()
.stream().map(nestedAggsFields -> nestedAggsFields.getFieldName()).findFirst().get(), fieldNameAndPath.getFieldValues())));
}
if(fieldNameAndPath.getAggType().equals("terms")){
parentAggregationBuilder.subAggregation(AggregationBuilders.terms(fieldNameAndPath.getFieldName())
.field(fieldNameAndPath.getFieldPath()));
}
if(fieldNameAndPath.getAggType().equals("sum")){
parentAggregationBuilder.subAggregation(AggregationBuilders.
sum(fieldNameAndPath.getFieldName()).field(fieldNameAndPath.getFieldPath()));
}
return parentAggregationBuilder;
}
SearchCriteria class:
#Data
public class SearchCriteria {
Map<String, List<FieldNameAndPath>> stringListMap;
private List<String> searchFields;
}
And the DTO FieldNameAndPath:
public class FieldNameAndPath{
private String aggType;
private String fieldName;
private String fieldPath;
private List<String> fieldValues;
private List<NestedAggsFields> nestedAggs;
private int order;
}
And the query output from the above code is:
{
"aggregations": {
"parent": {
"sampler": {
"shard_size": 100
},
"aggregations": {
"exceptionIds": {
"filter": {
"terms": {
"exceptionIdsMatch": [
"1"
],
"boost": 1
}
}
},
"by_exceptionId": {
"terms": {
"field": "recommendations.exceptionId.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_item": {
"terms": {
"field": "recommendations.item.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_destination": {
"terms": {
"field": "recommendations.location.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_trans": {
"terms": {
"field": "recommendations.transportMode.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_sourcelocation": {
"terms": {
"field": "recommendations.sourceLocation.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_shipdate": {
"terms": {
"field": "recommendations.shipDate",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"by_arrival": {
"terms": {
"field": "recommendations.arrivalDate",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
"quantity": {
"sum": {
"field": "recommendations.transferQuantity"
}
},
"transfercost": {
"sum": {
"field": "recommendations.transferCost"
}
},
"revenueRecovered": {
"sum": {
"field": "recommendations.revenueRecovered"
}
}
}
}
}
}
Expected Query is:
{
"size": 0,
"aggregations": {
"exceptionIds": {
"nested": {
"path": "recommendations"
},
"aggregations": {
"exceptionIdsMatch": {
"filter": {
"terms": {
"recommendations.exceptionId.keyword": [
"1"
],
"boost": 1
}
},
"aggregations": {
"by_exceptionId": {
"terms": {
"field": "recommendations.exceptionId.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_item": {
"terms": {
"field": "recommendations.item.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_destination": {
"terms": {
"field": "recommendations.location.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_trans": {
"terms": {
"field": "recommendations.transportMode.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_sourcelocation": {
"terms": {
"field": "recommendations.sourceLocation.keyword",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_shipdate": {
"terms": {
"field": "recommendations.shipDate",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"by_arrival": {
"terms": {
"field": "recommendations.arrivalDate",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"quantity": {
"sum": {
"field": "recommendations.transferQuantity"
}
},
"transfercost": {
"sum": {
"field": "recommendations.transferCost"
}
},
"revenueRecovered": {
"sum": {
"field": "recommendations.revenueRecovered"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

elasticsearch how to group by repetitive items in array without distinct

I'm trying to get the counts group by the repetitive items in array without distinct, use aggs terms but not work
GET /my_index/_search
{
"size": 0,
"aggs": {
"keywords": {
"terms": {
"field": "keywords"
}
}
}
}
documents like:
"keywords": [
"value1",
"value1",
"value2"
],
but the result is:
"buckets": [
{
"key": "value1",
"doc_count": 1
},
{
"key": "value2",
"doc_count": 1
}
]
how can i get the result like:
"buckets": [
{
"key": "value1",
"doc_count": 2
},
{
"key": "value2",
"doc_count": 1
}
]

finally I modify the mapping use nested:
"keywords": {
"type": "nested",
"properties": {
"count": {
"type": "integer"
},
"keyword": {
"type": "keyword"
}
}
},
and query:
GET /my_index/_search
{
"size": 0,
"aggs": {
"keywords": {
"nested": {
"path": "keywords"
},
"aggs": {
"keyword_name": {
"terms": {
"field": "keywords.keyword"
},
"aggs": {
"sums": {
"sum": {
"field": "keywords.count"
}
}
}
}
}
}
}
}
result:
"buckets": [{
"key": "value1",
"doc_count": 495,
"sums": {
"value": 609
}
},
{
"key": "value2",
"doc_count": 440,
"sums": {
"value": 615
}
},
{
"key": "value3",
"doc_count": 319,
"sums": {
"value": 421
}
},
...]

Java Jolt bazaarvoice nested array

I am using Jolt to transform data from
{
"Data": {
"ROOT": {
"MODIFIED_DATE": "2018-06-27T13:53:47.8",
"A1": [
{
"FLD1": "BB",
"A2": [
{
"FLD2": 1
}
]
},
{
"FLD1": "AA",
"A2": [
{
"FLD2": 2
}
]
}
]
}
},
"metaData": {
"FLD3": "5f3c4"
}
}
To
{
"modifiedDate": "2018-06-27T13:53:47.8",
"a1": [
{
"fld1": "BB",
"a2": [
{
"fld2": 1
}
]
},
{
"fld1": "AA",
"a2": [
{
"fld2": 2
}
]
}
],
fld3: "5f3c4"
}
My spec is
[
{
"operation": "shift",
"spec": {
"Data": {
"ROOT": {
"MODIFIED_DATE": "modifiedDate",
"A1": {
"*": {
"FLD1": "a1[&1]",
"A2": {
"*": {
"FLD2": "a2[&2].fld2"
}
}
}
}
}
},
"metaData": {
"FLD3": "fld3"
}
}
},
{
"operation": "default",
"spec": {}
}
]
But it doesn't work properly. What have i missed?

Figured it out.
[
{
"operation": "shift",
"spec": {
"Data": {
"ROOT": {
"MODIFIED_DATE": "modifiedDate",
"A1": {
"*": {
"FLD1": "a1[&1].fld1",
"A2": {
"*": {
"FLD2": "a1[&3].a2[&1].fld2"
}
}
}
}
}
},
"metaData": {
"FLD3": "fld3"
}
}
},
{
"operation": "default",
"spec": {}
}
]

Categories

Resources