I want the jolt transform for the given input . Your help in this is highly appreciated . thanks i am providing the input and expected output. in input json Photos array is dynamic in nature. Here it is 3 , it can be 3 or 4 or5 any .
Input JSON :
{
"Entity": {
"card": {
"cardNo":"123456789",
"cardStatus":"10",
"cardAddress":"UK",
"cardAddress1":"US",
"cardCity":"mk" ,
"name": "RAM",
"lastName": "ABU",
"name1": "RAM1",
"lastName1": "ABU1"
},
"Photos": [
{
"Id": 327703,
"Caption": "TEST>> photo 1",
"Url": "http://bob.com/0001/327703/photo.jpg"
},
{
"Id": 327704,
"Caption": "TEST>> photo 2",
"Url": "http://bob.com/0001/327704/photo.jpg"
},
{
"Id": 327704,
"Caption": "TEST>> photo 2",
"Url": "http://bob.com/0001/327704/photo.jpg"
}
]
}
}
Used Jolt Spec :
[
{
"operation": "shift",
"spec": {
"Entity": {
"card": {
"cardNo": "tab.text",
"cardAddress": "address[0].add",
"cardAddress1": "address[0].add2",
"cardC*": "address[0].mk",
"Id1": "Photos.no",
"#http.1.com": "Photos.caption2",
"Id2": "Photos.no",
"#http.2.com": "Photos.caption2"
},
"Photos": {
"*": {
"Id": "Photos.no",
"Caption": "Photos.caption2"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"tab": "&",
"address": "&",
"Photos": {
"*": {
"*": {
"#": "&3[&1].&2"
}
}
}
}
}
]
Current Output :
{
"tab": {
"text": "123456789"
},
"address": [
{
"add": "UK",
"add2": "US",
"mk": "mk"
}
],
"Photos": [
{
"caption2": "http.1.com",
"no": 222444
},
{
"caption2": "http.2.com",
"no": 222444
},
{
"caption2": "TEST>> photo 1",
"no": 327703
},
{
"caption2": "TEST>> photo 2",
"no": 327704
},
{
"caption2": "TEST>> photo 2",
"no": 327704
}
]
}
Expected Output ( i need "no" in string like below ) :
{
"tab": {
"text": "123456789"
},
"address": [
{
"add": "UK",
"add2": "US",
"mk": "mk"
}
],
"Photos": [
{
"caption2": "http.1.com",
"no": "222444"
},
{
"caption2": "http.2.com",
"no": "222444"
},
{
"caption2": "TEST>> photo 1",
"no": "327703"
},
{
"caption2": "TEST>> photo 2",
"no": "327704"
},
{
"caption2": "TEST>> photo 2",
"no": "327704"
}
]
}
You can add a modify transformation to the current spec along with toString function such as
,
{
"operation": "modify-overwrite-beta",
"spec": {
"Photos": {
"*": {
"no": "=toString"
}
}
}
}
Related
I am pretty new to Jolt. I have a JSON payload that represents a logical condition such as AND/OR which can contain an array of conditions called "conditionPredicates". This can result in nested conditions of the form AND(OR(a, b ,c), OR(d,e)) I want to replace the field "conditionPredicates" as simply "conditions". Any help please?
Sample Input:
{
"type": "Condition.Aggregate.AND",
"payload": {
"conditionPredicates": [
{
"type": "Condition.Apple",
"payload": {
"fruit": "apple"
}
},
{
"type": "Condition.Aggregate.AND",
"payload": {
"conditionPredicates": [
{
"type": "Condition.Orange",
"payload": {
"fruit": "orange"
}
}
]
}
}
]
}
}
Expected Output
{
"type": "Condition.Aggregate.AND",
"payload": {
"conditions": [
{
"type": "Condition.Apple",
"payload": {
"fruit": "apple"
}
},
{
"type": "Condition.Aggregate.AND",
"payload": {
"conditions": [
{
"type": "Condition.Orange",
"payload": {
"fruit": "orange"
}
}
]
}
}
]
}
}
You can use a shift transformation like this one :
[
{
"operation": "shift",
"spec": {
"*": "&",
"payload": {
"conditionPredicates": {
"*": {
"*": "&3.conditions[&1].&",
"payload": {
"*": "&4.conditions[&2].&1.&",
"conditionPredicates": {
"*": {
"#": "&3.conditions[&4].&3.conditions[]"
}
}
}
}
}
}
}
}
]
I'm trying to get the counts group by the repetitive items in array without distinct, use aggs terms but not work
GET /my_index/_search
{
"size": 0,
"aggs": {
"keywords": {
"terms": {
"field": "keywords"
}
}
}
}
documents like:
"keywords": [
"value1",
"value1",
"value2"
],
but the result is:
"buckets": [
{
"key": "value1",
"doc_count": 1
},
{
"key": "value2",
"doc_count": 1
}
]
how can i get the result like:
"buckets": [
{
"key": "value1",
"doc_count": 2
},
{
"key": "value2",
"doc_count": 1
}
]
finally I modify the mapping use nested:
"keywords": {
"type": "nested",
"properties": {
"count": {
"type": "integer"
},
"keyword": {
"type": "keyword"
}
}
},
and query:
GET /my_index/_search
{
"size": 0,
"aggs": {
"keywords": {
"nested": {
"path": "keywords"
},
"aggs": {
"keyword_name": {
"terms": {
"field": "keywords.keyword"
},
"aggs": {
"sums": {
"sum": {
"field": "keywords.count"
}
}
}
}
}
}
}
}
result:
"buckets": [{
"key": "value1",
"doc_count": 495,
"sums": {
"value": 609
}
},
{
"key": "value2",
"doc_count": 440,
"sums": {
"value": 615
}
},
{
"key": "value3",
"doc_count": 319,
"sums": {
"value": 421
}
},
...]
I've the below JSON data.
{
"faceDetails": [
{
"boundingBox": {
"width": 0.36888888,
"height": 0.2777778,
"left": 0.4814815,
"top": 0.4422222
},
"emotions": [
{
"type": "SAD",
"confidence": 40.245743
},
{
"type": "CONFUSED",
"confidence": 15.142041
},
{
"type": "SURPRISED",
"confidence": 1.9677103
}
],
"smile": {
"value": false,
"confidence": 90.49947
}
},
{
"boundingBox": {
"width": 0.36888888,
"height": 0.2777778,
"left": 0.4814815,
"top": 0.4422222
},
"emotions": [
{
"type": "SAD",
"confidence": 40.245743
},
{
"type": "CONFUSED",
"confidence": 15.142041
},
{
"type": "SURPRISED",
"confidence": 1.9677103
}
],
"smile": {
"value": false,
"confidence": 90.49947
}
},
{
"boundingBox": {
"width": 0.36888888,
"height": 0.2777778,
"left": 0.4814815,
"top": 0.4422222
},
"emotions": [
{
"type": "SAD",
"confidence": 40.245743
},
{
"type": "CONFUSED",
"confidence": 15.142041
},
{
"type": "SURPRISED",
"confidence": 1.9677103
}
],
"smile": {
"value": false,
"confidence": 90.49947
}
},
{
"boundingBox": {
"width": 0.36888888,
"height": 0.2777778,
"left": 0.4814815,
"top": 0.4422222
},
"emotions": [
{
"type": "Happy",
"confidence": 40.245743
},
{
"type": "CONFUSED",
"confidence": 15.142041
},
{
"type": "SURPRISED",
"confidence": 1.9677103
}
],
"smile": {
"value": false,
"confidence": 90.49947
}
}
]
}
From this data I need to get the emotion with the top confidence score, create a variable and assign this score to that. For ex. from the above data, My output should be like below.
Happy: 1
Sad: 3
Here the order of printing doesn't matter. I'm asking you this because, I'm unable to understand on how to create a variable on the go. i.e. If there is another variable the type is cool, then I need a variable named cool and its count.
Also I'm unable to understand how to capture the emotions value
Below is the code that I tried
private static void getTheResultBasedOnEmotions(String inputText)
throws JsonParseException, JsonMappingException, IOException {
ObjectMapper mapper = new ObjectMapper();
Map<String, Object> map = mapper.readValue(inputText, new TypeReference<Map<String, Object>>() {
});
List faceCount = (List) map.get("faceDetails");
System.out.println(faceCount.toString());
}
and the result that I get is as below
[
{
"boundingBox": {
"width": 0.36888888,
"height": 0.2777778,
"left": 0.4814815,
"top": 0.4422222
},
"emotions": [
{
"type": "SAD",
"confidence": 40.245743
},
{
"type": "CONFUSED",
"confidence": 15.142041
},
{
"type": "SURPRISED",
"confidence": 1.9677103
}
],
"smile": {
"value": false,
"confidence": 90.49947
}
},
{
"boundingBox": {
"width": 0.36888888,
"height": 0.2777778,
"left": 0.4814815,
"top": 0.4422222
},
"emotions": [
{
"type": "SAD",
"confidence": 40.245743
},
{
"type": "CONFUSED",
"confidence": 15.142041
},
{
"type": "SURPRISED",
"confidence": 1.9677103
}
],
"smile": {
"value": false,
"confidence": 90.49947
}
},
{
"boundingBox": {
"width": 0.36888888,
"height": 0.2777778,
"left": 0.4814815,
"top": 0.4422222
},
"emotions": [
{
"type": "SAD",
"confidence": 40.245743
},
{
"type": "CONFUSED",
"confidence": 15.142041
},
{
"type": "SURPRISED",
"confidence": 1.9677103
}
],
"smile": {
"value": false,
"confidence": 90.49947
}
},
{
"boundingBox": {
"width": 0.36888888,
"height": 0.2777778,
"left": 0.4814815,
"top": 0.4422222
},
"emotions": [
{
"type": "Happy",
"confidence": 40.245743
},
{
"type": "CONFUSED",
"confidence": 15.142041
},
{
"type": "SURPRISED",
"confidence": 1.9677103
}
],
"smile": {
"value": false,
"confidence": 90.49947
}
}
]
Please let me know how can I do this.
Here I use jackson file to parse.
Thanks
Here's my approach:
Parse JSON into POJO with help of Gson and Online POJO generator
PS.You can use any library, like jackson, to parse JSON to POJO.
//classes generated by Json2POJO generator
class FaceDetail {
private Object boundingBox;
private List<Emotion> emotions;
private Object smile;
public List<Emotion> getEmotions() {
return emotions;
}
}
class Emotion {
private String type;
private Double confidence;
public String getType() {
return type;
}
}
//Parse Json to POJO
List<FaceDetail> faceDetails = gson.fromJson(JSONData, ArrayList<FaceDetail>.class);
Count Emotion number by type with Java stream API
//Flap map all emotion to an ArrayList
List<Emotion> allEmotions = faceDetails.stream()
.map(FaceDetail::getEmotions)
.flatMap(emotions -> emotions.stream())
.collect(Collectors.toList());
//count
long countSAD = allEmotions.stream()
.filter(emotion -> emotion.getType().equals("SAD"))
.count();
Hope this could help:)
I have created Elastic search mapping as below.
PUT indexcloud
{
"mappings": {
"_default_": {
"_all": {
"enabled": false
},
"_source": {
"compressed": true
},
"properties": {
"term": {
"fields": {
"raw": {
"index": "not_analyzed",
"analyzer": "lowercase_analyzer",
"type": "string"
}
},
"analyzer": "concat_all_alpha",
"type": "string"
},
"relation": {
"type": "nested",
"properties": {
"term": {
"type": "string",
"analyzer": "concat_all_alpha",
"fields": {
"raw": {
"index": "not_analyzed",
"analyzer": "lowercase_analyzer",
"type": "string"
}
}
}
}
}
}
}
},
"settings": {
"index": {
"analysis": {
"analyzer": {
"concat_all_alpha": {
"char_filter": [
"only_alphanum"
],
"filter": [
"lowercase"
],
"tokenizer": "keyword"
},
"uppercase_analyzer": {
"filter": "uppercase",
"tokenizer": "keyword"
},
"lowercase_analyzer": {
"filter": "lowercase",
"tokenizer": "keyword"
}
},
"char_filter": {
"only_alphanum": {
"pattern": "[^A-Z^a-z^0-9]|\\^",
"replacement": "",
"type": "pattern_replace"
}
}
},
"max_result_window": "1000000"
}
}
}
Sample index doc
POST indexcloud/skill
{"term":"Java Language","relation":[{"term":"java8"},{"term":"struct"},{"term":"j2ee"},{"term":"Progamming Language"}]}
I want to search using filtered query as below
GET indexcloud/_search
{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"term" : "Java Language"
}
}
}
}
}
But this is not working. How can i achieve this ?. Note : i dont want like below
GET indexcloud/_search
{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"term" : "javalanguage"
}
}
}
}
}
Because i want to search, the way i index.
My mapping is as follows:
{
"mappings": {
"person": {
"properties": {
"lastUpdated": {
"type": "long"
},
"isDeleted": {
"type": "boolean"
},
"person": {
"properties": {
"car": {
"type": "nested",
"properties": {
"model": {
"type": "string"
},
"make": {
"type": "string"
}
}
},
"last_name": {
"type": "string"
},
"first_name": {
"type": "string"
}
}
}
}
}
}
}
I have two documents:
{
"person": {
"first_name": "Bob",
"last_name": "Doe",
"car": [
{
"make": "Saturn",
"model": "Imprezza"
},
{
"make": "Honda",
"model": "Accord"
}
]
},
"isDeleted": false,
"lastUpdated": 1433257051959
}
and
{
"person": {
"first_name": "Zach",
"last_name": "Foobar",
"car": [
{
"make": "Saturn",
"model": "SL"
},
{
"make": "Subaru",
"model": "Imprezza"
}
]
},
"isDeleted": false,
"lastUpdated": 1433257051959
}
I wanted to query the car.make field and so, I wrote the following query:
{
"query": {
"nested": {
"path": "person.person.car",
"query": {
"match": {
"car.make": "Saturn"
}
},
"inner_hits": {}
}
}
}
However, I am not getting anything back results back in return. When I remove the person level object and try to search, then it works. Any idea how to go about doing multi-level nested queries?
EDIT: On the other hand, when I structure my data like this and query then it works.
{
"mappings": {
"person": {
"properties": {
"car": {
"type": "nested",
"properties": {
"model": {
"type": "string"
},
"make": {
"type": "string"
}
}
},
"last_name": {
"type": "string"
},
"first_name": {
"type": "string"
}
}
}
}
}
{
"first_name": "Zach",
"last_name": "Foobar",
"car": [
{
"make": "Saturn",
"model": "SL"
},
{
"make": "Subaru",
"model": "Imprezza"
}
]
}
{
"first_name": "Bob",
"last_name": "Doe",
"car": [
{
"make": "Saturn",
"model": "Imprezza"
},
{
"make": "Honda",
"model": "Accord"
}
]
}
{
"query": {
"nested": {
"path": "person.car",
"query": {
"match": {
"car.make": "Honda"
}
},
"inner_hits": {}
}
}
}
This way the query works. I feel like this has something to do with multi-level nesting. Multi-level nesting is not working.
The nested path attribute needs to be "person.car".
Add "type": "nested", above the (2nd level) person properties line if you wish person to be a nested field type, which is required for Nested Query searches. The default field type is object field.
The naming you are using is confusing, try to rename your mapping not to use person twice.
{
"query": {
"nested": {
"path": "person.car",
"query": {
"match": {
"make": "Saturn"
}
},
"inner_hits": {}
}
}
}