I have this query I made using sense, I'm breaking my head how to transform it into Java.
I can manage the aggs part, the real pain is the "constant_score"
GET /xxxx/yyyy/_search
{
"size": 0,
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : [
{ "term" : {"userId" : 275}},
{ "range" :{"logDate" : { "gte" : "2016-04-30", "lte" : "now/d" }}}
]
}
}
}
},
"aggs" : {
"datebucket" : {
"date_histogram" : {
"field" : "logDate",
"interval": "day",
"format": "yyyy-MM-dd",
"min_doc_count": 0
},
"aggs": {
"info": {
"filters": {
"filters" : [
{"term": { "logAction": "sleep" }},
{"term": { "logAction": "stop" }}
]
}
}
}
}
}
}
I had a more simple query I managed to do it like this
SearchResponse res = client.prepareSearch("xxxx").setTypes("yyyy")
.setSize(0)
.setQuery(QueryBuilders.termQuery("userId", 95))
.addAggregation(
AggregationBuilders.dateHistogram("date_histogram")
.field("logDate")
.interval(DateHistogramInterval.DAY)
.format("yyyy-MM-dd")
.minDocCount(0)
).execute().get();
Well, I found the solution. I hope this will help someone
String query = "{\"constant_score\" : "
+ "{ \"filter\" : "
+ "{\"bool\" : "
+ "{\"must\" : "
+ "[{ \"term\" : {\"userId\" : " + userID + "}}, "
+ "{ \"range\" :{\"logDate\" : { \"gte\" : \"" + startdate + "\", \"lte\" : \"" + enddate + "\" }}}]"
+ "}"
+ "}"
+ "}"
+ "}";
SearchResponse res = client.prepareSearch(xxxx).setTypes(yyyy)
.setQuery(query).addAggregation(
AggregationBuilders.dateHistogram("date_histogram")
.field("logDate")
.interval(DateHistogramInterval.DAY)
.format("dd-MM-yyyy")
.minDocCount(0)
.subAggregation(AggregationBuilders.filters("info")
.filter(QueryBuilders.termQuery("logAction", "click"))
.filter(QueryBuilders.termQuery("logAction", "view")))
).setSize(0).execute().get();
Related
I have an array of Jsonnodes. something like that
[
{
"coupon": "VAR",
"currency": "USD",
"sip": "94989WAX5",
"lastModifiedDate": "2022-09-23T08:16:25Z"
},
{
"coupon": "VAR1",
"currency": "USD",
"sip": "94989WAX5",
"lastModifiedDate": "2022-09-21T08:16:25Z"
},
{
"coupon": "VAR3",
"currency": "USD",
"sip": "XHBRYWEB1",
"lastModifiedDate": "2022-09-20T08:16:25Z"
}
]
I have a requirement, if the sip value of two nodes are same then I need to pick only that sip which lastModifiedDate is latest. In above example the final output should be remaining two nodes.
[
{
"coupon": "VAR",
"currency": "USD",
"sip": "94989WAX5",
"lastModifiedDate": "2022-09-23T08:16:25Z"
},
{
"coupon": "VAR3",
"currency": "USD",
"sip": "XHBRYWEB1",
"lastModifiedDate": "2022-09-20T08:16:25Z"
}
]
I was try to solve it by creating HashMap<String,JsonNode> where Sip is the key and the JsonNode is complete node. It doesn't seems to be a cleaner way. is there any other way to achieve it. I am using fasterxml.jackson.databind.JsonNode
Map<String, JsonNode> map =
jsonNodeList.stream()
.collect(
toMap(
jsonNode -> jsonNode.get("sip").asText(),
jsonNode -> jsonNode,
(jsonNode1, jsonNode2) -> {
boolean after =
LocalDateTime.parse(String.valueOf(jsonNode1.get("lastModifiedDate")))
.isAfter(
LocalDateTime.parse(
String.valueOf(jsonNode2.get("lastModifiedDate"))));
return after ? jsonNode1 : jsonNode2;
},
HashMap::new));
https://github.com/octomix/josson
Deserialization
Josson josson = Josson.fromJsonString(
"[" +
" {" +
" \"coupon\": \"VAR\"," +
" \"currency\": \"USD\"," +
" \"sip\": \"94989WAX5\"," +
" \"lastModifiedDate\": \"2022-09-23T08:16:25Z\"" +
" }," +
" {" +
" \"coupon\": \"VAR1\"," +
" \"currency\": \"USD\"," +
" \"sip\": \"94989WAX5\"," +
" \"lastModifiedDate\": \"2022-09-21T08:16:25Z\"" +
" }," +
" {" +
" \"coupon\": \"VAR3\"," +
" \"currency\": \"USD\"," +
" \"sip\": \"XHBRYWEB1\"," +
" \"lastModifiedDate\": \"2022-09-20T08:16:25Z\"" +
" }" +
" ]");
Transformation
JsonNode node = josson.getNode("group(sip)#.elements.findByMax(lastModifiedDate)");
System.out.println(node.toPrettyString());
Output
[ {
"coupon" : "VAR",
"currency" : "USD",
"sip" : "94989WAX5",
"lastModifiedDate" : "2022-09-23T08:16:25Z"
}, {
"coupon" : "VAR3",
"currency" : "USD",
"sip" : "XHBRYWEB1",
"lastModifiedDate" : "2022-09-20T08:16:25Z"
} ]
How to keep other fields in the Jolt transform JSON array, I am trying to use wildcard but fields are not added in the final output?
Here is the example input I am using
[
{
"foundduring": "D-DC",
"user_type": "type1",
"location": "location1"
},
{
"foundduring": "D-DG",
"user_type": "type2",
"location": "location2"
},
{
"foundduring": "D-DI",
"user_type": "type3",
"location": "location3"
}
]
I am using the following Jolt transformation and also trying wildcard:
[
{
"operation": "shift",
"spec": {
"*": {
"foundduring": {
"D-DC": {
"#CycleCount": "[&3].foundduring"
},
"D-DG": {
"#Pick": "[&3].foundduring"
},
"D-DI": {
"#Issue": "[&3].foundduring"
}
},
"#": "&"
}
}
}
]
Following is my expected output where shift operation happened and then need to keep all other fields as it it
[
{
"foundduring" : "CycleCount",
"user_type" : "type1",
"location" : "location1"
},
{
"foundduring" : "Pick",
"user_type" : "type2",
"location" : "location2"
},
{
"foundduring" : "Issue",
"user_type" : "type3",
"location" : "location3"
}
]
Actual Output coming:
[
{
"foundduring": "CycleCount"
},
{
"foundduring": "Pick"
},
{
"foundduring": "Issue"
}
]
Consider using "*" wildcard as else case instead of "#" such as
[
{
"operation": "shift",
"spec": {
"*": {
"foundduring": {
"D-DC": {
"#CycleCount": "[&3].&2"
},
"D-DG": {
"#Pick": "[&3].&2"
},
"D-DI": {
"#Issue": "[&3].&2"
}
},
"*": "[&1].&"
}
}
}
]
Btw, no need to get the key name "foundduring", just use &2 substitution to go 2 level up from the current branch and grab that value.
The demo on the site http://jolt-demo.appspot.com/ is
You may consider another library Josson.
https://github.com/octomix/josson
Deserialization
Josson josson = Josson.fromJsonString(
"[" +
" {" +
" \"foundduring\": \"D-DC\"," +
" \"user_type\": \"type1\"," +
" \"location\": \"location1\"" +
" }," +
" {" +
" \"foundduring\": \"D-DG\"," +
" \"user_type\": \"type2\"," +
" \"location\": \"location2\"" +
" }," +
" {" +
" \"foundduring\": \"D-DI\"," +
" \"user_type\": \"type3\"," +
" \"location\": \"location3\"" +
" }" +
"]");
Transformation
JsonNode node = josson.getNode(
"field(foundduring.caseValue('D-DC','CycleCount','D-DG','Pick','D-DI','Issue'))");
System.out.println(node.toPrettyString());
Output
[ {
"foundduring" : "CycleCount",
"user_type" : "type1",
"location" : "location1"
}, {
"foundduring" : "Pick",
"user_type" : "type2",
"location" : "location2"
}, {
"foundduring" : "Issue",
"user_type" : "type3",
"location" : "location3"
} ]
I have XML which can be of any structure like below. Therefore there is no POJO class for instantiation. As you can see I have nested arrays (in this case reservations is an array of two reservation and every reservation have simple elements and arrays of rooms - but that is only example - there may be more arrays and nested arrays).
<reservations>
<reservation>
<id>1318504</id>
<add_date>2020-12-10 12:48:09</add_date>
<rooms>
<room>
<id>28902</id>
<floor>2</floor>
</room>
<room>
<id>28903</id>
<floor>3</floor>
</room>
</rooms>
</reservation >
<reservation>
<id>1318501</id>
<add_date>2021-05-07 07:47:05</add_date>
<rooms>
<room>
<id>5</id>
<floor>25</floor>
</room>
<room>
<id>6</id>
</room>
</rooms>
</reservation>
</reservations>
I need to convert it to Json as:
{
"reservations":[
{
"id":"1318504",
"add_date":"2020-12-10 12:48:09",
"rooms":[
{
"id":"28902",
"floor":2
},
{
"id":"28903",
"floor":3
}
]
},
{
"id":"1318501",
"add_date":"2021-05-07 07:47:05",
"rooms":[
{
"id":"5",
"floor":25
},
{
"id":"6"
}
]
}
]
}
I have read lots of topic about converting XML to Json. I tried something like:
json.org
JSONObject xmlJSONObj = XML.toJSONObject(xmlString);
jackson
JsonNode node = new XmlMapper().readTree(xmlContent);
underscore-java
String jsonString = U.xmlToJson(xmlContent)
and so on. Bu the result, in best case is like:
{
"reservations":{
"reservation":[
{
"rooms":{
"room":[
{
"id":28902,
"floor":2
},
{
"id":28903,
"floor":3
}
]
},
"add_date":"2020-12-10 12:48:09",
"id":1318504
},
{
"rooms":{
"room":[
{
"id":5,
"floor":25
},
{
"id":6
}
]
},
"add_date":"2021-05-07 07:47:05",
"id":1318501
}
]
}
}
I don't need nodes like reservation and room. Could you help me how can I solve this problem? I can manipulate with xml or json, but the form without these nodes is my target.
The solution:
import com.github.underscore.U;
String xml = "<reservations>\n"
+ " <reservation>\n"
+ " <id>1318504</id>\n"
+ " <add_date>2020-12-10 12:48:09</add_date>\n"
+ " <rooms> \n"
+ " <room>\n"
+ " <id>28902</id>\n"
+ " <floor>2</floor>\n"
+ " </room> \n"
+ " <room>\n"
+ " <id>28903</id>\n"
+ " <floor>3</floor>\n"
+ " </room>\n"
+ " </rooms>\n"
+ " </reservation >\n"
+ " <reservation>\n"
+ " <id>1318501</id>\n"
+ " <add_date>2021-05-07 07:47:05</add_date>\n"
+ " <rooms>\n"
+ " <room>\n"
+ " <id>5</id>\n"
+ " <floor>25</floor>\n"
+ " </room>\n"
+ " <room>\n"
+ " <id>6</id>\n"
+ " </room>\n"
+ " </rooms>\n"
+ " </reservation>\n"
+ "</reservations>";
Map<String, Object> map = U.fromXmlMap(xml);
U.set(map, "reservations", U.get(map, "reservations.reservation"));
List<Map<String, Object>> list = U.get(map, "reservations");
for (Map<String, Object> item : list) {
U.set(item, "rooms", U.get(item, "rooms.room"));
}
System.out.println(U.toJson(map));
Result:
{
"reservations": [
{
"id": "1318504",
"add_date": "2020-12-10 12:48:09",
"rooms": [
{
"id": "28902",
"floor": "2"
},
{
"id": "28903",
"floor": "3"
}
]
},
{
"id": "1318501",
"add_date": "2021-05-07 07:47:05",
"rooms": [
{
"id": "5",
"floor": "25"
},
{
"id": "6"
}
]
}
],
"#omit-xml-declaration": "yes"
}
You may be able to use json:Array="true" in the array - and the JsonConvert will interpret it correctly.
https://www.newtonsoft.com/json/help/html/ConvertXmlToJsonForceArray.htm
I have a big problem with XMLSerializer... I use JSON lib (net.sf.json) and this is my code:
public static void main(String[] args) throws Exception {
String xmlInputMsg = "<state>\n" +
" <appId>newID</appId>\n" +
" <newDocuments>\n" +
" <nameValues>\n" +
" <name>test1</name>\n" +
" <value>123456</value>\n" +
" </nameValues>\n" +
" <nameValues>\n" +
" <name>test2</name>\n" +
" <value>987654</value>\n" +
" </nameValues>\n" +
" </newDocuments>\n" +
" <oldDocument>\n" +
" <oldAppId>true</oldAppId>\n" +
" <name>App</name>\n" +
" <source>NA</source>\n" +
" </oldDocument>\n" +
"</state>";
String result = null;
XMLSerializer xmlSerializer = new XMLSerializer();
xmlSerializer.setForceTopLevelObject(true);
result = XMLSerializer.class.toGenericString();
System.out.println(result);
}
My current output is:
"state": {
"appId": "newID",
"newDocuments": [
{
"name": "test1",
"value": "123456"
},
{
"name": "test2",
"value": "987654"
}
],
"oldDocument": {
"oldAppId": "true",
"name": "App",
"source": "NA"
}
}}
But I would like have this msg:
{
"state": {
"appId": "newID",
"newDocuments": {
"nameValues": [
{
"name": "test1",
"value": "123456"
},
{
"name": "test2",
"value": "987654"
}
]
},
"oldDocument": {
"oldAppId": "true",
"name": "App",
"source": "NA"
}
}}
Unfortunately now I missing tag nameValues. Could someone help me with this?
i want to get the maximun id form a subfield of an aptitude object,
{
"mappings": {
"aptitude": {
"dynamic": "strict",
"properties": {
"id": {
"type": "long"
},
"es": {
"type": "text"
},
"en": {
"type": "text"
},
"behaviors": {
"properties": {
"id": {
"type": "long"
},
"es": {
"type": "text"
},
"en": {
"type": "text"
}
}
}
}
}
}
as you can see the aptitude have an array of behaviors who in turn have an id, afaik i should use the maxAggregation from Jest, but cant find a decent example of how to do it in java, can someone help?
i found the way like this:
String query = "{\n"
+" \"query\" : {\n"
+" \"match\" : {\"id\":" + aptitudeId + "}\n"
+" },\n"
+" \"aggs\" : {\n"
+" \"max1\" : {\n"
+" \"max\" : {\n"
+" \"field\" : \"behaviors.id\"\n"
+" }\n"
+" }\n"
+" }\n"
+"}";
i was looking into the aggregation builders from jest but doing it via query was much easier.
the return looks like this:
Search search = new Search.Builder(query)
.addIndex(aptitudeIndexName)
.addType(aptitudeTypeName)
.build();
try {
SearchResult result = client.execute(search);
MaxAggregation max1 = result.getAggregations().getMaxAggregation("max1");
Double max = max1.getMax();
return max.longValue() + 1;//so it would add 1 to the current maximum id