I am new to JSON and trying to manipulate JSON for some validation
My JSON looks like this . I need to pick the JSON object based on the refcode and then get count of different object in that, and navigate deeper inside to get the key value pair. Can someone guide me how I can navigate.
{
"components": [
{
"id": 12,
"text": "ABC",
"refCode": "CO_ABC",
"patternCode": "0",
"components": [
{
"id": 1234,
"text": "types",
"refCode": "CO_TYPES",
"questions": [
{
"questionId": 122324,
"questionText": "Is this you",
"questionSequence": 1,
"questionRefCode": "QN_STAY",
"hasPreselectedAnswer": false,
"responsesMetadata": {
"cardinality": "single",
"patternCode": "5",
"dataType": "STRING",
"numberMin": null,
"numberMax": null
},
"choices": [
{
"choiceId": 5456,
"choiceRefCode": "YES",
"choiceText": "Yes",
"sequence": 1
},
{
"choiceId": 8798,
"choiceRefCode": "NO",
"choiceText": "No",
"sequence": 2
}
],
"editable": true,
"accessible": true
}
]
},
{
"id": 13,
"text": "State of stay",
"refCode": "CO_STATE",
"questions": [
{
"questionId": 1,
"questionText": "Which state do you stay",
"questionSequence": 2,
"questionRefCode": "QN_STATE",
"hasPreselectedAnswer": false,
"responsesMetadata": {
"cardinality": "multiple",
"patternCode": "1",
"dataType": "STRING",
"numberMin": null,
"numberMax": null
},
"choices": [
{
"choiceId": 1,
"choiceRefCode": "CH_AZ",
"choiceText": "Arizona",
"sequence": 1
},
{
"choiceId": 2,
"choiceRefCode": "CH_PA",
"choiceText": "Pennsylvania",
"sequence": 2
}
],
"accessible": true
}
]
}
]
}
]
}
I need to do sortByCount and return all the fields instead of just _id and count.
sortByCount returns:
{ "_id" : "1", "count" : 4 }
{ "_id" : "2", "count" : 3 }
{ "_id" : "3", "count" : 2 }
{ "_id" : "4", "count" : 2 }
{ "_id" : "5", "count" : 1 }
But, I need a complete document like below:
{
"_id": 1,
"title": "The Pillars of Society",
"artist": "Grosz",
"year": 1926,
"tags": ["painting", "satire", "Expressionism", "caricature"]
} {
"_id": 2,
"title": "Melancholy III",
"artist": "Munch",
"year": 1902,
"tags": ["woodcut", "Expressionism"]
} {
"_id": 3,
"title": "Dancer",
"artist": "Miro",
"year": 1925,
"tags": ["oil", "Surrealism", "painting"]
} {
"_id": 4,
"title": "The Great Wave off Kanagawa",
"artist": "Hokusai",
"tags": ["woodblock", "ukiyo-e"]
} {
"_id": 5,
"title": "The Persistence of Memory",
"artist": "Dali",
"year": 1931,
"tags": ["Surrealism", "painting", "oil"]
}
Is there any way to replace root after sortByCount? In Java, I don't see any push method after sortByCount
$sortByCount is essentially a combination of $group followed by $sort on count field of group stage. If you really want the entire documents, you can try this:
db.collection.aggregate([
{
"$group": {
"_id": {
"id": "$_id"
},
"count": {
$sum: 1
},
"reqItems": {
$push: {
"title": "$title",
"artist": "$artist"
}
}
}
},
{
$sort: {
count: -1
}
}
])
Playground link
I need to collect an entity records as from date to date, based on one attribute's value. If the value of type attribute is sequentially same based on date, it should be grouped by date. Since date mentioned as sequentially, should be ordered. Even if the value of type attribute of a record is different, rest of the records also should be under the same day. See the visual representation. I've tried this;
Map<LocalDate, List<Entity>> collection = entities.stream().collect(Collectors.groupingBy(Entity::getDate))
.entrySet().stream().sorted(Map.Entry.comparingByKey()).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue, (oldValue, newValue) -> oldValue, LinkedHashMap::new));
In my implementation, I am able to only collect by dates, but I want to collect by DateRange. I want to achieve some thing like this;
Map<DateRange, List<Entity>> collection = entities.stream()...// implementation
Entity
[
{
"id": 1,
"date": "2020-01-01",
"type": 5
},
{
"id": 2,
"date": "2020-01-01",
"type": 5
},
{
"id": 1,
"date": "2020-01-02",
"type": 5
},
{
"id": 2,
"date": "2020-01-02",
"type": 5
},
.
.
.
]
Example
Date range changes based on the value of the type attribute. For example, if type=5 for all dates, so all records should be in one range. Let's say there are only records for one year and I'm assuming there are only two unique id value(id=1, id=2), so in collection I should achieve this;
[{
"From: 2020-01-01, To: 2020-12-31": [{
"record1":
{
"id": 1,
"type": "5"
},
"record2":
{
"id": 2,
"type": "5"
}
}]
}]
Another example
If the type=5 for all dates except '2020-02-01' and in '2020-02-01' type=6 for the id=1, then ranges should be like the below. I'm still assuming, there are records only for one year and there are only two unique id value(id=1, id=2).
[
{
"From: 2020-01-01, To: 2020-01-31": [{
"record1":
{
"id": 1,
"type": "5"
},
"record2":
{
"id": 2,
"type": "5"
}
}],
},
{
"From: 2020-02-01, To: 2020-02-01": [{
"record1":
{
"id": 1,
"type": "6"
},
"record2":
{
"id": 2,
"type": "5"
}
}],
},
{
"From: 2020-02-02, To: 2020-12-31": [{
"record1":
{
"id": 1,
"type": "5"
},
"record2":
{
"id": 2,
"type": "5"
}
}]
}
]
I have a JSONArray within a JSONArray, I want to apply JSONPath expression on this in such a way that i get JSONObject or JSONArray as a result when a condition is satsified on the inner JSONArray.
Eg:
{
"A": [
{
"B": [
{
"id": 1
},
{
"id": 2
},
{
"id": 3
}
],
"C": {
"id": 10,
"name": "PQR"
},
"id": 25,
"name": "XYZ"
},
{
"B": [
{
"id": 4
},
{
"id": 5
},
{
"id": 6
}
],
"C": {
"id": 15,
"name": "PQR"
},
"id": 20,
"name": "XYZ"
}
]
}
if i want all elements of A where C.id = 10, I would use: $.A[?(#.C.id == 10)]
Now, What predicate is to be used to obtain all the objects within A, where B.id = 1? Note: B is an array of JSON objects.
I had success with $.A[?(#.B[?(#.id == 1)])]
but only when using Scala's Gatling implementation:
http://jsonpath.herokuapp.com/
The Jayway implementation seems to totally ignore the inner filter and according to an issue on their GitHub, that's a bug.
I have 999 documents which I am using for experimenting with elastic search.
There is a field f4 in my type mapping which is analyzed and has following settings for analyzer :
"myNGramAnalyzer" => [
"type" => "custom",
"char_filter" => ["html_strip"],
"tokenizer" => "standard",
"filter" => ["lowercase","standard","asciifolding","stop","snowball","ngram_filter"]
]
My filter is as below :
"filter" => [
"ngram_filter" => [
"type" => "edgeNGram",
"min_gram" => "2",
"max_gram" => "20"
]
]
I have value for field f4 as "Proj1", "Proj2", "Proj3"...... so on.
Now when I try to do search using cross fields for "proj1" string, I was expecting document with "Proj1" to be returned at the top of the response with max score. But it doesn't. Rest all the data is almost same in content.
Also I don't understand why it matches all 999 document?
Following is my search :
{
"index": "myindex",
"type": "mytype",
"body": {
"query": {
"multi_match": {
"query": "proj1",
"type": "cross_fields",
"operator": "and",
"fields": "f*"
}
},
"filter": {
"term": {
"deleted": "0"
}
}
}
}
My search response is :
{
"took": 12,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 999,
"max_score": 1,
"hits": [{
"_index": "myindex",
"_type": "mytype",
"_id": "42",
"_score": 1,
"_source": {
"f1": "396","f2": "125650","f3": "BH.1511AI.001",
"f4": "Proj42",
"f5": "BH.1511AI.001","f6": "","f7": "","f8": "","f9": "","f10": "","f11": "","f12": "","f13": "","f14": "","f15": "","f16": "09/05/16 | 01:02PM | User","deleted": "0"
}
}, {
"_index": "myindex",
"_type": "mytype",
"_id": "47",
"_score": 1,
"_source": {
"f1": "396","f2": "137946","f3": "BH.152096.001",
"f4": "Proj47",
"f5": "BH.1511AI.001","f6": "","f7": "","f8": "","f9": "","f10": "","f11": "","f12": "","f13": "","f14": "","f15": "","f16": "09/05/16 | 01:02PM | User","deleted": "0"
}
},
//.......
//.......
//MANY RECORDS IN BETWEEN HERE
//.......
//.......
{
"_index": myindex,
"_type": "mytype",
"_id": "1",
"_score": 1,
"_source": {
"f1": "396","f2": "142095","f3": "BH.705215.001",
"f4": "Proj1",
"f5": "BH.1511AI.001","f6": "","f7": "","f8": "","f9": "","f10": "","f11": "","f12": "","f13": "","f14": "","f15": "","f16": "09/05/16 | 01:02PM | User","deleted": "0"
}
//.......
//.......
//MANY RECORDS IN BETWEEN HERE
//.......
//.......
}]
}
}
Any thing that I am doing wrong or missing? (Apologies for lengthy question, but I thought to give all possible information discarding unnecessary other code).
EDITED :
Term vector response
{
"_index": "myindex",
"_type": "mytype",
"_id": "10",
"_version": 1,
"found": true,
"took": 9,
"term_vectors": {
"f4": {
"field_statistics": {
"sum_doc_freq": 5886,
"doc_count": 999,
"sum_ttf": 5886
},
"terms": {
"pr": {
"doc_freq": 999,
"ttf": 999,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
},
"pro": {
"doc_freq": 999,
"ttf": 999,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
},
"proj": {
"doc_freq": 999,
"ttf": 999,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
},
"proj1": {
"doc_freq": 111,
"ttf": 111,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
},
"proj10": {
"doc_freq": 11,
"ttf": 11,
"term_freq": 1,
"tokens": [{
"position": 0,
"start_offset": 0,
"end_offset": 6
}]
}
}
}
}
}
EDITED 2
Mappings for field f4
"f4" : {
"type" : "string",
"index_analyzer" : "myNGramAnalyzer",
"search_analyzer" : "standard"
}
I have updated to use standard analyzer for query time, which has improved the results but still not what I expected.
Instead of 999 (all documents) now it return 111 documents like "Proj1", "Proj11", "Proj111"......"Proj1", "Proj181"......... etc.
Still "Proj1" is in between the results and not at the top.
There is no index_analyzer (at least not from Elasticsearch version 1.7). For mapping parameters you can use analyzer and search_analyzer.
Try the following steps in order to make it work.
Create myindex with analyzer settings:
PUT /myindex
{
"settings": {
"analysis": {
"filter": {
"ngram_filter": {
"type": "edge_ngram",
"min_gram": 2,
"max_gram": 20
}
},
"analyzer": {
"myNGramAnalyzer": {
"type": "custom",
"tokenizer": "standard",
"char_filter": "html_strip",
"filter": [
"lowercase",
"standard",
"asciifolding",
"stop",
"snowball",
"ngram_filter"
]
}
}
}
}
}
Add mappings to mytype (to make it short I just mapped the relevant fields):
PUT /myindex/_mapping/mytype
{
"properties": {
"f1": {
"type": "string"
},
"f4": {
"type": "string",
"analyzer": "myNGramAnalyzer",
"search_analyzer": "standard"
},
"deleted": {
"type": "string"
}
}
}
Index some data:
PUT myindex/mytype/1
{
"f1":"396",
"f4":"Proj12" ,
"deleted": "0"
}
PUT myindex/mytype/2
{
"f1":"42",
"f4":"Proj22" ,
"deleted": "1"
}
Now try your query:
GET myindex/mytype/_search
{
"query": {
"multi_match": {
"query": "proj1",
"type": "cross_fields",
"operator": "and",
"fields": "f*"
}
},
"filter": {
"term": {
"deleted": "0"
}
}
}
It should return document #1. It worked for me with Sense. I am using Elasticsearch 2.X versions.
Hope I have managed to help :)
After hours of spending time to find a solution to this, I finally made it work.
So I kept everything same as mentioned in my question, using n gram analzyer while indexing data. The only thing I had to change was, to use the all field in my search query as a bool query with my existing multi-match query.
Now my result for search text Proj1 would return me results in an order such as Proj1, Proj121, Proj11, etc.
Although this does not return the exact order like Proj1, Proj11, Proj121, etc, but still it closely resembles the result that I wanted.