I am doing term aggregation based on field [type] like below but elastic is returning only 1 term count instead of 2 it is not doing nested object aggregation i.e under comments.data.comments[is a list] under this i have 2 type.
{
"aggs": {
"genres": {
"terms": {
"field": "comments.data.comments.type"
}
}
}
}
Gotta utilize the nested field type:
PUT events
{
"mappings": {
"properties": {
"events": {
"type": "nested",
"properties": {
"ecommerceData": {
"type": "nested",
"properties": {
"comments": {
"type": "nested",
"properties": {
"recommendationType": {
"type": "keyword"
}
}
}
}
}
}
}
}
}
}
POST events/_doc
{
"events": [
{
"eventId": "1",
"ecommerceData": [
{
"comments": [
{
"rank": 1,
"recommendationType": "abc"
},
{
"rank": 1,
"recommendationType": "abc"
}
]
}
]
}
]
}
GET events/_search
{
"size": 0,
"aggs": {
"genres": {
"nested": {
"path": "events.ecommerceData.comments"
},
"aggs": {
"nested_comments_recomms": {
"terms": {
"field": "events.ecommerceData.comments.recommendationType"
}
}
}
}
}
}
I'm trying to get the counts group by the repetitive items in array without distinct, use aggs terms but not work
GET /my_index/_search
{
"size": 0,
"aggs": {
"keywords": {
"terms": {
"field": "keywords"
}
}
}
}
documents like:
"keywords": [
"value1",
"value1",
"value2"
],
but the result is:
"buckets": [
{
"key": "value1",
"doc_count": 1
},
{
"key": "value2",
"doc_count": 1
}
]
how can i get the result like:
"buckets": [
{
"key": "value1",
"doc_count": 2
},
{
"key": "value2",
"doc_count": 1
}
]
finally I modify the mapping use nested:
"keywords": {
"type": "nested",
"properties": {
"count": {
"type": "integer"
},
"keyword": {
"type": "keyword"
}
}
},
and query:
GET /my_index/_search
{
"size": 0,
"aggs": {
"keywords": {
"nested": {
"path": "keywords"
},
"aggs": {
"keyword_name": {
"terms": {
"field": "keywords.keyword"
},
"aggs": {
"sums": {
"sum": {
"field": "keywords.count"
}
}
}
}
}
}
}
}
result:
"buckets": [{
"key": "value1",
"doc_count": 495,
"sums": {
"value": 609
}
},
{
"key": "value2",
"doc_count": 440,
"sums": {
"value": 615
}
},
{
"key": "value3",
"doc_count": 319,
"sums": {
"value": 421
}
},
...]
I am trying to validate a json payload against a swagger file that contains the service agreement. I am using the json-schema-validator(2.1.7) library to achieve this, but at the moment it's not validating against the specified patterns or min/max length.
Java Code:
public void validateJsonData(final String jsonData) throws IOException, ProcessingException {
ClassLoader classLoader = getClass().getClassLoader();
File jsonSchemaFile = new File (classLoader.getResource("coachingStatusUpdate.json").getFile());
String jsonSchema = new String(Files.readAllBytes(jsonSchemaFile.toPath()));
final JsonNode dataNode = JsonLoader.fromString(jsonData);
final JsonNode schemaNode = JsonLoader.fromString(jsonSchema);
final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
JsonValidator jsonValidator = factory.getValidator();
ProcessingReport report = jsonValidator.validate(schemaNode, dataNode);
System.out.println(report);
if (!report.toString().contains("success")) {
throw new ProcessingException (
report.toString());
}
}
Message I am sending through
{
"a": "b",
"c": "d",
"e": -1,
"f": "2018-10-30",
"g": "string" }
The swagger definition:
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "Test",
"termsOfService": "http://www.test.co.za",
"license": {
"name": "Test"
}
},
"host": "localhost:9001",
"basePath": "/test/",
"tags": [
{
"name": "controller",
"description": "Submission"
}
],
"paths": {
"/a": {
"put": {
"tags": [
"controller"
],
"summary": "a",
"operationId": "aPUT",
"consumes": [
"application/json;charset=UTF-8"
],
"produces": [
"application/json;charset=UTF-8"
],
"parameters": [
{
"in": "body",
"name": "aRequest",
"description": "aRequest",
"required": true,
"schema": {
"$ref": "#/definitions/aRequest"
}
}
],
"responses": {
"200": {
"description": "Received",
"schema": {
"$ref": "#/definitions/a"
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"408": {
"description": "Request Timeout"
},
"500": {
"description": "Generic Error"
},
"502": {
"description": "Bad Gateway"
},
"503": {
"description": "Service Unavailable"
}
}
}
}
},
"definitions": {
"aRequest": {
"type": "object",
"required": [
"a",
"b",
"c",
"d"
],
"properties": {
"a": {
"type": "string",
"description": "Status",
"enum": [
"a",
"b",
"c",
"d",
"e",
"f",
"g",
"h"
]
},
"aReason": {
"type": "string",
"description": "Reason",
"enum": [
"a",
"b",
"c",
"d",
"e",
"f",
"g",
"h",
"i",
"j",
"k",
"l",
"m",
"n"
]
},
"correlationID": {
"type": "integer",
"format": "int32",
"description": "",
"minimum": 1,
"maximum": 9999999
},
"effectiveDate": {
"type": "string",
"format": "date",
"description": ""
},
"f": {
"type": "string",
"description": "",
"minLength": 1,
"maxLength": 100
}
}
},
"ResponseEntity": {
"type": "object",
"properties": {
"body": {
"type": "object"
},
"statusCode": {
"type": "string",
"enum": [
"100",
"101",
"102",
"103",
"200",
"201",
"202",
"203",
"204",
"205",
"206",
"207",
"208",
"226",
"300",
"301",
"302",
"303",
"304",
"305",
"307",
"308",
"400",
"401",
"402",
"403",
"404",
"405",
"406",
"407",
"408",
"409",
"410",
"411",
"412",
"413",
"414",
"415",
"416",
"417",
"418",
"419",
"420",
"421",
"422",
"423",
"424",
"426",
"428",
"429",
"431",
"451",
"500",
"501",
"502",
"503",
"504",
"505",
"506",
"507",
"508",
"509",
"510",
"511"
]
},
"statusCodeValue": {
"type": "integer",
"format": "int32"
}
}
}
}
}
As you can see I am sending through a correlationID of -1, which should fail validation, but at the moment is's returning as successful:
com.github.fge.jsonschema.report.ListProcessingReport: success
I suggest using this library, which worked for me:
https://github.com/bjansen/swagger-schema-validator
Example:
invalid-pet.json
{
"id": 0,
"category": {
"id": 0,
"name": "string"
},
"named": "doggie",
"photoUrls": [
"string"
],
"tags": [
{
"id": 0,
"name": "string"
}
],
"status": "available"
}
My SchemaParser:
#Component
public class SchemaParser {
private Logger logger = LoggerFactory.getLogger(getClass());
public boolean isValid(String message, Resource schemaLocation) {
try (InputStream inputStream = schemaLocation.getInputStream()) {
SwaggerValidator validator = SwaggerValidator.forJsonSchema(new InputStreamReader(inputStream));
ProcessingReport report = validator.validate(message, "/definitions/Pet");
return report.isSuccess();
} catch (IOException e) {
logger.error("IOException", e);
return false;
} catch (ProcessingException e) {
e.printStackTrace();
return false;
}
}
}
A test:
#Test
void shouldFailValidateWithPetstoreSchema() throws IOException {
final Resource validPetJson = drl.getResource("http://petstore.swagger.io/v2/swagger.json");
try (Reader reader = new InputStreamReader(validPetJson.getInputStream(), UTF_8)) {
final String petJson = FileCopyUtils.copyToString(reader);
final boolean valid = schemaParser.isValid(petJson, petstoreSchemaResource);
assertFalse(valid);
}
}
json-schema-validator seems to work with pure JSON Schema only. OpenAPI Specification uses an extended subset of JSON Schema, so the schema format is different. You need a library that can validate specifically against OpenAPI/Swagger definitions, such as Atlassian's swagger-request-validator.
I have created Elastic search mapping as below.
PUT indexcloud
{
"mappings": {
"_default_": {
"_all": {
"enabled": false
},
"_source": {
"compressed": true
},
"properties": {
"term": {
"fields": {
"raw": {
"index": "not_analyzed",
"analyzer": "lowercase_analyzer",
"type": "string"
}
},
"analyzer": "concat_all_alpha",
"type": "string"
},
"relation": {
"type": "nested",
"properties": {
"term": {
"type": "string",
"analyzer": "concat_all_alpha",
"fields": {
"raw": {
"index": "not_analyzed",
"analyzer": "lowercase_analyzer",
"type": "string"
}
}
}
}
}
}
}
},
"settings": {
"index": {
"analysis": {
"analyzer": {
"concat_all_alpha": {
"char_filter": [
"only_alphanum"
],
"filter": [
"lowercase"
],
"tokenizer": "keyword"
},
"uppercase_analyzer": {
"filter": "uppercase",
"tokenizer": "keyword"
},
"lowercase_analyzer": {
"filter": "lowercase",
"tokenizer": "keyword"
}
},
"char_filter": {
"only_alphanum": {
"pattern": "[^A-Z^a-z^0-9]|\\^",
"replacement": "",
"type": "pattern_replace"
}
}
},
"max_result_window": "1000000"
}
}
}
Sample index doc
POST indexcloud/skill
{"term":"Java Language","relation":[{"term":"java8"},{"term":"struct"},{"term":"j2ee"},{"term":"Progamming Language"}]}
I want to search using filtered query as below
GET indexcloud/_search
{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"term" : "Java Language"
}
}
}
}
}
But this is not working. How can i achieve this ?. Note : i dont want like below
GET indexcloud/_search
{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"term" : "javalanguage"
}
}
}
}
}
Because i want to search, the way i index.
My mapping is as follows:
{
"mappings": {
"person": {
"properties": {
"lastUpdated": {
"type": "long"
},
"isDeleted": {
"type": "boolean"
},
"person": {
"properties": {
"car": {
"type": "nested",
"properties": {
"model": {
"type": "string"
},
"make": {
"type": "string"
}
}
},
"last_name": {
"type": "string"
},
"first_name": {
"type": "string"
}
}
}
}
}
}
}
I have two documents:
{
"person": {
"first_name": "Bob",
"last_name": "Doe",
"car": [
{
"make": "Saturn",
"model": "Imprezza"
},
{
"make": "Honda",
"model": "Accord"
}
]
},
"isDeleted": false,
"lastUpdated": 1433257051959
}
and
{
"person": {
"first_name": "Zach",
"last_name": "Foobar",
"car": [
{
"make": "Saturn",
"model": "SL"
},
{
"make": "Subaru",
"model": "Imprezza"
}
]
},
"isDeleted": false,
"lastUpdated": 1433257051959
}
I wanted to query the car.make field and so, I wrote the following query:
{
"query": {
"nested": {
"path": "person.person.car",
"query": {
"match": {
"car.make": "Saturn"
}
},
"inner_hits": {}
}
}
}
However, I am not getting anything back results back in return. When I remove the person level object and try to search, then it works. Any idea how to go about doing multi-level nested queries?
EDIT: On the other hand, when I structure my data like this and query then it works.
{
"mappings": {
"person": {
"properties": {
"car": {
"type": "nested",
"properties": {
"model": {
"type": "string"
},
"make": {
"type": "string"
}
}
},
"last_name": {
"type": "string"
},
"first_name": {
"type": "string"
}
}
}
}
}
{
"first_name": "Zach",
"last_name": "Foobar",
"car": [
{
"make": "Saturn",
"model": "SL"
},
{
"make": "Subaru",
"model": "Imprezza"
}
]
}
{
"first_name": "Bob",
"last_name": "Doe",
"car": [
{
"make": "Saturn",
"model": "Imprezza"
},
{
"make": "Honda",
"model": "Accord"
}
]
}
{
"query": {
"nested": {
"path": "person.car",
"query": {
"match": {
"car.make": "Honda"
}
},
"inner_hits": {}
}
}
}
This way the query works. I feel like this has something to do with multi-level nesting. Multi-level nesting is not working.
The nested path attribute needs to be "person.car".
Add "type": "nested", above the (2nd level) person properties line if you wish person to be a nested field type, which is required for Nested Query searches. The default field type is object field.
The naming you are using is confusing, try to rename your mapping not to use person twice.
{
"query": {
"nested": {
"path": "person.car",
"query": {
"match": {
"make": "Saturn"
}
},
"inner_hits": {}
}
}
}