Related
I am using ES 7.2.1 to store large amount of location based data and querying for near-by locations.
For location coordinates, I am using GeoPoint fields from my java codebase.
ES: 7.2.1
Spring Data Elasticsearch: 4.0.0.DATAES-690-SNAPSHOT
MVN org.elasticsearch: 7.2.1
Template:
curl -X PUT "localhost:9200/_template/store_locator_template?pretty"
{
"order": 1,
"index_patterns": [
"store_locator_*"
],
"settings": {
},
"mappings": {
"properties": {
"esId": {
"type": "keyword"
},
"geoPoint": {
"type": "geo_point"
},
"storeName": {
"type": "keyword"
}
}
}
}
When trying to insert data via bulkIndex(), I am getting this error:
org.springframework.data.elasticsearch.ElasticsearchException:
Bulk indexing has failures. Use ElasticsearchException.getFailedDocuments()
for detailed messages [{QObQeXEBqxAg6uMFyeNZ=ElasticsearchException
[Elasticsearch exception
[type=illegal_argument_exception, reason=
mapper [geoPoint] of different type,
current_type [geo_point], merged_type [ObjectMapper]]]}]
Entity:
#Getter
#Setter
#ToString
#EqualsAndHashCode(of = "esId", callSuper = false)
#NoArgsConstructor
#Document(indexName = "store_locator_index", replicas = 0, createIndex = false)
public class EsEntity {
#Id
#Field(type = FieldType.Text)
private String esId;
#GeoPointField
private GeoPoint geoPoint;
#Field(type = FieldType.Text)
private String storeName;
}
UPDATE:
If I use the below code, it works fine. it puts the mapping as required and spring data es does no complain!
//clazz -> entity class with #Document annotation
boolean indexCreated = false;
if (!elasticsearchOperations.indexExists(clazz)) {
indexCreated = elasticsearchOperations.createIndex(clazz);
}
if (indexCreated) {
elasticsearchOperations.refresh(clazz);
elasticsearchOperations.putMapping(clazz); --> Does the MAGIC
}
... And the mapping generated from the above code is:
{
"esentity":{ ---> Why is this here??
"properties":{
"esId":{
"type":"keyword",
"index":true
},
"geoPoint":{
"type":"geo_point"
}
}
}
}
It is adding a type, by the name of my entity class, to the mapping!
====================
Also.....
Everything seems to be working for:
ES: 6.4.3
Spring Data Elasticsearch: 3.1.X
I am able to (via template) insert document with GeoPoint field.
The index is generated automatically when doc is inserted via code.
The same set of code works fine with no error!!!!
Here's my template:
curl -X PUT "localhost:9200/_template/store_locator_template?pretty"
{
"order": 1,
"index_patterns": [
"store_locator_*"
],
"settings": {
},
"mappings": {
"store_locator_index": {
"properties": {
"esId": {
"type": "keyword"
},
"geoPoint": {
"type": "geo_point"
},
"storeName": {
"type": "keyword"
}
}
}
}
}
Here's the mapping:
{
"mapping": {
"properties": {
"esId": {
"type": "keyword"
},
"geoPoint": {
"type": "geo_point"
}
}
}
}
There are some things that don't match in the code you show:
In the first template you show, you define the storeName to be of type keyword, but on the entity you have it as type text.
A field annotated with #Id is always a type keyword, the #Field annotation defining it as type text is ignored.
I used the following versions: ES 7.3.0 (don't have 7.2.1 on my machine), Spring Data 4.0 current master, client libs set to 7.3.0.
When I don't have the template defined, but create the index with the code you showed:
boolean indexCreated = false;
Class<EsEntity> clazz = EsEntity.class;
if (!elasticsearchOperations.indexExists(clazz)) {
indexCreated = elasticsearchOperations.createIndex(clazz);
}
if (indexCreated) {
elasticsearchOperations.refresh(clazz);
elasticsearchOperations.putMapping(clazz);
}
I get the following index:
{
"store_locator_index": {
"aliases": {},
"mappings": {
"properties": {
"esId": {
"type": "keyword"
},
"geoPoint": {
"type": "geo_point"
},
"storeName": {
"type": "text"
}
}
},
"settings": {
"index": {
"refresh_interval": "1s",
"number_of_shards": "1",
"provided_name": "store_locator_index",
"creation_date": "1587073075464",
"store": {
"type": "fs"
},
"number_of_replicas": "0",
"uuid": "72aZqWDtS7KLDMwdkgVtag",
"version": {
"created": "7030099"
}
}
}
}
}
The mapping looks like it should, there is no type info in the mapping (this was written by the Spring Data Elasticsearch 3.2 version when using ES 6, but i not used anymore)
When I add the template you showed and then do a bulk insert with the following code:
EsEntity es1 = new EsEntity();
es1.setEsId("1");
es1.setGeoPoint(new GeoPoint(12, 34));
es1.setStoreName("s1");
IndexQuery query1 = new IndexQueryBuilder().withId("1").withObject(es1).build();
EsEntity es2 = new EsEntity();
es2.setEsId("2");
es2.setGeoPoint(new GeoPoint(56, 78));
es2.setStoreName("s2");
IndexQuery query2 = new IndexQueryBuilder().withId("2").withObject(es2).build();
elasticsearchOperations.bulkIndex(Arrays.asList(query1, query2), IndexCoordinates.of("store_locator_index"));
then the following index is created (note that store_name is type keywordnow, coming from the template):
{
"store_locator_index": {
"aliases": {},
"mappings": {
"properties": {
"_class": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"esId": {
"type": "keyword"
},
"geoPoint": {
"type": "geo_point"
},
"storeName": {
"type": "keyword"
}
}
},
"settings": {
"index": {
"creation_date": "1587073540386",
"number_of_shards": "1",
"number_of_replicas": "1",
"uuid": "LqzXMC5uRmKmImIzblFBOQ",
"version": {
"created": "7030099"
},
"provided_name": "store_locator_index"
}
}
}
}
and the two documents are inserted as they should:
{
"took": 22,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1.0,
"hits": [
{
"_index": "store_locator_index",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"_class": "com.sothawo.springdataelastictest.EsEntity",
"esId": "1",
"geoPoint": {
"lat": 12.0,
"lon": 34.0
},
"storeName": "s1"
}
},
{
"_index": "store_locator_index",
"_type": "_doc",
"_id": "2",
"_score": 1.0,
"_source": {
"_class": "com.sothawo.springdataelastictest.EsEntity",
"esId": "2",
"geoPoint": {
"lat": 56.0,
"lon": 78.0
},
"storeName": "s2"
}
}
]
}
}
So I cannot find an error in the code, but you should check the templates and existing indices if there are conflicting entries.
I'm having an issue in which the count keeps incrementing by 1 whenever I do an update to a document using save and the count's supposed to remain the same. If I create a document with save then the count is incremented by 2. Am I setting something wrong?
This is my settings for the ElasticSearch index:
{
"aliases": {
"case": {}
},
"mappings": {
"_doc": {
"dynamic": false,
"properties": {
"created": {
"index": true,
"type": "date"
},
"modified": {
"index": true,
"type": "date"
},
"type": {
"index": true,
"type": "keyword", "normalizer": "lower_case_normalizer"
},
"states": {
"type": "nested",
"properties": {
"from": {
"index": true,
"type": "keyword", "normalizer": "lower_case_normalizer"
},
"to": {
"index": true,
"type": "keyword", "normalizer": "lower_case_normalizer"
},
"event": {
"index": true,
"type": "keyword", "normalizer": "lower_case_normalizer"
}
}
}
}
}
},
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 2
},
"analysis": {
"normalizer": {
"lower_case_normalizer": {
"type": "custom",
"char_filter": [],
"filter": ["lowercase", "asciifolding"]
}
}
}
}
}
This is how I insert a document into ES:
public Case createCase(final Case case) throws UnableToGenerateUUIDException {
final UUID caseId = uuidService.getNowTimeUUID();
final Instant now = Instant.now();
case.setCreated(now);
case.setModified(now);
case.setId(caseId.toString());
return caseRepository.save(case);
}
This is the bug from the Chrome extension I used, not from Spring Data. I apologize for the mistake. I have verified that the count() from Spring Data reflects the correct number.
I have following Mongodb document. Would like to fetch document where participant = 'xxx' and message.lastmodifiedDate > dt and (message.touserid = 'xxx' or message.fromuserid = 'xxx').
{
"_id": {
"$oid": "575161ea02758f067057b8a8"
},
"_class": "com.idearealty.product.shopchat.persistence.model.Discussion",
"participants": "56d314a8e4b04d7f98cfd0c6,56d5d48ee4b0cc330f512a47,56d9d599e4b0cc330f512aaa,57130299e4b08c554c1092c7,56841002eceefce22f455c7f",
"messages": [
{
"_id": {
"$oid": "575161e802758f067057b8a4"
},
"formuserId": "56841002eceefce22f455c7f",
"fromuser": "9674642375",
"touserId": "56d314a8e4b04d7f98cfd0c6",
"touser": "debopam_r",
"message": "User Creating Discussion",
"isMute": false,
"index": 1,
"createDate": {
"$date": "2016-06-03T10:54:32.428Z"
},
"lastModifiedDate": {
"$date": "2016-06-03T10:54:32.428Z"
},
"createdBy": "9674642375",
"lastModifiedBy": "9674642375"
},
{
"_id": {
"$oid": "575161e902758f067057b8a5"
},
"formuserId": "56841002eceefce22f455c7f",
"fromuser": "9674642375",
"touserId": "56d5d48ee4b0cc330f512a47",
"touser": "Raushan",
"message": "User Creating Discussion",
"isMute": false,
"index": 2,
"createDate": {
"$date": "2016-06-03T10:54:33.006Z"
},
"lastModifiedDate": {
"$date": "2016-06-03T10:54:33.006Z"
},
"createdBy": "9674642375",
"lastModifiedBy": "9674642375"
},
{
"_id": {
"$oid": "575161e902758f067057b8a6"
},
"formuserId": "56841002eceefce22f455c7f",
"fromuser": "9674642375",
"touserId": "56d9d599e4b0cc330f512aaa",
"touser": "anirbanshop1",
"message": "User Creating Discussion",
"isMute": false,
"index": 3,
"createDate": {
"$date": "2016-06-03T10:54:33.572Z"
},
"lastModifiedDate": {
"$date": "2016-06-03T10:54:33.572Z"
},
"createdBy": "9674642375",
"lastModifiedBy": "9674642375"
},
{
"_id": {
"$oid": "575161ea02758f067057b8a7"
},
"formuserId": "56841002eceefce22f455c7f",
"fromuser": "9674642375",
"touserId": "57130299e4b08c554c1092c7",
"touser": "dummyshop",
"message": "User Creating Discussion",
"isMute": false,
"index": 4,
"createDate": {
"$date": "2016-06-03T10:54:34.208Z"
},
"lastModifiedDate": {
"$date": "2016-06-03T10:54:34.208Z"
},
"createdBy": "9674642375",
"lastModifiedBy": "9674642375"
}
],
"productId": "56841004eceefce22f455c9b",
"product": {
"_id": {
"$oid": "56841004eceefce22f455c9b"
},
"category": "Services",
"productName": "Driving School",
"imageurl": "service_icon.png",
"createDate": {
"$date": "2015-12-30T17:10:28.644Z"
},
"lastModifiedDate": {
"$date": "2015-12-30T17:10:28.644Z"
},
"createdBy": "UnAuntenticatedUser",
"lastModifiedBy": "UnAuntenticatedUser"
},
"userToRetailer": {
"57130299e4b08c554c1092c7": {
"_id": {
"$oid": "5713029ae4b08c554c1092c8"
},
"shopName": "dummyshop",
"user": {
"$ref": "IdeaRealtyUser",
"$id": {
"$oid": "57130299e4b08c554c1092c7"
}
}
},
"56d314a8e4b04d7f98cfd0c6": {
"_id": {
"$oid": "56d314a9e4b04d7f98cfd0c7"
},
"shopName": "Test Shop",
"user": {
"$ref": "IdeaRealtyUser",
"$id": {
"$oid": "56d314a8e4b04d7f98cfd0c6"
}
}
},
"56d9d599e4b0cc330f512aaa": {
"_id": {
"$oid": "56d9d59ae4b0cc330f512aab"
},
"shopName": "anirbanshop1",
"user": {
"$ref": "IdeaRealtyUser",
"$id": {
"$oid": "56d9d599e4b0cc330f512aaa"
}
}
},
"56d5d48ee4b0cc330f512a47": {
"_id": {
"$oid": "56d5d48fe4b0cc330f512a48"
},
"shopName": "Kolkata Shop1",
"user": {
"$ref": "IdeaRealtyUser",
"$id": {
"$oid": "56d5d48ee4b0cc330f512a47"
}
}
}
},
"messageCount": 4,
"createDate": {
"$date": "2016-06-03T10:54:34.215Z"
},
"lastModifiedDate": {
"$date": "2016-06-03T10:54:34.215Z"
},
"createdBy": "9674642375",
"lastModifiedBy": "9674642375"
}
I am using following java code to fetch the data, which translates to query
{ "aggregate" : "discussion" , "pipeline" : [ { "$match" : { "participants" : { "$regex" : "56d314a8e4b04d7f98cfd0c6"}}} , { "$unwind" : "$messages"} , { "$match" : { "$and" : [ { "messages.lastModifiedDate" : { "$gte" : { "$date" : "2016-02-28T16:06:11.960Z"}}} , { "$or" : [ { "messages.touserId" : "56d314a8e4b04d7f98cfd0c6"} , { "messages.formuserId" : "56d314a8e4b04d7f98cfd0c6"}]}]}} , { "$sort" : { "messages.lastModifiedDate" : -1}} , { "$skip" : 0} , { "$limit" : 10} , { "$group" : { "_id" : { "_id" : "$_id" , "productId" : "$productId"} , "data" : { "$push" : "$messages"}}} , { "$project" : { "productId" : "$_id.productId" , "data" : 1}}]}
But this query doesn't fecth any record. If I change the Date to ISODate it fetches expected result.
db.discussion.aggregate( [{ "$match" : { "participants" : { "$regex" : "56841002eceefce22f455c7f"}}} , { "$unwind" : "$messages"}, { "$match" : { "$and" : [ { "messages.lastModifiedDate" : { "$gte" : ISODate("2016-02-28T16:38:48.632Z")}} , { "$or" : [ { "messages.touserId" : "56841002eceefce22f455c7f"} , { "messages.formuserId" : "56841002eceefce22f455c7f"}]}]}}]);
Could you please let me know what changes is required so that it fetches using ISODate or while inserting the document it uses normal java.util.Date object? I tried with commented code as well but it didn't work.
public List<Discussion> findInbox(String userid,Date lastloginDate,int skip, int limit){
SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
Aggregation aggr = newAggregation(
match(Criteria.where("participants").regex(Pattern.compile(userid))),
unwind("messages"),
match(new Criteria().andOperator(Criteria.where("messages.lastModifiedDate").gte(lastloginDate),new Criteria().orOperator(Criteria.where("messages.touserId").is(userid),Criteria.where("messages.formuserId").is(userid)))),
//match(new Criteria().andOperator(Criteria.where("messages.lastModifiedDate").is(new BasicDBObject("$gte","ISODate("+format.format(lastloginDate)+")")),new Criteria().orOperator(Criteria.where("messages.touserId").is(userid),Criteria.where("messages.formuserId").is(userid)))),
sort(Direction.DESC, "messages.lastModifiedDate"),
skip(skip),
limit(limit),
group("_id","productId").push("messages").as("data"),
project("_id","productId","data")
//project("product","participants","messages")
);
AggregationResults<Discussion> results = mongoTemplate.aggregate(aggr, "discussion", Discussion.class);
List<Discussion> discussions = results.getMappedResults();
return discussions;
}
I was using a Date field from another document. Changing it to Calendar solved the problem.
// Calling method
Calendar cal = Calendar.getInstance();
cal.setTime(loginInfo.getCreateDate());
return customDiscussionRepository.findInbox(activeUser.getId(), cal.getTime(), pageNumber > 0?pageSize*(pageNumber-1):0, pageSize);
// Repository (lastloginDate is the method argument passed from calling method).
Criteria.where("messages.lastModifiedDate").gte(lastloginDate)
Query:
GET service/_search
{
"query":{
"match": {"id":1}
}
}
this query will end up with following result from the elastic search server . I would like to filter the search based on child property in specific based on subCategories. I have tried the following query but in vain what is wrong in it ? subCategories nod is an array list i mean jakson converted List is there anything wrong in the json conversion ?
GET service/_search
{
"query":
{
"match": {
"subCategories.name": "subname1"
}
}
}
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "service",
"_type": "service",
"_id": "1",
"_score": 1,
"_source": {
"id": 1,
"title": "title",
"searchTerms": null,
"description": "description",
"picUrl": "/imgurl",
"price": 65000,
"discount": 10,
"topservice": true,
"place": "100,200",
"status": null,
"subCategories": [
{
"id": 1,
"name": "subname1",
"subCategoryGroup": {
"id": 1,
"name": "Engineering",
"category": {
"id": 1,
"name": "Education"
}
}
},
{
"id": 2,
"name": "subname2",
"subCategoryGroup": {
"id": 1,
"name": "Engineering",
"category": {
"id": 1,
"name": "Education"
}
}
},
{
"id": 3,
"name": "subname3",
"subCategoryGroup": {
"id": 1,
"name": "Engineering",
"category": {
"id": 1,
"name": "Education"
}
}
},
],
"deleted": false
}
}
]
}
}
sub category mapping ; nothing fancy in it just a manytoMany mapping as follows
#Field(type= FieldType.Nested)
#ManyToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER)
#JoinTable(name = "service_subcategory", joinColumns = #JoinColumn(name = "service_id") , inverseJoinColumns = #JoinColumn(name = "subcategory_id") )
private List<SubCategory> subCategories;
I'm pretty sure the field "subCategories" is not a nested field. The behavior you want can be achieved by making "subCategories" a nested field. Read about nested types here and about how to query nested fields here.
Basically your mapping definition should look something like below:
{
"mappings": {
"<mapping_name>": {
"properties": {
... <-- Other fields like id, title, etc go here
"subCategories": {
"type": "nested", <-- This is important. This is missing in your current mapping definition
"properties": {
"id": {
"type":"integer"
},
"name": {
"type":"string"
},
... <-- Definition of subCategoryGroup goes here
}
}
}
}
}
}
And your query should look something like this.
{
"query": {
"nested": {
"path": "subCategories",
"query": {
"match": {
"subCategories.name": "subname1"
}
}
}
}
}
I have the below mapping structure for my Elasticsearch index.
{
"users": {
"mappings": {
"user-type": {
"properties": {
"lastModifiedBy": {
"type": "string"
},
"lastModifiedDate": {
"type": "date",
"format": "dateOptionalTime"
},
"details": {
"type": "nested",
"properties": {
"lastModifiedBy": {
"type": "string"
},
"lastModifiedDate": {
"type": "date",
"format": "dateOptionalTime"
},
"views": {
"type": "nested",
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"properties": {
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
},
"value": {
"type": "string"
}
}
}
}
}
}
}
}
}
}
}
}
Basically I want to retrieve ONLY the view object inside details based on index id & view id(details.views.id).
I have tried with the below java code.But seems to be not working.
SearchRequestBuilder srq = this.client.prepareSearch(this.indexName)
.setTypes(this.type)
.setQuery(QueryBuilders.termQuery("_id", sid))
.setPostFilter(FilterBuilders.nestedFilter("details.views",
FilterBuilders.termFilter("details.views.id", id)));
Below is the query structure for this java code.
{
"query": {
"term": {
"_id": "123"
}
},
"post_filter": {
"nested": {
"filter": {
"term": {
"details.views.id": "def"
}
},
"path": "details.views"
}
}
}
Since details is nested and view is nested inside details, you basically need two nested filters as well (one for each level) + the constraint on the _id field is best done with the ids query. The query DSL would look like this:
{
"query": {
"ids": {
"values": [
"123"
]
}
},
"post_filter": {
"nested": {
"filter": {
"nested": {
"path": "details.view",
"filter": {
"term": {
"details.views.id": "def"
}
}
}
},
"path": "details"
}
}
}
Translating this into Java code yields:
// 2nd-level nested filter
FilterBuilder detailsView = FilterBuilders.nestedFilter("details.views",
FilterBuilders.termFilter("details.views.id", id));
// 1st-level nested filter
FilterBuilder details = FilterBuilders.nestedFilter("details", detailsView);
// ids constraint
IdsQueryBuilder ids = QueryBuilders.idsQuery(this.type).addIds("123");
SearchRequestBuilder srq = this.client.prepareSearch(this.indexName)
.setTypes(this.type)
.setQuery(ids)
.setPostFilter(details);
PS: I second what #Paul said, i.e. always play around with the query DSL first and when you know you have zeroed in on the exact query you need, then you can translate it to the Java form.