I can't seem to get Nested Fields to return when I use the NativeSearchQueryBuilder.withFields(...) method.
Here is my parent Object:
#Document(indexName = "inventory")
public class Inventory
{
#Id
private String id;
#Field(type=FieldType.String)
private String name;
#Field(type=FieldType.Nested, index=FieldIndex.not_analyzed, store=true)
private List<Model> models;
}
And here is the nested Object:
public class Model
{
#Field(type=FieldType.String, index=FieldIndex.not_analyzed, store=true)
private String model;
#Field(type=FieldType.String, index=FieldIndex.not_analyzed, store=true)
private Set<String> series;
}
And the Query
NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
nativeSearchQueryBuilder.withQuery(QueryBuilders.matchAllQuery());
nativeSearchQueryBuilder.withFields("models.series");
NativeSearchQuery nativeSearchQuery = nativeSearchQueryBuilder.build();
FacetedPage<Inventory> results = inventoryRepository.search(nativeSearchQuery);
Resulting TotalElements = 529
But each Object in the Content Looks like this (in JSON format):
{
"id":"d5f82880-15bc-45ed-8abb-ff97d0e45da9",
"name": null,
"models": null
}
If I remove the withFields(...) setting, I get back:
{
"id":"d5f82880-15bc-45ed-8abb-ff97d0e45da9",
"name": "Cool Beans",
"models": [
{
"model" : "foo",
"series" : ["bar"]
}
]
}
I've tried models, models.model, models.series, model, series. I can't get withFields working with NestedFields.
Any thoughts?
My understanding of elastic search fields was incorrect. rahulroc tipped me off.
withFields is not the same as source filtering.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-fields.html
So i'm effectively telling Spring Data ES to do this:
curl localhost:9200/inventory/_search?pretty=true -d '
{
"fields" : ["models.series"],
"query" : {
"match" : {"name" : "cool"}
}
}'
When this is what I want
curl localhost:9200/inventory/_search?pretty=true -d '
{
"_source" : ["models.series"],
"query" : {
"match" : {"name" : "cool"}
}
}'
The withFields approach worked for what I was doing up till I started adding NestedFields. And the current implementation of Spring Data ES that I'm using does not support source filtering.
Source Filtering was just recently added to Spring Data ES 2.0.0.RC1
Related
I'm working with MongoDB 3.2.2 and Spring Data 2.1.8. I have the following document model with dynamic data field:
#Data
#Accessors(chain = true)
#Document(collection = "someCollection")
public class SomeEntity implements Serializable {
#Id
String id;
//some fields
Map<String, Object> data;
}
My goal is grouping my documents by specific key from the data field. For example, i have the following db content:
{
"_id": "5e5f8a89b70e4123a8285aa3",
"data": {
"someField": "someValue",
}
},
{
"_id": "5e5f72fcb70e4123a8285aa2",
"data": {
"someField": "someValue",
}
},
{
"_id": "5e5d22939ce87e2fccd80973",
"data": {
"someField": "otherValue",
}
}
I'd like to build the grouping aggregation using Spring Data like the following query for MongoDB:
$group: {
{
_id: "$data.someField",
count: {
$sum: 1
}
}
}
And I'd like to receive the following result:
{
_id: "someValue",
count: 2
},
{
_id: "otherValue",
count: 1
}
For this goal i'm using the next grouping with org.springframework.data.mongodb.core.aggregation.Aggregation:
Aggregation.group("$data.someField").count().as("count")
But i've got an error during execution of aggregation:
org.springframework.data.mapping.PropertyReferenceException: No property someField found for type Object! Traversed path: SomeEntity.data.
What was wrong? Could someone help me, please?
P.S.: i've also tried to use $replaceRoot for data field, so i could group documents by someField, but it's newer db version (New in version 3.4)
Could it be that you only have a little typo: No property someFiled?
Try the following:
group("$data.someField").count().as("count")
I am creating a new endpoint in springboot that will return simple stats on users generated from an aggregate query in a mongo database. However I get a PropertyReferenceException. I have read multiple stackoverflow questions about it, but didn't find one that solved this problem.
We have a mongo data scheme like this:
{
"_id" : ObjectId("5d795993288c3831c8dffe60"),
"user" : "000001",
"name" : "test",
"attributes" : {
"brand" : "Chrome",
"language" : "English" }
}
The database is filled with multiple users and we want using Springboot aggregate the stats of users per brand. There could be any number of attributes in the attributes object.
Here is the aggregation we are doing
Aggregation agg = newAggregation(
group("attributes.brand").count().as("number"),
project("number").and("type").previousOperation()
);
AggregationResults<Stats> groupResults
= mongoTemplate.aggregate(agg, Profile.class, Stats.class);
return groupResults.getMappedResults();
Which produces this mongo query which works:
> db.collection.aggregate([
{ "$group" : { "_id" : "$attributes.brand" , "number" : { "$sum" : 1}}} ,
{ "$project" : { "number" : 1 , "_id" : 0 , "type" : "$_id"}} ])
{ "number" : 4, "type" : "Chrome" }
{ "number" : 2, "type" : "Firefox" }
However when running a simple integration test we get this error:
org.springframework.data.mapping.PropertyReferenceException: No property brand found for type String! Traversed path: Profile.attributes.
From what I understand, it seems that since attributes is a Map<String, String> there might be a schematic problem. And in the mean time I can't modify the Profile object.
Is there something I am missing in the aggregation, or anything I could change in my Stats object?
For reference, here are the data models we're using, to work with JSON and jackson.
The Stats data model:
#Document
public class Stats {
#JsonProperty
private String type;
#JsonProperty
private int number;
public Stats() {}
/* ... */
}
The Profile data model:
#Document
public class Profiles {
#NotNull
#JsonProperty
private String user;
#NotNull
#JsonProperty
private String name;
#JsonProperty
private Map<String, String> attributes = new HashMap<>();
public Stats() {}
/* ... */
}
I found a solution, which was a combination of two problems:
The PropertyReferenceException was indeed caused because attributes is a Map<String, String> which means there is no schemes for Mongo.
The error message No property brand found for type String! Traversed path: Profile.attributes. means that the Map object doesn't have a brand property in it.
In order to fix that without touching my orginal Profile class, I had to create a new custom class which would map the attributes to an attributes object having the properties I want to aggreate on like:
public class StatsAttributes {
#JsonProperty
private String brand;
#JsonProperty
private String language;
public StatsAttributes() {}
/* ... */
}
Then I created a custom StatsProfile which would leverage my StatsAttributes and would be similar to the the original Profile object without modifying it.
#Document
public class StatsProfile {
#JsonProperty
private String user;
#JsonProperty
private StatsAttributes attributes;
public StatsProfile() {}
/* ... */
}
With that I made disapear my problem with the PropertyReferenceException using my new class StatsAggregation in the aggregation:
AggregationResults<Stats> groupResults
= mongoTemplate.aggregate(agg, StatsProfile.class, Stats.class);
However I would not get any results. It seems the query would not find any document in the database. That's where I realied that production mongo objects had the field "_class: com.company.dao.model.Profile" which was tied to the Profile object.
After some research, for the new StatsProfile to work it would need to be a #TypeAlias("Profile"). After looking around, I found that I also needed to precise a collection name which would lead to:
#Document(collection = "profile")
#TypeAlias("Profile")
public class StatsProfile {
/* ... */
}
And with all that, finally it worked!
I suppose that's not the prettiest solution, I wish I would not need to create a new Profile object and just consider the attributes as a StatsAttributes.class somehow in the mongoTemplate query. If anyone knows how to, please share 🙏
as title of the questions says; is it possible in DynamoDB to query for a particular attribute value within a nested JSON document? I've been looking for the answer on official docs, SO, forums, etc., however I'm still unable to provide a valid query.
Example of the entity objects in Java:
#DynamoDBTable(tableName = "Car")
public class Car {
#DynamoDBHashKey
String id;
String color;
String type;
Engine engine;
}
#DynamoDBDocument
public class Engine {
String manufacturer;
int hp;
String model;
}
Records in database:
{
"id" : "43-asdasda4214ads13-13",
"color" : "blue",
"type" : "sedan",
"engine" : {
"manufacturer" : "ford",
"hp" : 450,
"model" : "av15"
}
}
{
"id" : "24ads123-adad123-das1",
"color" : "red",
"type" : "cabrio",
"engine" : {
"manufacturer" : "peugeot",
"hp" : 130,
"model" : "ig31-2"
}
}
Can I query, let's say all cars which have engine provided by ford? If so, how could I do that? I've already tried keyConditionExpression in following format engine.manufacturer = ford, but no success. What I'd like to avoid is using FilterExpression.
Thanks in advance!
I have been trying to use a geospatial query to fetch data into a pojo with no success.
Here is an example data in my monogdb collection
{
"_id" : ObjectId("597b8c9a21871eeabd5a1cf5"),
"amb_id" : 10,
"amb_number" : "KL25 8945",
"driver_name" : "Amal Shaji",
"driver_licence_id" : "12/4562/2017",
"ownership" : "Amrita Institute of Medical Science",
"ownership_address" : "Peeliyadu Road, Ponekkara, Edappally, Ernakulam",
"location" : {
"type" : "Point",
"coordinates" : [
76.293485,
10.032871
]
}
}
The below mongo query works perfectly fine in the mongoshell
db.trial.find(
{ location :
{ $near :{
$geometry :{
type : "Point" ,
coordinates : [ 76.2 , 9.9 ] } ,
$maxDistance : 20000 }
}
}
)
.pretty();
Here is the pojo that I have been trying to fetch the data into
#Document(collection = "trial")
public class Ambulance {
#Id
String id;
#Field("amb_id")
String ambulanceId;
#Field("amb_number")
String licensePlateNumber;
#Field("driver_name")
String driverName;
#Field("driver_licence_id")
String driverLicenseNumber;
#Field("ownership")
String ownerShip;
#Field("ownership_address")
String ownerShipAddress;
#GeoSpatialIndexed(name="Location")
Double[] location;
//setters and getters
}
Here is the repository I have been using
#ComponentScan
#Repository
public interface AmbulanceRepo extends MongoRepository<Ambulance, String> {
GeoResults<Ambulance> findByLocationNear(Point p, Distance d);
}
and the controller
#RestController
#CrossOrigin(origins = "http://localhost:4200")
#RequestMapping("/")
public class NearbyAmbulanceController {
private AmbulanceRepo ambulanceRepo;
#Autowired
public NearbyAmbulanceController(AmbulanceRepo repo){
this.ambulanceRepo = repo;
}
#RequestMapping(value="/nearbyAmbulance",method = RequestMethod.POST)
#ResponseBody
public GeoResults<Ambulance> getAmbulanceDetails(
#RequestBody LocationRequest locationRequest){
System.out.println("lati "+locationRequest.getLatitude()+ " long "+locationRequest.getLongitude()+" d "+locationRequest.getDistance());
// List<Ambulance> ambulanceList=this.ambulanceRepo.findByLocationNear(new Point(Double.valueOf(locationRequest.getLongitude()),Double.valueOf(locationRequest.getLatitude())),new Distance(locationRequest.getDistance(), Metrics.KILOMETERS));
Point point = new Point(locationRequest.getLatitude(), locationRequest.getLongitude());
Distance distance = new Distance(locationRequest.getDistance(), Metrics.KILOMETERS);
GeoResults<Ambulance> ambulanceList=this.ambulanceRepo.findByLocationNear(point,distance);
System.out.println(ambulanceList);
return ambulanceList;
}
}
Every time I try it I get no results. I am sure that I have data in the given point and nearby locations and I was even able to use the mongoshell to fetch those. I have feeling that the problem is with the way that I have annotated the location field in the entity. Is there anything that I am missing? Any help is appreiciated.
Spring MongoDb Geo spatial query: It is tested well.
public List<MachineDetails> searchbylocation(String lat, String longt, String maxDistance) {
BasicQuery query1 = new BasicQuery("{geoLocation:{ $near: { $geometry: { type: 'Point',coordinates: ["+ lat+","+ longt+" ] }, $minDistance: 10, $maxDistance: "+maxDistance+"}}}");
List<MachineDetails> venues1 = mtemplate.find(query1, MachineDetails.class);
return venues1;
}
Seems like there were more than one database in mongodb that had collections by the name 'trial'. I tried removing all those and now it is working fine. Spring data wasn't able to figure out which collection to be queried on as there were multiple collection with the same name in different databases. Hope this helps someone, cause I have just wasted a couple of days on this.
I am working on a web application using Spring + Jersey + Mongo. I am using elemMatch from spring mongo for some specific result.
Here is my class that will be mapped with the output of the mongo query.
public class ABC {
private String id;
private String dId;
private String lId;
private String hId;
private String pId;
private List<GeneralData> generalRecords;
private List<String> fhistory;
private List<String> mhistory;
private List<String> specialNotes;
}
public class GeneralData {
private Object data;
private HistoryFilter dataType;
}
Now, this is how my data looks like in mongo :
{
"_id" : ObjectId("55ccda7fe4b03cc159c2ce17"),
"_class" : "com.dpdocter.collections.HistoryCollection",
"dId" : "5525ef96e4b077dfc168369b",
"lId" : "5525ef96e4b077dfc16836a1",
"hId : "5525ef96e4b077dfc16836a0",
"pId" : "55ccd9f1e4b03cc159c2ce0b",
"generalRecords" : [
{
"data" : "55ccdde7e4b03cc159c2ce18",
"dataType" : "REPORTS"
},
{
"data" : "55ccda63e4b03cc159c2ce14",
"dataType" : "CLINICAL_NOTES"
}
]
}
I want record based on dId, lId, hId, pId along with elemMatch on generalRecords for a specific dataType (say REPORTS).
And, below is my criteria code to do that.
Criteria matchCriteria = Criteria.where("dId").is(dId).and("lId").is(lId).and("hId").is(hId).and("pId").is(pId).and("generalRecords.dataType").is("REPORTS");
Criteria elementMatchCriteria = Criteria.where("generalRecords").elemMatch(Criteria.where("dataType").is("REPORTS"))
BasicQuery query = new BasicQuery(matchCriteria.getCriteriaObject(), elementMatchCriteria.getCriteriaObject());
ABC abc = mongoOperations.findOne(query, ABC.class);
Result of the above
ABC [id=55ccda7fe4b03cc159c2ce17, dId=null, lId=null, hId=null, pId=null, generalRecords=[GeneralData [data=55ccdde7e4b03cc159c2ce18, dataType=REPORTS]], fhistory=null, mhistory=null, specialNotes=null]
generalRecords is just perfect and as desired, but I just want to include dId, lId, hId, pId, fHistory, mHistory, and specialNotes also. I have already tried :
query.fields().include('fieldName')
also, some other weird query combos, but nothing helps.
Can anyone suggest something.