Develop Reference

How to aggregate in spring data mongo db a nested object and avoid a PropertyReferenceException?

I am creating a new endpoint in springboot that will return simple stats on users generated from an aggregate query in a mongo database. However I get a PropertyReferenceException. I have read multiple stackoverflow questions about it, but didn't find one that solved this problem.
We have a mongo data scheme like this:
{
"_id" : ObjectId("5d795993288c3831c8dffe60"),
"user" : "000001",
"name" : "test",
"attributes" : {
"brand" : "Chrome",
"language" : "English" }
}
The database is filled with multiple users and we want using Springboot aggregate the stats of users per brand. There could be any number of attributes in the attributes object.
Here is the aggregation we are doing
Aggregation agg = newAggregation(
group("attributes.brand").count().as("number"),
project("number").and("type").previousOperation()
);
AggregationResults<Stats> groupResults
= mongoTemplate.aggregate(agg, Profile.class, Stats.class);
return groupResults.getMappedResults();
Which produces this mongo query which works:
> db.collection.aggregate([
{ "$group" : { "_id" : "$attributes.brand" , "number" : { "$sum" : 1}}} ,
{ "$project" : { "number" : 1 , "_id" : 0 , "type" : "$_id"}} ])
{ "number" : 4, "type" : "Chrome" }
{ "number" : 2, "type" : "Firefox" }
However when running a simple integration test we get this error:
org.springframework.data.mapping.PropertyReferenceException: No property brand found for type String! Traversed path: Profile.attributes.
From what I understand, it seems that since attributes is a Map<String, String> there might be a schematic problem. And in the mean time I can't modify the Profile object.
Is there something I am missing in the aggregation, or anything I could change in my Stats object?
For reference, here are the data models we're using, to work with JSON and jackson.
The Stats data model:
#Document
public class Stats {
#JsonProperty
private String type;
#JsonProperty
private int number;
public Stats() {}
/* ... */
}
The Profile data model:
#Document
public class Profiles {
#NotNull
#JsonProperty
private String user;
#NotNull
#JsonProperty
private String name;
#JsonProperty
private Map<String, String> attributes = new HashMap<>();
public Stats() {}
/* ... */
}

I found a solution, which was a combination of two problems:
The PropertyReferenceException was indeed caused because attributes is a Map<String, String> which means there is no schemes for Mongo.
The error message No property brand found for type String! Traversed path: Profile.attributes. means that the Map object doesn't have a brand property in it.
In order to fix that without touching my orginal Profile class, I had to create a new custom class which would map the attributes to an attributes object having the properties I want to aggreate on like:
public class StatsAttributes {
#JsonProperty
private String brand;
#JsonProperty
private String language;
public StatsAttributes() {}
/* ... */
}
Then I created a custom StatsProfile which would leverage my StatsAttributes and would be similar to the the original Profile object without modifying it.
#Document
public class StatsProfile {
#JsonProperty
private String user;
#JsonProperty
private StatsAttributes attributes;
public StatsProfile() {}
/* ... */
}
With that I made disapear my problem with the PropertyReferenceException using my new class StatsAggregation in the aggregation:
AggregationResults<Stats> groupResults
= mongoTemplate.aggregate(agg, StatsProfile.class, Stats.class);
However I would not get any results. It seems the query would not find any document in the database. That's where I realied that production mongo objects had the field "_class: com.company.dao.model.Profile" which was tied to the Profile object.
After some research, for the new StatsProfile to work it would need to be a #TypeAlias("Profile"). After looking around, I found that I also needed to precise a collection name which would lead to:
#Document(collection = "profile")
#TypeAlias("Profile")
public class StatsProfile {
/* ... */
}
And with all that, finally it worked!
I suppose that's not the prettiest solution, I wish I would not need to create a new Profile object and just consider the attributes as a StatsAttributes.class somehow in the mongoTemplate query. If anyone knows how to, please share 🙏

spring data mongo geospatial query

I have been trying to use a geospatial query to fetch data into a pojo with no success.
Here is an example data in my monogdb collection
{
"_id" : ObjectId("597b8c9a21871eeabd5a1cf5"),
"amb_id" : 10,
"amb_number" : "KL25 8945",
"driver_name" : "Amal Shaji",
"driver_licence_id" : "12/4562/2017",
"ownership" : "Amrita Institute of Medical Science",
"ownership_address" : "Peeliyadu Road, Ponekkara, Edappally, Ernakulam",
"location" : {
"type" : "Point",
"coordinates" : [
76.293485,
10.032871
]
}
}
The below mongo query works perfectly fine in the mongoshell
db.trial.find(
{ location :
{ $near :{
$geometry :{
type : "Point" ,
coordinates : [ 76.2 , 9.9 ] } ,
$maxDistance : 20000 }
}
}
)
.pretty();
Here is the pojo that I have been trying to fetch the data into
#Document(collection = "trial")
public class Ambulance {
#Id
String id;
#Field("amb_id")
String ambulanceId;
#Field("amb_number")
String licensePlateNumber;
#Field("driver_name")
String driverName;
#Field("driver_licence_id")
String driverLicenseNumber;
#Field("ownership")
String ownerShip;
#Field("ownership_address")
String ownerShipAddress;
#GeoSpatialIndexed(name="Location")
Double[] location;
//setters and getters
}
Here is the repository I have been using
#ComponentScan
#Repository
public interface AmbulanceRepo extends MongoRepository<Ambulance, String> {
GeoResults<Ambulance> findByLocationNear(Point p, Distance d);
}
and the controller
#RestController
#CrossOrigin(origins = "http://localhost:4200")
#RequestMapping("/")
public class NearbyAmbulanceController {
private AmbulanceRepo ambulanceRepo;
#Autowired
public NearbyAmbulanceController(AmbulanceRepo repo){
this.ambulanceRepo = repo;
}
#RequestMapping(value="/nearbyAmbulance",method = RequestMethod.POST)
#ResponseBody
public GeoResults<Ambulance> getAmbulanceDetails(
#RequestBody LocationRequest locationRequest){
System.out.println("lati "+locationRequest.getLatitude()+ " long "+locationRequest.getLongitude()+" d "+locationRequest.getDistance());
// List<Ambulance> ambulanceList=this.ambulanceRepo.findByLocationNear(new Point(Double.valueOf(locationRequest.getLongitude()),Double.valueOf(locationRequest.getLatitude())),new Distance(locationRequest.getDistance(), Metrics.KILOMETERS));
Point point = new Point(locationRequest.getLatitude(), locationRequest.getLongitude());
Distance distance = new Distance(locationRequest.getDistance(), Metrics.KILOMETERS);
GeoResults<Ambulance> ambulanceList=this.ambulanceRepo.findByLocationNear(point,distance);
System.out.println(ambulanceList);
return ambulanceList;
}
}
Every time I try it I get no results. I am sure that I have data in the given point and nearby locations and I was even able to use the mongoshell to fetch those. I have feeling that the problem is with the way that I have annotated the location field in the entity. Is there anything that I am missing? Any help is appreiciated.

Spring MongoDb Geo spatial query: It is tested well.
public List<MachineDetails> searchbylocation(String lat, String longt, String maxDistance) {
BasicQuery query1 = new BasicQuery("{geoLocation:{ $near: { $geometry: { type: 'Point',coordinates: ["+ lat+","+ longt+" ] }, $minDistance: 10, $maxDistance: "+maxDistance+"}}}");
List<MachineDetails> venues1 = mtemplate.find(query1, MachineDetails.class);
return venues1;
}

Seems like there were more than one database in mongodb that had collections by the name 'trial'. I tried removing all those and now it is working fine. Spring data wasn't able to figure out which collection to be queried on as there were multiple collection with the same name in different databases. Hope this helps someone, cause I have just wasted a couple of days on this.

Spring Data Elastic Search Nested Field and NativeSearchQueryBuilder.withFields

I can't seem to get Nested Fields to return when I use the NativeSearchQueryBuilder.withFields(...) method.
Here is my parent Object:
#Document(indexName = "inventory")
public class Inventory
{
#Id
private String id;
#Field(type=FieldType.String)
private String name;
#Field(type=FieldType.Nested, index=FieldIndex.not_analyzed, store=true)
private List<Model> models;
}
And here is the nested Object:
public class Model
{
#Field(type=FieldType.String, index=FieldIndex.not_analyzed, store=true)
private String model;
#Field(type=FieldType.String, index=FieldIndex.not_analyzed, store=true)
private Set<String> series;
}
And the Query
NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
nativeSearchQueryBuilder.withQuery(QueryBuilders.matchAllQuery());
nativeSearchQueryBuilder.withFields("models.series");
NativeSearchQuery nativeSearchQuery = nativeSearchQueryBuilder.build();
FacetedPage<Inventory> results = inventoryRepository.search(nativeSearchQuery);
Resulting TotalElements = 529
But each Object in the Content Looks like this (in JSON format):
{
"id":"d5f82880-15bc-45ed-8abb-ff97d0e45da9",
"name": null,
"models": null
}
If I remove the withFields(...) setting, I get back:
{
"id":"d5f82880-15bc-45ed-8abb-ff97d0e45da9",
"name": "Cool Beans",
"models": [
{
"model" : "foo",
"series" : ["bar"]
}
]
}
I've tried models, models.model, models.series, model, series. I can't get withFields working with NestedFields.
Any thoughts?

My understanding of elastic search fields was incorrect. rahulroc tipped me off.
withFields is not the same as source filtering.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-fields.html
So i'm effectively telling Spring Data ES to do this:
curl localhost:9200/inventory/_search?pretty=true -d '
{
"fields" : ["models.series"],
"query" : {
"match" : {"name" : "cool"}
}
}'
When this is what I want
curl localhost:9200/inventory/_search?pretty=true -d '
{
"_source" : ["models.series"],
"query" : {
"match" : {"name" : "cool"}
}
}'
The withFields approach worked for what I was doing up till I started adding NestedFields. And the current implementation of Spring Data ES that I'm using does not support source filtering.
Source Filtering was just recently added to Spring Data ES 2.0.0.RC1

How to make an advanced search with Spring Data REST?

My task is to make an advanced search with Spring Data REST.
How can I implement it?
I managed to make a method to do a simple search, like this one:
public interface ExampleRepository extends CrudRepository<Example, UUID>{
#RestResource(path="searchByName", rel="searchByName")
Example findByExampleName(#Param("example") String exampleName);
}
This example works perfectly if I have to go simply to the url:
.../api/examples/search/searchByName?example=myExample
But what I have to do if there are more than one field to search?
For example, if my Example class has 5 fields, what implementation should I have to make an advanced search with all possibiles fileds?
Consider this one:
.../api/examples/search/searchByName?filed1=value1&field2=value2&field4=value4
and this one:
.../api/examples/search/searchByName?filed1=value1&field3=value3
What I have to do to implement this search in appropriate way?
Thanks.

Spring Data Rest has integrated QueryDSL with web support as well which you can use for your advanced search requirement. You need to change your repository to implement QueryDslPredicateExecutor and things will work out of the box.
Here is a sample from the blog article about the feature:
$ http :8080/api/stores?address.city=York
{
"_embedded": {
"stores": [
{
"_links": {
…
},
"address": {
"city": "New York",
"location": { "x": -73.938421, "y": 40.851 },
"street": "803 W 181st St",
"zip": "10033-4516"
},
"name": "Washington Hgts/181st St"
},
{
"_links": {
…
},
"address": {
"city": "New York",
"location": { "x": -73.939822, "y": 40.84135 },
"street": "4001 Broadway",
"zip": "10032-1508"
},
"name": "168th & Broadway"
},
…
]
},
"_links": {
…
},
"page": {
"number": 0,
"size": 20,
"totalElements": 209,
"totalPages": 11
}
}

I managed to implement this using Query by Example.
Let's say you have the following models:
#Entity
public class Company {
#Id
#GeneratedValue
Long id;
String name;
String address;
#ManyToOne
Department department;
}
#Entity
public class Department {
#Id
#GeneratedValue
Long id;
String name;
}
And the repository:
#RepositoryRestResource
public interface CompanyRepository extends JpaRepository<Company, Long> {
}
(Note that JpaRepository implements QueryByExampleExecutor).
Now you implement a custom controller:
#RepositoryRestController
#RequiredArgsConstructor
public class CompanyCustomController {
private final CompanyRepository repository;
#GetMapping("/companies/filter")
public ResponseEntity<?> filter(
Company company,
Pageable page,
PagedResourcesAssembler assembler,
PersistentEntityResourceAssembler entityAssembler
){
ExampleMatcher matcher = ExampleMatcher.matching()
.withIgnoreCase()
.withStringMatcher(ExampleMatcher.StringMatcher.CONTAINING);
Example example = Example.of(company, matcher);
Page<?> result = this.repository.findAll(example, page);
return ResponseEntity.ok(assembler.toResource(result, entityAssembler));
}
}
And then you can make queries like:
localhost:8080/companies/filter?name=google&address=NY
You can even query nested entities like:
localhost:8080/companies/filter?name=google&department.name=finances
I omitted some details for brevity, but I created a working example on Github.

The implementation of query methods is widely documented in Spring reference documentation and tons of technical blogs, though quite a bunch are outdated.
Since your question is probably "How can I perform a multi-parameter search with any combination of fields without declaring an awful lot of findBy* methods?", the answer is Querydsl, which is supported by Spring.

I have found a working solution for this task.
#RepositoryRestResource(excerptProjection=MyProjection.class)
public interface MyRepository extends Repository<Entity, UUID> {
#Query("select e from Entity e "
+ "where (:field1='' or e.field1=:field1) "
+ "and (:field2='' or e.field2=:field2) "
// ...
+ "and (:fieldN='' or e.fieldN=:fieldN)"
Page<Entity> advancedSearch(#Param("field1") String field1,
#Param("field2") String field2,
#Param("fieldN") String fieldN,
Pageable page);
}
With this solution, using this base url:
http://localhost:8080/api/examples/search/advancedSearch
We can make advanced searches with all the fields that we need.
Some examples:
http://localhost:8080/api/examples/search/advancedSearch?field1=example
// filters only for the field1 valorized to "example"
http://localhost:8080/api/examples/search/advancedSearch?field1=name&field2=surname
// filters for all records with field1 valorized to "name" and with field2 valorized to "surname"

I guess You can try following:
List<Person> findDistinctPeopleByLastnameOrFirstname(#Param("lastName")String lastname, #Param("firstName")String firstname);
and examples/search/searchByLastnameOrFirstname?firstName=value1&lastName=value2
Check out: http://docs.spring.io/spring-data/jpa/docs/current/reference/html/#repositories.query-methods.query-creation

Spring Data Mongo Custom Repository Query with ObjectID

I have a mongo query I want to implement as a Spring Mongo Repository
db.collection.find({ 'items':
{ $elemMatch: {
'refund.$id' : ObjectId('5638cab2e4b07ff212618d7e')
}
}
})
My Repository interface is
#Query("{ 'items': { $elemMatch: { 'refund.$id' : ObjectId(?0) } } }")
RMA findRMAByItemRefund(String refundId);
It throws JSONParseException
Caused by: com.mongodb.util.JSONParseException:
{ 'items': { $elemMatch: { 'refund.$id' : ObjectId("_param_0") } } }
^
at com.mongodb.util.JSONParser.parse(JSON.java:216)

As the accepted solution did not yield any results in my case, I had to find another solution, that was:
Instead of using the autogenerated query functionality defined via #Query(...) I opted for manually building the DBObject used to query mongo, and defining the functionality as a default implementation in the interface, thus keeping it "clean"
(" - because had to introduce a _query method)
Note: This solution only works in Java 1.8+
public interface SomeRepository extends MongoRepository<SomeEntity, String> {
#Query("{'nestedEntity._id': ?0}")
SomeEntity findByNestedEntityId_DoesntWork(String nestedEntityId);
#Query("?0")
SomeEntity _query(DBObject query);
default SomeEntity findByNestedEntityId(String nestedEntityId) {
DBObject queryObject = BasicDBObjectBuilder
.start("nestedEntity._id", new ObjectId(nestedEntityId))
.get();
return this._query(queryObject);
}
}

Try this
#Query("{ 'items': { $elemMatch: { 'refund.id' : ?0 } } }")
RMA findRMAByItemRefund(String refundId);
From my experience, ?0 should standalone and not used as a mongo function parameter.
Also, is $id your own assumption or the field is actually stored as $id. If not, I will go with refund.id

I guess my problem was a little different, but since I couldn't find answer anywhere, I believe it's worth mentioning.
Basically I wanted to do the search by ObjectId AND userId, so here's what I've done:
#Query("{ '_id': ?0, 'userId': ?1 }")
T findByObjectIdAndUserId(final ObjectId objectId, final Long userId);

This is the example object on mongo document:
{
"_id" : ObjectId("5c052a43f14008522c1c90c8"),
"_class" : "com.gdn.payment.report.entity.ReportDocument",
"status" : "DONE",
"report" : {
"id" : ObjectId("5c0508bcf14008460015eb07"),
"startDate" : NumberLong(1542795843188),
"endDate" : NumberLong(1543568857157),
"fields" : [
"customerInfo.billingAddress.streetAddress",
"customerInfo.name",
"items[*].name",
"orderId",
"type"
]
}
}
You can make the repository method for searching report.id only by using ObjectId data type, I've tried using just string and it cannot work, here are my repository method:
Page<ReportDocument> findByReport_id(ObjectId reportId, Pageable pageable)
for creating ObjectId you can do it using:
new ObjectId({input-reportId-string})

I'd suggest using Spring Data Derived Queries. You can also use QueryDsl on top.
Then you'd be able to write much more complex queries without having to deal with secondary issues such as query syntax and parsing.