Develop Reference

How to aggregate in spring data mongo db a nested object and avoid a PropertyReferenceException?

I am creating a new endpoint in springboot that will return simple stats on users generated from an aggregate query in a mongo database. However I get a PropertyReferenceException. I have read multiple stackoverflow questions about it, but didn't find one that solved this problem.
We have a mongo data scheme like this:
{
"_id" : ObjectId("5d795993288c3831c8dffe60"),
"user" : "000001",
"name" : "test",
"attributes" : {
"brand" : "Chrome",
"language" : "English" }
}
The database is filled with multiple users and we want using Springboot aggregate the stats of users per brand. There could be any number of attributes in the attributes object.
Here is the aggregation we are doing
Aggregation agg = newAggregation(
group("attributes.brand").count().as("number"),
project("number").and("type").previousOperation()
);
AggregationResults<Stats> groupResults
= mongoTemplate.aggregate(agg, Profile.class, Stats.class);
return groupResults.getMappedResults();
Which produces this mongo query which works:
> db.collection.aggregate([
{ "$group" : { "_id" : "$attributes.brand" , "number" : { "$sum" : 1}}} ,
{ "$project" : { "number" : 1 , "_id" : 0 , "type" : "$_id"}} ])
{ "number" : 4, "type" : "Chrome" }
{ "number" : 2, "type" : "Firefox" }
However when running a simple integration test we get this error:
org.springframework.data.mapping.PropertyReferenceException: No property brand found for type String! Traversed path: Profile.attributes.
From what I understand, it seems that since attributes is a Map<String, String> there might be a schematic problem. And in the mean time I can't modify the Profile object.
Is there something I am missing in the aggregation, or anything I could change in my Stats object?
For reference, here are the data models we're using, to work with JSON and jackson.
The Stats data model:
#Document
public class Stats {
#JsonProperty
private String type;
#JsonProperty
private int number;
public Stats() {}
/* ... */
}
The Profile data model:
#Document
public class Profiles {
#NotNull
#JsonProperty
private String user;
#NotNull
#JsonProperty
private String name;
#JsonProperty
private Map<String, String> attributes = new HashMap<>();
public Stats() {}
/* ... */
}

I found a solution, which was a combination of two problems:
The PropertyReferenceException was indeed caused because attributes is a Map<String, String> which means there is no schemes for Mongo.
The error message No property brand found for type String! Traversed path: Profile.attributes. means that the Map object doesn't have a brand property in it.
In order to fix that without touching my orginal Profile class, I had to create a new custom class which would map the attributes to an attributes object having the properties I want to aggreate on like:
public class StatsAttributes {
#JsonProperty
private String brand;
#JsonProperty
private String language;
public StatsAttributes() {}
/* ... */
}
Then I created a custom StatsProfile which would leverage my StatsAttributes and would be similar to the the original Profile object without modifying it.
#Document
public class StatsProfile {
#JsonProperty
private String user;
#JsonProperty
private StatsAttributes attributes;
public StatsProfile() {}
/* ... */
}
With that I made disapear my problem with the PropertyReferenceException using my new class StatsAggregation in the aggregation:
AggregationResults<Stats> groupResults
= mongoTemplate.aggregate(agg, StatsProfile.class, Stats.class);
However I would not get any results. It seems the query would not find any document in the database. That's where I realied that production mongo objects had the field "_class: com.company.dao.model.Profile" which was tied to the Profile object.
After some research, for the new StatsProfile to work it would need to be a #TypeAlias("Profile"). After looking around, I found that I also needed to precise a collection name which would lead to:
#Document(collection = "profile")
#TypeAlias("Profile")
public class StatsProfile {
/* ... */
}
And with all that, finally it worked!
I suppose that's not the prettiest solution, I wish I would not need to create a new Profile object and just consider the attributes as a StatsAttributes.class somehow in the mongoTemplate query. If anyone knows how to, please share 🙏

Using Couchbase SDK in Java

I am trying to map the result of a couchbase query to a java reference type, so far I have found no way to do this.
How can I capture the following as a java reference type:
N1qlQueryResult result = couchbaseBucket.query(
N1qlQuery.simple("SELECT * FROM customers LIMIT 1"));
JsonObject cust = result.allRows().get(0).value();
How can I cast this 'cust' to a java object? What would be the best way of doing this, doesnt the couchbase SDK provide some solution to this?

There was a blog post published yesterday that shows you how to do this with couchbase spring-boot and spring data.
I'm not a Java expert at all, but it looks like you start by creating an entity class like this:
#Document
#Data
#AllArgsConstructor
#NoArgsConstructor
#EqualsAndHashCode
public class Building {
#NotNull
#Id
private String id;
#NotNull
#Field
private String name;
#NotNull
#Field
private String companyId;
// ... etc ...
}
Then, create a repository class.
#N1qlPrimaryIndexed
#ViewIndexed(designDoc = "building")
public interface BuildingRepository extends CouchbasePagingAndSortingRepository<Building, String> {
List<Building> findByCompanyId(String companyId);
// ... etc ...
}
Finally, you can use #Autowired in a service class or wherever to instantiate a BuildingRepository and start calling the methods on it. The full documentation for Spring Data Couchbase is available on docs.spring.io

IllegalArgumentException using Spring Data and MongoDB

We're using MongoDB and Spring Data in our Spring Boot app. Another developer has earlier written a service, using MongoTemplate's findAndModify method, and some additional code that uses that service. It worked fine, even in production.
I've branched and went on adding some new features (completely new code) and needed to call that service. Service doesn't work for me, even though I didn't even touch the service code. Moreover, even the code the other guy wrote before (it's a REST controller calling the service) doesn't work now. On the master branch, everything works like it should.
Here's the service:
#Autowired
private MongoTemplate mongo;
public void addRetention(Date when, String userId, Platform platform) {
LocalDateTime dateTime = LocalDateTime.ofEpochSecond(when.getTime()/1000, 0, ZoneOffset.UTC);
int yyyymm = 100 * dateTime.getYear() + dateTime.getMonthValue();
Retention r = mongo.findAndModify( // the exception is coming from this line
new Query(Criteria.where("userId").is(userId).and("yyyymm").is(yyyymm)),
new Update().inc("count", 1).set("platform", platform),
new FindAndModifyOptions().upsert(true).returnNew(true), Retention.class
);
addRealTimeRetention(userId, r, yyyymm);
}
How I call the service:
retentionService.addRetention(timeService.timeToUtil(usage2.getDate()),
usage2.getUserId(), platform);
And the stack trace on PasteBin. The exception message is:
IllegalArgumentException: Target bean is not of type of the persistent entity!
EDIT: Here's the Retention.java class:
#Document
#JsonInclude(JsonInclude.Include.NON_NULL)
#CompoundIndexes({
#CompoundIndex(name = "userid_yyyymm_idx", def = "{ userId: 1, yyyymm: 1 }")
})
public class Retention {
#Id
private String id;
#Indexed
private String userId;
#Indexed
private int yyyymm;
private int count;
private Platform platform;
// setters and getters...
}
FINAL EDIT: I solved this by deleting spring devtools dependency in pom.xml. I'm not sure how that dependency has anything to do with this exception. Found the solution here. Thank you to everyone who helped out.

How do I get distinct fields from MongoRepository/QueryDSL?

My Document is
#QueryEntity #Data #Document(collection = "MyCol") public class MyCol {
#Id private String _id;
private String version;
I want to get all distinct version stored in the db.
My attempts:
public interface MyColDao extends MongoRepository<MyCol, String>, QueryDslPredicateExecutor<MyCol> {
#Query("{ distinct : 'MyCol', key : 'version'}")
List<String> findDistinctVersion();
}
Or just findDistinctVersion without the query annotation.
Most of the examples of github have a By-field like
List<Person> findDistinctPeopleByLastnameOrFirstname(String lastname, String firstname);
I don't need a By field.
Another example I found here.
#Query("{ distinct : 'channel', key : 'game'}")
public JSONArray listDistinctGames();
This doesn't seem to work for me.
I can't seem to find queryDSL/Morphia's documentation to do this.

public interface MyColDao extends MongoRepository<MyCol, String>, QueryDslPredicateExecutor<MyCol> {
#Query("{'yourdbfieldname':?0}")
List<String> findDistinctVersion(String version);
}
here version replaces your your db field name
more you can see here

This spring documentation provide the details, how to form a expression when you are want to fetch distinct values.
Link

I had a similar problem, but I couldn't work out how to do it within the MongoRepository (as far as I can tell, it's not currently possible) so ended up using MongoTemplate instead.
I believe the following would meet your requirement.
#AutoWired
MongoTemplate mongoTemplate
public List<String> getVersions(){
return mongoTemplate.findDistinct("version", MyCol.class, String.class);
}

Ignore JsonIgnore in Elasticsearch

I am developing an application which uses Spring-boot, a relational database and Elasticsearch.
I use JSON serialization at 2 differents places in the code:
In the response of the REST API.
When the code interacts with Elasticsearch.
There are some properties that I need in Elasticsearch but that I want to hide to the application user (e.g. internal ids coming from the relational database).
Here is an example of entity :
#Document
public class MyElasticsearchEntity {
#Id
private Long id; //I want to hide this to the user.
private String name;
private String description;
}
Problem : When the object it persisted in Elasticsearch, it gets serialized as JSON. Hence, fields with #JsonIgnore are ignored when serialized to Elasticsearch.
Up to now, I found 2 unsatisfying solutions :
Solution 1 : Use #JsonProperty like this :
#Id
#JsonProperty(access = JsonProperty.Access.READ_ONLY)
private Long id;
The id gets written in Elasticsearch and is nullified in the JSON response :
{
"id" : null,
"name" : "abc",
"description" : null
}
So it works but the application user still sees that this property exists. This is messy.
Solution 2 : Cutomize the object mapper to ignore null values
Spring-boot has a built-in option for that :
spring.jackson.serialization-inclusion=NON_NULL
Problem : it suppresses all non-null properties, not only those that I want to ignore. Suppose that the field description of the previous entity is empty, the JSON response will be :
{
"name" : "abc"
}
And this is problematic for the UI.
So is there a way to ignore such field only in the JSON response?

You could use Jackson JsonView for your purpose. You can define one view which will be used to serialize pojo for the application user :
Create the views as class, one public and one private:
class Views {
static class Public { }
static class Private extends Public { }
}
Then uses the view in your Pojo as an annotation:
#Id
#JsonView(Views.Private.class) String name;
private Long id;
#JsonView(Views.Public.class) String name;
private String publicField;
and then serialize your pojo for the application user using the view:
objectMapper.writeValueUsingView(out, beanInstance, Views.Public.class);
This is one example of many others on how view can fit your question. Eg. you can use too objectMapper.configure(SerializationConfig.Feature.DEFAULT_VIEW_INCLUSION, false); to exclude field without view annotation and remove the Private view.