I have the following JSON object which I would like to convert to a Criteria object so that I can use in Aggregation.match() query.
{
"_filter": {
"$and": [
{
"$or": [
{
"country": "India"
},
{
"age": 20
}
]
}
]
},
"_page": {
"pageNum": 0,
"recordsPerPage": 1
}
}
I have seen that we have BasicQuery object which can be constructed from the contents of the _filter field. However, I am not able to deduce Criteria object from that.
Is there any method/utility in Spring Data MongoDB that does this?
Actually Criteria is a wrapper class to abstract you from the MongoDB syntax and build the queries in an elegant way.
I would like to convert to a Criteria object so that I can use in Aggregation.match() query
There is no need to do so. Try this:
AggregationOperation match = ctx => new Document("$match", Document.parse(your_json).get("_filter"));
...
Aggregation agg = Aggregation.newAggregation(match)
.withOptions(Aggregation.newAggregationOptions().allowDiskUse(Boolean.TRUE).build());
mongotemplate.aggregate();
mongoTemplate.aggregate(agg, inputType, outputType).getMappedResults();
Looking at com.mongodb.reactivestreams.client.MongoCollection interface we see that aggregation can be invoked using list of Bson elements.
public interface MongoCollection<TDocument> {
...
AggregatePublisher<TDocument> aggregate(List<? extends Bson> list);
}
It is clear how to use it when aggregation steps are JSONs (see Example with simple JSONs)
Unfortunately, when any aggregation step contains a function (which is allowed by native MongoDB query), for instance, $accumulator the same approach cannot be applied due to it causes violation of Bson format (org.bson.json.JsonParseException) (see Example with functions)
What is the best way to convert a native MongoDB aggregation query into a result in Java?
(suppose that queries are complex and it is not expedient to rewrite them with Mongo aggregation builders in Java)
Example with simple JSONs:
ReactiveMongoOperations mongo = /* ... */;
var match = BasicDBObject.parse("{ $match: {name: \"Jack\"} }");
var project = BasicDBObject.parse("{ $project: {_id: 0, age: 1, name: 1} }";
var queryParts = List.of(match, project);
Flux<PersonInfo> infoFlux = mongo
.getCollection("person")
.flatMapMany(person -> person.aggregate(queryParts).toFlux())
.map(it -> objectMapper.readValue(it.toJson(), PersonInfo.class))
.collectList()
Example with functions:
// here for conciseness it is just a counting accumulator; generally functions are more complex
var match = BasicDBObject.parse("""
{
$group: {
_id: "$token",
count: {
$accumulator: {
init: function() {
return {owned: 0, total: 0}
},
accumulate: function(state, owner) {
return {
total: state.total + 1
}
},
accumulateArgs: ["$owner"],
merge: function(a, b) {
return {
total: a.total + b.total
}
},
lang: "js"
}
},
minPriceEth: {$min: "$priceEth"}
}
}
""");
Am querying ElasticSearch using Java API and am getting lot of duplicate values. I want to get only the unique values from the query (distinct value). How can we get the distinct values from the Query Builder.
Please find my java code below, which is giving duplicate values.
QueryBuilder qb2=null;
List<Integer> link_id_array=new ArrayList<Integer>();
for(Replacement link_id:linkIDList) {
link_id_array.add(link_id.getLink_id());
}
qb2 = QueryBuilders.boolQuery()
.must(QueryBuilders.termsQuery("id", link_id_array));
Am using elastic search 6.2.3 version with RestHighLevelClient
Way 1: You need to use the so-called aggregation API :
Sample query to get distinct emails client :
{
"query" : {
"match_all" : { }
},
"aggregations" : {
"label_agg" : {
"terms" : {
"field" : "Email_client",
"size" : 100
}
}
}
}
Java code sample=>
SearchRequestBuilder aggregationQuery =
client.prepareSearch("emails")
.setQuery(QueryBuilders.matchAllQuery())
.addAggregation(AggregationBuilders.terms("label_agg")
.field("Email_client").size(100));
SearchResponse response = aggregationQuery.execute().get();
Aggregation aggregation = response.getAggregations().get("label_agg");
StringTerms st = (StringTerms) aggregation;
return st.getBuckets().stream()
.map(bucket -> bucket.getKeyAsString())
.collect(toList());
Way 2 :
Use cardinality of aggregation Api:
Sample elasticquery:
{
"size": 0,
"aggs": {
"distinct": {
"cardinality": {
"field": "Email_client",
"size" : 100
}
}
}
Java code sample=>
AggregationBuilder agg11 = AggregationBuilders.cardinality("distinct").field("Email_client");
SearchResponse response11 = client.prepareSearch("emails")// we can give multiple index names here
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(query11)
.addAggregation(agg11)
.setExplain(true)
.setSize(0)
.get();
The database MongoDB I have stored documents in the format:
{
"achievement": [
{
"userFromId":"max",
"userToId":"peter",
"date":"2016-01-25",
"pointCount":1,
"description":"good work",
"type":"THANKS"
}
]
}
How to get the number of records in the database (if any) for the a certain date, in which people are thanking the other people.
I created a query to retrieve data:
DBObject clause1 = new BasicDBObject("userFromId", userFromId);
DBObject clause2 = new BasicDBObject("userToId", userToId);
DBObject clause3 = new BasicDBObject("sendDate", localDate);
DBObject clause4 = new BasicDBObject("type", Thanks);
BasicDBList or = new BasicDBList();
or.add(clause1);
or.add(clause2);
or.add(clause3);
or.add(clause4);
DBObject query = new BasicDBObject("$or", or);
But I do not know how to get the number of records and how can rewrite the query using aggregation?
For example:
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.group("userFromId")
.first("userFromId").as("userFromId")
.sum("pointCount").as("pointCount"));
I do not know how to add a few more parameters.
What the return request if the data to the database does not exist?
Thanks for any help
You can use something like this. This will count all the number of documents matching the below criteria.
Regular Query
db.collection.count({ $or: [ { "userFromId": userFromId }, { "userToId": userToId } ] });
Using Aggregation
db.collection.aggregate( [
{ $match: { $or: [ { "userFromId": userFromId }, { "userToId": userToId } ] } },
{ $group: { _id: null, count: { $sum: 1 } } }
] );
I'm using Spring-Data MongoDB aggregation framework.
Here is a code sample:
Aggregation agg = Aggregation.newAggregation(
match(Criteria.where("type").is("PROMO")),
group("locale")//.count().as("counts")
);
AggregationResults<Message> results = mongoTemplate.aggregate(agg, "message", Message.class);
return results.getMappedResults();
Throw:
org.springframework.core.convert.ConversionFailedException: Failed to convert from type java.lang.String to type java.math.BigDecimal for value 'CL'; nested exception is java.lang.NumberFormatException
CL is a value on locale field, but i dont understand why throw that exception.
Im using a similar example from the documentation.
SOLVED:
Aggregation agg = Aggregation.newAggregation(
match(Criteria.where("type").is("PROMO")),
group("created", "text").addToSet("locale").as("countries").addToSet("device.deviceType").as("platforms").count().as("count")
);
I try a simple example on mongo console. After that, map the operations to the builder.
I dont undesrtand why dont work before. If someone can clear the problem will be great.
The model "message":
{ "_id" : "90.0", "device" : { "_id" : "5faf92fd-37f2-4d42-a01a-dd1abce0c1af", "deviceType" : "iPhone", "countryId" : "AR" }, "text" : "Text", "created" : ISODate("2014-01-03T15:56:27.096Z"), "status" : "SENT", "type" : "PROMO" }
My simple answer is avoid using the MongoDB Spring Data classes directly for aggregation and use the standard MongoDB Java objects e.g. DBObject / AggregationOutput. The reason for that is I have lost several hours trying to get anything but basic aggregation queries working in MongoDB Spring data (and that is using the latest which as of today is spring-data-mongodb 1.5.0.RELEASE).
However, constructing aggregation queries using the standard MongoDB Java objects can be a pain (especially if nested / complex) as you end up creating countless DBObject groupFields = new BasicDBObject("_id", null); and the code looks a mess.
I recommend adding the following 3 wrapper methods to your code.
protected DBObject dbObj (String key, Object value) {
return new BasicDBObject (key, value);
}
protected DBObject dbObj (Object ... objs) {
DBObject dbObj = new BasicDBObject();
if (objs.length % 2 == 0) {
for (int i = 0; i < objs.length; i+=2) {
dbObj.put((String)objs[i], objs[i+1]);
}
}
return dbObj;
}
protected DBObject dbList (Object ... objs) {
BasicDBList dbList = new BasicDBList();
for (Object obj : objs) {
dbList.add(obj);
}
return (DBObject)dbList;
}
This enables easy translation between your JSON based queries and your Java code. e.g. if you had the following complex query (taken from http://docs.mongodb.org/manual/tutorial/aggregation-zip-code-data-set/)
db.zipcodes.aggregate(
{
$group: {
_id: { state: "$state", city: "$city" },
pop: { $sum: "$pop" }
}
},{
$sort: { pop: 1 }
},{
$group: {
_id: "$_id.state",
biggestCity: { $last: "$_id.city" },
biggestPop: { $last: "$pop" },
smallestCity: { $first: "$_id.city" },
smallestPop: { $first: "$pop" }
}
},{
$project: {
_id: 0,
state: "$_id",
biggestCity: {
name: "$biggestCity",
pop: "$biggestPop"
},
smallestCity: {
name: "$smallestCity",
pop: "$smallestPop"
}
}
});
... then your Java code would look something like this ...
List<DBObject> aggregation = Arrays.asList (
dbObj ("$group", dbObj (
"_id", dbObj ("state", "$state", "city", "$city"),
"pop", dbObj ("$sum", "$post")
)),
dbObj ("$sort", dbObj ("pop", 1)),
dbObj ("$group", dbObj (
"_id", "$_id.state",
"biggestCity", dbObj ("$last", "$_id.city"),
"biggestPop", dbObj ("$last", "$pop"),
"smallestCity", dbObj ("$first", "$_id.city"),
"smallestPop", dbObj ("$first", "$pop")
)),
dbObj ("$project", dbObj (
"_id", 0,
"state", "$_id",
"biggestCity", dbObj ("name", "$biggestCity", "pop", "$biggestPop"),
"smallestCity", dbObj ("name", "$smallestCity", "pop", "$smallestPop")
))
);
// Run aggregation query
DBCollection collection = mongoTemplate.getCollection(COLLECTION_NAME);
AggregationOutput output = collection.aggregate (aggregation);
Doing it this way, if it works in your editor (e.g. RoboMongo) it will work in your Java code although you will have to manually convert the objects from the result, which isn't too painful i.e.
List<MyResultClass> results = new ArrayList<MyResultClass>();
Iterator<DBObject> it = output.results().iterator();
while (it.hasNext()) {
DBObject obj = it.next();
MyResultClass result = mongoTemplate.getConverter().read(MyResultClass.class, obj);
results.add(result);
}
However, you may find the Spring Data Aggregation stuff does work OK for you. I love Spring and I do use Mongo Spring Data in various parts of my code, it is the aggregation support that lets it down e.g. doing a "$push" inside a "$group" with multiple items doesn't seem to work. I'm sure it will improve with time (and better documentation). Other people have echoed these thoughts e.g. http://movingfulcrum.tumblr.com/post/61693014502/spring-data-and-mongodb-a-mismatch-made-in-hell - see section 4.
Happy coding!