I'm using Spring-Data MongoDB aggregation framework.
Here is a code sample:
Aggregation agg = Aggregation.newAggregation(
match(Criteria.where("type").is("PROMO")),
group("locale")//.count().as("counts")
);
AggregationResults<Message> results = mongoTemplate.aggregate(agg, "message", Message.class);
return results.getMappedResults();
Throw:
org.springframework.core.convert.ConversionFailedException: Failed to convert from type java.lang.String to type java.math.BigDecimal for value 'CL'; nested exception is java.lang.NumberFormatException
CL is a value on locale field, but i dont understand why throw that exception.
Im using a similar example from the documentation.
SOLVED:
Aggregation agg = Aggregation.newAggregation(
match(Criteria.where("type").is("PROMO")),
group("created", "text").addToSet("locale").as("countries").addToSet("device.deviceType").as("platforms").count().as("count")
);
I try a simple example on mongo console. After that, map the operations to the builder.
I dont undesrtand why dont work before. If someone can clear the problem will be great.
The model "message":
{ "_id" : "90.0", "device" : { "_id" : "5faf92fd-37f2-4d42-a01a-dd1abce0c1af", "deviceType" : "iPhone", "countryId" : "AR" }, "text" : "Text", "created" : ISODate("2014-01-03T15:56:27.096Z"), "status" : "SENT", "type" : "PROMO" }
My simple answer is avoid using the MongoDB Spring Data classes directly for aggregation and use the standard MongoDB Java objects e.g. DBObject / AggregationOutput. The reason for that is I have lost several hours trying to get anything but basic aggregation queries working in MongoDB Spring data (and that is using the latest which as of today is spring-data-mongodb 1.5.0.RELEASE).
However, constructing aggregation queries using the standard MongoDB Java objects can be a pain (especially if nested / complex) as you end up creating countless DBObject groupFields = new BasicDBObject("_id", null); and the code looks a mess.
I recommend adding the following 3 wrapper methods to your code.
protected DBObject dbObj (String key, Object value) {
return new BasicDBObject (key, value);
}
protected DBObject dbObj (Object ... objs) {
DBObject dbObj = new BasicDBObject();
if (objs.length % 2 == 0) {
for (int i = 0; i < objs.length; i+=2) {
dbObj.put((String)objs[i], objs[i+1]);
}
}
return dbObj;
}
protected DBObject dbList (Object ... objs) {
BasicDBList dbList = new BasicDBList();
for (Object obj : objs) {
dbList.add(obj);
}
return (DBObject)dbList;
}
This enables easy translation between your JSON based queries and your Java code. e.g. if you had the following complex query (taken from http://docs.mongodb.org/manual/tutorial/aggregation-zip-code-data-set/)
db.zipcodes.aggregate(
{
$group: {
_id: { state: "$state", city: "$city" },
pop: { $sum: "$pop" }
}
},{
$sort: { pop: 1 }
},{
$group: {
_id: "$_id.state",
biggestCity: { $last: "$_id.city" },
biggestPop: { $last: "$pop" },
smallestCity: { $first: "$_id.city" },
smallestPop: { $first: "$pop" }
}
},{
$project: {
_id: 0,
state: "$_id",
biggestCity: {
name: "$biggestCity",
pop: "$biggestPop"
},
smallestCity: {
name: "$smallestCity",
pop: "$smallestPop"
}
}
});
... then your Java code would look something like this ...
List<DBObject> aggregation = Arrays.asList (
dbObj ("$group", dbObj (
"_id", dbObj ("state", "$state", "city", "$city"),
"pop", dbObj ("$sum", "$post")
)),
dbObj ("$sort", dbObj ("pop", 1)),
dbObj ("$group", dbObj (
"_id", "$_id.state",
"biggestCity", dbObj ("$last", "$_id.city"),
"biggestPop", dbObj ("$last", "$pop"),
"smallestCity", dbObj ("$first", "$_id.city"),
"smallestPop", dbObj ("$first", "$pop")
)),
dbObj ("$project", dbObj (
"_id", 0,
"state", "$_id",
"biggestCity", dbObj ("name", "$biggestCity", "pop", "$biggestPop"),
"smallestCity", dbObj ("name", "$smallestCity", "pop", "$smallestPop")
))
);
// Run aggregation query
DBCollection collection = mongoTemplate.getCollection(COLLECTION_NAME);
AggregationOutput output = collection.aggregate (aggregation);
Doing it this way, if it works in your editor (e.g. RoboMongo) it will work in your Java code although you will have to manually convert the objects from the result, which isn't too painful i.e.
List<MyResultClass> results = new ArrayList<MyResultClass>();
Iterator<DBObject> it = output.results().iterator();
while (it.hasNext()) {
DBObject obj = it.next();
MyResultClass result = mongoTemplate.getConverter().read(MyResultClass.class, obj);
results.add(result);
}
However, you may find the Spring Data Aggregation stuff does work OK for you. I love Spring and I do use Mongo Spring Data in various parts of my code, it is the aggregation support that lets it down e.g. doing a "$push" inside a "$group" with multiple items doesn't seem to work. I'm sure it will improve with time (and better documentation). Other people have echoed these thoughts e.g. http://movingfulcrum.tumblr.com/post/61693014502/spring-data-and-mongodb-a-mismatch-made-in-hell - see section 4.
Happy coding!
Related
I tried to write a group aggregation query using the year value from a date object as a key, but for some reason I'm getting this exception.
org.springframework.data.mapping.PropertyReferenceException: No property year(invoiceDate)
Here is the mongo query which I'm trying to replicate:
db.collection.aggregate([
{
$match:
{
"status": "Active"
}
},
{
$group:
{
"_id":{$year:"$invoiceDate"}
}
},
{
$sort:
{
"_id" : -1
}
}
])
And this is my Java implementation:
Aggregation aggregation = Aggregation.newAggregation(
match(new Criteria().andOperator(criteria())),
Aggregation.group("year(invoiceDate)")
).withOptions(newAggregationOptions().allowDiskUse(true).build());
I also didn't find a way how I can apply the sorting on the results from the grouping.
You're basically looking for extractYear() which maps to the $year operator with MongoDB:
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(new Criteria().andOperator(criteria())),
Aggregation.project().and("invoiceDate").extractYear().as("_id"),
Aggregation.group("_id"),
Aggregation.sort(Sort.Direction.DESC, "_id)
)
This generally needs to go into a $project in order to make the helpers happy.
If you really want the expression within the $group then you can add a custom operation expression:
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(new Criteria().andOperator(criteria())),
new AggregationOperation() {
#Override
public Document toDocument(AggregationOperationContext aggregationOperationContext) {
return new Document("$group",
new Document("_id", new Document("$year","$invoiceDate") )
);
}
},
Aggregation.sort(Sort.Direction.DESC, "_id)
)
The database MongoDB I have stored documents in the format:
{
"achievement": [
{
"userFromId":"max",
"userToId":"peter",
"date":"2016-01-25",
"pointCount":1,
"description":"good work",
"type":"THANKS"
}
]
}
How to get the number of records in the database (if any) for the a certain date, in which people are thanking the other people.
I created a query to retrieve data:
DBObject clause1 = new BasicDBObject("userFromId", userFromId);
DBObject clause2 = new BasicDBObject("userToId", userToId);
DBObject clause3 = new BasicDBObject("sendDate", localDate);
DBObject clause4 = new BasicDBObject("type", Thanks);
BasicDBList or = new BasicDBList();
or.add(clause1);
or.add(clause2);
or.add(clause3);
or.add(clause4);
DBObject query = new BasicDBObject("$or", or);
But I do not know how to get the number of records and how can rewrite the query using aggregation?
For example:
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.group("userFromId")
.first("userFromId").as("userFromId")
.sum("pointCount").as("pointCount"));
I do not know how to add a few more parameters.
What the return request if the data to the database does not exist?
Thanks for any help
You can use something like this. This will count all the number of documents matching the below criteria.
Regular Query
db.collection.count({ $or: [ { "userFromId": userFromId }, { "userToId": userToId } ] });
Using Aggregation
db.collection.aggregate( [
{ $match: { $or: [ { "userFromId": userFromId }, { "userToId": userToId } ] } },
{ $group: { _id: null, count: { $sum: 1 } } }
] );
I have a collection of documents like the following:
{ name : "John" ,
age : 25.0 ,
bornIn : "Milan" ,
city : [ {
name : "Roma" ,
state : "IT" ,
mayor : "John"
}]
}
{ name : "Jim" ,
age : 35.0 ,
bornIn : "Madrid" ,
city : [ {
name : "Madrid" ,
state : "ESP" ,
mayor : "Jim"
}]
}
I want to retrieve all the documents that have the field $bornIn equal to the field $city.name. I need to do this as an intermediate stage of a pipeline, so I can't use the $where operator.
I searched online and I found a suggestion to implement something like this:
{ $project:
{ matches:
{ $eq:[ '$bornIn', '$city.name' ] }
}
},
{ $match:
{ matches:true }
} )
But it didn't work neither via shell nor via Java driver as it marks the fields as different.
For the sake of completeness I report my code:
final DBObject eq = new BasicDBObject();
LinkedList eqFields = new LinkedList();
eqFields.add("$bornIn");
eqFields.add("$city.name");
eq.put("$eq", eqFields);
projectFields.put("matches", eq);
final DBObject proj = new BasicDBObject("$project", projectFields);
LinkedList agg = new LinkedList();
agg.add(proj);
final AggregationOutput aggregate = table.aggregate( agg);
Do you have any suggestion? I'm using MongoDB 3.2, and I need to do this via Java Driver.
Thanks!!
PS. It is not relevant but actually the documents above are the output of a $lookup stage among collections "cities" and "persons", with join on $name/$mayor.. it is super cool!! :D :D
I'm a little rusty on how Mongo deals with deep equality searching arrays of objects, but this is definitely doable with $unwind
db.foo.aggregate([
{$unwind: "$city"},
{ $project:
{ matches:
{ $eq:[ '$bornIn', '$city.name' ] }
}
},
{ $match:
{ matches:true }
}
]);
I'm not on a computer with Mongo right now, so my syntax might be off a bit.
How do I make a MongoDB query using BasicDBObjects in Java, when I wish to find all documents that contain an array of nested documents, where one of those nested documents meets all the specified criteria?
Taking the example data:
[
{
"_id":"blood_0",
"type":"O",
"list":[
{
"firstname":"John",
"lastname":"Smith",
"zipcode":"12345"
},
{
"firstname":"John",
"lastname":"Hamilton",
"zipcode":"54627"
},
{
"firstname":"Ben",
"lastname":"Brick",
"zipcode":"12345"
},
{
"firstname":"William",
"lastname":"Tell",
"zipcode":"15487"
}
]
},
{
"_id":"blood_1",
"type":"AB",
"list":[
{
"firstname":"Mary",
"lastname":"Smith",
"zipcode":"12345"
},
{
"firstname":"John",
"lastname":"Henry",
"zipcode":"54624"
},
{
"firstname":"Jacob",
"lastname":"Tell",
"zipcode":"19283"
},
{
"firstname":"William",
"lastname":"Dirk",
"zipcode":"15999"
}
]
}
]
If I only want to return the objects that contain a contact in the list that meets the criteria of firstname = William, lastname = Tell how would I go about doing that? The queries I am doing are not grouping the criteria, so I would get two results where I actually only should be getting one.
How would I do the same query but also checking for type = AB, as well as the other criteria, which would return no results?
You are looking for the $elemMatch operator. It restricts the query operators to a single element within the array of values.
In the shell your query will look like:
db.people.find( { list : { $elemMatch : { lastName:"Smith", firstName: "John" } } } )
To add the blood type:
db.people.find( {
type : "AB",
list : { $elemMatch : { lastName:"Smith", firstName: "John" } }
} )
This gets a bit verbose using the Java Driver.
DBObject elemMatch = new BasicDBObject();
elemMatch.put("lastName","Smith");
elemMatch.put("firstName","John");
DBObject query = new BasicDBObject();
query.append( "type", "AB");
query.append( "list", elemMatch);
Pass that query to one of the find() methods on the collection and you should get the documents you are looking for.
Note that the $elemMatch query operator will return the entire document, including all of the elements in the array. There is a similarly named projection operator to limit the array elements returned to only those matched.
HTH - Rob.
First things first. I really think your model is utterly wrong. Nested arrays which potentially grow indefinetly are bad for multiple reasons:
If the document exceeds it's padding when new members are written to this array, the document needs to be migrated within a data file. That is a pretty costly operation and you want to prevent it as much as you can.
BSON documents are limited to 16MB. So per blood type you could only have a limited number of people.
All queries tend to be a bit more complicated, the according code more bloated and hence slower.
So how to do it? Take these documents:
{
_id: ObjectId(),
firstName: "Mary",
lastName: "Smith",
zip: "12345",
bt: "AB"
},
{
_id: ObjectId(),
firstName: "John",
lastName: "Smith",
zip: "12345",
bt: "0"
}
With indices set like
db.people.ensureIndex({lastName:1,firstName:1})
db.people.ensureIndex({bt:1})
on the MongoDB shell, you can get what you want with
db.people.find({ lastName:"Smith", firstName: "John"})
or
db.people.find({ bt: "AB" })
This query for example translates to the following
MongoClient client = new MongoClient("localhost");
DB db = client.getDB("yourDB");
DBCollection coll = db.getCollection("yourCollection");
BasicDBOBject query = new BasicDBObject("bt","AB");
DBCursor cursor = coll.find(query);
try {
while( cursor.hasNext() ) {
System.out.println( cursor.next() );
}
} finally {
cursor.close();
}
You might find the MongoDB introduction for working with a Java driver interesting.
I am trying to use the $project operator in the aggregation-framework with MongoDB for Java.
DBObject fields = new BasicDBObject("example", 1);
fields.put("timestamp", $timestampField);
The above results in "exception: Unrecognized pipeline stage name: 'timestamp'" , "code" : 16436 , "ok" : 0.0}
If what you are trying to $project is basically something that serializes like this:
{ "$project": {
"example": 1,
"timestamp": "$timestameField"
}}
Then you construct your BSON accordingly, and pretty much exactly as shown:
DBObject project = new BasicDBObject(
"$project", new BasicDBObject(
"example", 1
).append(
"timestamp", "$timestamp"
)
);
It is the .append() method that adds additional field content. The .put() method "replaces" the content in the BasicDBObject.