Develop Reference

How to fetch data in a single go from mongodb based on multiple filters?

I am new to mongodb and aggregation framework.
We have a class UserMetaData and a list of UserMetaData. I need to fetch data according to the userMetaDataList that is passed to the method solve().
Currently I am iterating the list and one by one fetching the corresponding collection from the monogdb. Since the db calls are made for each element in the list, this becomes a highly expensive operation.
Is there any way to fetch all the required data from mongodb in one shot(more like a bulk fetch operation).
mongodb - perform batch query the solution provided in this does not fulfill the requirements of the current scenario.
Please help!!
This is how I am doing currently.
class UserMetaData{
String userId;
String vehicleId;
String vehicleColour;
String orderId;
}
public List<String> getOrderIds(List<UserMetaData> userMetaDataList) {
List<String> orderIds = new ArrayList<>();
for (UserMetaData userMetadata : userMetaDataList) {
try {
BasicDBObject matchDBObject = new BasicDBObject("user_id", new BasicDBObject("$eq", userMetadata.getUserId()));
matchDBObject.append("vehicle_id", new BasicDBObject("$eq", userMetadata.getVehicleID()));
matchDBObject.append("vehicle_colour", new BasicDBObject("$in", ImmutableSet.of("WHITE", "BLACK")));
Document document = eventCollection.find(matchDBObject)
.projection(new BasicDBObject("order_id", "1"))
.first();
orderIds.add(document.get("order_id").toString());
} catch (Exception e) {
log.info("Exception occurred while fetching order id for user_id: {} asset_id:{} - {}", metadata.getUserId(), metadata.getAssetID(), e);
}
}
return ordersIds;
}
I want to fetch all the corresponding data in a single query.
Requesting help.

You can join all filters with $OR condition and fetch the full list at once ...

I want to fetch all the corresponding data in a single query.
You can use this approach and perform the query as a single operation (avoids the for-loop).
Consider sample documents in the collection test:
{ "_id" : ObjectId("621762e2cda7c6394d557f37"), "userid" : 1, "name" : "ijk", "orderid" : "11" }
{ "_id" : ObjectId("621762efcda7c6394d557f38"), "userid" : 12, "name" : "abc", "orderid" : "99" }
{ "_id" : ObjectId("621762fccda7c6394d557f39"), "userid" : 13, "name" : "xyz", "orderid" : "100" }
The array of objects to filter:
var DOCS = [
{ userid: 12, name: "abc" },
{ userid: 13, name: "xyz" }
]
The query to filter by DOCS:
db.test.find(
{
$expr: {
$in: [ { userid: "$userid", name: "$name" }, DOCS ]
}
},
{
orderid: 1
}
)
The output has documents with userids 12 and 13.
[ EDIT - ADD ]
This aggregation an improvement over the find:
db.test.aggregate([
// This matches the 'userid' and 'name' fields with the input list 'DOCS'
{
$match: {
$expr: {
$in: [ { userid: "$userid", name: "$name" }, DOCS ]
}
}
},
// The grouping will select only the first matching for the 'userid' and 'name'
// (this is as per the question post's code: `.first()`)
{
$group: {
_id: {
userid: "$userid",
name: "$name"
},
orderid: {
$first: "$orderid"
}
}
},
// Remove the '_id' field
// Now the result has just the 'orderid' field only
{
$unset: "_id"
}
])

Is it possible to rename _id field after mongo's group aggregation?

I have a query like this (simplified):
db.collection.aggregate([
{ $match: { main_id: ObjectId("58f0f67f50c6af16709fd2c7") } },
{
$group: {
_id: "$name",
count: { $sum: 1 },
sum: { $sum: { $add: ["$P31", "$P32"] } }
}
}
])
I do this query from Java, and I want to map it on my class, but I don't want _id to be mapped on name field. Because if I do something like this:
#JsonProperty("_id")
private String name;
then when I save this data back to mongo (after some modification) the data is saved with name as _id while I want a real Id to be generated.
So, how can I rename _id after $group operation?

You can achieve this by adding a $project stage at the end of your pipeline like this :
{ $project: {
_id: 0,
name: "$_id",
count: 1,
sum: 1
}
}
try it online: mongoplayground.net/p/QpVyh-0I-bP

From mongo v3.4 you could use $addFields in conjunction with $project to avoid to write all the fields in $project that could be very tedious.
This happen in $project because if you include specifically a field, the other fields will be automatically excluded.
Example:
{
$addFields: { my_new_id_name: "$_id" }
},
{
$project: { _id: 0 }
}

db.report.aggregate(
{
$group: {_id: '$name'}
},
{
$project:{
name:"$_id",
_id:false} }
)

Starting in Mongo 4.2, you can use a combination of $set / $unset stages:
// { x: 1, z: "a" }
// { x: 2, z: "b" }
db.collection.aggregate([
{ $set: { y: "$x" } },
{ $unset: "x" }
])
// { y: 1, z: "a" }
// { y: 2, z: "b" }
The $set stage adds the new field to documents and the $unset stage removes/excludes the field to be renamed from documents.

if you are using find method you can't do this, but if you using aggregation it is very easy like this:
db.collectionName.aggregate([
{
$project: {
newName: "$existingKeyName"
}
}
]);

As all of the answers are written the solution in MongoDB query despite the question seeks the solution in Java, posting my approach using Java for posterities.
After the grouping, we can rename the _id fieldname using
Projections.computed("<expected field name>", "$_id")))
To Transform the core part of the query mentioned in the question to Java
Bson mainIdMatch = match(eq("main_id", new ObjectId("58f0f67f50c6af16709fd2c7")));
Bson group = Aggregates.group("$name", Accumulators.sum("count", 1L));
Bson project = Aggregates.project(Projections.fields(Projections.excludeId(),
Projections.computed("name", "$_id")));
reportMongoCollection.aggregate(Arrays.asList(mainIdMatch, group, project))
.into(new ArrayList<>());
To answer specifically, I have added an excerpt from the above code snippet, where I am renaming _id field value as name using Projections.computed("name", "$_id") which map the values of _id which we got as a result of grouping to the field called name. Also, we should exclude the id using Projections.excludeId().
Aggregates.project(Projections.fields(Projections.excludeId(),
Projections.computed("name", "$_id")))

Group by Field MongoDB Java Driver 3+

I would like to apply a group operation on my entire Document using MongoDB Java Driver 3.0
My query is something like:
db.coll.group( { key: {"field": 1}, cond: {}, reduce: function(curr, result){}, initial: {} } )
Results are:
{
"field" : "A61038968K16X275KNWCEIHr"
},
{
"field" : "AH1038968716P3210C6NiQpD"
},
{
"field" : "AV1038968F16Q321DCxY7T6w"
},
{
"field" : "A71038968K165321PLiEhbGJ"
},
{
"field" : "AY1038968N16w321a537co1U"
},
{
"field" : "AJ1038968E16S3212MJpeNNV"
}
I'm trying things in Java like : collection.aggregate(group("field")) but it doesn't work. Sorry if it's easy but I can't find anything googling.
Thanks!

So, what I was looking for was a 'distinct' method.
My solution was:
collection.distinct(<string field>, <query>, <result class>);
For example, in my case, my result class is a String.
collection.distinct("field_grouping",gte("field_query", 1000),String.class);
After that you can iterate your results and do whatever you want:
Iterator<String> iterator = collection.distinct("field_grouping",gte("field_query", 1000),String.class).iterator();

MongoDB compare document fields in aggregation pipeline

I have a collection of documents like the following:
{ name : "John" ,
age : 25.0 ,
bornIn : "Milan" ,
city : [ {
name : "Roma" ,
state : "IT" ,
mayor : "John"
}]
}
{ name : "Jim" ,
age : 35.0 ,
bornIn : "Madrid" ,
city : [ {
name : "Madrid" ,
state : "ESP" ,
mayor : "Jim"
}]
}
I want to retrieve all the documents that have the field $bornIn equal to the field $city.name. I need to do this as an intermediate stage of a pipeline, so I can't use the $where operator.
I searched online and I found a suggestion to implement something like this:
{ $project:
{ matches:
{ $eq:[ '$bornIn', '$city.name' ] }
}
},
{ $match:
{ matches:true }
} )
But it didn't work neither via shell nor via Java driver as it marks the fields as different.
For the sake of completeness I report my code:
final DBObject eq = new BasicDBObject();
LinkedList eqFields = new LinkedList();
eqFields.add("$bornIn");
eqFields.add("$city.name");
eq.put("$eq", eqFields);
projectFields.put("matches", eq);
final DBObject proj = new BasicDBObject("$project", projectFields);
LinkedList agg = new LinkedList();
agg.add(proj);
final AggregationOutput aggregate = table.aggregate( agg);
Do you have any suggestion? I'm using MongoDB 3.2, and I need to do this via Java Driver.
Thanks!!
PS. It is not relevant but actually the documents above are the output of a $lookup stage among collections "cities" and "persons", with join on $name/$mayor.. it is super cool!! :D :D

I'm a little rusty on how Mongo deals with deep equality searching arrays of objects, but this is definitely doable with $unwind
db.foo.aggregate([
{$unwind: "$city"},
{ $project:
{ matches:
{ $eq:[ '$bornIn', '$city.name' ] }
}
},
{ $match:
{ matches:true }
}
]);
I'm not on a computer with Mongo right now, so my syntax might be off a bit.

How to count items in nested array of mongo document using spring data?

I have to count the number of properties my Product has. I am using following document structure:
{
"_id": ObjectId("0000000000000000000002"),
"name" : "Test",
"properties" : [
ObjectId("0000000000000000000003"),
ObjectId("0000000000000000000004")
]
}
How to do it?
Update:
My try:
...
#Autowired
private MongoOperations mongoOperations;
...
mongoOperations.aggregate(
new BasicQuery("{$match: {\"_id\" : " + new ObjectId(productId) + "}}," +
"{$unwind: \"$properties\"}," +
"{$project: {count:{$add:1}}}," +
"{$group: {_id: null, number: {$sum: \"$count\" }}} "),
Product.class);
I am getting error from IDE:
aggregate in MaongoOperations cannot be applied to or.springframework.data.mongodb.core.query.BasicQuery
How to fix it? Is it the easiest way to count properties using Sparing data?

If you are using MongoDB version 2.6 or later, you can make user of the $size aggregation operator to get the array size:
db.products.aggregate(
[
{
$project: {
name: 1,
numberOfProperties: { $size: "$properties" }
}
}
]
)