Develop Reference

How to fetch data in a single go from mongodb based on multiple filters?

I am new to mongodb and aggregation framework.
We have a class UserMetaData and a list of UserMetaData. I need to fetch data according to the userMetaDataList that is passed to the method solve().
Currently I am iterating the list and one by one fetching the corresponding collection from the monogdb. Since the db calls are made for each element in the list, this becomes a highly expensive operation.
Is there any way to fetch all the required data from mongodb in one shot(more like a bulk fetch operation).
mongodb - perform batch query the solution provided in this does not fulfill the requirements of the current scenario.
Please help!!
This is how I am doing currently.
class UserMetaData{
String userId;
String vehicleId;
String vehicleColour;
String orderId;
}
public List<String> getOrderIds(List<UserMetaData> userMetaDataList) {
List<String> orderIds = new ArrayList<>();
for (UserMetaData userMetadata : userMetaDataList) {
try {
BasicDBObject matchDBObject = new BasicDBObject("user_id", new BasicDBObject("$eq", userMetadata.getUserId()));
matchDBObject.append("vehicle_id", new BasicDBObject("$eq", userMetadata.getVehicleID()));
matchDBObject.append("vehicle_colour", new BasicDBObject("$in", ImmutableSet.of("WHITE", "BLACK")));
Document document = eventCollection.find(matchDBObject)
.projection(new BasicDBObject("order_id", "1"))
.first();
orderIds.add(document.get("order_id").toString());
} catch (Exception e) {
log.info("Exception occurred while fetching order id for user_id: {} asset_id:{} - {}", metadata.getUserId(), metadata.getAssetID(), e);
}
}
return ordersIds;
}
I want to fetch all the corresponding data in a single query.
Requesting help.

You can join all filters with $OR condition and fetch the full list at once ...

I want to fetch all the corresponding data in a single query.
You can use this approach and perform the query as a single operation (avoids the for-loop).
Consider sample documents in the collection test:
{ "_id" : ObjectId("621762e2cda7c6394d557f37"), "userid" : 1, "name" : "ijk", "orderid" : "11" }
{ "_id" : ObjectId("621762efcda7c6394d557f38"), "userid" : 12, "name" : "abc", "orderid" : "99" }
{ "_id" : ObjectId("621762fccda7c6394d557f39"), "userid" : 13, "name" : "xyz", "orderid" : "100" }
The array of objects to filter:
var DOCS = [
{ userid: 12, name: "abc" },
{ userid: 13, name: "xyz" }
]
The query to filter by DOCS:
db.test.find(
{
$expr: {
$in: [ { userid: "$userid", name: "$name" }, DOCS ]
}
},
{
orderid: 1
}
)
The output has documents with userids 12 and 13.
[ EDIT - ADD ]
This aggregation an improvement over the find:
db.test.aggregate([
// This matches the 'userid' and 'name' fields with the input list 'DOCS'
{
$match: {
$expr: {
$in: [ { userid: "$userid", name: "$name" }, DOCS ]
}
}
},
// The grouping will select only the first matching for the 'userid' and 'name'
// (this is as per the question post's code: `.first()`)
{
$group: {
_id: {
userid: "$userid",
name: "$name"
},
orderid: {
$first: "$orderid"
}
}
},
// Remove the '_id' field
// Now the result has just the 'orderid' field only
{
$unset: "_id"
}
])

How to parse MongoDB aggregation step containing functions?

Looking at com.mongodb.reactivestreams.client.MongoCollection interface we see that aggregation can be invoked using list of Bson elements.
public interface MongoCollection<TDocument> {
...
AggregatePublisher<TDocument> aggregate(List<? extends Bson> list);
}
It is clear how to use it when aggregation steps are JSONs (see Example with simple JSONs)
Unfortunately, when any aggregation step contains a function (which is allowed by native MongoDB query), for instance, $accumulator the same approach cannot be applied due to it causes violation of Bson format (org.bson.json.JsonParseException) (see Example with functions)
What is the best way to convert a native MongoDB aggregation query into a result in Java?
(suppose that queries are complex and it is not expedient to rewrite them with Mongo aggregation builders in Java)
Example with simple JSONs:
ReactiveMongoOperations mongo = /* ... */;
var match = BasicDBObject.parse("{ $match: {name: \"Jack\"} }");
var project = BasicDBObject.parse("{ $project: {_id: 0, age: 1, name: 1} }";
var queryParts = List.of(match, project);
Flux<PersonInfo> infoFlux = mongo
.getCollection("person")
.flatMapMany(person -> person.aggregate(queryParts).toFlux())
.map(it -> objectMapper.readValue(it.toJson(), PersonInfo.class))
.collectList()
Example with functions:
// here for conciseness it is just a counting accumulator; generally functions are more complex
var match = BasicDBObject.parse("""
{
$group: {
_id: "$token",
count: {
$accumulator: {
init: function() {
return {owned: 0, total: 0}
},
accumulate: function(state, owner) {
return {
total: state.total + 1
}
},
accumulateArgs: ["$owner"],
merge: function(a, b) {
return {
total: a.total + b.total
}
},
lang: "js"
}
},
minPriceEth: {$min: "$priceEth"}
}
}
""");

Translate mongodb aggregate query into Java/Kotlin Spring Data

So I came up with a query that I'm not really sure how to translate into Spring Data. This is the query:
db.collection.aggregate([{
{
$group: {
_id: "$field",
count: {
$sum: "$count"
},
data: {
"$addToSet": "$$ROOT"
}
}
},
{
$sort: {
count: -1
}
},
{
$limit: 10
},
{
$unwind: "$data"
}
])
The problematic part is unwind. In this raw query I'm putting the original document("$$ROOT") that is being processed into data, so after the set is being processed I can just unwind it and end up with just the documents that I really want. This works fine from the shell. The problem is that I'm not seeing how to do the same operation in Java/Kotlin, I can just add to the set without having a way to reference that data later. Could someone help me out and write how this would look in Java/Kotlin with Spring Data?

You can try this
public List<Object> test() {
Aggregation aggregation = Aggregation.newAggregation(
group("field")
.sum("count").as("count")
.addToSet("$$ROOT").as("data"),
sort(Sort.Direction.DESC, "count"),
limit(10),
unwind("data")
).withOptions(AggregationOptions.builder().allowDiskUse(Boolean.TRUE).build());
return mongoTemplate.aggregate(aggregation, mongoTemplate.getCollectionName(YOUR_COLLECTION.class), Object.class).getMappedResults();
}

java mongodb - get array length without downloading all data

I'm using mongodb to store data for my java program and I have a collection with an array field that has a lot of things in it but i want only to get the length, without all the other data.
Now i'm using this to get it:
((UUID[])document.get("customers")).length
How can I make this not to download all the array?
A possible answer is to create an int that counts the pushes and the pulls of the array but it's not the cleanest method.

You are looking for aggregation framework where you can use the $size operator in your pipeline, this counts and returns the total the number of items in an array:
db.collection.aggregate([
{
"$project": {
"_id": 0, "customer_count": { "$size": "$customers" }
}
}
]);
where the Java equivalent:
DBObject projectFields = new BasicDBObject("_id", 0);
projectFields.put("customer_count", new BasicDBObject( "$size", "$customers" ));
DBObject project = new BasicDBObject("$project", projectFields);
AggregationOutput output = db.getCollection("collectionName").aggregate(project);
System.out.println("\n" + output);

You can use MongoDB's Aggregation Framework to get the size of the array. For example, given the following document structure:
> db.macross.findOne()
{
"_id" : "SDF1",
"crew" : [
"Rick",
"Minmay",
"Roy",
"Max",
"Misa",
"Milia"
]
}
get the size of the array
> db.macross.aggregate(
{ $match: { _id: "SDF1" } },
{ $unwind: "$crew" },
{ $group: { _id: "", count: { $sum: 1 } } },
{ $project: { _id: 0, count: 1 } }
)
{ "count" : 6 }
More detailed and interesting examples are available in the docs.

unmarshall aggregate result with Jongo

I am using the MongoDB aggregate framework to query a document, the results is the following:
{
"result" : [
{
"_id" : "luke",
"times" : 8
},
{
"_id" : "albert",
"times" : 4
},
{
"_id" : "matt",
"times" : 4
}
],
"ok" : 1
}
As you can see from the result above, the query works in the mongoDB shell, but I have a problem when getting the results with Jongo:
Aggregationoutput =
gamesCollection.aggregate(
"{ ... }"
).as(Aggregation.class);
output.results().iterator().hasNext();
The main problem seems to be that Jongo doesn't allow me to use AggregationOutput? he wants instead Aggregation ... but can't find any example available on how to use it
EDIT:
I am a bit frustrated that I can't make Jongo to work with the aggregate. I had to write the query with DBObjects as specified in the MongoDB Java driver, but the code looks really ugly..
EDIT2:
Just to complete the information, this is the original aggregate I was using with Jongo which could not unmarshall to ResultObject
List<ResultObject> output =
gamesCollection.aggregate(
"{ $match: { 'playersList.playerid': 'bob' }},"
+"{ $unwind: '$playersList' },"
+"{ $match: { 'playersList.playerid': { $ne: 'bob' } } },"
+"{ $group: { _id: '$playersList.playerid', times: { $sum : 1} } },"
+"{ $sort: { times: -1 } }"
).as(ResultObject.class);
class ResultObject{
String _id;
int times;
}
}

You can use aggregate feature like find/findOne... Results are automatically unmarshalled into Pojo :
List<Email> emails = collection.aggregate("{$project:{sender:1}}")
.and("{$match:{tags:'read'}}")
.and("{$limit:10}")
.as(Email.class);
You can find more examples here : https://github.com/bguerout/jongo/blob/master/src/test/java/org/jongo/AggregateTest.java

I just saw this question now, but I hope it can help others.
You can create an inner class like this:
private static class AggregateResult {
String _id;
int time;
}
And call the aggregate function as following:
List<AggregateResult> res = gamesCollection.aggregate(
"{ ... }"
).as(AggregateResult.class);
Then you can iterate over the results in the res list.