Develop Reference

Java Spring Mongo DB Bulk Upsert Performance

I am trying to upsert a POJO list about 50000 items using bulk operation api in Spring.
I am using following code block to upsert the records, however it is taking about 15 minutes to update the whole collection.
public void saveHistory(List<History> history) {
List<Pair<Query, Update>> updates = new ArrayList<>(history.size());
history.forEach(item -> {
Query query = Query.query(Criteria.where("Id").is(item.getUserId()));
Update update = new Update();
update.set("_id", item.getUserId());
update.set(“auditHistoryMap", item.getAuditHistoryMap());
updates.add(Pair.of(query, update));
});
BulkOperations bulkOperations = mongoTemplate.bulkOps(BulkOperations.BulkMode.UNORDERED, Historyclass);
bulkOperations.upsert(updates);
bulkOperations.execute();
}
However if the POJO were converted to Document type prior to upsert whole collection can be upserted in about 3 minutes .
public <T> void bulkUpdate(String collectionName, List<T> documents, Class<T> tClass) {
BulkOperations bulkOps = mongoTemplate.bulkOps(BulkOperations.BulkMode.UNORDERED, tClass, collectionName);
for (T document : documents) {
Document doc = new Document();
mongoTemplate.getConverter().write(document, doc);
Query query = new Query(Criteria.where("_id").is(doc.get("_id")));
Document updateDoc = new Document();
updateDoc.append("$set", doc);
Update update = Update.fromDocument(updateDoc, "_id");
bulkOps.upsert(query, update);
}
bulkOps.execute();
}
What is the reason for such disparity ?

How to distinct query in MongoDB with projection and filters?

I have list of attributes in mongo and I am querying some nested field. Here is my code,
public List<Brand> searchBrands(Request request) {
final MongoCollection<Document> collection = mongoDatabase.getCollection("shop");
final Document query = new Document();
final Document projection = new Document();
final List<Brand> brandList = new ArrayList<>();
query.append("_id", request.getId());
query.append("isActive", true);
if (request.Year() != null) {
query.append("attributes.name", "myYear");
query.append("attributes.value", request.getYear());
}
projection.append("brand.code", 1.0);
projection.append("brand.description", 1.0);
projection.append("_id", 0.0);
Block<Document> processBlock = document -> brandList.
add(Brand.builder().code(document.get("brand",Document.class).getString("code"))
.description(document.get("brand",Document.class).getString("description"))
.build());
collection.find(query).projection(projection).forEach(processBlock);
return brandList;}
Above code return results correctly, 72 item with same brand.code. But I want to fetch distinct according to brand.code How can I do that?

I'm not sure which mongodb client library you're using to create queries for mongodb;
I'm sharing the query that you can run in mongodb console to get the results you want. I hope you know how to create this query using your mongodb client library
db.shop.distinct('brand.code', myQuery)
//Replace myQuery with your query e.g. {isActive: true}

using $$ROOT inside Spring Data Mongodb for retrieving whole document

using mongodb shell, I am able to perform an aggregation query that retrieves the whole document.
In order to do that I use the $$ROOT variable.
db.reservations.aggregate([
{ $match : { hotelCode : "0360" } },
{ $sort : { confirmationNumber : -1 , timestamp: -1 } },
{ $group : {
_id : "$confirmationNumber",
timestamp :{$first : "$timestamp"},
fullDocument :{$first : "$$ROOT"}
}}
])
It retrieves objects whose content is confirmationNumber, timestamp, fullDocument.
fullDocument is the whole document.
I am wondering if it is possible to do the same with Spring-Data and the aggregation framework.
My java code is:
TypedAggregation<ReservationImage> aggregation = newAggregation(
ReservationImage.class,
match(where("hotelCode").is(hotelCode)),
sort(Direction.DESC,"confirmationNumber","timestamp"),
group("confirmationNumber").
first("timestamp").as("timestamp").
first("$$ROOT").as("reservationImage"));
List<myClass> items = mongoTemplate.aggregate(
aggregation,
myClass.class).getMappedResults();
the error is :
org.springframework.data.mapping.PropertyReferenceException: No property $$ found for type myClass
Do you have any ideas?
Thanks.

We created https://jira.spring.io/browse/DATAMONGO-954 to track the support for accessing System Variables from MongoDB Pipeline expressions.
Once that is in place, you should be able to write:
Aggregation agg = newAggregation( //
match(where("hotelCode").is("0360")), //
sort(Direction.DESC, "confirmationNumber", "timestamp"), //
group("confirmationNumber") //
.first("timestamp").as("timestamp") //
.first(Aggregation.ROOT).as("reservationImage") //
);

I've seen this sort of thing before and it is not just limited to variable names such as $$ROOT. Spring data has it's own ideas about how to map "properties" of the document in the pipeline. Another common problem is simply projecting a new or calculated field that essentially has a new "property" name to it that does not get recognized.
Probably the best approach is to "step down" from using the helper classes and methods and construct the pipeline as BSON documents. You can even get the underlying collection object and the raw output as a BSON document, yet still cast to your typed List at the end.
Mileage may vary to your actual approach,but essentially:
DBObject match = new BasicDBObject(
"$match", new BasicDBObject(
"hotelCode", "0360"
)
);
DBObject sort = new BasicDBObject(
"$sort", new BasicDBObject(
"cofirmationNumber", -1
).append("timestamp", -1)
);
DBObject group = new BasicDBObject(
"$group", new BasicDBObject(
"_id", "confirmationNumber"
).append(
"timestamp", new BasicDBObject(
"$first", "$timestamp"
)
).append(
"reservationImage", new BasicDBObject(
"$first", "$$ROOT"
)
)
);
List<DBObject> pipeline = Arrays.asList(match,sort,group);
DBCollection collection = mongoOperation.getCollection("collection");
DBObject rawoutput = (DBObject)collection.aggregate(pipeline);
List<myClass> items = new AggregationResults(List<myClass>, rawoutput).getMappedResults();
The main thing is moving away from the helpers that are getting in the way and constructing the pipeline as it should be free of the imposed restrictions.

How to insert multiple documents at once in MongoDB through Java

I am using MongoDB in my application and was needed to insert multiple documents inside a MongoDB collection .
The version I am using is of 1.6
I saw an example here
http://docs.mongodb.org/manual/core/create/
in the
Bulk Insert Multiple Documents Section
Where the author was passing an array to do this .
When I tried the same , but why it isn't allowing , and please tell me how can I insert multiple documents at once ??
package com;
import java.util.Date;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.MongoClient;
public class App {
public static void main(String[] args) {
try {
MongoClient mongo = new MongoClient("localhost", 27017);
DB db = mongo.getDB("at");
DBCollection collection = db.getCollection("people");
/*
* BasicDBObject document = new BasicDBObject();
* document.put("name", "mkyong"); document.put("age", 30);
* document.put("createdDate", new Date()); table.insert(document);
*/
String[] myStringArray = new String[] { "a", "b", "c" };
collection.insert(myStringArray); // Compilation error at this line saying that "The method insert(DBObject...) in the type DBCollection is not applicable for the arguments (String[])"
} catch (Exception e) {
e.printStackTrace();
}
}
}
Please let me know what is the way so that I can insert multiple documents at once through java .

DBCollection.insert accepts a parameter of type DBObject, List<DBObject> or an array of DBObjects for inserting multiple documents at once. You are passing in a string array.
You must manually populate documents(DBObjects), insert them to a List<DBObject> or an array of DBObjects and eventually insert them.
DBObject document1 = new BasicDBObject();
document1.put("name", "Kiran");
document1.put("age", 20);
DBObject document2 = new BasicDBObject();
document2.put("name", "John");
List<DBObject> documents = new ArrayList<>();
documents.add(document1);
documents.add(document2);
collection.insert(documents);
The above snippet is essentially the same as the command you would issue in the MongoDB shell:
db.people.insert( [ {name: "Kiran", age: 20}, {name: "John"} ]);

Before 3.0, you can use below code in Java
DB db = mongoClient.getDB("yourDB");
DBCollection coll = db.getCollection("yourCollection");
BulkWriteOperation builder = coll.initializeUnorderedBulkOperation();
for(DBObject doc :yourList)
{
builder.insert(doc);
}
BulkWriteResult result = builder.execute();
return result.isAcknowledged();
If you are using mongodb version 3.0 , you can use
MongoDatabase database = mongoClient.getDatabase("yourDB");
MongoCollection<Document> collection = database.getCollection("yourCollection");
collection.insertMany(yourDocumentList);

As of MongoDB 2.6 and 2.12 version of the driver you can also now do a bulk insert operation. In Java you could use the BulkWriteOperation. An example use of this could be:
DBCollection coll = db.getCollection("user");
BulkWriteOperation bulk = coll.initializeUnorderedBulkOperation();
bulk.find(new BasicDBObject("z", 1)).upsert().update(new BasicDBObject("$inc", new BasicDBObject("y", -1)));
bulk.find(new BasicDBObject("z", 1)).upsert().update(new BasicDBObject("$inc", new BasicDBObject("y", -1)));
bulk.execute();

Creating Documents
There're two principal commands for creating documents in MongoDB:
insertOne()
insertMany()
There're other ways as well such as Update commands. We call these operations, upserts. Upserts occurs when there're no documents that match the selector used to identify documents.
Although MongoDB inserts ID by it's own, We can manually insert custom IDs as well by specifying _id parameter in the insert...() functions.
To insert multiple documents we can use insertMany() - which takes an array of documents as parameter. When executed, it returns multiple ids for each document in the array. To drop the collection, use drop() command. Sometimes, when doing bulk inserts - we may insert duplicate values. Specifically, if we try to insert duplicate _ids, we'll get the duplicate key error:
db.startup.insertMany(
[
{_id:"id1", name:"Uber"},
{_id:"id2", name:"Airbnb"},
{_id:"id1", name:"Uber"},
]
);
MongoDB stops inserting operation, if it encounters an error, to supress that - we can supply ordered:false parameter. Ex:
db.startup.insertMany(
[
{_id:"id1", name:"Uber"},
{_id:"id2", name:"Airbnb"},
{_id:"id1", name:"Airbnb"},
],
{ordered: false}
);

Your insert record format like in MongoDB that query retire from any source
EG.
{
"_id" : 1,
"name" : a
}
{
"_id" : 2,
"name" : b,
}
it is mongodb 3.0
FindIterable<Document> resulutlist = collection.find(query);
List docList = new ArrayList();
for (Document document : resulutlist) {
docList.add(document);
}
if(!docList.isEmpty()){
collectionCube.insertMany(docList);
}

How to upsert with mongodb-java-driver

How can I upsert data into mongodb collection with java-driver?
I try (with empty collection):
db.getCollection(collection).update(new BasicDBObject("_id", "12"), dbobject, true, false);
But document was created with _id == ObjectID(...). Not with "12" value.
This code (js) add document with _id = "12" as expected
db.metaclass.update(
{ _id:12},
{
$set: {b:1}
},
{ upsert: true }
)
mongo-java-driver-2.11.2

If you are using mongo-java driver 3, following .updateOne() method with {upsert, true} flag works.
void setLastIndex(MongoClient mongo, Long id, Long lastIndexValue) {
Bson filter = Filters.eq("_id", id);
Bson update = new Document("$set",
new Document()
.append("lastIndex", lastIndexValue)
.append("created", new Date()));
UpdateOptions options = new UpdateOptions().upsert(true);
mongo.getDatabase(EventStreamApp.EVENTS_DB)
.getCollection(EventCursor.name)
.updateOne(filter, update, options);
}

You cannot set _id if dbobject is just a document and does not contain an update operator eg: $set, $setOnInsert.
Just passing a document will replace the whole document meaning it doesn't set an _id a falls back to ObjectId
So your example works if you use an update operator eg:
db.getCollection(collection).update(
new BasicDBObject("_id", "12"),
new BasicDBObject("$set", new BasicDBObject("Hi", "world")), true, false)

You can use the replaceOne method and specify the ReplaceOptions (since 3.7) :
private static final ReplaceOptions REPLACE_OPTIONS
= ReplaceOptions.createReplaceOptions(new UpdateOptions().upsert(true));
db.getCollection(collection).replaceOne(new BasicDBObject("_id", "12"), dbobject, REPLACE_OPTIONS);
For older versions you can directly pass the UpdateOptions to the replaceOne method :
private static final UpdateOptions UPDATE_POLICY = new UpdateOptions().upsert(true);
db.getCollection(collection).replaceOne(new BasicDBObject("_id", "12"), dbobject, UPDATE_POLICY);
As mentioned in the documentation :
replaceOne() replaces the first matching document in the collection
that matches the filter, using the replacement document.
If upsert: true and no documents match the filter, replaceOne()
creates a new document based on the replacement document.

This is to upsert with scala driver which i couldnot find in web
con.updateOne(
equal("vendor_id", vendorId),
inc("views_count", f.views),
UpdateOptions().upsert(true))
to do so import the following
import org.mongodb.scala.model.UpdateOptions