Develop Reference

Retrieve only the queried element in an object array in MongoDB using java

Suppose we have the following documents in a MongoDB collection:
{
"_id":ObjectId("562e7c594c12942f08fe4192"),
"shapes":[
{
"shape":"square",
"color":"blue"
},
{
"shape":"circle",
"color":"red"
}
]
},
{
"_id":ObjectId("562e7c594c12942f08fe4193"),
"shapes":[
{
"shape":"square",
"color":"black"
},
{
"shape":"circle",
"color":"green"
}
]
}
And the MongoDB query is
db.test.find({"shapes.color": "red"}, {_id: 0, 'shapes.$': 1});
Can someone tell me how to write it in Java?
I am using:
List<BasicDBObject> obj = new ArrayList<>();
obj1.add(new BasicDBObject("shapes.color", "red"));
List<BasicDBObject> obj1 = new ArrayList<>();
obj2.add(new BasicDBObject("shapes.$", "1"));
BasicDBObject parameters1 = new BasicDBObject();
parameters1.put("$and", obj1);
DBCursor cursor = table.find(parameters1,obj2).limit(500);
and I am not getting anything.

The syntax of the Mongo Shell find function is:
db.collection.find(query, projection)
query document Optional. Specifies selection filter using query operators. To return all documents in a collection, omit this parameter or pass an empty document ({}).
projection document Optional. Specifies the fields to return in the documents that match the query filter.
When translating this for execution by the Mongo Java driver you need to construct separate BasicDBObject instances for;
the query
the projection
Here's an example:
MongoCollection<Document> table = ...;
// {"shapes.color": "red"}
BasicDBObject query = new BasicDBObject("shapes.color", "red");
// {_id: 0, 'shapes.$': 1}
BasicDBObject projection = new BasicDBObject("shapes.$", "1").append("_id", 0);
FindIterable<Document> documents = table
// assign the query
.find(query)
// assign the projection
.projection(projection);
System.out.println(documents.first().toJson());
Given the sample documents included in your question the above code will print out:
{
"shapes": [
{
"shape": "circle",
"color": "red"
}
]
}
This is identical to the output from db.test.find({"shapes.color": "red"}, {_id: 0, 'shapes.$': 1});.

Java/Grails - MongoDB aggregation 16MB buffer size limit

I am trying to run mongo db aggregate query from java, but buffer size is exceeding 16MB. Is there any way to adjust the buffer size or any other workaround. I do not have the option to create collection in mongo server side and also I do not have any mongo utility like mongo.exe or mongoExport.exe in my client system.
Here is little part of code
if (!datasetObject?.isFlat && jsonFor != 'collection-grid'){
//mongoPipeline = new AggregateArgs (Pipeline = pipeline, AllowDiskUse = true, OutputMode = AggregateOutputMode.Cursor)
output= dataSetCollection.aggregate(pipeline)
}else{
output= dataSetCollection.aggregate(project)
}
I have 100K records with 30 field. When I query for 5 fields for all 100K records I get result(Success). But when I make a query for 100K records with all fields its throwing below error.
Issue is when I am trying to access all documents from collection including all fields of document its exceeding 16Mb limit size.
Actual Error:
com.mongodb.CommandFailureException: { "serverUsed" : "127.0.0.1:27017" , "errmsg" : "exception: aggregation result exceeds maximum document size (16MB)" , "code" : 16389 , "ok" : 0.0
How to resolve this issue?
Using MongoDB-3.0.6
Note: GridFS is not suitable for my criteria. Because I need to retrieve all documents in one request not one document.

When running the aggregation you can tell mongo to return a cursor. With the new APIs in the 3.0 Java driver that would look like this:
// Assuming MongoCollection
dataSetCollection.aggregate(pipeline).useCursor(true)
You might also need to tell it to use disk space on the server rather than doing it all in memory:
// Assuming MongoCollection
dataSetCollection.aggregate(pipeline).useCursor(true).allowDiskUse(true)
If you're using an older driver (or the old API in the new driver) those two options would look like this:
// Assuming DBCollection
dataSetCollection.aggregate(pipeline, AggregationOptions.builder()
.allowDiskUse(true)
.useCursor(true)
.build())
.useCursor(true)

There are two options to resolve this issue
1) use of $out which creates new collection and write result, Which is not good idea because this process is time consuming and complex to implement.
public class JavaAggregation {
public static void main(String args[]) throws UnknownHostException {
MongoClient mongo = new MongoClient();
DB db = mongo.getDB("databaseName");
DBCollection coll = db.getCollection("dataset");
/*
MONGO SHELL :
db.dataset.aggregate([
{ "$match": { isFlat : true } },
{ "$out": "datasetTemp" }
])
*/
DBObject match = new BasicDBObject("$match", new BasicDBObject("isFlat", true));
DBObject out = new BasicDBObject("$out", "datasetTemp");
AggregationOutput output = coll.aggregate(match, out);
DBCollection tempColl = db.getCollection("datasetTemp");
DBCursor cursor = tempColl.find();
try {
while(cursor.hasNext()) {
System.out.println(cursor.next());
}
} finally {
cursor.close();
}
}
}
2. Use of allowDiskUse(true) is very simple to implement and not even time consuming.
public class JavaAggregation {
public static void main(String args[]) throws UnknownHostException {
MongoClient mongo = new MongoClient();
DB db = mongo.getDB("databaseName");
DBCollection coll = db.getCollection("dataset");
/*
MONGO SHELL :
db.dataset.aggregate([
{ "$match": { isFlat : true } },
{ "$out": "datasetTemp" }
])
*/
DBObject match = new BasicDBObject("$match", new BasicDBObject("isFlat", true));
def dbObjArray = new BasicDBObject[1]
dbObjArray[0]= match
List<DBObject> flatPipeline = Arrays.asList(dbObjArray)
AggregationOptions aggregationOptions = AggregationOptions.builder()
.batchSize(100)
.outputMode(AggregationOptions.OutputMode.CURSOR)
.allowDiskUse(true)
.build();
def cursor = dataSetCollection.aggregate(flatPipeline,aggregationOptions)
try {
while(cursor.hasNext()) {
System.out.println(cursor.next());
}
}
finally {
cursor.close();
}
}
For more see here and here

Spring Data MongoDB and Bulk Update

I am using Spring Data MongoDB and would like to perform a Bulk Update just like the one described here: http://docs.mongodb.org/manual/reference/method/Bulk.find.update/#Bulk.find.update
When using regular driver it looks like this:
The following example initializes a Bulk() operations builder for the items collection, and adds various multi update operations to the list of operations.
var bulk = db.items.initializeUnorderedBulkOp();
bulk.find( { status: "D" } ).update( { $set: { status: "I", points: "0" } } );
bulk.find( { item: null } ).update( { $set: { item: "TBD" } } );
bulk.execute()
Is there any way to achieve similar result with Spring Data MongoDB ?

Bulk updates are supported from spring-data-mongodb 1.9.0.RELEASE. Here is a sample:
BulkOperations ops = template.bulkOps(BulkMode.UNORDERED, Match.class);
for (User user : users) {
Update update = new Update();
...
ops.updateOne(query(where("id").is(user.getId())), update);
}
ops.execute();

You can use this as long as the driver is current and the server you are talking to is at least MongoDB, which is required for bulk operations. Don't believe there is anything directly in spring data right now (and much the same for other higher level driver abstractions), but you can of course access the native driver collection object that implements the access to the Bulk API:
DBCollection collection = mongoOperation.getCollection("collection");
BulkWriteOperation bulk = collection.initializeOrderedBulkOperation();
bulk.find(new BasicDBObject("status","D"))
.update(new BasicDBObject(
new BasicDBObject(
"$set",new BasicDBObject(
"status", "I"
).append(
"points", 0
)
)
));
bulk.find(new BasicDBObject("item",null))
.update(new BasicDBObject(
new BasicDBObject(
"$set", new BasicDBObject("item","TBD")
)
));
BulkWriteResult writeResult = bulk.execute();
System.out.println(writeResult);
You can either fill in the DBObject types required by defining them, or use the builders supplied in the spring mongo library which should all support "extracting" the DBObject that they build.

public <T> void bulkUpdate(String collectionName, List<T> documents, Class<T> tClass) {
BulkOperations bulkOps = mongoTemplate.bulkOps(BulkOperations.BulkMode.UNORDERED, tClass, collectionName);
for (T document : documents) {
Document doc = new Document();
mongoTemplate.getConverter().write(document, doc);
org.springframework.data.mongodb.core.query.Query query = new org.springframework
.data.mongodb.core.query.Query(Criteria.where(UNDERSCORE_ID).is(doc.get(UNDERSCORE_ID)));
Document updateDoc = new Document();
updateDoc.append("$set", doc);
Update update = Update.fromDocument(updateDoc, UNDERSCORE_ID);
bulkOps.upsert(query, update);
}
bulkOps.execute();
}
Spring Mongo template is used to perform the update. The above code will work if you provide the _id field in the list of documents.

How to represent the following query(mongodb) in java

I have a JSON in MongoDB with the following structure:
{
id:"_234234",
"stationId":"ALM",
"storageData": {
}
}
To retrieve JSON with storageData equal to null, in MongoDB I query as:
db.collection.find({"storageData":{"$gt" : {}}})
It gives me list of JSON bodies with empty storageData. So how do I represent that in Java
BasicDBObject obj=new BasicDDBObject("storageData", new BasicDBObject("$gt",{}));
collection.find(obj);
I am getting an error near BasicDBObject("$gt",{}))...
How do I represent ("$gt",{}) in Java??

First understand that null is a valid value. This would be valid:
{
id:"_234234",
StationId:"ALM",
StorageData: null
}
and retrieving the document, asking for storagedata which is null would retrieve the doc with the id _234234.
If what you need is to check which documents DON'T have the key "storagedata" then use the $exist keyword or construct the query in this way:
db.yourcollection.find("this.storagedata == null")
I would do it via query, and not in Java because it would alleviate cpu time and memory.

All you want to to here is represent an empty object:
BasicDBObject query = new BasicDBObject(
"storageData", new BasicDBObject(
"$gt",new BasicDBObject()
)
);
Which of course produces the query:
{ "storageData" : { "$gt" : { } } }
So that's it, just call BasicDBObject without any arguments.

How to insert multiple documents at once in MongoDB through Java

I am using MongoDB in my application and was needed to insert multiple documents inside a MongoDB collection .
The version I am using is of 1.6
I saw an example here
http://docs.mongodb.org/manual/core/create/
in the
Bulk Insert Multiple Documents Section
Where the author was passing an array to do this .
When I tried the same , but why it isn't allowing , and please tell me how can I insert multiple documents at once ??
package com;
import java.util.Date;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.MongoClient;
public class App {
public static void main(String[] args) {
try {
MongoClient mongo = new MongoClient("localhost", 27017);
DB db = mongo.getDB("at");
DBCollection collection = db.getCollection("people");
/*
* BasicDBObject document = new BasicDBObject();
* document.put("name", "mkyong"); document.put("age", 30);
* document.put("createdDate", new Date()); table.insert(document);
*/
String[] myStringArray = new String[] { "a", "b", "c" };
collection.insert(myStringArray); // Compilation error at this line saying that "The method insert(DBObject...) in the type DBCollection is not applicable for the arguments (String[])"
} catch (Exception e) {
e.printStackTrace();
}
}
}
Please let me know what is the way so that I can insert multiple documents at once through java .

DBCollection.insert accepts a parameter of type DBObject, List<DBObject> or an array of DBObjects for inserting multiple documents at once. You are passing in a string array.
You must manually populate documents(DBObjects), insert them to a List<DBObject> or an array of DBObjects and eventually insert them.
DBObject document1 = new BasicDBObject();
document1.put("name", "Kiran");
document1.put("age", 20);
DBObject document2 = new BasicDBObject();
document2.put("name", "John");
List<DBObject> documents = new ArrayList<>();
documents.add(document1);
documents.add(document2);
collection.insert(documents);
The above snippet is essentially the same as the command you would issue in the MongoDB shell:
db.people.insert( [ {name: "Kiran", age: 20}, {name: "John"} ]);

Before 3.0, you can use below code in Java
DB db = mongoClient.getDB("yourDB");
DBCollection coll = db.getCollection("yourCollection");
BulkWriteOperation builder = coll.initializeUnorderedBulkOperation();
for(DBObject doc :yourList)
{
builder.insert(doc);
}
BulkWriteResult result = builder.execute();
return result.isAcknowledged();
If you are using mongodb version 3.0 , you can use
MongoDatabase database = mongoClient.getDatabase("yourDB");
MongoCollection<Document> collection = database.getCollection("yourCollection");
collection.insertMany(yourDocumentList);

As of MongoDB 2.6 and 2.12 version of the driver you can also now do a bulk insert operation. In Java you could use the BulkWriteOperation. An example use of this could be:
DBCollection coll = db.getCollection("user");
BulkWriteOperation bulk = coll.initializeUnorderedBulkOperation();
bulk.find(new BasicDBObject("z", 1)).upsert().update(new BasicDBObject("$inc", new BasicDBObject("y", -1)));
bulk.find(new BasicDBObject("z", 1)).upsert().update(new BasicDBObject("$inc", new BasicDBObject("y", -1)));
bulk.execute();

Creating Documents
There're two principal commands for creating documents in MongoDB:
insertOne()
insertMany()
There're other ways as well such as Update commands. We call these operations, upserts. Upserts occurs when there're no documents that match the selector used to identify documents.
Although MongoDB inserts ID by it's own, We can manually insert custom IDs as well by specifying _id parameter in the insert...() functions.
To insert multiple documents we can use insertMany() - which takes an array of documents as parameter. When executed, it returns multiple ids for each document in the array. To drop the collection, use drop() command. Sometimes, when doing bulk inserts - we may insert duplicate values. Specifically, if we try to insert duplicate _ids, we'll get the duplicate key error:
db.startup.insertMany(
[
{_id:"id1", name:"Uber"},
{_id:"id2", name:"Airbnb"},
{_id:"id1", name:"Uber"},
]
);
MongoDB stops inserting operation, if it encounters an error, to supress that - we can supply ordered:false parameter. Ex:
db.startup.insertMany(
[
{_id:"id1", name:"Uber"},
{_id:"id2", name:"Airbnb"},
{_id:"id1", name:"Airbnb"},
],
{ordered: false}
);

Your insert record format like in MongoDB that query retire from any source
EG.
{
"_id" : 1,
"name" : a
}
{
"_id" : 2,
"name" : b,
}
it is mongodb 3.0
FindIterable<Document> resulutlist = collection.find(query);
List docList = new ArrayList();
for (Document document : resulutlist) {
docList.add(document);
}
if(!docList.isEmpty()){
collectionCube.insertMany(docList);
}