Develop Reference

MongoDB - find() calls getting stuck sometimes - Timeout

We are using MongoDb for saving and fetching data.
All calls that are putting data into collections are working fine and are through common method.
All calls that are fetching data from collections are working fine sometimes and are through common method.
But Sometimes, only for one of the collection, i get my calls being stuck for forever, consuming CPU usage. I have to manually kill the threads otherwise it consumes my whole CPU.
Mongo Connection
MongoClient mongo = new MongoClient(hostName , Integer.valueOf(port));
DB mongoDb = mongo.getDB(dbName);
Code To fetch
DBCollection collection = mongoDb.getCollection(collectionName);
DBObject dbObject = new BasicDBObject("_id" , key);
DBCursor cursor = collection.find(dbObject);
Though i have figured out the collection for which it is causing issues, but how can i improve upon this, since it is occurring for this particular collection and sometimes.
EDIT
Code to save
DBCollection collection = mongoDb.getCollection(collectionName);
DBObject query = new BasicDBObject("_id" , key);
DBObject update = new BasicDBObject();
update.put("$set" , JSON.parse(value));
collection.update(query , update , true , false);
Bulk Write / collection
DB mongoDb = controllerFactory.getMongoDB();
DBCollection collection = mongoDb.getCollection(collectionName);
BulkWriteOperation bulkWriteOperation = collection.initializeUnorderedBulkOperation();
Map<String, Object> dataMap = (Map<String, Object>) JSON.parse(value);
for (Entry<String, Object> entrySet : dataMap.entrySet()) {
BulkWriteRequestBuilder bulkWriteRequestBuilder = bulkWriteOperation.find(new BasicDBObject("_id" ,
entrySet.getKey()));
DBObject update = new BasicDBObject();
update.put("$set" , entrySet.getValue());
bulkWriteRequestBuilder.upsert().update(update);
}
How can i set timeout for fetch calls..??

A different approach is to use the proposed method for MongoDB 3.2 Driver. Keep in mind that you have to update your .jar libraries (if you haven't) to the latest version.
public final MongoClient connectToClient(String hostName, String port) {
try {
MongoClient client = new MongoClient(hostName, Integer.valueOf(port));
return client;
} catch(MongoClientException e) {
System.err.println("Cannot connect to Client.");
return null;
}
}
public final MongoDatabase connectToDB(String databaseName) {
try {
MongoDatabase db = client.getDatabase(databaseName);
return db;
} catch(Exception e) {
System.err.println("Error in connecting to database " + databaseName);
return null;
}
public final void closeConnection(MongoClient client) {
client.close();
}
public final void findDoc(MongoDatabase db, String collectionName) {
MongoCollection<Document> collection = db.getCollection(collectionName);
try {
FindIterable<Document> iterable = collection
.find(new Document("_id", key));
Document doc = iterable.first();
//For an Int64 field named 'special_id'
long specialId = doc.getLong("special_id");
} catch(MongoException e) {
System.err.println("Error in retrieving document.");
} catch(NullPointerException e) {
System.err.println("Document with _id " + key + " does not exist.");
}
}
public final void insertToDB(MongoDatabase db, String collectioName) {
try {
db.getCollection(collectionName).insertOne(new Document()
.append("special_id", 5)
//Append anything
);
catch(MongoException e) {
System.err.println("Error in inserting new document.");
}
}
public final void updateDoc(MongoDatabase db, String collectionName, long id) {
MongoCollection<Document> collection = db.getCollection(collectionName);
try {
collection.updateOne(new Document("_id", id),
new Document("$set",
new Document("special_id",
7)));
catch(MongoException e) {
System.err.println("Error in updating new document.");
}
}
public static void main(String[] args) {
String hostName = "myHost";
String port = "myPort";
String databaseName = "myDB";
String collectionName = "myCollection";
MongoClient client = connectToClient(hostName, port);
if(client != null) {
MongoDatabase db = connectToDB(databaseName);
if(db != null) {
findDoc(db, collectionName);
}
client.closeConnection();
}
}
EDIT: As the others suggested, check from the command line if the procedure of finding the document by its ID is slow too. Then maybe this is a problem with your hard drive. The _id is supposed to be indexed but for better or for worse, re-create the index on the _id field.

The answers posted by others are great, but did not solve my purpose.
Actually issue was in my existing code itself , my cursor was waiting in while loop infinite time.
I was missing few checks which has been resolved now.

Just some possible explanations/thoughts.
In general "query by id" has to be fast since _id is supposed to be indexed, always. The code snippet looks correct, so probably the reason is in mongo itself. This leads me to a couple of suggestions:
Try to connect to mongo directly from the command line and run the "find" from there. The chances are that you'll still be able to observe occasional slowness.
In this case:
Maybe its about the disks (maybe this particular server is deployed on the slow disk or at least there is a correlation with some slowness of accessing the disk).
Maybe your have a sharded configuration and one shard is slower than others
Maybe its a network issue that occurs sporadically. If you run mongo locally/on staging env. with the same collection does this reproduce?
Maybe (Although I hardly believe that) the query runs in sub un-optimal way. In this case you can use "explain()" as someone has already suggested here.
If you happen to have replica set, please figure out what is the [Read Preference]. Who knows, maybe you prefer to get this id from the sub-optimal server

Mongodb Java cmd find().pretty() not working

I have Mongo set up and running. I have a problem where I cannot see any data being created in Mongo client in cmd prompt.
I am using Mongo 3.2.1 Java driver and Android Studio.
I have connected the port, address and working. Using the code below to create a database and collection works. But when I save the data it's not being saved (sometimes it works and sometimes not) and I don't know why.
MongoClient mongoClient = new MongoClient("xxx.xxx.x.x", xxxxx);
MongoDatabase database = mongoClient.getDatabase("testdb");
try {
database.createCollection("cars");
} catch (MongoCommandException e) {
database.getCollection("cars").drop();
}
List<Document> writes = new ArrayList<>();
MongoCollection<Document> carsCol = database.getCollection("cars");
Document d1 = new Document("_id", 1);
d1.append("name", "Audi");
d1.append("price", 52642);
writes.add(d1);
Document d2 = new Document("_id", 2);
d2.append("name", "Mercedes");
d2.append("price", 57127);
writes.add(d2);
Document d3 = new Document("_id", 3);
d3.append("name", "Skoda");
d3.append("price", 9000);
writes.add(d3);
Document d4 = new Document("_id", 4);
d4.append("name", "Volvo");
d4.append("price", 29000);
writes.add(d4);
Document d5 = new Document("_id", 5);
d5.append("name", "Bentley");
d5.append("price", 350000);
writes.add(d5);
Document d6 = new Document("_id", 6);
d6.append("name", "Citroen");
d6.append("price", 21000);
writes.add(d6);
Document d7 = new Document("_id", 7);
d7.append("name", "Hummer");
d7.append("price", 41400);
writes.add(d7);
Document d8 = new Document("_id", 8);
d8.append("name", "Volkswagen");
d8.append("price", 21600);
writes.add(d8);
carsCol.insertMany(writes);
MongoIterable<String> dbs = mongoClient.listDatabaseNames();
System.out.println("The Following are your Databases!");
for(String checkDBS : dbs){
System.out.println(checkDBS);
}
mongoClient.close();
}
After running the above I check the data by querying them but it doesn't seem to do anything. Sometimes it shows and at times, the commands don't do anything.
UPDATED ===============================================================
It does actually stores the data but I still cant query using cmd but I tried querying with Java and it worked
iterable.forEach(new Block<Document>() {
#Override
public void apply(final Document document) {
System.out.println(document);
}
});
I don't know why cmd does not work.

In your MongoShell you are using different DB than in your program.
You are trying to list all records in the cars database. But the collection cars is in the testdb database.
Proper sequence is:
use testdb
db.cars.find().pretty()
and you'll see your records.

Avoiding duplicate entries in a mongoDB with Java and JSON objects

I´m developing an analyzing program for Twitter Data.
I´m using mongoDB and at the moment. I try to write a Java program to get tweets from the Twitter API and put them in the database.
Getting the Tweets already works very well, but I have a problem when I want to put them in the database. As the Twitter API often returns just the same Tweets, I have to place some kind of index in the database.
First of all, I connect to the database and get the collection related to the search-term, or create this collection if this doesn´t exist.
public void connectdb(String keyword)
{
try {
// on constructor load initialize MongoDB and load collection
initMongoDB();
items = db.getCollection(keyword);
BasicDBObject index = new BasicDBObject("tweet_ID", 1);
items.ensureIndex(index);
} catch (MongoException ex) {
System.out.println("MongoException :" + ex.getMessage());
}
}
Then I get the tweets and put them in the database:
public void getTweetByQuery(boolean loadRecords, String keyword) {
if (cb != null) {
TwitterFactory tf = new TwitterFactory(cb.build());
Twitter twitter = tf.getInstance();
try {
Query query = new Query(keyword);
query.setCount(50);
QueryResult result;
result = twitter.search(query);
System.out.println("Getting Tweets...");
List<Status> tweets = result.getTweets();
for (Status tweet : tweets) {
BasicDBObject basicObj = new BasicDBObject();
basicObj.put("user_name", tweet.getUser().getScreenName());
basicObj.put("retweet_count", tweet.getRetweetCount());
basicObj.put("tweet_followers_count", tweet.getUser().getFollowersCount());
UserMentionEntity[] mentioned = tweet.getUserMentionEntities();
basicObj.put("tweet_mentioned_count", mentioned.length);
basicObj.put("tweet_ID", tweet.getId());
basicObj.put("tweet_text", tweet.getText());
if (mentioned.length > 0) {
// System.out.println("Mentioned length " + mentioned.length + " Mentioned: " + mentioned[0].getName());
}
try {
items.insert(basicObj);
} catch (Exception e) {
System.out.println("MongoDB Connection Error : " + e.getMessage());
loadMenu();
}
}
// Printing fetched records from DB.
if (loadRecords) {
getTweetsRecords();
}
} catch (TwitterException te) {
System.out.println("te.getErrorCode() " + te.getErrorCode());
System.out.println("te.getExceptionCode() " + te.getExceptionCode());
System.out.println("te.getStatusCode() " + te.getStatusCode());
if (te.getStatusCode() == 401) {
System.out.println("Twitter Error : \nAuthentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect.\nEnsure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.");
} else {
System.out.println("Twitter Error : " + te.getMessage());
}
loadMenu();
}
} else {
System.out.println("MongoDB is not Connected! Please check mongoDB intance running..");
}
}
But as I mentioned before, there are often the same tweets, and they have duplicates in the database.
I think the tweet_ID field is a good field for an index and should be unique in the collection.

Set the unique option on your index to have MongoDb enforce uniqueness:
items.ensureIndex(index, new BasicDBObject("unique", true));
Note that you'll need to manually drop the existing index and remove all duplicates or you won't be able to create the unique index.

This question is already answered but I would like to contribute a bit since MongoDB API 2.11 offers a method which receives unique option as a parameter:
public void ensureIndex(DBObject keys, String name, boolean unique)
A minor remind to someone who would like to store json documents on MongoDBNote is that uniqueness must be applied to a BasicObject key and not over values. For example:
BasicDBObject basicObj = new BasicDBObject();
basicObj.put("user_name", tweet.getUser().getScreenName());
basicObj.put("retweet_count", tweet.getRetweetCount());
basicObj.put("tweet_ID", tweet.getId());
basicObj.put("tweet_text", tweet.getText());
basicObj.put("a_json_text", "{"info_details":{"info_id":"1234"},"info_date":{"year":"2012"}, {"month":"12"}, {"day":"10"}}");
On this case, you can create unique index only to basic object keys:
BasicDBObject index = new BasicDBObject();
int directionOrder = 1;
index.put("tweet_ID", directionOrder);
boolean isUnique = true;
items.ensureIndex(index, "unique_tweet_ID", isUnique);
Any index regarding JSON value like "info_id" would not work since it´s not a BasicObject key.
Using indexes on MongDB is not as easy as it sounds. You may also check MongoDB docs for more details here Mongo Indexing Tutorials and Mongo Index Concepts. Direction order might be pretty important to understand once you need a composed index which is well explained here Why Direction order matter.

No exception for duplicates with gmongo

I have tried to insert duplicates into a mongodb database with both Groovy and Java. Java raises an exception while the groovy version just quietly ignores to insert the duplicate.
Here is some example code that illustrate this. I am running on OsX, Mongodb version 2.4.4. Can anyone enlighten me on this? Most grateful!
package chapter3
import com.gmongo.GMongo
import com.mongodb.BasicDBObject
import com.mongodb.DB
import com.mongodb.DBCollection
import com.mongodb.MongoClient
class TweetArchiveWithJava {
static void main(String[] args) {
new TweetArchiveWithJava()
}
private static final int ASCENDING = 1
TweetArchiveWithJava() {
duplicateInsertGroovy()
duplicateInsertJava()
}
def duplicateInsertJava() {
println "Inserting duplicates with Java"
MongoClient mongoClient = new MongoClient()
DB db = mongoClient.getDB("twitter-archive")
DBCollection tweets = db.getCollection("tweets")
tweets.remove(new BasicDBObject())
tweets.ensureIndex(new BasicDBObject("last_name", ASCENDING), "unique_index", true)
BasicDBObject insertedRecord = new BasicDBObject("last_name", "jones")
BasicDBObject duplicate = new BasicDBObject("last_name", "jones")
tweets.insert(insertedRecord)
System.out.println("Inserted first one")
printAllTweets(tweets)
tweets.insert(duplicate)
System.out.println("What?!! Should not be able to insert duplicates.")
printAllTweets(db)
}
private void duplicateInsertGroovy() {
println "Inserting duplicates with Groovy"
def mongo = new GMongo("127.0.0.1", 27017)
def db = mongo.getDB("twitter-archive")
DBCollection tweets = db.getCollection("tweets")
tweets.remove([:])
tweets.ensureIndex(new BasicDBObject("last_name", ASCENDING), "unique_index", true)
def jones = [last_name: "jones"]
tweets.insert(jones)
println "Inserted first Jones"
def duplicate = [last_name: "jones"]
tweets.insert(duplicate)
println "Succeeded inserting duplicate"
println "But only one record is found"
printAllTweets(tweets)
println "\n"
}
def printAllTweets(tweets) {
def cursor = tweets.find()
cursor.each { println it }
}
}

First, You you can use WriteResult for error indication:
def jones = [last_name: "jones"]
WriteResult rs = tweets.insert(jones)
println rs.error
//println "Inserted first Jones"
def duplicate = [last_name: "jones"]
rs = tweets.insert(duplicate)
//println "Succeeded inserting duplicate"
println rs.error
Prints:
null
E11000 duplicate key error index: twitter.tweets.$unique_index dup key: { : "jones" }
The reason for the difference is the WriteConcern:
The java method code uses {w=1} (acknowledged)
Whereas groovy method code uses {w=0} (normal)
Form DBTCPConnector
if ( concern.callGetLastError() ){
return _checkWriteError( db , port , concern );
}
else {
return new WriteResult( db , port , concern );
}
and in WriteConcern :
public boolean callGetLastError(){
if (_w instanceof Integer)
return (Integer) _w > 0;
return _w != null;
}
So the java method calls _checkWriteError behind the scenes - where an exception is thrown - and the groovy method just returns the write result.
The reason that you have different write concerns is initialization code:
You initialize MongoClient mongoClient = new MongoClient() in your java method, which internally uses WriteConcern.ACKNOWLEDGED.
In your groovy method, the GMongo constructor uses Mongo deprecated constructor which internally uses WriteConcern.NORMAL.
If you change tweets.insert(duplicate) to tweets.insert(duplicate, WriteConcern.ACKNOWLEDGED), you'll get an exception for the groovy method too

Not sure, but what happens if you change the groovy version to:
def jones = [last_name: "jones"] as BasicDBObject
tweets.insert(jones)
println "Inserted first Jones"
def duplicate = [last_name: "jones"] as BasicDBObject
tweets.insert(duplicate)

You can also use com.gmongo.GMongoClient. It has the same constructors as com.mongodb.MongoClient.

Combine for loop and sql in java

I have 2 input dates: myStartDate,myEndDate and a table TEST_TABLE with columns
TEST_ID, TEST_USER,TEST_START, TEST_END
I need to check if the range of dates between myStartDate and myEndDate have corresponding records in the TEST_TABLE.
I also need to ensure that I don't retrieve duplicate records.
Here's a sample of the logic I have so far:
Assuming,
myStartDate=06/06/2012;myEndDate=06/09/2012
int diff = myEndDate - myStartDate; //In this case = 3
String myQuery = "SELECT * FROM TEST_TABLE WHERE"+ myStartDate +"BETWEEN TEST_START AND TEST_END OR "+ (myStartDate +1) +" BETWEEN TEST_START AND TEST_END OR"+ (myStartDate+2)+"BETWEEN TEST_START AND TEST_END OR"+(myStartDate+3)+"BETWEEN TEST_START AND TEST_END";
List <TestTableData> myList = new List();
//Exceute query & save results into myList using add method
Want to know if there's any way to test the range of dates between myStartDate &myEndDate using a for loop in java code, instead of the approach used above in myQuery.Also, how can I avoid duplicates.
New to Java so any help would be appreciated!

Use a ResultSet to iterate over the output, like the code below.
while (res.next()) {
String col1= res.getString("col1");
String col2 = res.getString("col2");
}
If you use an Array implementation , it does not allow for duplicate elements and hence there is no need to check for one.
But if you must use a list then , you could use the following code to remove any duplicate elements.
public static void removeDuplicates(List list)
{
Set set = new HashSet();
List newList = new ArrayList();
for (Iterator iter = list.iterator(); iter.hasNext(); ) {
Object element = iter.next();
if (set.add(element))
newList.add(element);
}
list.clear();
list.addAll(newList);
}

I think what you are asking are some generic questions about how to read a database and how to handle dates in java. I will give you some sample code below. But I suggest you look at the java database tutorial http://docs.oracle.com/javase/tutorial/jdbc/index.html and the java.util.Date api doc http://docs.oracle.com/javase/1.5.0/docs/api/java/sql/Date.html for more info.
Here is some sample code that specifically demonstrates how to implement your question:
// get the input dates
// they are hard coded in this example
// but would probably normally be passed in
String startDateStr = "2/3/03";
String endDateStr = "3/1/03";
// unfortunately, there are 2 "Date" classes in this code and you need to differentiate
// java.util.Date is the standard java class for manipulating dates
// java.sql.Date is used to handle dates in the database
// name conflicts like this are rare in Java
SimpleDateFormat dateFmt = new SimpleDateFormat("M/d/yy");
java.util.Date myStartDate = dateFmt.parse(startDateStr);
java.util.Date myEndDate = dateFmt.parse(endDateStr);
// conneect to the database
// I am using mysql and its driver class is "com.mysql.jdbc.Driver"
// if using a different database, you would use its driver instead
// make sure the jar containing the driver is in your classpath (library list)
// you also have to know the url string that connects to your database
Class.forName("com.mysql.jdbc.Driver").newInstance(); // loads the driver
Connection dbConn = DriverManager.getConnection(
"jdbc:mysql://localhost/testdb", "(db user)", "(db password)"
);
// get the database rows from the db table
// my table is named "testtable"
// my columns are named "DateStart" and "DateEnd"
Statement st = dbConn.createStatement();
String sqlStr = "Select * from testtable";
ResultSet rs = st.executeQuery(sqlStr);
// loop through the rows until you find a row with the right date range
boolean foundRange = false;
while (rs.next()) {
java.util.Date dbStartDate = rs.getDate("DateStart");
java.util.Date dbEndDate = rs.getDate("DateEnd");
if (myStartDate.before(dbStartDate)) continue;
if (myEndDate.after(dbEndDate)) continue;
foundRange = true;
break;
}
if (foundRange) {
// code that executes when range is found in db
} else {
// code that executes if range not found in db
}
dbConn.close();
Hope this helps you get started.