This is my first shot at using Mongodb with the java drivers. I can query the database via command line using javascript and the Date() object, however, I am having trouble using the driver. Based on my query, can anybody see what the problem is? Thanks
Date current = new Date();
DBCollection coll = db.getCollection("messages");
BasicDBObject query = new BasicDBObject("created_on", new BasicDBObject("$gte", new Date(current.getYear(), current.getMonth(), current.getDate())).
append("created_on", new BasicDBObject("$lt", new Date(current.getYear(), current.getMonth() - 1, current.getDate()))));
System.out.println("Query: " + query);
DBCursor cursor = coll.find(query);
Query: { "created_on" : { "$gte" : { "$date" :
"2012-12-06T05:00:00.000Z"} , "created_on" : { "$lt" : { "$date" :
"2012-11-06T05:00:00.000Z"}}}}
P.S. In case it is not obvious, I'm trying to find all of the records within the last month.
Seems like you are constructing the query wrong. Please try the below one:
BasicDBObject query = new BasicDBObject("created_on", //
new BasicDBObject("$gte", new DateTime().toDate()).append("$lt", new DateTime().toDate()));
Datetime object is a library which simplies date manipulation in java. You can check that out.
http://joda-time.sourceforge.net/
Also morphia is a nice java object-document-mapper (ODM) framework for working with mongodb through java driver. It simplifies querying through java.
https://github.com/jmkgreen/morphia
Based on the query that was output, you are looking for a document with a field created_on that also has a child named created_on. I assume no such document exists. In other words, you query is not correctly formed.
Your query object should look like this:
BasicDBObject dateRange = new BasicDBObject ("$gte", new Date(current.getYear(), current.getMonth(), current.getDate());
dateRange.put("$lt", new Date(current.getYear(), current.getMonth() - 1, current.getDate());
BasicDBObject query = new BasicDBObject("created_on", dateRange);
Also, as a sidebar, you probably should avoid using the three-argument constructor of the java.util.Date class, as it is deprecated. When working with dates in the MongoDB Java driver, I typically use the java.util.Calendar class, and its getTime() method.
I have not used the Java driver for mongo before, but it seems that the query you have created is not correct.
Query: { "created_on" : { "$gte" : { "$date" : "2012-12-06T05:00:00.000Z"} , "created_on" : { "$lt" : { "$date" : "2012-11-06T05:00:00.000Z"}}}}
The query should in fact end up looking like:
Query: { "created_on" : {$gte: start, $lt: end}}
Where start and end are dates. It seems like the second time you refer to "created_on" is unnecessary and in fact might be breaking your query.
NOTE: I have not had the chance to test out this theory, but I am working from http://cookbook.mongodb.org/patterns/date_range/ which seems to be very relevant to the question at hand.
Jodatime lib is very userful, Please make use of DateTimeZone.UTC for timezone parameter of DateTime. Once you set timezone, you will get accurate results. Try this
Calendar cal = Calendar.getInstance();
//get current year,month & day using Calender
int year=cal.get(Calendar.YEAR);
int monthNumber=cal.get(Calendar.MONTH);
int dateNumber=cal.get(Calendar.DAY_OF_MONTH);
monthNumber+=1;
BasicDBObject query = new BasicDBObject("dateCreated",new BasicDBObject("$gte", new DateTime(year, monthNumber, dateNumber, 0, 0,DateTimeZone.UTC).toDate()).append("$lte",new DateTime(year, monthNumber, dateNumber, 23, 59,DateTimeZone.UTC).toDate()));
System.out.println("formed query: "+query);
DBCursor cursor = collection.find(query);
while(cursor.hasNext())
{
System.out.println("found doc in given time range: "+cursor.next().toString());
}
Related
I am using Hibernate Criteria API and using projection for my result shown below:
projList.add(Projections.property("router"), "router");
projList.add(Projections.property("date"), "date");
criteria.setProjection(Projections.distinct(projList));
Output:
2017-01-10 19:47:33.0 Router1
2017-01-11 20:45:59.0 Router1
2017-01-10 21:58:49.0 Router2
2017-01-10 21:59:00.0 Router2
This code works as expected but i want to run distinct function on the basis of unique date records, meaning distinct function should not consider time value but just the day value. So out put should be like:
2017-01-10 Router1
2017-01-11 Router1
2017-01-10 Router2
Any idea how to do that?
I found a solution, though not so clean..
a) Create a POJO class in which the projections results will be stored
public class Proj{
private Date date;
private String router;
// .. getters / setters
}
b) Use an 'sqlProjections' for the date property in your criteria API (native sql here .. im on MySql):
projList.add(Projections.sqlProjection("DATE_FORMAT(date, '%Y-%m-%d') as date", new String[]{"date"},new Type[] {new StringType()}));
This is enough to get the distinct values you are seeking though the date will be in a String format, if you want to transform it to date then:
c) Add a ResultTransformer to the criteria api:
.setResultTransformer(new ResultTransformer() {
public Object transformTuple(Object[] tuple, String[] aliases) {
SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd");
java.util.Date parsed = null;
try {
parsed = format.parse((String) tuple[1]);
} catch (ParseException e) {}
Proj p = new Proj();
p.setDate(parsed);
p.setRouter((String)tuple[0]);
return p;
}
)
Like i said this is not the prettiest solution but it worked for me.
The criteria call now returns an objects which encapsulated the results you seek.
I found a way for doing it:
Instead of
projList.add(Projections.property("date"), "date");
Use
projList.add(Projections.sqlProjection( "date(date) as date", new String[] {"date"}, new Type[] {StandardBasicTypes.STRING} ));
Where expression "date(date) as date" first "date" is mysql Function, "(date)" is HBM column name of date, as "as date" is return column name from mysql and type "date" will be bean name that will be populated.
I'm trying to create an aggregate query using mongotemplate where there's a grouping by date (i.e 2016-03-01) instead of datetime (i.e 2016-03-01 16:40:12).
The dateToString operation exists in the mongodb documentation it can be used to extract the date from the datetime using formatting:
https://docs.mongodb.org/manual/reference/operator/aggregation/dateToString/
but I get get it to work with mongotemplate - I get a NullPointerException.
(my db version is 3.2)
List<AggregationOperation> aggregationOperations = new ArrayList<AggregationOperation>();
aggregationOperations.add(
Aggregation.project("blabla", ...).
andExpression("dateToString('%Y-%m-%d',timeCreated).as("date"));
aggregationOperations.add(Aggregation.group("date").sum("blabla").as("blabla"));
AggregationResults<?> aggregationResults = this.mongoTemplate.aggregate(
Aggregation.newAggregation(aggregationOperations),
collectionName,
resultClass);
When I use dayOfMonth(timeCreated) to extract the day, there's no exception, but I couldn't find and example of how to make this work with dateToString. I tried without '' for the date format, and it also didn't work...
This is the exception I get:
java.lang.NullPointerException
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:226)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:194)
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:255)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:194)
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:255)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:194)
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:255)
at org.bson.BasicBSONEncoder.putIterable(BasicBSONEncoder.java:324)
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:263)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:194)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:136)
at com.mongodb.DefaultDBEncoder.writeObject(DefaultDBEncoder.java:36)
at com.mongodb.OutMessage.putObject(OutMessage.java:289)
at com.mongodb.OutMessage.writeQuery(OutMessage.java:211)
at com.mongodb.OutMessage.query(OutMessage.java:86)
at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:81)
at com.mongodb.DB.command(DB.java:320)
at com.mongodb.DB.command(DB.java:299)
at com.mongodb.DB.command(DB.java:374)
at com.mongodb.DB.command(DB.java:246)
at org.springframework.data.mongodb.core.MongoTemplate$2.doInDB(MongoTemplate.java:357)
at org.springframework.data.mongodb.core.MongoTemplate$2.doInDB(MongoTemplate.java:355)
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:442)
at org.springframework.data.mongodb.core.MongoTemplate.executeCommand(MongoTemplate.java:355)
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1497)
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1432)
EDIT:
Eventually we decided here on a different solution than what was suggested below, I'm writing it here in case anyone else finds it useful:
In addition to the "timeCreated" field which holds the datetime, we saved another field in the document: "date", that holds just the date (as long).
For example if "timeCreated" = "2015-12-24 16:36:06.657+02:00", then date is "2015-12-24 00:00:00", and we save 1449180000000.
Now we can simply group by "date".
You could try projecting the fields first by using the SpEL andExpression in the projection operation and then group by the new fields in the group operation:
Aggregation agg = newAggregation(
project()
.andExpression("year(timeCreated)").as("year")
.andExpression("month(timeCreated)").as("month")
.andExpression("dayOfMonth(timeCreated)").as("day"),
group(fields().and("year").and("month").and("day"))
.sum("blabla").as("blabla")
);
AggregationResults<BlaBlaModel> result =
mongoTemplate.aggregate(agg, collectionName, BlaBlaModel.class);
List<BlaBlaModel> resultList = result.getMappedResults();
You could try using the DateOperators.DateToString class
aggregationOperations.add(
Aggregation.project("blabla", ...).
and(DateOperators.DateToString.dateOf("timeCreated").toString("%Y-%m-%d"));
There are many similar questions asked. But not exactly similar to the issue i am facing. I have seen almost all the questions and answers around it
So the problem is
I got to insert a date field in my mongo collection
But I can't access the collection directly. I got to use a service. The service takes a string and returns me oid.
So once i construct the BasicDBObject I call toString on it and pass it on to my service.. I even tried inserting it directly in a test collection and mongo is complaining.
BasicDBObject document = new BasicDBObject();
long createdAtSinceEpoch = 0;
long expiresAtSinceEpoch = 0;
createdAtSinceEpoch = System.nanoTime();
Date createdAt = new Date(TimeUnit.NANOSECONDS.toMillis(createdAtSinceEpoch));
document.append("createdAt", createdAt);
expiresAtSinceEpoch = createdAtSinceEpoch + +TimeUnit.SECONDS.toNanos(30);
Date expiresAt = new Date(TimeUnit.NANOSECONDS.toMillis(expiresAtSinceEpoch));
document.append("expiresAt", expiresAt);
service.storeRecord(document.toString());
and the generated JSON String looks like
{
"createdAt": {
"$date": "2015-09-01T20:05:21.641Z"
},
"expiresAt": {
"$date": "2015-09-01T20:05:51.641Z"
}
and Mongo complains that
Unable to parse JSON : Date expecting integer milliseconds, at (3,17)
So If i pass milliseconds alone instead of date object in the document.append() method then it DOES NOT recognize this field as date and considers it as String but inserts into the collection
I need 2 things
1) I want the data to be inserted
2) I am planning to expire that row by adding an index to the expiresAt field. So I want mongo to recognize that its a date field
JSON makes a difference between a numeric field and a text field containing a number. The latter one is only recognized as a String; I assume that this is what you did when you thought you were giving your service the date as an integer. Unfortunately you didn’t show us the relevant code.
When I save the Date info as a non String format, I annotate the field in my DTO as below. This helps the MongoDB know that the field is to be treated as an ISO date which then would be useful for making range search etc.,
#DateTimeFormat(iso = ISO.DATE_TIME) private Date date;
Date date = new Date();
BasicDBObject date= new BasicDBObject("date", date);
Data.insert(date);
I have created an Elastic search index from a Mongo database.
The documents in Mongo have the following structure:
{
"_id" : ObjectId("525facace4b0c1f5e78753ea"),
"time" : ISODate("2013-10-17T09:23:56.131Z"),
"type" : "A",
"url" : "www.google.com",
"name" : "peter",
}
The index was created (apparently) without any problems.
Now, I am trying to use Elastic Search to retrieve the documents in the index between two dates. I have read that I have to use range queries, but I have tried many times things like
MatchQueryBuilder queryBuilder = QueryBuilders.matchQuery("name", "peter").type(Type.PHRASE).minimumShouldMatch("99%");
LocalDateTime toLocal = new LocalDateTime(2013,12,18, 0, 0);
Date to = toLocal.toDate();
LocalDateTime fromLocal = new LocalDateTime(2013,12,17, 0, 0);
Date from = fromLocal.toDate();
RangeQueryBuilder queryDate = QueryBuilders.rangeQuery("time").to(to).from(from);
FilterBuilder filterDate = FilterBuilders.queryFilter(queryDate);
srb = esH.client.prepareSearch("my_index");
srb.setQuery(queryBuilder);
srb.setFilter(filterDate);
sr = srb.execute().actionGet();
and I get 0 hits although there should be many results. I have tried to enter strings instead of dates, but same results.
When I perform a basic query without filters such as:
MatchQueryBuilder queryBuilder = QueryBuilders.matchQuery("name", "peter").type(Type.PHRASE).minimumShouldMatch("99%");
SearchRequestBuilder srb = esH.client.prepareSearch("my_index");
rb.setQuery(queryBuilder);
SearchResponse sr = srb.execute().actionGet();
I get hits with that look like this:
{
"_index" : "my_index",
"_type" : "type",
"_id" : "5280d3c2e4b05e95aa703e34",
"_score" : 1.375688, "_source" : {"type":["A"],"time":["Mon Nov 11 13:55:30 CET 2013"],"name":["peter"]}
}
Where the field time does not have the format ISODate("2013-10-17T09:23:56.131Z")anymore.
To sum up, what would be the Java code (and types) for querying between two dates (and times), taking into account the format?
You are probably passing the wrong field name to the range query at this line:
RangeQueryBuilder queryDate = QueryBuilders.rangeQuery("time").to(to).from(from);
It should probably be #timestamp (or the field you're using to store your timestamp) instead of time. Additionally, it seems that there is no time field in Elasticsearch for the example document you included. This also points to the issue that the time field wasn't converted correctly from Mongo to Elasticsearch.
Can you try
FilterBuilders.rangeFilter("#timestamp").from("from time").to("toTime")
This will work -
You can pass in Long timestamps to the gte and lte params.
QueryBuilders.rangeQuery("time").gte(startTime).lte(endTime);
Make sure to add an "L" at the end of the startTime and endTime, so that it knows its a long and not an int.
I'm using the MongoDB Java driver, and can't seem to get a query to work. I have a collection named "Questions", which has entries that look like:
{
"question" : "how are you",
"category" : "personal",
"processed" : false,
"training" : true
}
When running the Mongo command line client, the query
db.questions.find()
or
db.questions.find({"processed" : false, "training" : true})
results are returned as expected; however, my java code does the following:
DBObject queryObj = new BasicDBObject();
queryObj.put("processed", false);
queryObj.put("training", isTrain);
DBObject updateObj = new BasicDBObject();
queryObj.put("processed", true);
DBCursor cursor = mongoCollection.find(queryObj).limit(NUM_TO_LOAD);
mongoCollection.update(queryObj, updateObj);
and the cursor that is returned to me is empty/the update doesn't make any changes. If I remove the queryObj argument from the call to find, results are once again returned as expected. Am I doing something wrong here?
Thanks,
Chris Covert
looks like, you used the wrong variable on line 6. Shouldn't it be updateObj, instead of queryObj?
Hope isTrain is a boolean, and got correctly initiated with a value (true/false).