I'm trying to create an aggregate query using mongotemplate where there's a grouping by date (i.e 2016-03-01) instead of datetime (i.e 2016-03-01 16:40:12).
The dateToString operation exists in the mongodb documentation it can be used to extract the date from the datetime using formatting:
https://docs.mongodb.org/manual/reference/operator/aggregation/dateToString/
but I get get it to work with mongotemplate - I get a NullPointerException.
(my db version is 3.2)
List<AggregationOperation> aggregationOperations = new ArrayList<AggregationOperation>();
aggregationOperations.add(
Aggregation.project("blabla", ...).
andExpression("dateToString('%Y-%m-%d',timeCreated).as("date"));
aggregationOperations.add(Aggregation.group("date").sum("blabla").as("blabla"));
AggregationResults<?> aggregationResults = this.mongoTemplate.aggregate(
Aggregation.newAggregation(aggregationOperations),
collectionName,
resultClass);
When I use dayOfMonth(timeCreated) to extract the day, there's no exception, but I couldn't find and example of how to make this work with dateToString. I tried without '' for the date format, and it also didn't work...
This is the exception I get:
java.lang.NullPointerException
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:226)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:194)
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:255)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:194)
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:255)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:194)
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:255)
at org.bson.BasicBSONEncoder.putIterable(BasicBSONEncoder.java:324)
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:263)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:194)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:136)
at com.mongodb.DefaultDBEncoder.writeObject(DefaultDBEncoder.java:36)
at com.mongodb.OutMessage.putObject(OutMessage.java:289)
at com.mongodb.OutMessage.writeQuery(OutMessage.java:211)
at com.mongodb.OutMessage.query(OutMessage.java:86)
at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:81)
at com.mongodb.DB.command(DB.java:320)
at com.mongodb.DB.command(DB.java:299)
at com.mongodb.DB.command(DB.java:374)
at com.mongodb.DB.command(DB.java:246)
at org.springframework.data.mongodb.core.MongoTemplate$2.doInDB(MongoTemplate.java:357)
at org.springframework.data.mongodb.core.MongoTemplate$2.doInDB(MongoTemplate.java:355)
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:442)
at org.springframework.data.mongodb.core.MongoTemplate.executeCommand(MongoTemplate.java:355)
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1497)
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1432)
EDIT:
Eventually we decided here on a different solution than what was suggested below, I'm writing it here in case anyone else finds it useful:
In addition to the "timeCreated" field which holds the datetime, we saved another field in the document: "date", that holds just the date (as long).
For example if "timeCreated" = "2015-12-24 16:36:06.657+02:00", then date is "2015-12-24 00:00:00", and we save 1449180000000.
Now we can simply group by "date".
You could try projecting the fields first by using the SpEL andExpression in the projection operation and then group by the new fields in the group operation:
Aggregation agg = newAggregation(
project()
.andExpression("year(timeCreated)").as("year")
.andExpression("month(timeCreated)").as("month")
.andExpression("dayOfMonth(timeCreated)").as("day"),
group(fields().and("year").and("month").and("day"))
.sum("blabla").as("blabla")
);
AggregationResults<BlaBlaModel> result =
mongoTemplate.aggregate(agg, collectionName, BlaBlaModel.class);
List<BlaBlaModel> resultList = result.getMappedResults();
You could try using the DateOperators.DateToString class
aggregationOperations.add(
Aggregation.project("blabla", ...).
and(DateOperators.DateToString.dateOf("timeCreated").toString("%Y-%m-%d"));
Related
Working with queryDSL and mySQL:
I have a field from in the time_check table to which I want to add timetrackAutoStop from the location table as hours and then compare it with the current time.
We managed to get queryDSL to understand the following code:
if (data.getAutostopTimeExceeded() != null) {
QLocationEntity location = QLocationEntity.locationEntity;
query.join(TIME_CHECK.location, location);
Expression<LocalDateTime> currentTimeExpr = Expressions.asDateTime(LocalDateTime.now());
TIME_CHECK.from, Expressions.asNumber(location.timetrackAutoStop));
predicates.add(
Expressions.predicate(Ops.GT,
currentTimeExpr,
Expressions.dateTimeOperation(LocalDateTime.class,
Ops.DateTimeOps.ADD_HOURS,
TIME_CHECK.from,
Expressions.asNumber(location.timetrackAutoStop)
)
)
);
}
However it then generates SQL which causes the mySQL database to complain that the ADD_HOURS function doesn't exist.
Is there are way to get what I'm looking for? I.E. all time_check_records older than now() minus timeTrackAutoStop ?
Update:
We tried the following, but get an error message stating: expecting CLOSE, found locationEntity:
QLocationEntity qLocationEntity = QLocationEntity.locationEntity;
query.join(TIME_CHECK.location, qLocationEntity);
Expression<LocalDateTime> currentTimeExpr = Expressions.asDateTime(LocalDateTime.now());
Expression<LocalDateTime> subExpr = Expressions.template(LocalDateTime.class,
"DATE_ADD({0}, INTERVAL {1} HOUR)",
TIME_CHECK.from,
qLocationEntity.timetrackAutoStop
);
predicates.add(Expressions.predicate(Ops.GT, currentTimeExpr, subExpr));
The MySQL database does not have a built-in function for adding hours to a date. However, you can use the DATE_ADD() function to add an interval of hours to a date. The syntax for this function is as follows:
DATE_ADD(date, INTERVAL expr unit)
Where date is the date to which you want to add the interval, expr is the number of hours to add, and unit is the unit of time (in this case, 'hour').
This is the code which I am using to fill the column in the db.
DateTimeFormatter dateFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSSSS");
JSONObject publishedObj = jsonObject.optJSONObject("created");
if(publishedObj != null){
String dateStr = publishedObj.getString("value");
book.setPublishedDate(LocalDate.parse(dateStr,dateFormat));
}
Below is the instance variable of the column where the data needs to go:
#Column("published_date")
#CassandraType(type = CassandraType.Name.DATE)
private LocalDate publishedDate;
Error Message which i am getting:
com.datastax.oss.driver.api.core.type.codec.CodecNotFoundException: Codec not found for requested operation: [TEXT <-> java.time.LocalDate]
Can please someone help.
Thankyou!!
I can reproduce that error with your code above. To remedy it, I have ALTERed the book_by_id table with two new columns:
ALTER TABLE book_by_id ADD pubdate2 TEXT;
ALTER TABLE book_by_id ADD pubdate3 DATE;
My BookEntity class for those columns looks like this:
#Column("pubdate2")
#CassandraType(type = CassandraType.Name.TEXT)
private String publishedDate2;
#Column("pubdate3")
#CassandraType(type = CassandraType.Name.DATE)
private LocalDate publishedDate3;
The code to parse and set the date looks like this:
DateTimeFormatter dateFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSSSS");
String dateStr = "2022-03-03T09:52:33.235555";
musicBook.setPublishedDate2(LocalDate.parse(dateStr,dateFormat).toString());
musicBook.setPublishedDate3(LocalDate.parse(dateStr,dateFormat));
template.insert(musicBook);
tl;dr;
Redefine published_date as a DATE type, and it will work. Besides, dates/times should be stored in date/time types in databases.
Note that Cassandra won't allow you to modify a column's data type. Also, the process of dropping and adding a column with the same name in quick succession has proven to be problematic with Cassandra in the past. I'd advise adding a newly named column of a DATE type, and reloading its data. Or recreate the table (with the correct data types) and reload the data.
I am using Hibernate Criteria API and using projection for my result shown below:
projList.add(Projections.property("router"), "router");
projList.add(Projections.property("date"), "date");
criteria.setProjection(Projections.distinct(projList));
Output:
2017-01-10 19:47:33.0 Router1
2017-01-11 20:45:59.0 Router1
2017-01-10 21:58:49.0 Router2
2017-01-10 21:59:00.0 Router2
This code works as expected but i want to run distinct function on the basis of unique date records, meaning distinct function should not consider time value but just the day value. So out put should be like:
2017-01-10 Router1
2017-01-11 Router1
2017-01-10 Router2
Any idea how to do that?
I found a solution, though not so clean..
a) Create a POJO class in which the projections results will be stored
public class Proj{
private Date date;
private String router;
// .. getters / setters
}
b) Use an 'sqlProjections' for the date property in your criteria API (native sql here .. im on MySql):
projList.add(Projections.sqlProjection("DATE_FORMAT(date, '%Y-%m-%d') as date", new String[]{"date"},new Type[] {new StringType()}));
This is enough to get the distinct values you are seeking though the date will be in a String format, if you want to transform it to date then:
c) Add a ResultTransformer to the criteria api:
.setResultTransformer(new ResultTransformer() {
public Object transformTuple(Object[] tuple, String[] aliases) {
SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd");
java.util.Date parsed = null;
try {
parsed = format.parse((String) tuple[1]);
} catch (ParseException e) {}
Proj p = new Proj();
p.setDate(parsed);
p.setRouter((String)tuple[0]);
return p;
}
)
Like i said this is not the prettiest solution but it worked for me.
The criteria call now returns an objects which encapsulated the results you seek.
I found a way for doing it:
Instead of
projList.add(Projections.property("date"), "date");
Use
projList.add(Projections.sqlProjection( "date(date) as date", new String[] {"date"}, new Type[] {StandardBasicTypes.STRING} ));
Where expression "date(date) as date" first "date" is mysql Function, "(date)" is HBM column name of date, as "as date" is return column name from mysql and type "date" will be bean name that will be populated.
There are many similar questions asked. But not exactly similar to the issue i am facing. I have seen almost all the questions and answers around it
So the problem is
I got to insert a date field in my mongo collection
But I can't access the collection directly. I got to use a service. The service takes a string and returns me oid.
So once i construct the BasicDBObject I call toString on it and pass it on to my service.. I even tried inserting it directly in a test collection and mongo is complaining.
BasicDBObject document = new BasicDBObject();
long createdAtSinceEpoch = 0;
long expiresAtSinceEpoch = 0;
createdAtSinceEpoch = System.nanoTime();
Date createdAt = new Date(TimeUnit.NANOSECONDS.toMillis(createdAtSinceEpoch));
document.append("createdAt", createdAt);
expiresAtSinceEpoch = createdAtSinceEpoch + +TimeUnit.SECONDS.toNanos(30);
Date expiresAt = new Date(TimeUnit.NANOSECONDS.toMillis(expiresAtSinceEpoch));
document.append("expiresAt", expiresAt);
service.storeRecord(document.toString());
and the generated JSON String looks like
{
"createdAt": {
"$date": "2015-09-01T20:05:21.641Z"
},
"expiresAt": {
"$date": "2015-09-01T20:05:51.641Z"
}
and Mongo complains that
Unable to parse JSON : Date expecting integer milliseconds, at (3,17)
So If i pass milliseconds alone instead of date object in the document.append() method then it DOES NOT recognize this field as date and considers it as String but inserts into the collection
I need 2 things
1) I want the data to be inserted
2) I am planning to expire that row by adding an index to the expiresAt field. So I want mongo to recognize that its a date field
JSON makes a difference between a numeric field and a text field containing a number. The latter one is only recognized as a String; I assume that this is what you did when you thought you were giving your service the date as an integer. Unfortunately you didn’t show us the relevant code.
When I save the Date info as a non String format, I annotate the field in my DTO as below. This helps the MongoDB know that the field is to be treated as an ISO date which then would be useful for making range search etc.,
#DateTimeFormat(iso = ISO.DATE_TIME) private Date date;
Date date = new Date();
BasicDBObject date= new BasicDBObject("date", date);
Data.insert(date);
One of the Oracle tables looks like this, I dont have the option of changing this:
REPORTING_PERIOD | REPORTING_DATE (Oracle DATE type)
-------------------------------
1140 01-FEB-12
1139 01-JAN-12
The JPA entity (with Hibernate as the provider) which looks like this :
#Column(name="REPORTING_PERIOD")
private long reportingPeriod;
#Temporal( TemporalType.DATE)
#Column(name="REPORT_DATE")
private Date reportDate; //java.util.Date
Now, let us go through my unit tests:
(I am using Spring Data JPA for repositories)
The below line queries the DB by REPORTING_PERIOD column
ReportingPeriod period1 = reportingPeriodRepository.findByMaxReportingPeriod();
assertNotNull(period1); // works fine and entity is fetched
System.out.println(period1.getReportDate());
The out put of SOP is 2012-02-01 - Notice the automatic conversion from value in db 01-FEB-12
Now, If I query directly by date using '01-FEB-12', as I am doing below, I dont get any results:
ReportingPeriod period2 = reportingPeriodRepository.findByReportDate(period1.getReportDate());
assertNotNull(period2);
Notice that, I am using date field from the same entity which I could successfully fetch in the previous statement.
Nor this works :
DateFormat df = new SimpleDateFormat("yyyy-MM-dd");
ReportingPeriod period3 = reportingPeriodRepository.findByReportDate(df.parse("2012-02-01"));
assertNotNull(period3);
Any help on how can I query ( with HQL will also be ok) by REPORTING_DATE as the param when the value in db is 01-FEB-12 is greatly appreciated.
I think there is some explicit date format conversion while obtaining the result in reportingPeriodRepository.findByMaxReportingPeriod();
Hence we can check whether we get the data using the same format as database format
Change
DateFormat df = new SimpleDateFormat("yyyy-MM-dd");
ReportingPeriod period3 = reportingPeriodRepository.findByReportDate(df.parse("2012-02-01"))
to
DateFormat df = new SimpleDateFormat("dd-MMM-yy");
ReportingPeriod period3 = reportingPeriodRepository.findByReportDate(df.parse("01-FEB-12"))