Develop Reference

Calculate difference between 2 times in MongoDB

I have a field called Last Modified At with value like '2016/04/12 20:24:18'. It is not an ISO Date but a normal String value stored via a java process in MongoDb.
I am trying to write a shell script to calculate the difference between '2016/04/12 20:24:18' and say '2016/04/12 16:24:18'. The difference could be either in days or hours or mins or secs. I tried couple of things including converting to ISO dates but it doesnt work out. Is there an easy way to find out like Oracle.
Any help would be appreciated?
Thanks,
Ram

I'm not exactly sure how you plan on running the shell script, but it is possible in the mongo shell to parse dates (using Javascript) and calculate time between two dates as you have asked. Assume we have a document in the things database as follows:
{
"_id" : ObjectId("570f1e528163383227ace761"),
"lastModifiedAt" : "2016/04/12 20:24:18"
}
We can run the following script to get the difference between the lastModifiedDate of a document and a hard-coded date, such as 2016/04/12 16:24:18:
db.things.find({}).forEach(function(thing) {
var date1 = new Date(thing.lastModifiedAt);
var date2 = new Date('2016/04/12 16:24:18');
var dateDiff = date1.getTime() - date2.getTime();
printjson({_id:thing._id,lastModifiedAt:thing.lastModifiedAt,dateDiff:dateDiff});
});
This results in:
{
"_id" : ObjectId("570f1e528163383227ace761"),
"lastModifiedAt" : "2016/04/12 20:24:18",
"dateDiff" : 14400000
}
where dateDiff is milliseconds and 14400000 milliseconds = 4 hours.
If you provide more information on how you plan on making this call, and where the second date is coming from I would be happy to expand upon this answer.

How to expire gelocations on redis

I am using geo-support on Redis.
Adding new geolocations this way:
"GEOADD" "report-geo-set" "30.52439985197" "50.56539003041" "john"
I want to expire john key from report-geo-set after X hours.
Any suggestions doing that?
Thank you,
ray.

Not possible with built-in commands. Keep in mind that geo-support based on zset and your question is look`s like "How to use TTL for individual keys in ZSET".
You may use something like that:
Add "john" to additional special timeout ZSET with time() + X hours score.
From time to time run script/worker which get all obsolete keys from timeout zset and execute ZREM for your "john" key.
Example of given suggestion. Add items:
MULTI
GEOADD report-geo-set 30.52439985197 50.56539003041 john
ZADD geo-timeout 1452600528 john //1452600528 is unix time stamp current + X hours
EXEC
Clean up script called from time to time (with LUA):
local currentTime = redis.call('TIME');
local list = redis.call('ZRANGEBYSCORE', 'geo-timeout', 0, currentTime[0]);
local keysRemoved = 0;
for i, name in ipairs(list) do
redis.call('ZREM', 'geo-timeout', name);
redis.call('ZREM', 'report-geo-set', name);
keysRemoved = keysRemoved + 1;
end
return keysRemoved;

Elasticsearch aggregation query not giving expected output

I am trying to do aggregation on documents which contains datetime and CPU time and server name. I want to find the avg CPU time on latest date. I have the following query which partially works it gives me the avg CPU time but not on latest date it just randomly chooses date.
client.prepareSearch("myindex").
setTypes("mytype").
setQuery(
QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),
FilterBuilders.andFilter(FilterBuilders.termFilter("server","x")))).
addAggregation(AggregationBuilders.avg("cpu_agg")
.field("dt_time").field("cpu_time"))
.get()
Please guide. I want avg cpu time on latest date say today's date. I am new to ElasticSearch. Thanks in advance.

client.prepareSearch("myindex").
setTypes("mytype").
setQuery(
QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),
FilterBuilders.andFilter(FilterBuilders.termFilter("server","x")))).
addAggregation(AggregationBuilders.avg("cpu_agg")
.field("dt_time").field("cpu_time"))
.get()
look at the portion (where field is set to dt_time at first and replaced to cpu_time), which means aggregation is build for cpu_time,
If you want to get cpu time in today's date then one way is use date filter ,
FilterBuilders.andFilter(FilterBuilders.termFilter("server","x"),FilterBuilders.rangeFilter("dt_time").to(to).from(from))))
For your problem, from = to = today_date (or last date)
so finally,
client.prepareSearch("myindex").
setTypes("mytype").
setQuery(
QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),
FilterBuilders.andFilter(FilterBuilders.termFilter("server", "x"), FilterBuilders.rangeFilter("dt_time").to("to").from("from")))).
addAggregation(AggregationBuilders.avg("cpu_agg")
.field("cpu_time"))
.get();

More Efficient Way of Doing This SQL Query? A time comparison query?

I have this SQL query which queries the database every 5 seconds to determine who is currently actively using the software. Active users have pinged the server in the last 10 seconds. (The table gets updated correctly on user activity and a I have a thread evicting entries on session timeouts, that all works correctly).
What I'm looking for is a more efficient/quicker way to do this, since it gets called frequently, about every 5 seconds. In addition, there may be up to 500 users in the database. The language is Java, but the question really pertains to any language.
List<String> r = new ArrayList<String>();
Calendar c = Calendar.getInstance();
long threshold = c.get(Calendar.SECOND) + c.get(Calendar.MINUTE)*60 + c.get(Calendar.HOUR_OF_DAY)*60*60 - 10;
String tmpSql = "SELECT user_name, EXTRACT(HOUR FROM last_access_ts) as hour, EXTRACT(MINUTE FROM last_access_ts) as minute, EXTRACT(SECOND FROM last_access_ts) as second FROM user_sessions";
DBResult rs = DB.select(tmpSql);
for (int i=0; i<rs.size(); i++)
{
Map<String, Object> result = rs.get(i);
long hour = (Long)result.get("hour");
long minute = (Long)result.get("minute");
long second = (Long)result.get("second");
if (hour*60*60 + minute*60 + second > threshold)
r.add(result.get("user_name").toString());
}
return r;

If you want this to run faster, then create an index on user_sessions(last_access_ts, user_name), and do the date logic in the query:
select user_name
from user_sessions
where last_access_ts >= now() - 5/(24*60*60);
This does have a downside. You are, presumably, updating the last_access_ts field quite often. An index on the field will also have to be updated. On the positive side, this is a covering index, so the index itself can satisfy the query without resorting to the original data pages.

I would move the logic from Java to DB. This mean you translate if into where, and just select the name of valid result.
SELECT user_name FROM user_sessions WHERE last_access_ts > ?
In your example the c represent current time. It is highly possible that result will be empty.
So your question should be more about date time operation on your database.

Just let the database do the comparison for you by using this query:
SELECT
user_name
FROM user_sessions
where TIMESTAMPDIFF(SECOND, last_access_ts, current_timestamp) > 10
Complete example:
List<String> r = new ArrayList<String>();
Calendar c = Calendar.getInstance();
long threshold = c.get(Calendar.SECOND) + c.get(Calendar.MINUTE)*60 + c.get(Calendar.HOUR_OF_DAY)*60*60 - 10;
// this will return all users that were inactive for longer than 10 seconds
String tmpSql = "SELECT
user_name
FROM user_sessions
where TIMESTAMPDIFF(SECOND, last_access_ts, current_timestamp) > 10";
DBResult rs = DB.select(tmpSql);
for (int i=0; i<rs.size(); i++)
{
Map<String, Object> result = rs.get(i);
r.add(result.get("user_name").toString());
}
return r;
SQLFiddle

The solution is to remove the logic from your code to the sql query to only get the active users from that select, using a where clause.
It is faster to use the sql built-in functions to get fewer records and iterate less in your code.
Add this to your sql query to get the active users only:
Where TIMESTAMPDIFF(SECOND, last_access_ts, current_timestamp) > 10
This will get you all the records whose date is 10 seconds ago or sooner.

Try the MySQL TimeDiff function in your select. This way you can select only the results that are active without having to do any other calculations.
Link: MySQL: how to get the difference between two timestamps in seconds

If I get you right, then you got only 500 entries in your user_sessions table. In this case I wouldn't even care about indexes. Throw them away. The DB engine probably won't use them anyway for such a low record count. The performance gain due to not updating the indexes on every record update could be probably higher than the query overhead.
If you care about DB stress, then lengthen the query/update intervals to 1 minute or more, if your application allows this. Gordon Linoff's answer should give you the best query performance though.
As a side note (because it has bitten me before): If you don't use the same synchronized time for all user callbacks, then your "active users logic" is flawed by design.

Database timestamps not matching

I have an action in struts2 that will query the database for an object and then copy it with a few changes. Then, it needs to retrieve the new objectID from the copy and create a file called objectID.txt.
Here is relevant the code:
Action Class:
ObjectVO objectVOcopy = objectService.searchObjects(objectId);
//Set the ID to 0 so a new row is added, instead of the current one being updated
objectVOcopy.setObjectId(0);
Date today = new Date();
Timestamp currentTime = new Timestamp(today.getTime());
objectVOcopy.setTimeStamp(currentTime);
//Add copy to database
objectService.addObject(objectVOcopy);
//Get the copy object's ID from the database
int newObjectId = objectService.findObjectId(currentTime);
File inboxFile = new File(parentDirectory.getParent()+"\\folder1\\folder2\\"+newObjectId+".txt");
ObjectDAO
//Retrieve identifying ID of copy object from database
List<ObjectVO> object = getHibernateTemplate().find("from ObjectVO where timeStamp = ?", currentTime);
return object.get(0).getObjectId();
The problem is that more often than not, the ObjectDAO search method will not return anything. When debugging I've noticed that the Timestamp currentTime passed to it is usually about 1-2ms off the value in the database. I have worked around this bug changing the hibernate query to search for objects with a timestamp within 3ms of the one passed, but I'm not sure where this discrepancy is coming from. I'm not recalculating the currentTime; I'm using the same one to retrieve from the database as I am to write to the database. I'm also worried that when I deploy this to another server the discrepancy might be greater. Other than the objectID, this is the only unique identifier so I need to use it to get the copy object.
Does anyone know why this is occuring and is there a better work around than just searching through a range? I'm using Microsoft SQL Server 2008 R2 btw.
Thanks.

Precision in SQL Server's DATETIME data type does not precisely match what you can generate in other languages. SQL Server rounds to the nearest 0.003 - this is why you can say:
DECLARE #d DATETIME = '20120821 23:59:59.997';
SELECT #d;
Result:
2012-08-21 23:59:59.997
Then try:
DECLARE #d DATETIME = '20120821 23:59:59.999';
SELECT #d;
Result:
2012-08-22 00:00:00.000
Since you are using SQL Server 2008 R2, you should make sure to use the DATETIME2 data type instead of DATETIME.
That said, #RedFilter makes a good point - why are you relying on the time stamp when you can use the generated ID instead?

This feels wrong.
Other than the objectID, this is the only unique identifier
Databases have the concept of a unique identifier for a reason. You should really use that to retrieve an instance of your object.
You can use the get method on the Hibernate session and take advantage of the session and second level caches as well.
With your approach you execute a query everytime you retrieve your object.