I'm trying to correlate the timing information obtain from a java job and linux performance monitoring tool perf (specifically perf stat).
The timing information from java is obtained using
String tstamp0 = String.valueOf(System.currentTimeMillis()); (This is essentially time in milliseconds from epoch)
whereas perf gives the time the process has began and the subsequent recording only show the time elapsed.
What I would like to do is, convert the timing information obtained from the perf stat to milliseconds, and here is where I'm failing. I'm approaching this problem in Python.
This piece of code is giving me the timing information from perf
tailit = "head -n 1 " + dataset_path
process = subprocess.Popen(tailit, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = process.communicate()
date_time = out.split("\n")[0].split(" ")[4:]
date = date_time[3] + "-" + date_time[0] + "-" + date_time[1]
time = date_time[2]
#TIMESTAMP
INIT_TIME = datetime.datetime.strptime(date + ' ' + time, "%Y-%B-%d %H:%M:%S") + datetime.timedelta(seconds=0.01)
#df is pandas data frame
df['STAMPME'] = df['TCOUNT'].apply(lambda x: foobar(datetime.timedelta(seconds=x) + INIT_TIME))
here foobar is the following to convert a string to timestamp in milliseconds, but it doesn't make sense.
def foobar(INIT_TIME):
d = datetime.datetime.strptime(str(INIT_TIME), "%Y-%m-%d %H:%M:%S.%f").strftime('%s')
d_in_ms = int(d)*1000
return (d_in_ms)
Any help will be appreciated.
EDIT: Prior questions were not addressing the problem of correlating the java timestamp (currentTimeMillis()) to the datetime with milliseconds.
For instance: with the function foobar:
with INIT_TIME set as 2017-05-11 10:56:54.203, the return value is 1494493014000 when it instead should be 1494500214203
I think figured out the problem.
Looks like foobar function is returning time in GMT+2, whereas the java job was returning time in GMT. so with a timedelta of +2, I could solve it.
Related
I want to check how fast the CRUD Operations are executing on a MongoDB.
Therefore I recorded the time with the following code:
long start = System.nanoTime();
FindIterable<Document> datasetFindIterable = this.collection.find(filter);
long finish = System.nanoTime();
long timeElapsed = finish - start;
I am aware, that the FindIterable Object comes with "executionStats" and "executionTimeMillis":
JSONObject jsonobject = (JSONObject) parser.parse(datasetFindIterable.explain().toJson())
JSONObject executionStats = (JSONObject) jsonobject.get("executionStats");
Long executionTimeMillis = (Long) executionStats.get("executionTimeMillis");
However I am a bit confused, I get the following results:
start (ns)
finish (ns)
timeElapsed (ns)
executionTimeMillis (ms)
582918161918004
582918161932511
14507
1234
14507 ns are 0.014507 ms
How can it be, that the executionTimeMillis (1234 ms) is that much larger than the difference between the System.nanoTime() (=0.014507 ms). Shouldn't it be the other way around, since the System.nanoTime() does also need some time to execute itself?
If I recall correctly, there are asynchronous and synchronous MongoDB Drivers available.
If you use an asynchronous driver, it could be the issue, that the
"long finish = System.nanoTime();"
command does not wait for the
"FindIterable<Document> datasetFindIterable = this.collection.find(filter);"
command to return with a value, therefore the time difference could be lower than the execution time stored in the FindIterable variable.
I am working on a django and java project in which I need to compare the time in django to the time in current time in java.
I am storing the enbled_time in models as :
enabled_time = models.DateTimeField(auto_now = True, default=timezone.now())
The time gets populated in the db in the form :
2017-02-26 14:54:02
Now in my java project a cron is running which checks whether enabled_time plus an expiry time is greater than the current time something as:
Long EditedTime = db.getEnabledTime() + (expiryTime*60*1000); //expiryTime is in mins
if (System.currentTimeMillis() - EditedTime > 0) {
//do something
}
Here db is the database entity for that table.
But db.getEnabledTime() gives a result '2017'. What am I doing wrong?
PS: I am storing time as Long which seems unsuitable to me. Can someone suggest which datatype should I choose or does it work fine?
I have a field called Last Modified At with value like '2016/04/12 20:24:18'. It is not an ISO Date but a normal String value stored via a java process in MongoDb.
I am trying to write a shell script to calculate the difference between '2016/04/12 20:24:18' and say '2016/04/12 16:24:18'. The difference could be either in days or hours or mins or secs. I tried couple of things including converting to ISO dates but it doesnt work out. Is there an easy way to find out like Oracle.
Any help would be appreciated?
Thanks,
Ram
I'm not exactly sure how you plan on running the shell script, but it is possible in the mongo shell to parse dates (using Javascript) and calculate time between two dates as you have asked. Assume we have a document in the things database as follows:
{
"_id" : ObjectId("570f1e528163383227ace761"),
"lastModifiedAt" : "2016/04/12 20:24:18"
}
We can run the following script to get the difference between the lastModifiedDate of a document and a hard-coded date, such as 2016/04/12 16:24:18:
db.things.find({}).forEach(function(thing) {
var date1 = new Date(thing.lastModifiedAt);
var date2 = new Date('2016/04/12 16:24:18');
var dateDiff = date1.getTime() - date2.getTime();
printjson({_id:thing._id,lastModifiedAt:thing.lastModifiedAt,dateDiff:dateDiff});
});
This results in:
{
"_id" : ObjectId("570f1e528163383227ace761"),
"lastModifiedAt" : "2016/04/12 20:24:18",
"dateDiff" : 14400000
}
where dateDiff is milliseconds and 14400000 milliseconds = 4 hours.
If you provide more information on how you plan on making this call, and where the second date is coming from I would be happy to expand upon this answer.
I am using geo-support on Redis.
Adding new geolocations this way:
"GEOADD" "report-geo-set" "30.52439985197" "50.56539003041" "john"
I want to expire john key from report-geo-set after X hours.
Any suggestions doing that?
Thank you,
ray.
Not possible with built-in commands. Keep in mind that geo-support based on zset and your question is look`s like "How to use TTL for individual keys in ZSET".
You may use something like that:
Add "john" to additional special timeout ZSET with time() + X hours score.
From time to time run script/worker which get all obsolete keys from timeout zset and execute ZREM for your "john" key.
Example of given suggestion. Add items:
MULTI
GEOADD report-geo-set 30.52439985197 50.56539003041 john
ZADD geo-timeout 1452600528 john //1452600528 is unix time stamp current + X hours
EXEC
Clean up script called from time to time (with LUA):
local currentTime = redis.call('TIME');
local list = redis.call('ZRANGEBYSCORE', 'geo-timeout', 0, currentTime[0]);
local keysRemoved = 0;
for i, name in ipairs(list) do
redis.call('ZREM', 'geo-timeout', name);
redis.call('ZREM', 'report-geo-set', name);
keysRemoved = keysRemoved + 1;
end
return keysRemoved;
Here is my code:
long treatmentTimeBeginDay;
if ( effectiveBegin.after(businessClosing) ) {
LOGGER.debug("Compute treatment time for beginDay = 0: the effective begin {} is after business closing {}",
config.formatLogDate(effectiveBegin),config.formatLogDate(businessClosing));
treatmentTimeBeginDay = 0;
} else {
LOGGER.debug("Compute treatment time for beginDay between {} and {}",config.formatLogDate(effectiveBegin),config.formatLogDate(businessClosing));
treatmentTimeBeginDay = businessClosing.getTime() - effectiveBegin.getTime();
}
Preconditions.checkState( treatmentTimeBeginDay >= 0 , "Internal bug! treatmentTimeBeginDay="+treatmentTimeBeginDay );
effectiveBegin and businessClosing are not null, also checked by Guava preconditions and you can see it in the logs...
It runs fine in most cases but in production we have these errors:
Caused by: java.lang.IllegalStateException: Internal bug!
treatmentTimeBeginDay=-852
I don't give you the rest of the stack/code because it should be enough...
The exception is explicitly raised by my Guava checkState call.
I also have the log:
DEBUG [BusinessHoursUtils.java:257] llairie - Compute treatment time
for beginDay between 7/19/12 8:00 PM and 7/19/12 8:00 PM
(I can't have log with millies for now)
What i'm trying to understand is.
If i get the log i gave you, this means that the test if ( effectiveBegin.after(businessClosing) ) is false, so effectiveBegin should be before or equals to the businessClosing.
In this case effectiveBegin timestamp should be lower than the businessClosing timestamp.
So when i do businessClosing.getTime() - effectiveBegin.getTime(); i would expect to have a positive number.
So please can someone tell me why i have -852 milliseconds in my exception message? How is this possible?
Edit: I was suspecting a tricky case where the after/before method wouldn't work for milliseconds and it seems that's the problem since i could reproduce it locally.
The 2 dates at runtime are:
businessClosing = {java.util.Date#503}"Thu Jul 19 20:00:00 CEST 2012"
fastTime = 1342720800000
cdate = null
effectiveBegin = {java.sql.Timestamp#498}"2012-07-19 20:00:00.999"
nanos = 999000000
fastTime = 1342720800000
cdate = {sun.util.calendar.Gregorian$Date#512}"2012-07-19T20:00:00.000+0200"
With these runtime objects, effectiveBegin.after(businessClosing) = false
If i set in DB effectiveBegin = 2012-07-19 20:00:01.000, 1 millisecond later, then the test = true
In both cases i would expect to have effectiveBegin.after(businessClosing) = true
It seems, like ametren suspected, that my dates are different.
So in the end, what's the problem?
Aren't we supposed to be able to compare the 2 date instance with a millisecond precision?
Even if they are subclasses of the java.util.Date?
The problem here is that you mix Timestamp and Date. Timestamp.after(Date) compares only the milliseconds of the Date components, which are both 1342720800000 in your example.
However, Timestamp.getTime() does also consider the Nanoseconds (999000000ns = 999ms) that are stored within the Timestamp and will return 1342720800999. So businessClosing.getTime() - effectiveBegin.getTime() will return -999 as result.
To solve this issue, you could modify the if-statement to if(effectiveBegin.compareTo(businessClosing) > 0) or you could convert the businessClosingDate to a Timestamp before the if-statement.