Develop Reference

Formatting UTC date-time's partial seconds

After parsing JSON UTC date-time data from a server, I was presented with
2017-03-27 16:27:45.567
... is there any way to format this without using tedious amount of String manipulation so that the seconds part is rounded up to 46 prior to passing it in as a DateTimeFormat pattern of say, "yyyy-MM-dd HH:mm:ss"?

You can round the second up like this:
DateTime dateTime = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS")
.withZoneUTC()
.parseDateTime("2017-03-27 16:27:45.567")
.secondOfMinute()
.roundCeilingCopy();
System.out.println(dateTime);
// 2017-03-27T16:27:46.000Z

Have you looked at (and could you use) the MomentJS library? I had issues with reading various date formats from the server and making sense of them in JavaScript code (which led me here). Since then, I've used MomentJS and working with dates/times in JavaScript has been much easier.
Here is an example:
<script>
try
{
var myDateString = "2017-03-27 16:27:45.567";
var d = moment(myDateString);
var result = d.format('YYYY/MM/DD HH:mm:ss');
alert("Simple Format: " + result);
// If we have millliseconds, increment to the next second so that
// we can then get its 'floor' by using the startOf() function.
if(d.millisecond() > 0)
d = d.add(1, 'second');
result = d.startOf('second').format('YYYY/MM/DD HH:mm:ss');
alert("Rounded Format: " + result);
}
catch(er)
{
console.log(er);
}
</script>
But of course, you'll probably want to wrap this logic into a function.

Accessing field called Date and Time in Event Viewer through Java

I would like to access the field called Date and time from Event Viewer through Java. I managed to get the EventID :
EVENTLOGRECORD record = new EVENTLOGRECORD(pevlr);
if (record.EventID.shortValue() == 4624) {
System.out.println("Successful log in: " + " Event ID: " + record.EventID.shortValue());
}
if (record.EventID.shortValue() == 4634) {
System.out.println("Successful log out: " + " Event ID: " + record.EventID.shortValue());
}
And now the next step would be to get the Date and time
If anyone could help i would really appreciate it !

Meanwhile I figured out how to get the Date and time field. Just used :
long timestamp = record.TimeGenerated.longValue() * 1000;
Date d = new Date(timestamp);

Dataframes are slow to parse through small amount of data

I have 2 classes doing a similar task in Apache Spark but the one using data frame is many times slower than the "regular" one using RDD. (30x)
I would like to use data frame since it will eliminate a lot of code and classes we have but obviously I can't have it be that much slower.
The data set is nothing big. We have 30 some files with json data in each about events triggered from activities in another piece of software. There are between 0 to 100 events in each file.
A data set with 82 events will take about 5 minutes to be processed with data frames.
Sample code:
public static void main(String[] args) throws ParseException, IOException {
SparkConf sc = new SparkConf().setAppName("POC");
JavaSparkContext jsc = new JavaSparkContext(sc);
SQLContext sqlContext = new SQLContext(jsc);
conf = new ConfImpl();
HashSet<String> siteSet = new HashSet<>();
// last month
Date yesterday = monthDate(DateUtils.addDays(new Date(), -1)); // method that returns the date on the first of the month
Date startTime = startofYear(new Date(yesterday.getTime())); // method that returns the date on the first of the year
// list all the sites with a metric file
JavaPairRDD<String, String> allMetricFiles = jsc.wholeTextFiles("hdfs:///somePath/*/poc.json");
for ( Tuple2<String, String> each : allMetricFiles.toArray() ) {
logger.info("Reading from " + each._1);
DataFrame metric = sqlContext.read().format("json").load(each._1).cache();
metric.count();
boolean siteNameDisplayed = false;
boolean dateDisplayed = false;
do {
Date endTime = DateUtils.addMonths(startTime, 1);
HashSet<Row> totalUsersForThisMonth = new HashSet<>();
for (String dataPoint : Conf.DataPoints) { // This is a String[] with 4 elements for this specific case
try {
if (siteNameDisplayed == false) {
String siteName = parseSiteFromPath(each._1); // method returning a parsed String
logger.info("Data for site: " + siteName);
siteSet.add(siteName);
siteNameDisplayed = true;
}
if ( dateDisplayed == false ) {
logger.info("Month: " + formatDate(startTime)); // SimpleFormatDate("yyyy-MM-dd")
dateDisplayed = true;
}
DataFrame lastMonth = metric.filter("event.eventId=\"" + dataPoint + "\"").filter("creationDate >= " + startTime.getTime()).filter("creationDate < " + endTime.getTime()).select("event.data.UserId").distinct();
logger.info("Distinct for last month for " + dataPoint + ": " + lastMonth.count());
totalUsersForThisMonth.addAll(lastMonth.collectAsList());
} catch (Exception e) {
// data does not fit the expected model so there is nothing to print
}
}
logger.info("Total Unique for the month: " + totalStudentForThisMonth.size());
startTime = DateUtils.addMonths(startTime, 1);
dateDisplayed = false;
} while ( startTime.getTime() < commonTmsMetric.monthDate(yesterday).getTime());
// reset startTime for the next site
startTime = commonTmsMetric.StartofYear(new Date(yesterday.getTime()));
}
}
There are a few things that are not efficient in this code but when I look at the logs it only adds a few seconds to the whole processing.
I must be missing something big.
I have ran this with 2 executors and 1 executor and the difference is 20 seconds on 5 minutes.
This is running with Java 1.7 and Spark 1.4.1 on Hadoop 2.5.0.
Thank you!

So there a few things, but its hard to say without seeing the breakdown of the different tasks & their time. The short version is you are doing way to much work in the driver and not taking advantage of Spark's distributed capabilities.
For example, you are collecting all of the data back to the driver program (toArray() and your for loop). Instead you should just point Spark SQL at the files in needs to load.
For the operators, it seems like your doing many aggregations in the driver, instead you could use the driver to generate the aggregations and have Spark SQL execute them.
Another big difference between your in-house code and the DataFrame code is going to be Schema inference. Since you've already created classes to represent your data, it seems likely that you know the schema of your JSON data. You can likely speed up your code by adding the schema information at read time so Spark SQL can skip inference.
I'd suggest re-visiting this approach and trying to build something using Spark SQL's distributed operators.

Difference between two times always returns 2 or bigger

I have 2 events in my application. I need the time difference between the two of them so I store the time in a sharedpreferences using this line of code :
public static void PutStatus(Context ctx,boolean stat)
{
if (ctx != null)
{
SharedPreferences shr = ctx.getSharedPreferences("test", Context.MODE_WORLD_READABLE);
shr.edit().putBoolean("SHIFT", stat).commit();
if (stat) shr.edit().putLong("START_TIME", System.currentTimeMillis()).commit();
}
}
And then I calculate the difference between the two date and times using this :
SharedPreferences shr = getApplicationContext().getSharedPreferences("test", Context.MODE_WORLD_READABLE);
long time_stamp = System.currentTimeMillis() - shr.getLong("START_TIME", 0);
Date data = new Date(time_stamp);
The minutes are working great but the hours are 2 hour ahead.
Why is that does it have any connection with timezones ?

The proper way to get a duration with Java is a little bit complicated. You would create it the following way:
import javax.xml.datatype.Duration;
import javax.xml.datatype.DatatypeFactory
...
Duration d = DatatypeFactory.newInstance().newDuration(time_stamp);
System.out.println(d.getHours() + "h:" + d.getMinutes() + "m: " + d.getSeconds() + "s");
For documentation see Duration and DatatypeFactory.

If you need time interval, what about just doing some modulo computations? An hour is 3600 * 1000 milliseconds, day is 86400 * 1000. When you create date this way, you are creating date object relative to 1.1.1970 UTC - this may be as well 2 hours off from timezone where your phone thinks to be.

I would suggest to use two date objects, something like
int diffInDays = (newerDate.getTime() - olderDate.getTime()) / (1000 * 60 * 60 * 24)

Calculate client-server time difference in Borland Starteam server 8

Problem. I need a way to find Starteam server time through Starteam Java SDK 8.0. Version of server is 8.0.172 so method Server.getCurrentTime() is not available since it was added only in server version 9.0.
Motivation. My application needs to use views at specific dates. So if there's some difference in system time between client (where the app is running) and server then obtained views are not accurate. In the worst case the client's requested date is in the future for server so the operation results in exception.

After some investigation I haven't found any cleaner solution than using a temporary item. My app requests the item's time of creation and compares it with local time. Here's the method I use to get server time:
public Date getCurrentServerTime() {
Folder rootFolder = project.getDefaultView().getRootFolder();
Topic newItem = (Topic) Item.createItem(project.getTypeNames().TOPIC, rootFolder);
newItem.update();
newItem.remove();
newItem.update();
return newItem.getCreatedTime().createDate();
}

If your StarTeam server is on a Windows box and your code will be executing on a Windows box, you could shell out and execute the NET time command to fetch the time on that machine and then compare it to the local time.
net time \\my_starteam_server_machine_name
which should return:
"Current time at \\my_starteam_server_machine_name is 10/28/2008 2:19 PM"
"The command completed successfully."

We needed to come up with a way of finding the server time for use with CodeCollab. Here is a (longish) C# code sample of how to do it without creating a temporary file. Resolution is 1 second.
static void Main(string[] args)
{
// ServerTime replacement for pre-2006 StarTeam servers.
// Picks a date in the future.
// Gets a view, sets the configuration to the date, and tries to get a property from the root folder.
// If it cannot retrieve the property, the date is too far in the future. Roll back the date to an earlier time.
DateTime StartTime = DateTime.Now;
Server s = new Server("serverAddress", 49201);
s.LogOn("User", "Password");
// Getting a view - doesn't matter which, as long as it is not deleted.
Project p = s.Projects[0];
View v = p.AccessibleViews[0]; // AccessibleViews saves checking permissions.
// Timestep to use when searching. One hour is fairly quick for resolution.
TimeSpan deltaTime = new TimeSpan(1, 0, 0);
deltaTime = new TimeSpan(24 * 365, 0, 0);
// Invalid calls return faster - start a ways in the future.
TimeSpan offset = new TimeSpan(24, 0, 0);
// Times before the view was created are invalid.
DateTime minTime = v.CreatedTime;
DateTime localTime = DateTime.Now;
if (localTime < minTime)
{
System.Console.WriteLine("Current time is older than view creation time: " + minTime);
// If the dates are so dissimilar that the current date is before the creation date,
// it is probably a good idea to use a bigger delta.
deltaTime = new TimeSpan(24 * 365, 0, 0);
// Set the offset to the minimum time and work up from there.
offset = minTime - localTime;
}
// Storage for calculated date.
DateTime testTime;
// Larger divisors converge quicker, but might take longer depending on offset.
const float stepDivisor = 10.0f;
bool foundValid = false;
while (true)
{
localTime = DateTime.Now;
testTime = localTime.Add(offset);
ViewConfiguration vc = ViewConfiguration.CreateFromTime(testTime);
View tempView = new View(v, vc);
System.Console.Write("Testing " + testTime + " (Offset " + (int)offset.TotalSeconds + ") (Delta " + deltaTime.TotalSeconds + "): ");
// Unfortunately, there is no isValid operation. Attempting to
// read a property from an invalid date configuration will
// throw an exception.
// An alternate to this would be proferred.
bool valid = true;
try
{
string testname = tempView.RootFolder.Name;
}
catch (ServerException)
{
System.Console.WriteLine(" InValid");
valid = false;
}
if (valid)
{
System.Console.WriteLine(" Valid");
// If the last check was invalid, the current check is valid, and
// If the change is this small, the time is very close to the server time.
if (foundValid == false && deltaTime.TotalSeconds <= 1)
{
break;
}
foundValid = true;
offset = offset.Add(deltaTime);
}
else
{
offset = offset.Subtract(deltaTime);
// Once a valid time is found, start reducing the timestep.
if (foundValid)
{
foundValid = false;
deltaTime = new TimeSpan(0,0,Math.Max((int)(deltaTime.TotalSeconds / stepDivisor), 1));
}
}
}
System.Console.WriteLine("Run time: " + (DateTime.Now - StartTime).TotalSeconds + " seconds.");
System.Console.WriteLine("The local time is " + localTime);
System.Console.WriteLine("The server time is " + testTime);
System.Console.WriteLine("The server time is offset from the local time by " + offset.TotalSeconds + " seconds.");
}
Output:
Testing 4/9/2009 3:05:40 PM (Offset 86400) (Delta 31536000): InValid
Testing 4/9/2008 3:05:40 PM (Offset -31449600) (Delta 31536000): Valid
...
Testing 4/8/2009 10:05:41 PM (Offset 25200) (Delta 3): InValid
Testing 4/8/2009 10:05:38 PM (Offset 25197) (Delta 1): Valid
Run time: 9.0933426 seconds.
The local time is 4/8/2009 3:05:41 PM
The server time is 4/8/2009 10:05:38 PM
The server time is offset from the local time by 25197 seconds.

<stab_in_the_dark>
I'm not familiar with that SDK but from looking at the API if the server is in a known timezone why not create and an OLEDate object whose date is going to be the client's time rolled appropriately according to the server's timezone?
</stab_in_the_dark>