Develop Reference

Looping stops when assertion fails

There is an excelsheet where all URLs (16) are listed in one column. Now once page gets loaded need to verify whether page title is matching with the expected title which is already stored in excel. I am able to perform it using for loop. It runs all URls if all are passed but stops when it fails. I need to run it completely and give a report which passed and which failed. I written the below code.
rowCount = suite_pageload_xls.getRowCount("LoadURL");
for(i=2,j=2;i<=rowCount;i++,j++) {
String urlData = suite_pageload_xls.getCellData("LoadURL", "URL", i);
Thread.sleep(3000);
long start = System.currentTimeMillis();
APP_LOGS.debug(start);
driver.navigate().to(urlData);
String actualtitle = driver.getTitle();
long finish = System.currentTimeMillis();
APP_LOGS.debug(finish);
APP_LOGS.debug(urlData+ "-----" +driver.getTitle());
long totalTime = finish - start;
APP_LOGS.debug("Total time taken is "+totalTime+" ms");
String expectedtitle = suite_pageload_xls.getCellData("LoadURL", "Label", j);
Assert.assertEquals(actualtitle, expectedtitle);
if (actualtitle.equalsIgnoreCase(expectedtitle)) {
APP_LOGS.debug("PAGE LABEL MATCHING....");
String resultpass = "PASS";
APP_LOGS.debug(resultpass);
APP_LOGS.debug("***********************************************************");
} else {
APP_LOGS.debug("PAGE LABEL NOT MATCHING....");
String resultfail = "FAIL";
APP_LOGS.debug(resultfail);
APP_LOGS.debug("***********************************************************");
}
}
Kindly help me in this regard.

This is the correct behavior of the assertion, it throws an Exception when the assertion is wrong.
You could store the actualTitles and expectedTitles in arrays and perform the assertions all at once.
For better assertions I suggest you try AssertJ, you could directly compare 2 lists, the actual and the expected, and it will return the complete difference.

Dataframes are slow to parse through small amount of data

I have 2 classes doing a similar task in Apache Spark but the one using data frame is many times slower than the "regular" one using RDD. (30x)
I would like to use data frame since it will eliminate a lot of code and classes we have but obviously I can't have it be that much slower.
The data set is nothing big. We have 30 some files with json data in each about events triggered from activities in another piece of software. There are between 0 to 100 events in each file.
A data set with 82 events will take about 5 minutes to be processed with data frames.
Sample code:
public static void main(String[] args) throws ParseException, IOException {
SparkConf sc = new SparkConf().setAppName("POC");
JavaSparkContext jsc = new JavaSparkContext(sc);
SQLContext sqlContext = new SQLContext(jsc);
conf = new ConfImpl();
HashSet<String> siteSet = new HashSet<>();
// last month
Date yesterday = monthDate(DateUtils.addDays(new Date(), -1)); // method that returns the date on the first of the month
Date startTime = startofYear(new Date(yesterday.getTime())); // method that returns the date on the first of the year
// list all the sites with a metric file
JavaPairRDD<String, String> allMetricFiles = jsc.wholeTextFiles("hdfs:///somePath/*/poc.json");
for ( Tuple2<String, String> each : allMetricFiles.toArray() ) {
logger.info("Reading from " + each._1);
DataFrame metric = sqlContext.read().format("json").load(each._1).cache();
metric.count();
boolean siteNameDisplayed = false;
boolean dateDisplayed = false;
do {
Date endTime = DateUtils.addMonths(startTime, 1);
HashSet<Row> totalUsersForThisMonth = new HashSet<>();
for (String dataPoint : Conf.DataPoints) { // This is a String[] with 4 elements for this specific case
try {
if (siteNameDisplayed == false) {
String siteName = parseSiteFromPath(each._1); // method returning a parsed String
logger.info("Data for site: " + siteName);
siteSet.add(siteName);
siteNameDisplayed = true;
}
if ( dateDisplayed == false ) {
logger.info("Month: " + formatDate(startTime)); // SimpleFormatDate("yyyy-MM-dd")
dateDisplayed = true;
}
DataFrame lastMonth = metric.filter("event.eventId=\"" + dataPoint + "\"").filter("creationDate >= " + startTime.getTime()).filter("creationDate < " + endTime.getTime()).select("event.data.UserId").distinct();
logger.info("Distinct for last month for " + dataPoint + ": " + lastMonth.count());
totalUsersForThisMonth.addAll(lastMonth.collectAsList());
} catch (Exception e) {
// data does not fit the expected model so there is nothing to print
}
}
logger.info("Total Unique for the month: " + totalStudentForThisMonth.size());
startTime = DateUtils.addMonths(startTime, 1);
dateDisplayed = false;
} while ( startTime.getTime() < commonTmsMetric.monthDate(yesterday).getTime());
// reset startTime for the next site
startTime = commonTmsMetric.StartofYear(new Date(yesterday.getTime()));
}
}
There are a few things that are not efficient in this code but when I look at the logs it only adds a few seconds to the whole processing.
I must be missing something big.
I have ran this with 2 executors and 1 executor and the difference is 20 seconds on 5 minutes.
This is running with Java 1.7 and Spark 1.4.1 on Hadoop 2.5.0.
Thank you!

So there a few things, but its hard to say without seeing the breakdown of the different tasks & their time. The short version is you are doing way to much work in the driver and not taking advantage of Spark's distributed capabilities.
For example, you are collecting all of the data back to the driver program (toArray() and your for loop). Instead you should just point Spark SQL at the files in needs to load.
For the operators, it seems like your doing many aggregations in the driver, instead you could use the driver to generate the aggregations and have Spark SQL execute them.
Another big difference between your in-house code and the DataFrame code is going to be Schema inference. Since you've already created classes to represent your data, it seems likely that you know the schema of your JSON data. You can likely speed up your code by adding the schema information at read time so Spark SQL can skip inference.
I'd suggest re-visiting this approach and trying to build something using Spark SQL's distributed operators.

How to use all the cores of Solr in solrj

I have downloaded solr 5.2.0 and have started using $solr_home/bin/solr start
The Logs stated:
Waiting to see Solr listening on port 8983 [/]
Started Solr server on port 8983 (pid=17330). Happy searching!
Then I visited http://localhost:8983/solr and created a new core using Core Admin / new Core as Core1 ( using solr-5.2.0/server/solr/configsets/data_driven_schema_configs/config for the new core)
Then I have written a java code that indexes a few records
the Code is as follows:
public static void main(String[] args) throws IOException, SolrServerException {
HttpSolrClient solrClient = new HttpSolrClient("http://localhost:8983/solr/core1"); //$NON-NLS-1$4
//HttpSolrClient solrClient = new HttpSolrClient(args[0]); //$NON-NLS-1$4
// Empty the database...
solrClient.deleteByQuery( "*:*" );// delete everything! //$NON-NLS-1$
System.out.println("cleared"); //$NON-NLS-1$
ArrayList<SolrInputDocument> docs = new ArrayList<>();
long starttime = System.currentTimeMillis();
for (int i = 0; i < 1000000; ++i) {
SolrInputDocument doc = new SolrInputDocument();
doc.addField("bat", "book"+i); //$NON-NLS-1$ //$NON-NLS-2$
doc.addField("id", "book id -" + i); //$NON-NLS-1$ //$NON-NLS-2$
doc.addField("name", "The Legend of the Hobbit part 1 " + i); //$NON-NLS-1$ //$NON-NLS-2$
doc.addField("id1", "book id -" + i); //$NON-NLS-1$ //$NON-NLS-2$
doc.addField("name1", "The Legend of the Hobbit part 2 " + i); //$NON-NLS-1$ //$NON-NLS-2$
doc.addField("id2", "book id -" + i); //$NON-NLS-1$ //$NON-NLS-2$
docs.add(doc);
if (i % 250000 == 0) {
solrClient.add(docs);
System.out.println("added "+ i + "documents"); //$NON-NLS-1$ //$NON-NLS-2$
docs.clear();
}
}
solrClient.add(docs);
System.out.println("completed adding to Solr. Now commiting.. Please wait"); //$NON-NLS-1$
solrClient.commit();
long endtime = System.currentTimeMillis();
System.out.println("process completed in "+(endtime-starttime)/1000+" seconds"); //$NON-NLS-1$ //$NON-NLS-2$
}
After running the code , I check
http://localhost:8983/solr/#/~cores/core1
and seen 1000000 documents indexed.
Then I added another core (named core2 in similar fashion to core 1 ) and then again ran the job, this time , I saw that core 2 didnot show any documents , only core 1 still showed.
Can anybody suggest how to use both the cores of solr to distribute and store the document so that I can index data faster, My assumption is that If I increase the number of cores, the speed of Indexing should Increase.
Kindly anybody let me know if any one has tried and succedded in utilizing both the cores and seen performance improvement.
Thanks.

here in your code it still points to core1.
HttpSolrClient solrClient = new HttpSolrClient("http://localhost:8983/solr/core1"
In case you want to have the indexex for core2
you need to change here
HttpSolrClient solrClient = new HttpSolrClient("http://localhost:8983/solr/core2"
after this change try run the job, it will index for the core2.

How to extend expiration time java json web token?

I try to create Json Web Token in Java with jjwt library
But I have problem when I try to extend the expiration time.
I try it by the code below.
public class Main {
public static void main(String args[]) {
byte[] key = new byte[64];
new SecureRandom().nextBytes(key);
Date date = new Date();
long t = date.getTime();
Date expirationTime = new Date(t + 5000l); // set 5 seconds
String compact = Jwts.builder().setSubject("Joe").setExpiration(expirationTime).signWith(SignatureAlgorithm.HS256, key).compact();
System.out.println("compact : " + compact);
try {
String unpack = Jwts.parser().setSigningKey(key).parseClaimsJws(compact).getBody().getSubject();
System.out.println("unpackage 0 : " + unpack);
// check if the expiration work.
Thread.sleep(3000);
System.out.println("unpackage 1 : " + Jwts.parser().setSigningKey(key).parseClaimsJws(compact).getBody().getSubject());
//extend the expration time.
Date date1 = new Date();
long t1 = date1.getTime();
Date expirationTime1 = new Date(t1 + 5000l); //prolongation 5 seconds
Jwts.parser().setSigningKey(key).parseClaimsJws(compact).getBody().setExpiration(expirationTime1).getSubject();
// check if the extend expiration work.
Thread.sleep(3000);
System.out.println("unpackage 2 : " + Jwts.parser().setSigningKey(key).parseClaimsJws(compact).getBody().getSubject());
} catch (InterruptedException | ExpiredJwtException ex) {
System.out.println("exception : " + ex.getMessage());
Thread.currentThread().interrupt();
}
}
The result is :
compact : eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJKb2UiLCJleHAiOjE0Mjk2NjU1MjB9.oMY2mDHvNoMZqBfic41LbiKvAyi93wIfu_WgIADb9Wc
unpackage 0 : Joe
unpackage 1 : Joe
exception : JWT expired at 2015-04-22T08:18:40+0700. Current time: 2015-04-22T08:18:42+0700
So it mean, the unpackage2 cant run, Because it was expiration.
I trying to extend the expiration time.
Because I apply the code on web application.
If user still connect with my application, He should not get token timeout.
I have found another question like mine.

The problem is with the parsing code:
Jwts.parser().setSigningKey(key).parseClaimsJws(compact).getBody().setExpiration(expirationTime1).getSubject();
In this line, you're modifying the JWT that is returned by the parser. In other words, the above is equivalent to this:
Jws<Claims> jws = Jwts.parser().setSigningKey(key).parseClaimsJws(compact);
jws.getBody().setExpiration(expirationTime1).getSubject();
Notice how this code modifies the JWT returned by the parser? It does not - and cannot - modify the JWT represented by the original compact String.
Your next line of code after that tries to parse the original (unmodified) compact String:
// check if the extend expiration work.
Thread.sleep(3000);
System.out.println("unpackage 2 : " + Jwts.parser().setSigningKey(key).parseClaimsJws(compact).getBody().getSubject());
But we know this won't work because changing the state of the JWT returned by the parser does not have any effect on the original compact String.
If your user presents a JWT to your web application and you want to 'extend the life' of the token so it won't expire, you must generate a new JWT and send that JWT back to the user. The user should send the new JWT back on future requests. You keep repeating this process for as long as you want to allow the user to keep talking to your web application without having to re-login again.
I should point out that if you don't want to worry about any of this stuff, Stormpath can perform user and JWT token authentication between the browser and your app automatically for you - you don't have to build any of this yourself (disclosure: I'm Stormpath's CTO).
Finally, you might be interested to know that JJWT's test suite already validates the correct behavior in numerous places for both expired and premature token use cases:
https://github.com/jwtk/jjwt/blob/0.4/src/test/groovy/io/jsonwebtoken/JwtParserTest.groovy#L163-L189
https://github.com/jwtk/jjwt/blob/0.4/src/test/groovy/io/jsonwebtoken/JwtParserTest.groovy#L307-L335
https://github.com/jwtk/jjwt/blob/0.4/src/test/groovy/io/jsonwebtoken/JwtParserTest.groovy#L421-L454
But, you don't have to take my word for it :) Here is your code, modified so that your expiration modifications function as described:
public class Main {
public static void main(String args[]) {
byte[] key = new byte[64];
new SecureRandom().nextBytes(key);
Date date = new Date();
long t = date.getTime();
Date expirationTime = new Date(t + 5000l); // set 5 seconds
String compact = Jwts.builder().setSubject("Joe").setExpiration(expirationTime).signWith(SignatureAlgorithm.HS256, key).compact();
System.out.println("compact : " + compact);
try {
String unpack = Jwts.parser().setSigningKey(key).parseClaimsJws(compact).getBody().getSubject();
System.out.println("unpackage 0 : " + unpack);
// check if the expiration work.
Thread.sleep(3000);
System.out.println("unpackage 1 : " + Jwts.parser().setSigningKey(key).parseClaimsJws(compact).getBody().getSubject());
//Create a *new* token that reflects a longer extended expiration time.
Date date1 = new Date();
long t1 = date1.getTime();
Date expirationTime1 = new Date(t1 + 5000l); //prolongation 5 seconds
String compact2 = Jwts.builder().setSubject("Joe").setExpiration(expirationTime1).signWith(SignatureAlgorithm.HS256, key).compact();
// check if the extend expiration work.
Thread.sleep(3000);
System.out.println("unpackage 2 : " + Jwts.parser().setSigningKey(key).parseClaimsJws(compact2).getBody().getSubject());
Thread.sleep(1000);
} catch (InterruptedException | ExpiredJwtException ex) {
System.out.println("exception : " + ex.getMessage());
Thread.currentThread().interrupt();
}
}
}
Notice that a 2nd new JWT (compact2) needed to be generated to reflect the new/latest expiration time. You cannot modify a parsed JWT and expect the changes to apply to the original compact value.
In summary, use Jwts.parser() when you need to parse a JWT string to get a nice Java object representation of the JWT. Use Jwts.builder() when you need to create or modify a JWT to produce a new compact String representation.
I hope that helps!

Calculate client-server time difference in Borland Starteam server 8

Problem. I need a way to find Starteam server time through Starteam Java SDK 8.0. Version of server is 8.0.172 so method Server.getCurrentTime() is not available since it was added only in server version 9.0.
Motivation. My application needs to use views at specific dates. So if there's some difference in system time between client (where the app is running) and server then obtained views are not accurate. In the worst case the client's requested date is in the future for server so the operation results in exception.

After some investigation I haven't found any cleaner solution than using a temporary item. My app requests the item's time of creation and compares it with local time. Here's the method I use to get server time:
public Date getCurrentServerTime() {
Folder rootFolder = project.getDefaultView().getRootFolder();
Topic newItem = (Topic) Item.createItem(project.getTypeNames().TOPIC, rootFolder);
newItem.update();
newItem.remove();
newItem.update();
return newItem.getCreatedTime().createDate();
}

If your StarTeam server is on a Windows box and your code will be executing on a Windows box, you could shell out and execute the NET time command to fetch the time on that machine and then compare it to the local time.
net time \\my_starteam_server_machine_name
which should return:
"Current time at \\my_starteam_server_machine_name is 10/28/2008 2:19 PM"
"The command completed successfully."

We needed to come up with a way of finding the server time for use with CodeCollab. Here is a (longish) C# code sample of how to do it without creating a temporary file. Resolution is 1 second.
static void Main(string[] args)
{
// ServerTime replacement for pre-2006 StarTeam servers.
// Picks a date in the future.
// Gets a view, sets the configuration to the date, and tries to get a property from the root folder.
// If it cannot retrieve the property, the date is too far in the future. Roll back the date to an earlier time.
DateTime StartTime = DateTime.Now;
Server s = new Server("serverAddress", 49201);
s.LogOn("User", "Password");
// Getting a view - doesn't matter which, as long as it is not deleted.
Project p = s.Projects[0];
View v = p.AccessibleViews[0]; // AccessibleViews saves checking permissions.
// Timestep to use when searching. One hour is fairly quick for resolution.
TimeSpan deltaTime = new TimeSpan(1, 0, 0);
deltaTime = new TimeSpan(24 * 365, 0, 0);
// Invalid calls return faster - start a ways in the future.
TimeSpan offset = new TimeSpan(24, 0, 0);
// Times before the view was created are invalid.
DateTime minTime = v.CreatedTime;
DateTime localTime = DateTime.Now;
if (localTime < minTime)
{
System.Console.WriteLine("Current time is older than view creation time: " + minTime);
// If the dates are so dissimilar that the current date is before the creation date,
// it is probably a good idea to use a bigger delta.
deltaTime = new TimeSpan(24 * 365, 0, 0);
// Set the offset to the minimum time and work up from there.
offset = minTime - localTime;
}
// Storage for calculated date.
DateTime testTime;
// Larger divisors converge quicker, but might take longer depending on offset.
const float stepDivisor = 10.0f;
bool foundValid = false;
while (true)
{
localTime = DateTime.Now;
testTime = localTime.Add(offset);
ViewConfiguration vc = ViewConfiguration.CreateFromTime(testTime);
View tempView = new View(v, vc);
System.Console.Write("Testing " + testTime + " (Offset " + (int)offset.TotalSeconds + ") (Delta " + deltaTime.TotalSeconds + "): ");
// Unfortunately, there is no isValid operation. Attempting to
// read a property from an invalid date configuration will
// throw an exception.
// An alternate to this would be proferred.
bool valid = true;
try
{
string testname = tempView.RootFolder.Name;
}
catch (ServerException)
{
System.Console.WriteLine(" InValid");
valid = false;
}
if (valid)
{
System.Console.WriteLine(" Valid");
// If the last check was invalid, the current check is valid, and
// If the change is this small, the time is very close to the server time.
if (foundValid == false && deltaTime.TotalSeconds <= 1)
{
break;
}
foundValid = true;
offset = offset.Add(deltaTime);
}
else
{
offset = offset.Subtract(deltaTime);
// Once a valid time is found, start reducing the timestep.
if (foundValid)
{
foundValid = false;
deltaTime = new TimeSpan(0,0,Math.Max((int)(deltaTime.TotalSeconds / stepDivisor), 1));
}
}
}
System.Console.WriteLine("Run time: " + (DateTime.Now - StartTime).TotalSeconds + " seconds.");
System.Console.WriteLine("The local time is " + localTime);
System.Console.WriteLine("The server time is " + testTime);
System.Console.WriteLine("The server time is offset from the local time by " + offset.TotalSeconds + " seconds.");
}
Output:
Testing 4/9/2009 3:05:40 PM (Offset 86400) (Delta 31536000): InValid
Testing 4/9/2008 3:05:40 PM (Offset -31449600) (Delta 31536000): Valid
...
Testing 4/8/2009 10:05:41 PM (Offset 25200) (Delta 3): InValid
Testing 4/8/2009 10:05:38 PM (Offset 25197) (Delta 1): Valid
Run time: 9.0933426 seconds.
The local time is 4/8/2009 3:05:41 PM
The server time is 4/8/2009 10:05:38 PM
The server time is offset from the local time by 25197 seconds.

<stab_in_the_dark>
I'm not familiar with that SDK but from looking at the API if the server is in a known timezone why not create and an OLEDate object whose date is going to be the client's time rolled appropriately according to the server's timezone?
</stab_in_the_dark>