Develop Reference

java.lang.IllegalArgumentException: Must supply a valid URL

Im trying to build a web crawler for my OOP class. The crawler needs to traverse 1000 wikipedia pages and collect the titles and words off the page. The current code I have will traverse a singular page and collect the required information but it also gives me the error code "java.lang.IllegalArgumentException: Must supply a valid URL:" Here is my crawlers code. Ive been using Jsoups libraries.
import java.util.HashMap;
import java.util.HashSet;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class crawler {
private static final int MAX_PAGES = 1000;
private final HashSet<String> titles = new HashSet<>();
private final HashSet<String> urlVisited = new HashSet<>();
private final HashMap<String, Integer> map = new HashMap<>();
public void getLinks(String startURL) {
if ((titles.size() < MAX_PAGES) && !urlVisited.contains(startURL)) {
urlVisited.add(startURL);
try {
Document doc = Jsoup.connect(startURL).get();
Elements linksFromPage = doc.select("a[href]");
String title = doc.select("title").first().text();
titles.add(title);
String text = doc.body().text();
CountWords(text);
for (Element link : linksFromPage) {
if(titles.size() <= MAX_PAGES) {
Thread.sleep(50);
getLinks(link.attr("a[href]"));
}
else {
System.out.println("URL couldn't visit");
System.out.println(startURL + ", " + urlVisited.size());
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
public void PrintAllTitles() {
for (String t : titles) {
System.out.println(t);
}
}
public void PrintAllWordsAndCount() {
for (String key : map.keySet()) {
System.out.println(key + " : " + map.get(key));
}
}
private void CountWords(String text) {
String[] lines = text.split(" ");
for (String word : lines) {
if (map.containsKey(word)) {
int val = map.get(word);
val += 1;
map.remove(word);
map.put(word, val);
} else {
map.put(word, 1);
}
}
}
}
The Driver function just uses c.getLinks(https://en.wikipedia.org/wiki/Computer)
as the starting URL.

The issue is in this line:
getLinks(link.attr("a[href]"));
link.attr(attributeName) is a method for getting an element's attribute by name. But a[href] is a CSS selector. So that method call returns a blank String (as there is no attribute in the element named a[href]), which is not a valid URL, and so you get the validation exception.
Before you call connect, you should log the URL you are about to hit. That way you will see the error.
You should change the line to:
getLinks(link.attr("abs:href"));
That will get the absolute URL pointed to by the href attribute. Most of the hrefs on that page are relative, so it's important to make them absolute before they are made into a URL for connect().
You can see the URLs that the first a[href] selector will return here. You should also think about how to only fetch HTML pages and not images (e.g., maybe filter out by filetype).
There is more detail and examples of this area in the Working with URLs article of jsoup.

Retrieving list of Tags from file properties

I would like to retrieve the list of Tags attached to a file in Windows 7 programatically. I am trying to create a mapping of file->tags that I can move across different platforms.
Is anyone aware of a library, or a way to get the 'Tags' values from command line? So far I have only been able to find ways to get basic file attributes such as Author, Date Created, etc.
I am unable to load PowerShell scripts on the computer unfortunately so am not able to make use of those features.
I tried using 'UserDefinedFileAttributeView' but that did not return any values, like so:
private LinkedList<String> windowsGetAllFileTags(File file) {
UserDefinedFileAttributeView fileAttributeView = Files.getFileAttributeView(file.toPath().toAbsolutePath(), UserDefinedFileAttributeView.class);
List<String> allAttributes = null;
try {
allAttributes = fileAttributeView.list();
} catch (IOException e) {
e.printStackTrace();
}
for(String attribute : allAttributes) {
System.out.println("Attribute = " + attribute);
}
return null;
}
An image of the Windows 7 Properties View

There is a Java library written and called as PE/COFF 4J on Github.
import java.io.IOException;
import org.boris.pecoff4j.PE;
import org.boris.pecoff4j.ResourceDirectory;
import org.boris.pecoff4j.ResourceEntry;
import org.boris.pecoff4j.constant.ResourceType;
import org.boris.pecoff4j.io.PEParser;
import org.boris.pecoff4j.io.ResourceParser;
import org.boris.pecoff4j.resources.StringFileInfo;
import org.boris.pecoff4j.resources.StringTable;
import org.boris.pecoff4j.resources.VersionInfo;
import org.boris.pecoff4j.util.ResourceHelper;
public class Main {
public static void main(String[] args) throws IOException {
PE pe = PEParser.parse("C:/windows/system32/notepad.exe");
ResourceDirectory rd = pe.getImageData().getResourceTable();
ResourceEntry[] entries = ResourceHelper.findResources(rd, ResourceType.VERSION_INFO);
for (int i = 0; i < entries.length; i++) {
byte[] data = entries[i].getData();
VersionInfo version = ResourceParser.readVersionInfo(data);
StringFileInfo strings = version.getStringFileInfo();
StringTable table = strings.getTable(0);
for (int j = 0; j < table.getCount(); j++) {
String key = table.getString(j).getKey();
String value = table.getString(j).getValue();
System.out.println(key + " = " + value);
}
}
}
}
Will print:
CompanyName = Microsoft Corporation
FileDescription = Notepad
FileVersion = 6.1.7600.16385 (win7_rtm.090713-1255)
InternalName = Notepad
LegalCopyright = © Microsoft Corporation. All rights reserved.
OriginalFilename = NOTEPAD.EXE
ProductName = Microsoft® Windows® Operating System
ProductVersion = 6.1.7600.16385
If you mention of obtaining tags of images or videos, #Drew Noakes has written Java library called as metadata-extractor for it.
Metadata metadata = ImageMetadataReader.readMetadata(imagePath);
To iterate all values in the file:
for (Directory directory : metadata.getDirectories()) {
for (Tag tag : directory.getTags()) {
System.out.println(tag);
}
}
You can also read specific values from specific directories:
// obtain the Exif SubIFD directory
ExifSubIFDDirectory directory
= metadata.getFirstDirectoryOfType(ExifSubIFDDirectory.class);
// query the datetime tag's value
Date date = directory.getDate(ExifSubIFDDirectory.TAG_DATETIME_ORIGINAL);
The library is available for Maven users too.

In Windows PowerShell, you could grab it with a bit of help from PresentationCore.dll:
function Get-ImageTags {
param(
[string]$Path
)
Add-Type -AssemblyName PresentationCore
try {
$FileStream = (Get-Item $Path).Open('Open','Read')
$BitmapFrame = [System.Windows.Media.Imaging.BitmapFrame]::Create($FileStream)
$Tags = #($BitmapFrame.Metadata.Keywords |%{ $_ })
}
catch {
throw
return
}
finally {
if($FileStream){
$FileStream.Dispose()
}
}
return $Tags
}
Then use like:
$Tags = Get-ImageTags -Path path\to\file.jpeg
The $Tags variable will now contain an array of tags

what about Files.getAttribute
I didn't tried that but probably this could work:
Files.getAttribute(Paths.get("/some/dir","file.txt"), "description:tags")

Need to pass test case in QC through Java

Could any one help me in below issue
I want to pass test cases in QC through Java, I used con4j and reached till test sets but I am unable to fetch the test cases under respective test set.
could any one please help me in how to pass test cases in QC through com4j
import com.qc.ClassFactory;
import com.qc.ITDConnection;
import com.qc.ITestLabFolder;
import com.qc.ITestSetFactory;
import com.qc.ITestSetTreeManager;
import com.qc.ITestSetFolder;
import com.qc.IList;
import com.qc.ITSTest;
import com.qc.ITestSet;
import com.qc.ITestFactory;
import com4j.*;
import com4j.stdole.*;
import com4j.tlbimp.*;
import com4j.tlbimp.def.*;
import com4j.tlbimp.driver.*;
import com4j.util.*;
import com4j.COM4J;
import java.util.*;
import com.qc.IRun;
import com.qc.IRunFactory;
public class Qc_Connect {
public static void main(String[] args) {
// TODO Auto-generated method stub
String url="http://abc/qcbin/";
String domain="abc";
String project="xyz";
String username="132222";
String password="Xyz";
String strTestLabPath = "Root\\Test\\";
String strTestSetName = "TestQC";
try{
ITDConnection itd=ClassFactory.createTDConnection();
itd.initConnectionEx(url);
System.out.println("COnnected To QC:"+ itd.connected());
itd.connectProjectEx(domain,project,username,password);
System.out.println("Logged into QC");
//System.out.println("Project_Connected:"+ itd.connected());
ITestSetFactory objTestSetFactory = (itd.testSetFactory()).queryInterface(ITestSetFactory.class);
ITestSetTreeManager objTestSetTreeManager = (itd.testSetTreeManager()).queryInterface(ITestSetTreeManager.class);
ITestSetFolder objTestSetFolder =(objTestSetTreeManager.nodeByPath(strTestLabPath)).queryInterface(ITestSetFolder.class);
IList its1 = objTestSetFolder.findTestSets(strTestSetName, true, null);
//IList ls= objTestSetFolder.findTestSets(strTestSetName, true, null);
System.out.println("No. of Test Set:" + its1.count());
ITestSet tst= (ITestSet) objTestSetFolder.findTestSets(strTestSetName, true, null).queryInterface(ITSTest.class);
System.out.println(tst.name());
//System.out.println( its1.queryInterface(ITestSet.class).name());
/* foreach (ITestSet testSet : its1.queryInterface(ITestSet.class)){
ITestSetFolder tsFolder = (ITestSetFolder)testSet.TestSetFolder;
ITSTestFactory tsTestFactory = (ITSTestFactory)testSet.TSTestFactory;
List tsTestList = tsTestFactory.NewList("");
}*/
/* Com4jObject comObj = (Com4jObject) its1.item(0);
ITestSet tst = comObj.queryInterface(ITestSet.class);
System.out.println("Test Set Name : " + tst.name());
System.out.println("Test Set ID : " + tst.id());
System.out.println("Test Set ID : " + tst.status());
System.out.println("Test Set ID : " );*/
System.out.println(its1.count());
System.out.println("TestSet Present");
Iterator itr = its1.iterator();
System.out.println(itr.hasNext());
while (itr.hasNext())
{
Com4jObject comObj = (Com4jObject) itr.next();
ITestSet sTestSet = comObj.queryInterface(ITestSet.class);
System.out.println(sTestSet.name());
Com4jObject comObj2 = sTestSet.tsTestFactory();
ITestSetFactory test = comObj2.queryInterface(ITestSetFactory.class);
}
// ITSTest tsTest=null;
// tsTest.
//its1.
/* comObj = (Com4jObject) its1.item(1);
ITSTest tst2=comObj.queryInterface(ITSTest.class);*/
// System.out.println( tst2.name());
/* foreach (ITSTest tsTest : tst2)
{
IRun lastRun = (IRun)tsTest.lastRun();
if (lastRun == null)
{
IRunFactory runFactory = (IRunFactory)tsTest.runFactory;
String date = "20160203";
IRun run = (IRun)runFactory.addItem( date);
run.status("Pass");
run.autoPost();
}
}*/
}
catch(Exception e){
e.printStackTrace();
}
}
}

I know the post is quite old. I have to struggle alot in OTA with Java and couldn't get a complete post for solving the issue.
Now i have running code after too much research.
so thought of sharing my code in case someone is looking for help.
Here is complete Solution.
`
ITestFactory sTestFactory = (connection.testFactory())
.queryInterface(ITestFactory.class);
ITest iTest1 = (sTestFactory.item(12081)).queryInterface(ITest.class);
System.out.println(iTest1.execDate());
System.out.println(iTest1.name());
ITestSetFactory sTestSetFactory = (connection.testSetFactory())
.queryInterface(ITestSetFactory.class);
ITestSet sTestSet = (sTestSetFactory.item(1402))
.queryInterface(ITestSet.class);
System.out.println(sTestSet.name() + "\n Test Set ID" + sTestSet.id());
IBaseFactory testFactory1 = sTestSet.tsTestFactory().queryInterface(
IBaseFactory.class);
testFactory1.addItem(iTest1);
System.out.println("Test case has been Added");
System.out.println(testFactory1.newList("").count());
IList tsTestlist = testFactory1.newList("");
ITSTest tsTest;
for (int tsTestIndex = 1; tsTestIndex <= tsTestlist.count(); tsTestIndex++) {
Com4jObject comObj = (Com4jObject) tsTestlist.item(tsTestIndex);
tsTest = comObj.queryInterface(ITSTest.class);
if (tsTest.name().equalsIgnoreCase("[3]TC_OTA_API_Test")) {
System.out.println("Hostname" + tsTest.hostName() + "\n"
+ tsTest.name() + "\n" + tsTest.status());
IRun lastRun = (IRun) tsTest.lastRun();
// IRun lastRun = comObjRun.queryInterface(IRun.class);
// don't update test if it may have been modified by someone
// else
if (lastRun == null) {
System.out.println("I am here last Run = Null");
runFactory = tsTest.runFactory().queryInterface(
IRunFactory.class);
System.out.println(runFactory.newList("").count());
String runName = "TestRun_Automated";
Com4jObject comObjRunForThisTS = runFactory
.addItem(runName);
IRun runObjectForThisTS = comObjRunForThisTS
.queryInterface(IRun.class);
runObjectForThisTS.status("Passed");
runObjectForThisTS.post();
runObjectForThisTS.refresh();
}
}
}
`

Why not build a client to access the REST API instead of passing through the OTA interface?
Once you build a basic client, you can post runs and update their status quite easily.

If you use c#/vb.net this has been easily completed. But you are working on java, I would suggest to provide interface above dlls to deal with operation. This will be much more easier than using com4j.
Similar query, probably following may help you. I would suggest to drop idea of using com4j and use solution provided in thread below which is proven,fail safe and auto-recoverable.
QC API JAR to connect using java
it was always been difficult to use com4j specially for HPQC/ALM. As dlls for QC are faulty and there are memory leaking/allocation problems which crashes dll executions frequently on certain platforms.

MongoDB Java driver: Undefined values are not shown

Open mongo shell and create a document with a undefined value:
> mongo
MongoDB shell version: 2.4.0
connecting to: test
> use mydb
switched to db mydb
> db.mycol.insert( {a_number:1, a_string:"hi world", a_null:null, an_undefined:undefined} );
> db.mycol.findOne();
{
"_id" : ObjectId("51c2f28a7aa5079cf24e3999"),
"a_number" : 1,
"a_string" : "hi world",
"a_null" : null,
"an_undefined" : null
}
As we can see, javascript translates the "undefined" value (stored in the db) to a "null" value, when showing it to the user. But, in the db, the value is still "undefined", as we are going to see with java.
Let's create a "bug_undefined_java_mongo.java" file, with the following content:
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.MongoClient;
public class bug_undefined_java_mongo
{
String serv_n = "myserver"; // server name
String db_n = "mydb"; // database name
String col_n = "mycol"; // collection name
public static void main(String[] args)
{
new bug_undefined_java_mongo().start();
}
public void start()
{
pr("Connecting to server ...");
MongoClient cli = null;
try
{
cli = new MongoClient( serv_n );
}
catch (Exception e)
{
pr("Can't connecto to server: " + e);
System.exit(1);
}
if (cli == null)
{
pr("Can't connect to server");
System.exit(1);
}
pr("Selecting db ...");
DB db_res = cli.getDB( db_n );
pr("Selecting collection ...");
DBCollection col = db_res.getCollection( col_n );
pr("Searching documents ...");
DBCursor cursor = null;
try
{
cursor = col.find( );
}
catch (Exception e)
{
pr("Can't search for documents: " + e);
System.exit(1);
}
pr("Printing documents ...");
try
{
while (cursor.hasNext())
{
Object doc_obj = cursor.next();
System.out.println("doc: " + doc_obj);
}
}
catch (Exception e)
{
pr("Can't browse documents: " + e);
return;
}
finally
{
pr("Closing cursor ...");
cursor.close();
}
}
public void pr(String cad)
{
System.out.println(cad);
}
}
After compiling and running it, we get this:
Connecting to server ...
Selecting db ...
Selecting collection ...
Searching documents ...
Printing documents ...
doc: { "_id" : { "$oid" : "51c2f0f85353d3425fcb5a14"} , "a_number" : 1.0 , "a_string" : "hi world" , "a_null" : null }
Closing cursor ...
We see that the "a_null:null" pair is shown, but... the "an_undefined:undefined" pair has disappeared! (both the key and the value).
Why? Is it a bug?
Thank you

Currently undefined is not supported by the java driver as there is no equivalent mapping in java.
Other drivers such as pymongo and the js shell handles this differently by casting undefined to None when representing the data, however it is a separate datatype and is deprecated in the bson spec.
If you need it in the java driver then you will have to code your own decoder factory and then set it like so:
collection.setDBDecoderFactory(MyDecoder.FACTORY);
A minimal example that has defined handling for undefined and factory is available on github in the horn of mongo repo.

I see, creating a factory could be a solution.
Anyway, probably many developers would find it useful the posibility of enabling a mapping in the driver to convert automatically "undefined" values to "null" value. For example, by calling a mapUndefToNull() method:
cli = new MongoClient( myserver );
cli.mapUndefToNull(true);
In my case, I'm running a MapReduce (it is Javascript code) on my collections, and I am having to explicitly convert the undefined values (generated when accessing to non existent keys) to null, in order to avoid Java driver to remove it:
try { value = this[ key ] } catch(e) {value = null}
if (typeof value == "undefined") value = null; // avoid Java driver to remove it
So, as a suggestion, I'd like the mapUndefToNull() method to be added to the Java driver. If possible.
Thank you

Android - getType(int) is undefined for the type Cursor

I am new on mobile development, so I decided to use Phonegap to develop my application. In my app I am using a SQLite plugin, because WebSQL is not attending my necessities. When I try to put one of the files required to use the plugin in my project, Eclipse return me errors in the following function:
public void processResults(Cursor cur, String query_id, String tx_id) {
String result = "[]";
// If query result has rows
if (cur.moveToFirst()) {
JSONArray fullresult = new JSONArray();
String key = "";
int colCount = cur.getColumnCount();
// Build up JSON result object for each row
do {
JSONObject row = new JSONObject();
try {
for (int i = 0; i < colCount; ++i) {
key = cur.getColumnName(i);
// for old Android SDK remove lines from HERE:
if(android.os.Build.VERSION.SDK_INT >= 11)
{
switch(cur.getType (i))
{
case Cursor.FIELD_TYPE_NULL:
row.put(key, null);
break;
case Cursor.FIELD_TYPE_INTEGER:
row.put(key, cur.getInt(i));
break;
case Cursor.FIELD_TYPE_FLOAT:
row.put(key, cur.getFloat(i));
break;
case Cursor.FIELD_TYPE_STRING:
row.put(key, cur.getString(i));
break;
case Cursor.FIELD_TYPE_BLOB:
row.put(key, cur.getBlob(i));
break;
}
}
else // to HERE.
{
row.put(key, cur.getString(i));
}
}
fullresult.put(row);
} catch (JSONException e) {
e.printStackTrace();
}
} while (cur.moveToNext());
result = fullresult.toString();
}
if(query_id.length() > 0)
this.sendJavascript(" SQLitePluginTransaction.queryCompleteCallback('" + tx_id + "','" + query_id + "', " + result + ");");
}
In the line 20 (when we have "switch(cur.getType (i))"), Eclipse returns the following error:
The method getType(int) is undefined for the type Cursor
When I saw this, I tried googling it, and on the Android's documentation says that getType is a Cursor's method.
Here is what this file imports:
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import com.phonegap.api.Plugin;
import com.phonegap.api.PluginResult;
import android.database.Cursor;
import android.database.sqlite.*;
import android.util.Log;
As don't know Java, I could not figure this out and I haven't found anything on my search, but I may be missing something. I hope someone can help me, thanks

Which version of Android are you targeting? The getType method was added in API 11 (Honeycomb).
You can see this on the right hand side of the header as ('Since: API Level 11') and you may find the 'Filter by API level' box at the top of the reference pages useful.