Develop Reference

Why is a blank Java Icon appearing when I parse a PDF file using Tabula?

I am working on an integration with Apache Drill which enables users to query PDF files directly using SQL. I'm about 80% done and really impressed with how well Tabula works for this.
However, when I execute the first Drill query that uses the Tabula libraries a Java icon pops up and I get the following text in the command line:
2020-10-25 15:06:55.770 java[71188:7121498] Persistent UI failed to open file file://localhost/Users/******/Saved%20Application%20State/net.java.openjdk.cmd.savedState/window_1.data: Permission denied (13)
I changed the permissions on that directory but I'm still getting the Java popup.
This is not normal behavior for Drill and my goal here was to integrate Tabula programmatically. Is Tabula trying to open a window or something like that and if so, is there a way to disable this? I noted that this does not occur in my unit tests.
Here are some relevant code snippets:
public static List<Table> extractTablesFromPDF(PDDocument document, ExtractionAlgorithm algorithm) {
NurminenDetectionAlgorithm detectionAlgorithm = new NurminenDetectionAlgorithm();
ExtractionAlgorithm algExtractor;
SpreadsheetExtractionAlgorithm extractor=new SpreadsheetExtractionAlgorithm();
ObjectExtractor objectExtractor = new ObjectExtractor(document);
PageIterator pages = objectExtractor.extract();
List<Table> tables= new ArrayList<>();
while (pages.hasNext()) {
Page page = pages.next();
algExtractor = algorithm;
/*if (extractor.isTabular(page)) {
algExtractor=new SpreadsheetExtractionAlgorithm();
}
else {
algExtractor = new BasicExtractionAlgorithm();
}*/
List<Rectangle> tablesOnPage = detectionAlgorithm.detect(page);
for (Rectangle guessRect : tablesOnPage) {
Page guess = page.getArea(guessRect);
tables.addAll(algExtractor.extract(guess));
}
}
return tables;
}
This doesn't happen in my unit tests.
Thanks in advance for your help!

Because some code is executed that does an operation that is usually, but technically not necessarily, involved in things that require so-called 'headful' mode (well, that's perhaps not really a term, but the opposite, 'headless' certainly is). This causes a few things to happen, including that icon showing up.
One easy way out of this is to force headless mode. But note that when you do this, any of these 'usually but technically not neccessarily headful' operations may either [1] work fine and no longer show that icon, or, [2] crash with a HeadlessException. Which one you end up with is not just dependent on which operation you're doing, but also which VM you are doing it on - as a rule once one of these ops works fine and no longer throws, later versions won't revert back to throwing (in other words, newer versions of java offer more things that work in headless mode).
To force headless mode, run java with java -Djava.awt.headless=true.
If you must do it from within java code, run System.setProperty("java.awt.headless", "true"); at least once, and before you do any of these 'usually causes headful mode' operations.
Presumably, the thing that is causes headful mode to occur is something graphics involved, such as rendering a JPG or PNG into an ImageBuffer. It's not surprising that Apache Drill is doing this to 'read' images, for example.
Another option is to just upgrade your VM, maybe that helps. As a general rule, features 'move downwards' on this line:
Requires headful mode; running it makes the VM go headful (icon appears); if java.awt.headless is set, the operation fails with a HeadlessException.
Causes headful mode; running it makes the VM go headful. However, if headless is set, it works fine and won't do that.
Completely freed. Running the code works fine and does not cause the VM to go headful. the headless flag has no bearing whatsoever on how the code operates.

Trying to suppress errors while attaching browser

EDIT: I'm using the LeanFT Java SDK 14.50
EDIT2: for text clarification
I'm writing test scripts for a web application that sometimes opens popup browsers for specific actions. So natually when that happens, I will attach the new browser using BrowserFactory.attach(...). The problem is that leanFT does not seem to have a way to validate that the browser exists before attaching it, and if I try to attach it too early, it will fail. And I don't like to use an arbitrairy wait/sleep time as I can never really know how much time it's going to take for the browser to get be ready. So my solution to this is below
private Browser attachPopUpBrowser(BrowserType bt, RegExpProperty url){
Browser browser = null;
int iteration = 0;
//TimeoutLimit.SHORT = 15000
while (browser == null && iteration < TimeoutLimit.SHORT.getLimit()) {
try {
Reporter.setReportLevel(ReportLevel.Off);
browser = BrowserFactory.attach(
new BrowserDescription.Builder()
.type(bt)
.url(url)
.build()
);
Reporter.setReportLevel(ReportLevel.All);
} catch (GeneralLeanFtException e) {
try {
Thread.sleep(1000);
iteration += 1000;
} catch (InterruptedException e1) {
}
}
}
return browser;
}
Now, this works wonderfully with one exception. It generates errors in the leanft test result. Errors that I want to ignore because I know that it will fail a few times before it will succeed. As you can see, I've tried changing the ReportLevel while doing this in order to suppress the error logging, but it doesn't work. I've tried using
Browser[] browsers = BrowserFactory.getallOpenBrowsers(BrowserDescription);
thinking that it will return an empty Array if it finds nothing, but I still get errors while the browser is not ready. Does anyone have suggestions as to how I could work around this?
TL;DR
I'm looking for a way to either suppress the errors generated within my While..Loop or to validate that the browser is ready before attaching it. All of that, so that I can have a nice and clean Run Result at the end of my test (because these errors will present false negatives in all nearly all of my tests)
Addendum
Also, when the attach fails for the first time, I get a an exception
com.hp.lft.sdk.ReplayObjectNotFoundException: attachApplication
as expected, but all subsequent failures are throwing
com.hp.lft.sdk.GeneralLeanFtException: Cannot read property 'match' of null
I've compared both stack traces and they are identical except for the last 2 lines which happen within the ReplayExceptionFactory.CreateDefault() so I think that there is something that gets corrupted during the exception generation, but that is within the leanft.sdk.internal package so there might not be a lot we can do about it right now.I'm guessing that if I did not get that second "cannot read property" exception, I would correctly get the ReplayObjectNotFoundException until the browser is correctly attached.

I'd rather not force an attach endlessly until it works. Even if we'd solve the false negatives, we'd still have a not so good approach to the problem.
The cleanest solution would be to see if there is anything to attach to in the first place.
And you can do just that by getting all the browser instances that meets your description.
Browser[] browsers = BrowserFactory.getAllOpenBrowsers(new BrowserDescription.Builder().build());
Any element in this collection is an already "attached" browser - you can start using it.
If the list doesn't contain your browser instance, rerun the query.

CommandExecuteIn Background throws a "Not an (encodable) value" error

I am currently trying to implement file exports in background so that the user can do some actions while the file is downloading.
I used the apache isis CommandExexuteIn:Background action attribute. However, I got an error
"Not an (encodable) value", this is an error thrown by the ScalarValueRenderer class.
This is how my method looks like:
#Action(semantics = SemanticsOf.SAFE,
command = CommandReification.ENABLED)
commandExecuteIn = CommandExecuteIn.BACKGROUND)
public Blob exportViewAsPdf() {
final Contact contact = this;
final String filename = this.businessName + " Contact Details";
final Map<String, Object> parameters = new HashMap<>();
parameters.put("contact", contact);
final String template = templateLoader.buildFromTemplate(Contact.class, "ContactViewTemplate", parameters);
return pdfExporter.exportAsPdf(filename, template);
}
I think the error has something to do with the command not actually invoking the action but returns the persisted background command.
This implementation actually worked on the method where there is no return type. Did I miss something? Or is there a way to implement background command and get the expected results?

interesting use case, but it's not one I anticipated when that part of the framework was implemented, so I'm not surprised it doesn't work. Obviously the error message you are getting here is pretty obscure, so I've raised a
JIRA ticket to see if we could at least improve that.
I'm interested to know in what user experience you think the framework should provide here?
In the Estatio application that we work on (that has driven out many of the features added to the framework over the last few years) we have a somewhat similar requirement to obtain PDFs from a reporting server (which takes 5 to 10 seconds) and then download them. This is for all the tenants in a shopping centre, so there could be 5 to 50 of these to generate in a single go. The design we went with was to move the rendering into a background command (similar to the templateLoader.buildFromTemplate(...) and pdfExporter.exportAsPdf(...) method calls in your code fragment, and to capture the output as a Document, via the document module. We then use the pdfbox addon to stitch all the document PDFs together as a single downloadable PDF for printing.
Hopefully that gives you some ideas of a different way to support your use case
Thx
Dan

Selenium sends incomplete text

I'm having a problem with Selenium when it comes to use .sendKeys(text). During the automation process, sometimes selenium is sending incomplete strings to the browser, which causes to create incorrect searchs.
i.e. I want to type "MY DROP", and it will type "Y DROP", or "ROP".
It does not always type the same way, so sometimes 2 letters might be missing, and sometimes the whole word is missing.
This only happens to dropdowns, where I have a specific method that handles the dropdown selection, as we are using angular I can't use the selenium select dropdown method.
I already tried to set Thread.Sleeps and waits on the dropdown selection but nothing seems to work, currently this is what I use to select a value:
public void select(String item) {
waitTillClicable();
WebElement element = getElement();
openDropDown(element);
element.sendKeys(item);
waitResultLoad();
selectResult(element);
}
This code was working perfectly until the last week. I'm thinking it has something to deal with the new Chrome version 45, as before it was not happening. I also tried to use different chromedriver versions, and running on a Linux machine, but nothing seems to have an effect.
Right now I created a workaround where I keep verifying if the string was typed correctly, and re-typing it until it is correct, but this makes the execution time increased, which I wanted to avoid.

Why are you using .sendKeys() to select a value in a SELECT? Use the provided methods for a Select: .selectByIndex(int), .selectByValue(String), or .selectByVisibleText(String). Some examples...
Select test = new Select(driver.findElement(By.id("dropdown")));
test.selectByIndex(1);
test.selectByValue("myValue");
test.selectByVisibleText("VisibleText");

See if the happens on Firefox driver or IE driver
The other thing is the method signature is
public void sendKeys(CharSequence... value)
can you try to send it like this sendKeys( "MY","DROP"); instad and see the result
Hope this may help.
Alan Mehio
London, UK

Saving a web page as image

As a hobby project I am exploring the ways to save a web page (HTML) as image, mostly programatically using c/c++/javascript/java. Till now I have come across the following ways:
Get the IHTMLElement of page body and use it to query for IHTMLElementRender and then use its DrawToDC method (Ref: http://www.codeproject.com/KB/IP/htmlimagecapture.aspx ). But the problem is that it did not work for all the pages (mostly pages having embedded iframes).
Another way which i can think of is to use some web browser component and when the pages is fully loaded then capture it using BitBlt (Ref: http://msdn.microsoft.com/en-us/library/dd183370%28VS.85%29.aspx ). But the problem is that the page I have requested may be longer than my screen size and it will not fit into the web browser component.
Any direction/suggestion to resolve above issues or an alternative approach is greatly appreciated.

If you use Python, there's pywebshot and webkit2png. Both of them have some dependencies, though.
Edit: Oops, Python is not in your list of preferred languages. I'll leave this answer here anyway, because you said "mostly" and not "exclusively".

Another (somewhat roundabout) option would be to run a server like Tomcat and use Java to call a command-line tool to take a screenshot. Googling for "command line screenshot windows" comes up with some reasonable-looking possibilities. Besides running a server, though, I don't know a good way to run local executables from javascript. This method would make it cross-browser, though, which is a plus (just make an ajax call to the script when you want a screenshot).
Unfortunately I don't actually know how to deploy war files. It might be more trouble to use Tomcat; I mentioned it because Java was a preferred language. It would be fairly simple to run XAMPP and use this PHP snippet, and you wouldn't really have to learn php:
<?php
exec("/path/to/exec args");
?>
EDIT
You know, I'm not sure that really answers your question. It's one way, but it's coming at it from the JavaScript end rather than the scripting end. If you want to do it via scripting, you could always use Selenium. It supports capturing screenshots of an entire page, and can be controlled via Java.

Well finally able to crack it by going through these two articles:
http://www.codeproject.com/KB/GDI-plus/WebPageSnapshot.aspx [c# code - IE]
http://www.codeproject.com/KB/graphics/IECapture.aspx [c++ & GDI - IE]
Can't share the code, but the above two articles will give you the best possible solution.
Also have a look at:
https://addons.mozilla.org/en-US/firefox/addon/3408/ [firefox + javascript]
Above things are still ok. BUT not guaranteed to work always. Check the below link:
How do I render the scrollable regions of a canvas with IViewObject::Draw?

If you are OK using javascript for it, I suggest going with phantomjs
Example from http://fcargoet.evolix.net/
var page = new WebPage(),
address = 'http://dev.sencha.com/deploy/ext-4.0.7-gpl/examples/feed-viewer/feed-viewer.html';
page.viewportSize = {
width : 800,
height : 600
};
// define the components we want to capture
var components = [{
output : 'feed-viewer-left.png',
//ExtJS has a nice component query engine
selector : 'feedpanel'
},{
output : 'feed-viewer-preview-btn.png',
selector : 'feeddetail > feedgrid > toolbar > cycle'
},{
output : 'feed-viewer-collapsed.png',
//executed before the rendering
before : function(){
var panel = Ext.ComponentQuery.query('feedpanel')[0];
panel.animCollapse = false; // cancel animation, no need to wait before capture
panel.collapse();
},
selector : 'viewport'
}];
page.open(address, function (status) {
if (status !== 'success') {
console.log('Unable to load the address!');
} else {
/*
* give some time to ExtJS to
* - render the application
* - load asynchronous data
*/
window.setTimeout(function () {
components.forEach(function(component){
//execute the before function
component.before && page.evaluate(component.before);
// get the rectangular area to capture
/*
* page.evaluate() is sandboxed
* so that 'component' is not defined.
*
* It should be possible to pass variables in phantomjs 1.5
* but for now, workaround!
*/
eval('function workaround(){ window.componentSelector = "' + component.selector + '";}')
page.evaluate(workaround);
var rect = page.evaluate(function(){
// find the component
var comp = Ext.ComponentQuery.query(window.componentSelector)[0];
// get its bounding box
var box = comp.el.getBox();
// box is {x, y, width, height}
// we want {top, left, width, height}
box.top = box.y;
box.left = box.x;
return box;
});
page.clipRect = rect;
page.render(component.output);
});
// job done, exit
phantom.exit();
}, 2000);
}
});