Develop Reference

Dos command to get IE Page source

Is it possible to get the source of a webpage, which is currently opened in IE or chrome from command line or using a java code? I believe there has to be a way. If yes, how could we fetch the exact info of it as chrome and IE support multiple tabs.
I am trying to process content from hundreds of webpages, some of them automatically refresh at regular 15 sec interval. And some do not.
Yes, i could get the webpage source by using sockets or by using an instance of URLConnection class. However, it doesn't provide the default refresh functionality of a browser. The only option will be to hit the URL multiple times, which could be avoided if the default browsers refresh functionality could be utilized.
Also, It would be great, if the reader could comment on how to fill in text boxes using a program and submit the request from the browser. Thanks.

There are several "scraping" frameworks in Java.
I personally like JSoup a lot, because it is lightweight and compact in code.
// get the source of a website in just 1 line of code.
Document doc = Jsoup.connect("http://www.google.com").get();
// print all hyperlink paths.
Elements links = doc.select("a[href$=.html]");
for (Element lnk : links) System.out.println(lnk.attr("href"));
However it does not render javascript or anything like that. It's simple, fast but stupid.
I think you may prefer to use HtmlUnit instead, which is more like an invisible webbrowser. It gives you the possibility to even simulate click events on buttons, execute javascript, ... etc. You can make it mimic Internet Explorer or Firefox.

You can use Selenium WebDrivers - set of modifications/add-on's for desktop and phone browsers that allow you to fully control them from your code - including getting source of currently loaded page(using getPageSource() method), filling input's and submitting forms, selecting text, clicking some points, and almost just every other thing's that can be done in browser.

You could use a simple HTTP Client in order to get the source of your page using commons-httpclient.
Having you set up your libraries, you can use the following code:
HttpClient client = new HttpClient();
HttpMethod method = new GetMethod(url); // http://www.google.com
client.executeMethod(method);
String result = method.getResponseBodyAsString();
In the result variable, you will get the source code of the page, in this case, the Google's main search page. So you can do whatever you want. For example, you can keep refreshing the page using a Java Thread and do whatever you want with the result.
You can find more information on Commons HTTP-Client Page

Wget for Windows may help, if you mean "terminal" and not specifically the DOS operating system. There's also something called bitsadmin (which I'm not familiar with), and I found this in a search too: Jaunt - Java Web Scraping & Automation, if that'll help. I'm not a Java guy, but hopefully that points you in the right direction.

Opening a Webpage at an Anchor in a Browser through Java

I have created an application in JSwing that has a button that I want to open the user manual (which is a html file) in a browser. I can successfully open the entire webpage, but I want to link to certain anchors in the document. For example I am trying to use this code:
URI uri = new URI("c:/Giggafriggin/user_manual/user_manual.html#h1_3");
Desktop.getDesktop().browse(uri);
But this causes an error, claiming the file cannot be found. But if leave off "#h1_3" it opens the page in a browser without a problem. The anchors work when i enter them into the browser manually. Any ideas?

You -could- have that linking to another html page which goes to the end uri. Unfortunately, Java is not a web browser.

Looks like this is a known issue you wouldn't run into if you were using HTTP instead of a local file.
One easy fix is simply to point to a version of the that's already online instead of on disk.
If you can't assume the content is available online, you could always spin up an embedded HTTP server like jetty inside your application and point to that instead.

Mask browser URL with GWT

My GWT app URL when a page is access looks like this:
http://127.0.0.1:8888/index.html?gwt.codesvr=127.0.0.1:9997#ViewPage;hash=6a
or
http://127.0.0.1:8888/index.html#ViewPage;hash=6a
However, this does not look good, is there a way for the GWT client side code to make it look like this, to "mask" it somehow to:
http://127.0.0.1:8888/6a
Where I have a servlet configured to forward this "shortened" URL to the long URL above so its very safe to make the URL "masked"? Without losing state and history.

The shortest possibility I know would be an URL like:
http://127.0.0.1:8888/#6a
Therefore you must configure index.html as default page. (It must be configured as default response).
You also have to rewrite the history-management. You have to remove the Place-Token from the URL.

I don't see any way to do this directly with GWT, since GWT needs access to the code fragment in the URL to manage browsing history and state.
One indirect way to do it would be to embed your GWT module inside an iframe that occupies the whole page area. The drawback is that the user would loose the ability to bookmark pages inside the GWT application.
By the way, I don't share your view that it "does not look good". Many popular Web applications use the URL like this, including Gmail. Most users don't care about what's in the URL.

Login to website with javascript submit button (Using Java)

VIA JAVA, I want to login to a website.
Authentication: The site has a javascript button that performs the redirection to the home page. My webcrawler can login programatically to sites that have html buttons, using Jsoup. But when I try to login to a website that has the submit in a javascript, I can't seem to get authenticated in any of the ways I discovered so far.
So far I've tried:
I've tried to log in using the native java api, using URLConnection, and OutputWriter. It fills the user and password fields with their proper values, but when I try to execute the javascript method, it simply doesn't work;
Jsoup. (It can log me in to any website containing html buttons. But since it doesn't support javascript, it won't help much;
I've tried HtmlUnit. Not only does it print a gazilion lines of output, it takes a long long time to run, and in the end still fails.
At last, I tried using Rhino (Which HtmlUnit is based on), got it to work in a long list of javascript methods. But cannot authenticate;
I already have tried Selenium, and got nowhere, also..
I'm running out of ideas.. Maybe I haven't explored all the solutions contained in one of these APIs, but I still can't login to a website containing a javascript button. Anyone has any ideas?

Using Selenium Webdriver, send javascript commands to the browser. I've successfully used it to reliably and repeatedly run hundreds of tests on complicated javascript/ajax procedures on the client.
If you target a specific web page, you can customize the script and make it quite small.
WebDriver driver; // Assigned elsewhere
JavascriptExecutor js = (JavascriptExecutor) driver;
// This is javascript, but can be done through Webdriver directly
js.executeScript("document.getElementById('theform').submit();");
Filling out the form is assumed to have been handled by using the Selenium Webdriver API. You can also send commands to click() the right button etcetera.
Using Selenium Webdriver, you could also write <script> tags to the browser, in order to load larger libraries. Remember that you may have to wait/sleep until the browser has loaded the script files - both your own and the one the original web page uses for the login procedures. It could be seconds to load and execute all of it. To avoid sleeping for too long, use the more reliable method of injecting a small script that will check if everything else has been loaded (checking web page script's status flags, browser status).

I suggest HtmlUnit:
HtmlUnit is a "GUI-Less browser for Java programs". It models HTML
documents and provides an API that allows you to invoke pages, fill
out forms, click links, etc... just like you do in your "normal"
browser.
It has fairly good JavaScript support (which is constantly improving)
and is able to work even with quite complex AJAX libraries, simulating
either Firefox or Internet Explorer depending on the configuration you
want to use.
It is typically used for testing purposes or to retrieve information
from web sites.

I had an issue that sounds similar (I had a login button that called a javascript method).
I used JMeter to observe what was being passed when I manually clicked the login button through a web browser (but I imagine you could do this with WireShark as well).
In my Java code, I created a PostMethod with all the parameters that were being sent.
PostMethod post = new PostMethod(WEB_URL); // URL of the login page
// first is the name of the field on the login page,
// then the value being submitted for that field
post.addParameter(FIELD_USERNAME, "username");
post.addParameter(FIELD_PASSWORD, "password");
I then used HttpClient (org.apache.commons.httpclient.HttpClient) to execute the Post request.
One thing to note, there were 'hidden' parameters that were being passed that I did not see by manually looking at the login page. These were revealed to me when I used JMeter.
I will be happy to clarify anything that seems unclear.

Proxy servlet and iframe to run site from another domain

For quite a long time I use proxy servlet, and from my experience in order for javascript to properly work, the proxy servlet has to be configured to run in the ROOT context. However, a site http://www.filestube.com/ from what I investigated, its running a site from another domain on a iframe.
I am wondering what kind of approach is this?

From what I can see on the target page it loads javascript from some 'local' domains (*.filestube.com) who then in turn load javascript from another domain (e.G. google or facebook in this case).
Also there is an IFrame on the site, but IFrames may come from everywhere anyway.
So the kind of approach is: Normal, don't think about user privacy and just load everything you think you might need from everywhere.