Develop Reference

Read full content of a web page in Java

I want to crawl the whole content of the following link with a Java program. The first page is no problem, but when I want to crawl the data of the next pages, there is the same source code as for page one. Therefore a simple HTTP Get does not help at all.
This is the link for the page I need to crawl.
The web site has active contents that need to be interpreted and executed by a HMTL/CSS/JavaScript rendering engine. Therefore I have a simple solution with PhantomJS, but it is sophisticated to run PhantomJS code in Java.
Is there any easier way to read the whole content of the page with Java code? I already searched for a solution, but could not find anything suitable.
Appreciate your help,
kind regards.

Using the Chrome network log (or a similar tool in any other browser) you can identify the XHR request that loads the actual data displayed on the page. I have removed some of the query parameters, but essentially the request looks like this:
GET https://www.blablacar.de/search_xhr?fn=frankfurt&fcc=DE&tn=muenchen&tcc=DE&sort=trip_date&order=asc&limit=10&page=1&user_bridge=0&_=1461181945520
Helpfully, the query parameters look quite easy to understand. The order=asc&limit=10&page=1 part looks like it would be easy to adjust to return your desired results. You could adjust the page parameter to crawl successive pages of data.
The response is JSON, for which there are a ton of libraries available.

Why does "../folder/image.jpg" work when the particular jsp is only run using 'right-click-> run on server'

my setup of folders in Eclipse are as below:
I have configured web.xml to open a welcome file as
<welcome-file>jsp/index.jsp</welcome-file>
the links, for example <img src="../images/image.jpg"/> works fine when i run like this:
i.e,. the individual jsp is clicked and ran on server.
But when I run the whole project on server, no links work(images, scripts, css nothing works)
Have tried jsp/index.jsp, ${pageContext.request.contextPath}, but cannot figure out a solution. help would be very much appreciated. :( :(

Purely because the page isn't parsed and reformatted until it's passed through the server and the jsp contents and processed.
Naturally speaking most browsers won't try and "be smart" about things and "auto look" when it "things" something "might" resemble HTML/whateverML
Once you've run the JSP through the server the serve then:
1) reads the xml and gets a bearing as to where it's "root is" i.e. / of you might be / on the hard drive whereas / for the web server could be /some/path/to/your/documents
2) reads the jsp page, parses, processes and executes whatever is in there. Usually per default TomCat (if thats what you're using) returns most output as HTML to the remote browser.
Thats why the image works... It's being seen as "part of an HTML" when it comes back from the server.

How can i extract a dynamic string/word from a website using Java

Hello everyone here is my problem.
I want to extract 2 words from a website, the words are "won" or "loss". If i can find those 2 words on the website i will be able to write the program i am working on.The problems i have are...
When i write a java program to get the html code from the site it only gives me the html code that is not changing ie: it doesnt giving the dynamic php code parts.
When i "inspect elements" on the website it gives me exactly what i want. It says i either won or loss in the html tags . However if i simply view source it doesn't show me that dynamic php code that u would see when inspecting elements.
Is there a way for me to write code that looks at "inspect elements" for the website and keep track of the part of the html code that is changing between "win" or "loss"?

I've had trouble with something like this before and since you lack details I will give you the best answer I can...
More information that will be helpful to know maybe if you edit include,
Code... Show me what you got
The html code
APIs or frameworks used in you application
So the issue seems like when you request the site the information is not there. Normally this doesn't happen since most webpage display information at load time.
These days we do a lot of stuff with Javascript so therefore that is probably the part you are having problems with. Javascript can load information onto the page dynamically at anytime. It need not me at load time and even if it looks like it by eye that its there when the page loads it may not be since it's too fast to notice.
Look into the javascript code and see if you can find a get, post, or put action and see if you can follow that to where it loads the page. Then mimic the request in your program.

Tapestry Web App on Tomcat occasionally spits out garbage

We have a Tapestry-Spring-Hibernate webapp running on Tomcat 6, handing about a few 1000 requests a second. Randomly, for no apparent reason, a page just displays a bunch of random characters on the browser. However, when the page is refreshed, it displays fine. Here is a screen-shot of the source of the garbled page on Chrome:
Here is what I have found so far:
It doesn't seem to be browser specific. I have witnessed this on Chrome and Firefox, but users have also reported this on IE 7 and up.
Load on the server seems to have no correlation to when this happens.
Refreshing the page displays the page normally, as if nothing ever happened.
I don't see anything relevant in the server or the application logs
The content-type tag for the page is <meta content="text/html; charset=utf-8" http-equiv="content-type"/>
There are a couple other webapps deployed on the same container, one being Alfresco, but they don't seem to exhibit this at all.
My question is, has someone run into this before, and if so, can they point me to where I should start looking? Is this a problem with the page having something like the incorrect content-type or the server not being able to handle it for some reason? Or could this be a framework bug in Tapestry or in the application itself? Any pointers are welcome. At this point, I am not sure where the problem is, so I wasn't sure if this goes on ServerFault or stays here.

It is most likely a bug in the application. (Most bugs are ... despite the natural tendency of programmers to blame something else.)
However, could be a bit tricky to track down this problem. I suggest that you start with the standard things:
Look at the server error logs to see if anything strange shows up at the time when one of these "events" occur.
Look at the server access logs to see if you can identify the request that produced the garbage data.
Enable your browser's debugger and see if you can track down the bad request that way.
If you can figure out what the request that produced the bad response was, you'll have more traction in finding the cause.
FWIW - that doesn't look like the result of a character encoding problem. That looks more like binary or compressed data.

Here's one situation that has led me to see a garbled page. On the error page Tapestry sets a response header called X-Tapestry-ErrorMessage. Evidently newlines aren't allowed in headers (at least on some browsers) so if that header has a newline in it, you get the gibberish. One error message we were setting happened to have a trailing new line. I changed it to remove any newlines before setting that header and then the error page showed correctly.

This seems to be related to gzip compression issues in the Tapestry framework (as suggested by #barnyr) and is possibly a regression bug in Tapestry 5.3. To quote Howard from a mailing list thread:
I believe this was a bug where under certain circumstances, a corrupt
GZIP stream of page content would be streamed to the client; this is
fixed in 5.2.6 for sure, but I thought it was fixed in 5.2.5 as well.
The quick fix is to add the following configuration symbol in the contributeApplicationDefaults method of the app's module class:
configuration.add(SymbolConstants.GZIP_COMPRESSION_ENABLED, "false");
This of course disables gzip compression, but might a trade-off worth making.
Possibly related issues:
GZip compression should be disabled if the request is over http 1.0
Is GZIP compression compatible with XmlHttpRequest?

Java android programming and java script

I'm developing an android application which takes its information from a site that use JavaScript.
I want to run one of the java script function through my android app.
in example: this site http://www.bgu.co.il/tremp.aspx has "Next page" in the bottom (in hebrew) that javascript function do. i want to get to the next page through my app.
How can i send the site the command "move to next page" or activate button's onClick event?
EDIT: I'm taking information from this site, using an XML parser (sax parser), I want to get to the next page of this site in order to parse it either. I hope I made myself clear now

You really need to explain a little more fully...
Are you opening that page and parsing it in your code, have you embedded a WebView or are you just creating an Intent which opens that page in the user's preferred web-browser?
If the latter, then you definately cannot do what you're suggesting.
If you're using a WebView I'm pretty sure you still can't access the DOM of the page in the way you want to
That means your solution is to load and parse the webpage in your code - extract the 'next' page and then do with that whatever you wish...

Check out the Using JavaScript in WebView section in the Android Developer Guide

OK - now we know you're parsing the page I can try to answer your question!
You can't "click" something on a parsed page because you have no Javascript engine - it's just an XML document - a collection of nodes made-up of text.
In this case you need to work-out how the paging system actually works - and looking at the page in something like Firebug makes that quite simple to do.
The paging system works through a FORM POST mechanism - when you click 'Next' or a page number like '2' it simply loads the same page from the server, but with POST variables set.
The key variable is "__EVENTTARGET" - it tells ASP which page to load (in Hebrew of course).
The snag is that you'll need to pass all the other POST variables from this page too - that's every INPUT with it's associated value. I notice there's one called "_EVENTVALIDATION" which appears to exist to stop people getting around this too-easily by just passing "_EVENTTARGET".
A tool like Fiddler would be useful here too - it will show what is POSTed to the server when you click NEXT - all you have to do is replicate that in your call to load the page again and you have the next page - and repeat that with each page until there's nothing left to load...
Alternatively, you could switch to using a WebView - loading the page into that would mean it does have a Javascript engine and you could automate the 'clicking' (although I don't know the specifics of how to do that, the other answers allude that it's possible?) ?