I have a Java program that I'm trying to interact with over the web.
I need to gather form data from the user on a Drupal site that I don't have any control over, send it to the Java program, and send the output back to the user. The Java program needs to load a lot of libraries every time it's run, so it needs to be up waiting for data from the user.
It'd be best for me to just have an HTML form for the input. What's the simplest way to deal with HTML form data using Java?
Also, I'm trying to call the Java program from a shell script. I want the program running in the background though so the libraries are loaded in advance. So ideally, I could use the server I set up for both applications.
Thanks for any help.
It sounds like you really just want to write a servlet (or use a higher level web framework, but a servlet would work fine). That makes it very easy to get web form data - you just ask for values by name, basically.
You could then "script" the application using curl, wget or something similar to make requests to the servlet.
Apologies if this doesn't answer your question - I'm finding it slightly tricky to understand exactly what you're trying to do, particularly as there are multiple layers of web UI involved, as far as I can see.
The easiest way to make POST requests with java is to use the Apache HttpClient or the more recent HttpComponents libraries.
Related
I'm building my own HTTP server in java, but i'm facing with a problem: I would like to build a page dynamically by creating every HTML object at runtime, the question is: how can i determine the screen dimension of the client's browser?
This information is not present in the HTTP header, so I was thinking about writing a "fake" webpage that runs a javascript that tells the server about the screen (it should redirect to something like www.website.com/w:1920,h:1080) but I don't know anything about cookies (that I suppose are essential to store those informations).
Do you think that I should learn somthng about cookies or there's another way?
BTW I'm not using servlets, just Socket, because that's what I know... should I use servlets?
Thanks for your time!
Matteo
Server knows nothing about client's screen until client send this information. Javascript is easiest way to determine screen size:
window.screen.availHeight
window.screen.availWidth
AJAX request can be used to send the information to the server where it can be stored in session data and backed in database for example if the user is logged in or identified somehow. In such case you don't need cookies. However solution with cookies is easier, check how to set them via javascript. But I'm afraid such solution would be a bit of non-standard, if your site is gonna depend on javascript why not to use it extensively and generate all objects on client side, get that lazy computer working and save your server's resources :) Just feed data by sending simplest HTML containing script doing the work.
Servlets? Can be really light-weight and done with minimal knowledge if you have time go for it.
I have a java program that runs on a couple different computers and I want to be able to be able to access them remotely. I don't need to do anything crazy, just some very basic input output (get and set type stuff). I could write a client type application and set each program to a server with RMI and do it that way but I will not always be on the same computer and do not want to have to carry the .java file around with me.
I'd prefer to write a simple php webpage that I can access from anywhere and have some very basic buttons that could send the commands to the different Applications and receive back some output.
Problems is that it would probably impossible to get my host to load anything other then php onto the server where I'd host my webpage.
Does anyone have any ideas? I've tried googling it and found a couple things out there that sort of sound like what I need but the more I look into them they seem to not be what I wanted after all.
If you really want to do this right I'd suggest looking at a Java web framework (I use Play! myself) to create a RESTful web service and then sending requests from your PHP code that will fetch JSON data.
A framework like Play will make it very easy to get a REST web service running. Just create some wrappers that invoke your existing java code and call renderJSON to output the result.
From the PHP code you'll issue the request using curl and then read the response using json_decode which will turn it into a handy php variable.
This also has the advantage that either end can be changed without the other noticing, as long as the json data format is kept the same.
Based on the way you described your setup you'll probably run into routing issues trying to reach your Java apps from your PHP host. I'd recommend you try something else: have your Java apps frequently check with your PHP app if there are any updates. You can simply use a Java URL connection and a timer:
http://download.oracle.com/javase/6/docs/api/java/util/Timer.html
http://download.oracle.com/javase/6/docs/api/java/net/URL.html
I am looking to develop an app that will take login details from the user, go to a website, login, return values on the web page and then display them to the user on the phone.
Does java have this functionallity? Will I need to use javascript instead maybe? do these answers depend on the website that I am trying to access?
In my head I figure that I could just read in the paramaters as strings or chars, parse the webpage for the appropriate form and "paste" the appropriate value into the form "box". However, I have never attempted anything like this with coding so I am completely new to the idea and dont really know where to start. I tried googling around but any information that I found was either irrelevant or conflicting.
I'm not looking for the code to do it because I will not really learn anythig from that but a finger in the right direction would be great. I really do want to try get better at programming so that's why I've started to give myself these little side projects
Any help that can be offered would be great
Ian,
You can try using http-client (http://hc.apache.org/httpclient-3.x/) lib from apache. It lets to pro grammatically access a website (from a Java code). You will need to do the following things
Use the http-client lib to POST the data to the web site.
Receive the html response.
Use some html parser or xpath to retrieve the values from the response html.
You would need a script which accesses the webpage and enters the data, but in my opinion this is illegal. Because you are accessing a secured area and are able to look into sensitive data. Also accessing the page via a script is "botting" - most pages have safety precautions to prevent the execution of scripts, because most of them are harmful.
In my opinion there is no legal and easy solution to this.
I am newbie in wkhtmltopdf. I am wondering how to use wkhtmltopdf with my Dynamic Web Project in Eclipse? How to integrate wkhtmltopdf with my Java dynamic web application?
Is there any tutorials available for beginners of wkhtmltopdf ?
(Basically, I would like to use wkhtmltopdf in my web application so that when user click a save button , the current page will be saved to PDF file).
First, a technical note: Because you want to use wkhtmltopdf in a web project, if and when you deploy to a Linux server machine that you access via ssh (i.e. over the network), you will need to either use the patched Qt version, or run an X server, e.g. the dummy X server xvfb. (I don't know what happens if you deploy to a server running an operating system other than Linux.)
Second, it should be really quite simple to use wkhtmltopdf from any language in a web project.
If you just want to save the server-generated version of the current page, i.e. without any changes which might have been made like the user filling on forms, or Javascript adding new DOM elements, you just need to have an extra optional argument like ?generate=pdf on the end of your URL, which will cause that page to be generated as a PDF, and then the PDF button will link to that URL. This may be a lot of work to add to each page manually if you are just using simple JSP or something, but depending on which web framework you are using, the web framework may offer some help to implement the same action on every page, if you need to implement that.
To implement this approach, you would probably want to capture the response by wrapping the response object and overridding its getWriter() and getOutputStream() methods.
Another approach is to have a button "submit and generate PDF" which will generate the next page as a PDF. This might make more sense if you have a form the user needs to fill in - I don't know. It's a design decision really.
A third approach is to use Javascript to upload the current state of the page back to the server, and process that using wkhtmltopdf. This will work on any page. (This can even be used on any site, not just yours, if you make it a bookmarklet. Just an idea that occurred to me - it may not be a good idea.)
A fourth approach is, because wkhtmltopdf can fetch URLs, to pass the URL of your page instead of the contents of the page (which will only work if the request was a HTTP GET, or if it's equivalent to a HTTP GET on the same URL). This has some small amount of overhead over capturing your own response output, but it will probably be negligible. You will also very likely need to copy the cookie(s) into a cookie jar with this approach, since presumably your user might be logged in or have an implicit session.
So as you can see there are quite a lot of choices!
Now, the question remains: when your server has the necessary HTML, from any of the above approaches, how to feed it into wkhtmltopdf? This is pretty simple. You will need to spawn an external process using either Runtime.getRuntime().exec(), or the newer API called ProcessBuilder - see http://www.java-tips.org/java-se-tips/java.util/from-runtime.exec-to-processbuilder.html for a comparison. If you are smart about it you should be able to do this without needing to create any temporary files.
One of the wkhtmltopdf websites is currently down, but the main README is available here, which explains the command line arguments.
This is merely an outline answer which gives some pointers. If you need more details, let us know what specifically you need to know.
Additional info:
If you do end up trying to call wkhtmltopdf in an external process from java (or for that matter, any language), please note that the "normal" output that you see when using wkhtmltopdf from the command line (i.e. what you would expect to see in STDOUT) is not not in STDOUT but in STDERR. I raised this issue in the project page
http://code.google.com/p/wkhtmltopdf/issues/detail?id=825
and was replied that this is by design because wkhtmltopdf supports giving the actual pdf output in STDOUT. Please see the link for more details and java code.
java-wkhtmltopdf-wrapper provides an easy API for using wkhtmltopdf in Java.
It also works out-of-the-box on a headless server with xvfb.
E.g., on a Ubuntu or Debian server:
aptitude install wkhtmltopdf xvfb
Then in Java:
Pdf pdf = new Pdf();
pdf.addPage("http://www.google.com", PageType.url);
pdf.saveAs("output.pdf");
See the examples on their Github page for more options.
I need to screen scrape some data from a website, because it isn't available via their web service. When I've needed to do this previously, I've written the Java code myself using Apache's HTTP client library to make the relevant HTTP calls to download the data. I figured out the relevant calls I needed to make by clicking through the relevant screens in a browser while using the Charles web proxy to log the corresponding HTTP calls.
As you can imagine this is a fairly tedious process, and I'm wodering if there's a tool that can actually generate the Java code that corresponds to a browser session. I expect the generated code wouldn't be as pretty as code written manually, but I could always tidy it up afterwards. Does anyone know if such a tool exists? Selenium is one possibility I'm aware of, though I'm not sure if it supports this exact use case.
Thanks,
Don
I would also add +1 for HtmlUnit since its functionality is very powerful: if you are needing behaviour 'as though a real browser was scraping and using the page' that's definitely the best option available. HtmlUnit executes (if you want it to) the Javascript in the page.
It currently has full featured support for all the main Javascript libraries and will execute JS code using them. Corresponding with that you can get handles to the Javascript objects in page programmatically within your test.
If however the scope of what you are trying to do is less, more along the lines of reading some of the HTML elements and where you dont much care about Javascript, then using NekoHTML should suffice. Its similar to JDom giving programmatic - rather than XPath - access to the tree. You would probably need to use Apache's HttpClient to retrieve pages.
The manageability.org blog has an entry which lists a whole bunch of web page scraping tools for Java. However, I do not seem to be able to reach it right now, but I did find a text only representation in Google's cache here.
You should take a look at HtmlUnit - it was designed for testing websites but works great for screen scraping and navigating through multiple pages. It takes care of cookies and other session-related stuff.
I would say I personally like to use HtmlUnit and Selenium as my 2 favorite tools for Screen Scraping.
A tool called The Grinder allows you to script a session to a site by going through its proxy. The output is Python (runnable in Jython).