.NET has the HttpWebRequest and WebClient classes for simulating a browser's requests.
I'd google it, but I'm not sure what keyword to use.
I want to write code that does does HTTP GETs and POSTs, along with cookies, in an applet or local .jar and gives me back the response in a text string or some other parseable structure.
HttpURLConnection is Java's equivalent of HttpWebRequest.
URL iurl = new URL(url);
HttpURLConnection uc = (HttpURLConnection)iurl.openConnection();
uc.connect();
if (uc.getContentType().equalsIgnoreCase("image/jpeg"))
{
result = true;
}
Apache HTTPClient has equivalent functionality, though the APIs are not exactly the same. Oakland Software has a table comparing their commercial product with various alternatives, including the Apache product. Apache's own opinion of the built-in HttpUrlConnection (quoted from the above linked-to page) is:
The jdk has the HttpUrlConnection
which is limited and in many ways
flawed.
Here's a link to the HTTPClient tutorial.
html unit for me.
i can simulate javascript (to a certain extent)
Verify Webclient in Apache Cx JaxRs Library.
Checkout this:
https://cxf.apache.org/javadoc/latest/org/apache/cxf/jaxrs/client/WebClient.html
Sample code looks below:
WebClient client = WebClient.create(url);
client.path(ADD_PATH).path("/books/2").accept("text/plain");
s = client.get(String.class);
System.out.println(s);
Related
I have a few questions about a specific REST call I'm making in JAVA. I'm quite the novice, so I've cobbled this together from several sources. The call itself looks like this:
String src = AaRestCall.subTrackingNum(trackingNum);
The Rest call class looks like this:
public class AaRestCall {
public static String subTrackingNum (Sting trackingNum) throws IOException {
URL url = new URL("https://.../rest/" + trackingNum);
String query = "{'TRACKINGNUM': trackingNum}";
//make connection
URLConnection urlc = url.openConnection();
//use post mode
urlc.setDoOutput(true);
urlc.setAllowUserInteraction(false);
//send query
PrintStream ps = new PrintStream(urlc.getOutputStream());
ps.print(query);
ps.close();
//get result
BufferedReader br = new BufferedReader(new InputStreamReader(urlc
.getInputStream()));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line=br.readLine())!=null) {
sb.append(line);
}
br.close();
return sb.toString();
}
}
Now, I have a few questions on top of the what is wrong with this in general.
1) If this rest call is returning a JSON object, is that going to get screwed up by going to a String?
2) What's the best way to parse out the JSON that is returning?
3) I'm not really certain how to format the query field. I assume that's supposed to be documented in the REST API?
Thanks in advance.
REST is a pattern applied on top of HTTP. From your questions, it seems to me that you first need to understand how HTTP (and chatty socket protocols in general) works and what the Java API offers for deal with it.
You can use whatever Json library out there to parse the HTTP response body (provided it's a 200 OK, that you need to check for, and also watch out for HTTP redirects!), but it's not how things are usually built.
If the service exposes a real RESTful interface (opposed to a simpler HTTP+JSON) you'll need to use four HTTP verbs, and URLConnection doesn't let you do so. Plus, you'll likely want to add headers for authentication, or maybe cookies (which in fact are just HTTP headers, but are still worth to be considered separately). So my suggestion is building the client-side part of the service with the HttpClient from Apache commons, or maybe some JAX-RS library with client support (for example Apache CXF). In that way you'll have full control of the communication while also getting nicer abstractions to work with, instead of consuming the InputStream provided by your URLConnection and manually serializing/deserializing parameters/responses.
Regarding the bit about how to format the query field, again you first need to grasp the basics of HTTP. Anyway, the definite answer depends on the remote service implementation, but you'll face four options:
The query string in the service URL
A form-encoded body of your HTTP request
A multipart body of your HTTP request (similar to the former, but the different MIME type is enough to give some headache) - this is often used in HTTP+JSON services that also have a website, and the same URL can be used for uploading a form that contains a file input
A service-defined (for example application/json, or application/xml) encoding for your HTTP body (again, it's really the same as the previous two points, but the different MIME encoding means that you'll have to use a different API)
Oh my. There are a couple of areas where you can improve on this code. I'm not even going to point out the errors since I'd like you to replace the HTTP calls with a HTTP client library. I'm also unaware of the spec required by your API so getting you to use the POST or GET methods properly at this level of abstraction will take more work.
1) If this rest call is returning a JSON object, is that going to get
screwed up by going to a String?
No, but marshalling that json into an obect is your job. A library like google gson can help.
2) What's the best way to parse out the JSON that is returning?
I like to use gson like I mentioned above, but you can use another marshal/unmarhal library.
3) I'm not really certain how to format the query field. I assume
that's supposed to be documented in the REST API?
Yes. Take a look at the documentation and come up with java objects that mirror the json structure. You can then parse them with the following code.
gson.fromJson(json, MyStructure.class);
Http client
Please take a look at writing your HTTP client using a library like apache HTTP client which will make your job much easier.
Testing
Since you seem to be new to this, I'd also suggest you take a look at a tool like Postman which can help you test your API calls if you suspect that the code you've written is faulty.
I think that you should use a REST client library instead of writing your own, unless it is for educational purposes - then by all means go nuts!
The REST service will respond to your call with a HTTP response, the payload may and may not be formatted as a JSON string. If it is, I suggest that you use a JSON parsing library to convert that String into a Java representation.
And yes, you will have to resort to the particular REST API:s documentation for details.
P.S. The java URL class is broken, use URI instead.
I need to find the HTTP response code of URLs in java. I know this can be done using URL & HTTPURLConnection API and have gone through previous questions like this
and this.
I need to do this on around 2000 links so speed is the most required attribute and among those I already have crawled 150-250 pages using crawler4j and don't know a way to get code from this library (due to which I will have to make connection on those links again with another library to find the response code).
In Crawler4J, the class WebCrawler has a method handlePageStatusCode, which is exactly what you are looking for and what you would also have found if you had looked for it. Override it and be happy.
The answer behind your first link contains everything you need:
How to get HTTP response code for a URL in Java?
URL url = new URL("http://google.com");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.connect();
int code = connection.getResponseCode();
The response code is the HTTP code returned by the server.
I am developing a client-side Java application that has a bit of functionality that requires getting data from some web services that transmit in JSON (some RESTful, some not). No JavaScript, no web browser, just a plain JAR file that will run locally with Swing for the GUI.
This is not a new or unique problem; surely there must be some open source libraries out there that will handle the JSON data transmission over HTTP. I've already found some that will parse JSON, but I'm having trouble finding any that will handle the HTTP communication to consume the JSON web service.
So far I've found Apache Axis2 apparently which might have at least part of the solution, but I don't see enough documentation for it to know if it will do what I need, or how to use it. Maybe part of the problem is that I don't have experience with web services so I'm not able to know a solution when I see it. I hope some of you can point me in the right direction. Examples would be helpful.
Apache HttpClient 4.0
is the best in the business and is moderately easy to learn.
If you want easier you could use HtmlUnit which imitates the behaviour of browsers so you could easily get the content (and parse it into Html, javascript and css, you could also execute javascript code on content so you could probably parse JSON files to using JSON.parse or any other equivalent functions) of any page on the web.
so for HtmlUnit here is a sample code:
WebClient wc = new WebClient(BrowserVersion.FIREFOX_3_6);
HtmlPage page = wc.getPage("http://urlhere");
page.executeJavaScript("JS code here");
but it maybe rather heavy for your requirements so a highly recommend the use of HttpClient library.
I'm sure you could find many JSON libraries for java but here is one for you json-lib
I did it using a simple Java JSON libary. Use the Google library..
URL url = new URL("http://www.siteconsortium.com/services/hello.php");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
JSONParser parser=new JSONParser();
Object object = parser.parse(in);
JSONArray array = (JSONArray) object;
JSONObject object2 = (JSONObject)array.get(0);
System.out.println(object2.get("hello"));
If the webservice uses OAuth and an access token you can't use the above example though.
Its great to see that your web services are RESTful. RESTful web services are pretty easy to develop and to consume.Well... you do not need to take any extra care to tranmit JSON data over the network... Data whether is in JSON on in XML format are embedded into the HTTP header..Following code snippet will help you understand the idea :
httpConnection = new HTTPConnectionManager(request);
HttpURLConnection httpURLConnection = httpConnection.connect();
int responseCode = httpURLConnection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
in = httpURLConnection.getInputStream();
int x;
StringBuilder stringBuilder = new StringBuilder();
while ((x = in.read()) != -1) {
stringBuilder.append((char) x);
}
XMLParser xmParser = new XMLParser();
....
....
}
In this code i am receiving data in XML format from web services.After receiving the data into a StringBuilder object,i am parsing the XML. In the same way you can call your web services using this code and can receive your JSON data. you can use javaJSON APIs,available Here, to extract the data from JSON notation.
Hope code will help you...
PS: HTTPConnectionManager,XMLParser and Request(request object) classes are not any standard APIs. they are written by my own account to handle multiple web service calls. This code snippet is just to give you my idea.
I have HTML based queries in my code and one specific kind seems to give rise to IOExceptions upon receiving 505 response from the server. I have looked up the 505 response along with other people who seemed to have similar problems. Apparently 505 stands for HTTP version mismatch, but when I copy the same query URL to any browser (tried firefox, seamonkey and Opera) there seems to be no problem. One of the posts I read suggested that the browsers might automatically handle the version mismatch problem..
I have tried to dig in deeper by using the nice developer tool that comes with Opera, and it looks like there is no mismatch in versions (I believe Java uses HTTP 1.1) and a nice 200 OK response is received. Why do I experience problems when the same query goes through my Java code?
private InputStream openURL(String urlName) throws IOException{
URL url = new URL(urlName);
URLConnection urlConnection = url.openConnection();
return urlConnection.getInputStream();
}
sample link: http://www.uniprot.org/uniprot/?query=mnemonic%3aNUGM_HUMAN&format=tab&columns=id,entry%20name,reviewed,organism,length
There has been some issues in Tomcat with URLs containing space in it. To fix the problem, you need to encode your url with URLEncoder.
Example (notice the space):
String url="http://example.org/test test2/index.html";
String encodedURL=java.net.URLEncoder.encode(url,"UTF-8");
System.out.println(encodedURL); //outputs http%3A%2F%2Fexample.org%2Ftest+test2%2Findex.html
AS a developer at www.uniprot.org I have the advantage of being able to look in the request logs. In the last year according to the logs we have not send a 505 response code. In any case our servers do understand http 1 requests as well as the default http1.1 (though you might not get the results that you expect).
That makes me suspect there was either some kind of data corruption on the way. Or you where affected by a hardware failure (lately we have had some trouble with a switch and a whole datacentre ;). In any case if you ever have questions or problems with uniprot.org please contact help#uniprot.org then we can see if we can help/fix the problem.
Your code snippet seems normal and should work.
Regards,
Jerven Bolleman
Are you behind a proxy? This code works for me and prints out the same text I see through a browser.
final URL url = new URL("http://www.uniprot.org/uniprot/?query=mnemonic%3aNUGM_HUMAN&format=tab&columns=id,entry%20name,reviewed,organism,length");
final URLConnection conn = url.openConnection();
final InputStream is = conn.getInputStream();
System.out.println(IOUtils.toString(is));
conn is an instance of HttpURLConnection
from the API documentation for the URL class:
The URL class does not itself encode or decode any URL components
[...]. It is the responsibility of the caller to encode any fields,
which need to be escaped prior to calling URL, and also to decode any
escaped fields, that are returned from URL.
so if you have any spaces in your url-str encode it before calling new URL(url-str)
#posdef I was having same HTTP error code 505 problem. When I pasted URL that I was using in Java code in Firefox, Chrome it worked. But through code was giving IOException. But at last I came to know that in url string there were brackets '(' and ')', by removing them it worked so it seems I needed URLEncodeing same like browsers.
I am trying to create a HttpServlet that forwards all incoming requests as is, to another serlvet running on a different domain.
How can this be accomplished? The RequestDispatcher's forward() only operates on the same server.
Edit: I can't introduce any dependencies.
You can't when it doesn't run in the same ServletContext or same/clustered webserver wherein the webapps are configured to share the ServletContext (in case of Tomcat, check crossContext option).
You have to send a redirect by HttpServletResponse.sendRedirect(). If your actual concern is reusing the query parameters on the new URL, just resend them along.
response.sendRedirect(newURL + "?" + request.getQueryString());
Or when it's a POST, send a HTTP 307 redirect, the client will reapply the same POST query parameters on the new URL.
response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT);
response.setHeader("Location", newURL);
Update as per the comments, that's apparently not an option as well since you want to hide the URL. In that case, you have to let the servlet play for proxy. You can do this with a HTTP client, e.g. the Java SE provided java.net.URLConnection (mini tutorial here) or the more convenienced Apache Commons HttpClient.
If it's GET, just do:
InputStream input = new URL(newURL + "?" + request.getQueryString()).openStream();
OutputStream output = response.getOutputStream();
// Copy.
Or if it's POST:
URLConnection connection = new URL(newURL).openConnection();
connection.setDoOutput(true);
// Set and/or copy request headers here based on current request?
InputStream input1 = request.getInputStream();
OutputStream output1 = connection.getOutputStream();
// Copy.
InputStream input2 = connection.getInputStream();
OutputStream output2 = response.getOutputStream();
// Copy.
Note that you possibly need to capture/replace/update the relative links in the HTML response, if any. Jsoup may be extremely helpful in this.
As others have pointed out, what you want is a proxy. Your options:
Find an open-source Java library that does this. There are a few out there, but I haven't used any of them, so I can't recommend any.
Write it yourself. Shouldn't be too hard, just remember to deal with stuff like passing along all headers and response codes.
Use the proxy module in Apache 2.2. This is the one I'd pick, because I already know that it works reliably.
Jetty has a sample ProxyServlet implementation that uses URL.openConnection() under the hood. Feel free to use as-is or to use as inspiration for your own implementation. ;-)
Or you can use Apache HttpClient, see the tutorial.