I want to use HttpURLConnection to connect to my webservice, POST an XML and get a result. I am using the following code:
URL url = new URL(urlString);
connection = (HttpURLConnection)url.openConnection();
connection.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
Problem is, when I call the setRequestProperty method, it fails with an IllegalStateException saying I am "already connected". Apparently, openConnection in fact opens the connection to the URL (in my debugger, I can see the connected boolean set to true). According to the URL documentation of Oracle though, it shouldn't. The Android docs are unclear about it.
How do I prevent openConnection to connect, so I can set extra properties?
Update it looks like the connection is in some pool and doesn't get disconnected, not even after calling connection.disconnect(), or even killing the server.
I do not think this is a duplicate of this question as it gives no real answer. Also, the documentation seems to be unclear.
openConnection() does not connect, and the code you have posted does not behave as yu have described. What opens the TCP connection is any of the following:
getInputStream()
getErrorStream()
getResponseCode()
Ergo you must have called one of those before trying to set a request property.
Related
I am currently trying to understand the logic behind each of the code below. Please let me know if I am correct and answer my confusions.
urlConnection = (HttpURLConnection) url.openConnection();
//connection object is created. However, do not understand
//"manipulate parameters that affect the connection to the remote resource."
urlConnection.setRequestMethod("GET");
//set the method for the URL request. Not sure what that means!
//there are other strings that could be used, but not sure what each means,
//and couldn't the Java documentations didn't seem to have explanations either
urlConnection.connect();
//the actual connection the the remote object is made
This question is not related to Java at all. What you need is to understand HTTP protocol and do some comprehensive reading. I recommend starting with
http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
and then read:
https://www.rfc-editor.org/rfc/rfc7230
I'm trying to implement a simple URL availability checker which basically checks if the link is available (No HTTP 403, 404 etc. returned).
I have more than 20,000 links(to different servers/websites) in my database for testing purposes but it doesn't seems to work when I try to create more than 10 Threads.
Here is the code I'm using for opening the connection and reading response code within each WorkerThread.
URL url = new URL(dto.getUrl());
httpUrlConnection = (HttpURLConnection) url.openConnection();
httpUrlConnection.setUseCaches(false);
// httpUrlConnection.setConnectTimeout(6000);
httpUrlConnection.setDoInput(true);
httpUrlConnection.setDoOutput(false);
httpUrlConnection.setRequestMethod("GET");
httpUrlConnection.setRequestProperty("Host", dto.getUrl().replace("http://", ""));
// httpUrlConnection.setRequestProperty("Connection",
// "Keep-Alive");
httpUrlConnection.setRequestProperty("User-Agent", USER_AGENT);
httpUrlConnection.setRequestProperty("Cache-Control", "no-cache");
httpUrlConnection.connect();
int code = httpUrlConnection.getResponseCode();
Few issues I have noticed when having multiple threads opening the connections:
1) Only first 100-200 connections seems to open with no problem, after that, I start getting "Read timeout", "Connection timeout", "Connection reset" etc. Although, if you try to run the code again the links which have thrown above exceptions will return proper response code (if they get processed in first 100).
2) The response code sometimes is not valid (especially if the link was processed after first 100 links). I have noticed that sometimes 404 is returned when in fact it should return 200 (I checked it by putting link in first 100).
I did try using Apache's Http client but it also fails to process links correctly with many threads.
So does anyone know a solution to this problem ? What is the maximum amount of connections you could open using HttpURLConnection using multiple threads ? Is there any other way to open many HTTP connections and check response codes ?
Thank you all in advance !
I need to find the HTTP response code of URLs in java. I know this can be done using URL & HTTPURLConnection API and have gone through previous questions like this
and this.
I need to do this on around 2000 links so speed is the most required attribute and among those I already have crawled 150-250 pages using crawler4j and don't know a way to get code from this library (due to which I will have to make connection on those links again with another library to find the response code).
In Crawler4J, the class WebCrawler has a method handlePageStatusCode, which is exactly what you are looking for and what you would also have found if you had looked for it. Override it and be happy.
The answer behind your first link contains everything you need:
How to get HTTP response code for a URL in Java?
URL url = new URL("http://google.com");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.connect();
int code = connection.getResponseCode();
The response code is the HTTP code returned by the server.
I'm trying to post data on a server using java HttpUrlConnection class. It seems that if I somehow read the response of the server, the post works fine, but if I don't, the information is never posted.
This kind of behaviour is not mentionned in the HttpUrlConnection doc and all examples I've seen of HttUrlConnection ask for a response from the server.
I want to know if I made a mistake in my code or if this is a normal behaviour of HttpUrlConnection, and if it's the case, can someone with a better understanding of how this class works explain to me why it's so?
Thanks a lot!
Here is the code, the commented line is the one that makes the POST either work or fail:
URL url=new URL("myUrl");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=utf-8");
connection.setRequestProperty("Host", "myHost");
connection.setDoOutput(true);
os.write(staffToPost.getBytes("utf-8"));
os.flush();
os.close();
//System.out.println(connection.getResponseCode()+" "+connection.getResponseMessage());
The URLCOnnection class needs a complete handshake for a POST to occur.This means, once you open a connection, you have to readback the response for the POST to actually take place.
You can explicitly call URLConnection.connect() or get any information about the HTTP response such as HttpURLConnection.getResponseCode etc to trigger HTTP request.
You should go through this comprehensive post on How to use java.net.URLConnection to fire and handle HTTP requests for better understanding.
Been working on this all day and have gotten no where with it.
My Java code looks like this:
final URL url = new URL(String.format("https://spreadsheets.google.com/feeds/download/spreadsheets/Export?key=%s&exportFormat=tsv&gid=0", spreadsheetId));
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("Authorization", "GoogleLogin auth=" + wiseAuth.getAuthToken());
conn.setRequestProperty("GData-Version", "3.0");
conn.setRequestMethod("GET");
conn.setDoOutput(true); // trouble here, see below
conn.setInstanceFollowRedirects(true);
conn.connect();
I always get a FileNotFound error when attempting to do conn.getInputStream(). I narrowed it down to being that the response code is 405 Method Not Allowed. The exception is returning me my URL and I can access the page just fine in my browser.
It was then that I discovered that setDoOutput(true) executes a POST internally. But if I remove that line, conn.getInputStream() is null, and conn.getOutputStream() appears to return nothing--though maybe I am setting it up wrong?
I don't recommend you to do it like this, even if you get it working now you cannot ensure you will get it working in the future if Google started changing it.
Instead, consider using Google Spreadsheet API. The provided Java examples are pretty straightforward and you should able to accomplish what you want.
I would recommend using a web debugger like Fiddler to see what exactly your application is sending in the GET request and compare it to your browser. You might be missing an important header or something, and Fiddler makes it really easy to slowly strip down your browser's request to the essential elements (just drag a request to clone it, then take out headers).