We are using HttpURLConnection API to invoke a REST API to the same provider often (kind of an aggregation usecase). We want to keep a pool of 5 connections always open to the provider host (always the same IP).
What is the proper solution? Here is what we tried:
System.setProperty("http.maxConnections", 5); // set globally only once
...
// everytime we need a connection, we use the following
HttpURLConnection conn = (HttpURLConnection) (new URL(url)).openConnection();
conn.setRequestMethod("GET");
conn.setDoInput(true);
conn.setDoOutput(false);
conn.setUseCaches(true);
...
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
...
At this point we read the input stream until the BufferedReader returns no more bytes. What do we do after that point if we want to reuse the underlying connection to the provider? We were under the impression that if the input stream is completely read, the connection is then added back to the pool.
It's been working for several weeks this way, but today it stopped working producing this exception: java.net.SocketException: Too many open files
We found numerous sockets in the CLOSE_WAIT state like this (by running lsof):
java 1814 root 97u IPv6 844702 TCP colinux:58517->123.123.254.205:www (CLOSE_WAIT)
Won't either conn.getInputStream().close() or conn.disconnect() completely close the connection and remove it from the pool?
We had this problem also on Java 5 and our solution is to switch to Apache HttpClient with pooled connection manager.
The keepalive implementation of Sun's URL handler for HTTP is very buggy. There is no maintenance thread to close idle connections.
Another bigger problem with keepalive is that you need to delete responses. Otherwise, the connection will be orphaned also. Most people don't handle error stream correctly. Please see my answer to this question for an example on how to read error responses correctly,
HttpURLConnection.getResponseCode() returns -1 on second invocation
From here:
The current implementation doesn't buffer the response body. Which means that the application has to finish reading the response body or call close() to abandon the rest of the response body, in order for that connection to be reused. Furthermore, current implementation will not try block-reading when cleaning up the connection, meaning if the whole response body is not available, the connection will not be reused.
I read this as if your solution should work, but that you are also free to call close and the connection will still be reused.
The reference cited by disown was what really helped.
We know Apache HttpClient is better, but that would require another jar and we might use this code in an applet.
Calling HttpURLConnection.connect() was unnecessary. I'm not sure if it prevents connection reuse, but we took it out. It is safe to close the stream, but calling disconnect() on the connection will prevent reuse. Also, setting sun.net.http.errorstream.enableBuffering=true helps.
Here is what we ended up using:
System.setProperty("http.maxConnections", String.valueOf(CONST.CONNECTION_LIMIT));
System.setProperty("sun.net.http.errorstream.enableBuffering", "true");
...
int responseCode = -1;
HttpURLConnection conn = null;
BufferedReader reader = null;
try {
conn = (HttpURLConnection) (new URL(url)).openConnection();
conn.setRequestProperty("Accept-Encoding", "gzip");
// this blocks until the connection responds
InputStream in = new GZIPInputStream(conn.getInputStream());
reader = new BufferedReader(new InputStreamReader(in));
StringBuffer sb = new StringBuffer();
char[] buff = new char[CONST.HTTP_BUFFER_SIZE];
int cnt;
while((cnt = reader.read(buff)) > 0) sb.append(buff, 0, cnt);
reader.close();
responseCode = conn.getResponseCode();
if(responseCode != HttpURLConnection.HTTP_OK) throw new IOException("abnormal HTTP response code:"+responseCode);
return sb.toString();
} catch(IOException e) {
// consume error stream, otherwise, connection won't be reused
if(conn != null) {
try {
InputStream in = ((HttpURLConnection)conn).getErrorStream();
in.close();
if(reader != null) reader.close();
} catch(IOException ex) {
log.fine(ex);
}
}
// log exception
String rc = (responseCode == -1) ? "unknown" : ""+responseCode;
log.severe("Error for HttpUtil.httpGet("+url+")\nServer returned an HTTP response code of '"+rc+"'");
log.severe(e);
}
Related
Should I close HttpUrlConnection and InputStream in this case? Only closing the connection will close the stream also? I feel that it's a bad practice but don't know exactly why.
Closing both:
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
try (AutoCloseable ac = con::disconnect) {
int responseCode = con.getResponseCode();
try (InputStream ins = responseCode >= 400 ? con.getErrorStream() : con.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(ins))) {
// receive response
}
}
Closing Connection only:
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
try (AutoCloseable ac = con::disconnect) {
int responseCode = con.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(ins)))
// ins will close automatically when con closes?
// receive response
}
When you disconnect the HttpURLConnection object, it MAY also close any InputStream or OutputStream that it has opened.
HttpURLConnection.disconnect() method description:
Calling the disconnect() method may close the underlying socket if a persistent connection is otherwise idle at that time.
You can read more here.
In turn, Socket.close() method description:
Closing this socket will also close the socket's InputStream and OutputStream.
If this socket has an associated channel then the channel is closed as well.
You can read more here.
But pay attention that "disconnecting" HttpURLConnection doesn’t mandatory close the Socket.
It have been already discussed quite well in that thread:
(When keepAlive == true)
If client called HttpURLConnection.getInputSteam().close(), the later call to HttpURLConnection.disconnect() will NOT close the Socket. i.e. The Socket is reused (cached)
If client does not call close(), call disconnect() will close the InputStream and close the Socket.
So in order to reuse the Socket, just call InputStream.close(). Do not call HttpURLConnection.disconnect().
On the other hand, in the official oracle tutorials they suggest to close InputStream explicitly in order to be sure that resources don’t leak.
I cannot comprehend why doesn't the following code does not put a packet onto wire (confirmed via wireshark). It is a fairly standard method of sending an HTTP POST request, as I believe. I don't intend to read anything just POST.
private void sendRequest() throws IOException {
String params = "param=value";
URL url = new URL(otherUrl.toString());
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setDoOutput(true);
con.setDoInput(true); //setting this to `false` does not help
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "text/plain");
con.setRequestProperty("Content-Length", "" + Integer.toString(params.getBytes().length));
con.setRequestProperty("Accept", "text/plain");
con.setUseCaches(false);
con.connect();
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(params);
wr.flush();
wr.close();
//Logger.getLogger("log").info("URL: "+url+", response: "+con.getResponseCode());
con.disconnect();
}
What happens is... actually nothing, unless I try to read anything. For example by uncommenting the above log line which reads the response code. Trying to read a response via con.getInputStream(); also works. There is no movement of packets. When I uncomment the getResponseCode, I can see that http POST is sent, and then 200 OK is sent back. The order is proper. I.e. I don't get some wild response before sending POST. Everything else looks exactly the same (I can attach wireshark screenshots if needed.). In the debugger the code executes (i.e. does not block anywhere).
I don't understand under what circumstances this can be happening. I belive it should be possible, to send a POST request with con.setDoInput(false);. Currently it doesn't send anything or fails (when trying to execute con.getResponseCode()) with an exception because I obviously promised I won't read anything.
It might be relevant, that before sendRequest I do request some data from the same site, but I trust I close everything properly. I.e:
public static String getData(String urlAddress) throws MalformedURLException, IOException {
URL url = new URL(urlAddress);
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setDoOutput(false);
InputStream in = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder data = new StringBuilder();
String line;
while((line = reader.readLine()) != null) {
data.append(line);
}
reader.close();
in.close();
con.getResponseCode();
con.disconnect();
return data.toString();
}
The server for url in both cases is the same, port also, so I believe it is possible to use the same socket for communication. The above code works and retrieves the data properly.
I am not sure, maybe I don't clean something, and it gets cached, so with out an explicit read the POST gets delayed. There is no other traffic on the socket.
Unless you're using fixed-length or chunked transfer mode, HttpURLConnection will buffer all your output until you call getInputStream() or getResponseCode(), so that it can send a correct Content-length header.
If you call getResponseCode() you should have a look at its value.
I would like to send a GET parameter to a server. I really do not need the InputStream (below), but the request is actually sent when I call "getInputStream". The problem is, this code hangs on getInputStream. The timeout does not apply because the connection is actually established (does not time-out).
What do I need to change so that I'm sending a clean GET to the server without hanging?
URL url = new URL("http://localhost:8888/abc?message=abc"); //[edit]
URLConnection uc = url.openConnection();
uc.setRequestProperty("Accept-Charset", "UTF-8");
uc.setConnectTimeout(1000);
InputStream in = uc.getInputStream();
in.close();
In case it matters, I'm testing with netcat -l as the server instead of using an actual web server. None the less, I would like this code to be very fail-safe so it the server can't adversely effect this code.
I basically gave up in using the URLConnection and wrote the code to use a socket instead. I'm still open for improvements, light-weight posting to a web server is very useful.
URL u = new URL("http://localhost:8888/abc?message=abc");
String get = "";
if (u.getPath() != null)
get += u.getPath();
if (u.getQuery() != null)
get += "?" + u.getQuery();
if (u.getRef() != null)
get += "#" + u.getRef();
Socket socket = new Socket();
socket
.connect(new InetSocketAddress(u.getHost(), u.getPort()),
750);
OutputStream out = socket.getOutputStream();
out.write(("GET " + get + "\n\n").getBytes());
out.close();
I am using HttpURLConnection for making POST requests. I observe always the same behaviour during tests:
first request runs very fast (miliseconds)
all following requests take one second + some miliseconds
So something is causing 1 second delay. What can it be? The delay is happening exactly in HttpURLConnection#getInputStream().
I replaced the implementation with HttpClient - then everything is OK, no second delays (so it is not the server fault). But I really don't want to use any external dependency, so I would like to fix the HttpURLConnection thing... Any ideas?
Below current implementation. I tried some tips from stackoverflow (adding headers to the request), but with no success.
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-Length", ""
+ (body == null ? 0 : body.length));
// Send post request
con.setDoOutput(true);
OutputStream wr = con.getOutputStream();
if (body != null) {
wr.write(body);
}
wr.flush();
wr.close();
BufferedReader rd = new BufferedReader(new InputStreamReader(
con.getInputStream()));
String line;
String result = "";
while ((line = rd.readLine()) != null) {
result += line;
}
rd.close();
con.disconnect();
return result;
PS: It is about jse, not android.
You're never closing the input stream from the connection - that may mean that the connection isn't eligible for pooling, and on the next attempt it's waiting for up to a second to see if the previous request's connection will become eligible for pooling.
At the very least, it would be a good idea to close the stream. Use a try-with-resources block if you're using Java 7 - and ditto for the writer.
As an aside, I suggest you explicitly state the encoding you expect when reading - or use a higher-level library which detects that automatically based on headers.
I'm writing a proxy and have the following code:
Socket conUser;
Socket conDest;
try{
ServerSocket ss = new ServerSocket(Integer.parseInt(p.getProperty("proxy.port")));
while(true){
//Connect to user
conUser = ss.accept();
BufferedReader inputFromUser = new BufferedReader(new InputStreamReader(conUser.getInputStream()));
BufferedWriter outputToUser = new BufferedWriter(new OutputStreamWriter(conUser.getOutputStream(), "UTF8"));
//Get user request
StringBuffer req = new StringBuffer();
getUserRequest(inputFromUser, req);
System.out.println("User requested the following:");
System.out.println(req);
//Connect to server
InetAddress a = InetAddress.getByName(determineHost(req));
conDest = new Socket(a,80);
//Send request to server
BufferedWriter outputToServer = new BufferedWriter(new OutputStreamWriter(conDest.getOutputStream(), "UTF8"));
InputStreamReader inputFromServer = new InputStreamReader(conDest.getInputStream(), "UTF8");
outputToServer.write(req.toString());
outputToServer.flush();
System.out.println("==============================");
System.out.println("Server replied with the following:");
//Read reply from the server
//=========================================
int chars;
while ((chars = inputFromServer.read()) != -1) {
System.out.print((char)chars);
outputToUser.write(chars);
outputToUser.flush();
//serverReply.append(chars);
}
//Relay reply to user
//outputToUser.write(serverReply.toString());
//System.out.println(serverReply);
//outputToUser.flush();
conUser.close();
conDest.close();
}
}
catch (Exception e) {
System.err.println(e);
}
What happens is: I make a connection and it succeeds. I also send the request, and that succeeds too. I also get a reply, and am able to load the entire page's HTML, except that the read doesn't seem to terminate when it reaches the end of the content.
Specifically, I was attempting to load Google's homepage and the chunked transfer reached 0 (that is- end of chanked transfer), and thus there should've been no more input to read, but this did not cause the loop to stop reading. What's also strange to me is that pretty much all code examples of proxies do use this loop, and assuming they work, I don't see much differences between their code and mine.
How do I make the loop terminate correctly?
EDIT: for the record, yes- I know that the TCP connection should be kept open to handle further connections. This is not relevant to the problem I'm having. I need to get this loop to terminate per response.
In general the connection is not closed at the end of each response. Creating TCP connections is relatively time-consuming so the connection is left open, ready for you to send your next request.
Here are a couple of explanatory links:
http://en.wikipedia.org/wiki/HTTP_persistent_connection
http://en.wikipedia.org/wiki/HTTP_pipelining
If you want to terminate connection correctly after receiving HTTP response, your simple loop is not enough. You have to determine the end of message as described in section 4.4 Message Length of RFC 2616 and then close the connection.
However, it would be better to use existing libraries, such as built-in URLConnection.