Develop Reference

What causes ch.ethz.ssh2.Connection to have a hang time?

I am currently using ch.ethz.ssh2.Connection to connect to my servers in java. sometimes it hangs on one server.(maybe like 10-15 seconds). I wanted to know what causes this hang time and how to avoid it.
Connection sample
conn = new ch.ethz.ssh2.Connection(serverName);
conn.connect();
boolean isAuthenticated = conn.authenticateWithPassword(user, pass);
logger.info("Connecting to " + server);
if (isAuthenticated == false) {
logger.info(server + " Please check credentials");
}
sess = conn.openSession();
// I am connecting to over 200 servers and closing them. What would be the best practice to loop thru all these servers in the minimal time.
//some servers quickly connects, while some takes some time.
why does this happen?

The main question is: Is it a code problem, a network problem or a server problem.
A code problem can be debugged - unfortunately ch.ethz.ssh2.Connection does not have any logging possibility to detect what is going inside.
May be you should thing about switching the ssh library (or use it for some tests with the problematic servers). From my experience sshj is very useful.
If it is a network problem or a server problem you can check what is going on via Wireshark. If network packets are sent but the response is delayed the problem is not the used client-side code.

My psychic debugging powers tell me that the server is doing a DNS lookup on the IP address of each client which connects. These DNS lookups are either taking a long time to complete, or they're failing entirely. The DNS lookup will block the authentication process until it finishes, successfully or not.
If the server is the OpenSSH server, this behavior is controlled by the sshd config "UseDNS" option.

java.net.SocketTimeoutException is not always thrown [duplicate]

Here is my code :
URL u = new URL("http://www.google.com");
URLConnection conn = u.openConnection();
conn.setConnectTimeout(3000);
conn.connect();
My network connection is sometimes unstable(I've connected to the wireless router but actually my router doesn't have Internet access). When that happens, this code will block for a lone time and finally throws UnknownHostException. Why setConnectTimeout(3000) doesn't work in this case? How to fix this?
Thanks!
------------update---------------
My guess is that conn.connect() will query DNS first but there's no time limit for this operation. I've tried Socket class and problem remains. setTimeout() seems do not work for DNS query.

I found a post that can work around it. Use another thread to query DNS to simulate timeout:
http://thushw.blogspot.sg/2009/11/resolving-domain-names-quickly-with.html

Some non-standard implmentation of this method may ignore the specified timeout.
See this setConnectTimeout

Java UrlConnection triggering "Connection reset" exceptions under high load. Why?

I'm using Java to stream files from Amazon S3, on Linux (Ubuntu 10) 64-bit servers.
I'm using a separate thread for each file, and each file opens an HttpURLConnection which downloads and processes each file concurrently.
Everything works beautifully until I reach a certain number of streams (usually around 2-300 concurrent streams). At irregular points after this, several (say 10) of the threads will start experiencing java.net.IOException: Connection reset errors simultaneously.
I am throttling the download speed, and am way below the 250mbit/s limit of an m1.large instance. There is also insignificant load on all other server aspects (e.g. CPU, load average and memory usage are all fine).
What could be causing this, or how could I track it down?

not trivial to guess what may happen but this is a couple of hints , may be some may apply into your context:
can you check your shell (linux bash /zsh or any other) to see if you raise up the standard limits restricting the number of file descriptors (but sockets too),
man ulimit with bash shell
did you close the streams explicitly in your Java code ? not closing streams may induce such clever problems
try to google for Linux TCP kernel tuning to try to see if your ubuntu server has a well suited stack for such load context...
HTH
Jerome

They might have spillover problem at VIPs because of number of con-current connections reached the limit. You may decrease the size and see...

The problem here is largely in your language. The high load is triggering the error condition, and the error condition results in the exception. Not the other way around.

One relatively common reason for problems like this is that an intermediate proxy (firewall, load balancer) drops what it deems inactive (or too long-lived) HTTP connection.
But beyond this general possibility, EC2 definitely has more kinks as others have suggested.

You are probably running out of ephemeral ports. This happens under load when many short lived connections are opened and closed rapidly. The standard Java HttpURLConnection is not going to get you the flexibility you need to set the proper socket options. I recommend going with the Apache HttpComponents project, and setting options like so...
...
HttpGet httpGet = new HttpGet(uri);
HttpParams params = new BasicHttpParams();
params.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 16 * 1000); // 16 seconds
params.setParameter(CoreConnectionPNames.SO_REUSEADDR, true); // <-- teh MOJO!
DefaultHttpClient httpClient = new DefaultHttpClient(connectionManager, params);
BasicHttpContext httpContext = new BasicHttpContext();
HttpResponse httpResponse = httpClient.execute(httpGet, httpContext);
StatusLine statusLine = httpResponse.getStatusLine();
if (statusLine.getStatusCode() >= HTTP_STATUS_CODE_300)
{
...
I've omitted some code, like the connectionManager setup, but you can grok that from their docs.
[Update]
You might also add params.setParameter(CoreConnectionPNames.SO_LINGER, 1); to keep ephemeral ports from lingering around before reclamation.

Understanding if URLConnection connections are really being pooled?

I'm using URLConnection for my http client. Using Wireshark, I can see that setting up an https connection can take up to four seconds. Therefore I would like to pool connections if possible to avoid the https setup time. My flow looks like this:
public String work(String url) {
HttpURLConnection conn = (HttpURLConnection)(new URL(url)).openConnection();
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String response = br.readResponseFromConnection();
conn.disconnect();
br.close();
return response;
}
public void onBtnClickTest() {
work("https://example.com/echo?param=abc");
}
With the above setup, I click my test button twice, but it looks like the https setup is done for each call, which makes me think the connection isn't really being reused (at least not in the way that I was hoping). Through Wireshark I see something like the following :
// first click
Client Hello
Server Hello
Certificate, Server Hello Done
Client Key Exchange, Change Cipher Spec, Encrypted...
Encrypted Handshake Message...
Application Data
// second click, 20 seconds after my first click.
Client Hello
Server Hello
Certificate, Server Hello Done
Client Key Exchange, Change Cipher Spec, Encrypted...
Encrypted Handshake Message...
Application Data
However, if I click the test button twice within a span of < 5 seconds, it looks like the handshake is skipped for the second run, and I immediately see the "Application Data" message. I think I remember reading somewhere that URLConnection only keeps connections pooled for 5 seconds.
So my questions:
Can URLConnection actually pool my connections in the way that I want, in that subsequent connections can skip the https handshake (if hitting the same domain)?
If the above is possible, is there a way to increase the duration that connections stay pooled? My application is unlikely to make http calls within 5 seconds of eachother.
I know HttpClient offers a pool manager to reuse a connection, but looks more complicated than URLConnection. Can it get me around # 1 & 2 if they won't do what I need?
Thank you

You can use Keep-Alive header property to persisit any httpconnection. Here is documentation on Http Keep-alive Http Keep Alive
section What can you do to help with Keep-Alive gives exact information what you are looking for. Quick search in SO gave me another link which is exactly like your question.
SO disucssion link

HttpURLConnection inside a loop

I'm trying to connect to one URL that I know that exist but I don't know when.
I don't have access to this server so I can't change anything to receive a event.
The actual code is this.
URL url = new URL(urlName);
for(int j = 0 ; j< POOLING && disconnected; j++){
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
int status = connection.getResponseCode();
if(status == HttpURLConnection.HTTP_OK || status == HttpURLConnection.HTTP_NOT_MODIFIED){
//Some work
}else{
//wait 3s
Thread.sleep(3000);
}
}
Java not is my best skill and I'm not sure if this code is good from the point of view of performance.
I'm opening a new connection every 3 seconds? or the connection is reused?
If I call to disconnect() I ensure that no new connections are open in the loop, but... it will impact in performance?.
Suggestions? What is the fast/best ways to know it a URL exist?

1) Do use disconnect, you don't want numerous open connections you don't use. Discarding resources you don't use is a basic practice in any language.
2) I don't know if opening and closing new network connection every 3 seconds will pollute system resources, the only way to check it is to try.
3) You may want to watch for 'ConnectException', if by "URL [does not] exist" you mean server is down.

This code is okay from a performance point of view, it will create a new connection each time. Anyway if you have a Thread.sleep(3000) in your loop, you shouldn't have to worry about performance ;)
If you're concerned about connection usage on the server side, you can look into apache HTTP client, it has a lot of features. I think it handles keep alive connections by default.