I have a java client that invokes a thread to hit a servlet and retrieves last few lines from logs at the server, and show the retrieved log lines on the client. Every once in a while, the log thread times out.
Application server is Tomcat, but the error is intermittently reproducible across both Tomcat and Websphere, with client on Windows and server on Windows. With client on Windows and server on AIX, this problem has not occurred till now. I must mention that the code was stable for quite a few iterations, and suddenly started giving these problems.
What I have tried so far
The log reading client invokes the thread every 0.1 sec (used a sleep). I tried increasing the sleep time to 5 sec in the code, but it did not help.
When creating URLConnection object, I set properties like connectTimeout and readTimeout. I don't think readTimeout can be a cause, because that would have thrown a Socket exception.
3 I tried working with Tomcat configuration.
Connector port="9962" protocol="HTTP/1.1" connectionTimeout="200000" redirectPort="8445" acceptCount="30"
4 . The url connection is "disconnected" after use.
5 The stack trace seems to imply that the request never reached the application server, could this be because of some OS layer limits on connection. But in that case, there would have been an entry in the Event viewer of Windows.
java.net.ConnectException: Connection timed out: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.<init>(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
How would you go about diagnosing this problem? The server logs don't show anything suspicious. The client and server do not have any other networking devices to the best of my knowledge so no proxy is required, and the firewall is switched off.
I have not used keep alive thus far.
It is difficult to predict what is causing this. However, your next step should be to try running a packet sniffer on the client and / or server to see if the TCP connection requests are making it to the windows machine.
If the problem occurs both with Tomcat and Websphere, that would imply that the cause is at a lower level; i.e. in the OSes TCP/IP stacks, firewall ... or in the network. (And if the server is running in a virtual, it could be a drop-out in the virtual networking.)
Related
I have a REST service and some clients get a "Connection reset" error. But SOAP is stateless, so why doesn't it just simply reconnect and resend the request? It actually sends multiple messages in my use case, but the very first fails, and that is just to get some config data from the server. Is this something I need to configure? Should the client programmatically try to resend the message? Some users tried multiple times with the same result.
It never happened in the last years but now I get some reports of this problem.
The client uses in implementation of javax.xml.ws.Service, not just a raw socket. But even though I use JAX I get the low level error. It is wrapped by a WebServiceException, but that doesn't really help me fixing this problem.
The clients all use Java 8. It's either Update 66 or Update 74.
I am not able to reproduce the problem myself, I only have log files from users.
Here's the complete stack trace:
javax.xml.ws.WebServiceException: java.net.SocketException: Connection reset
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(Unknown Source)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.createResponsePacket(Unknown Source)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(Unknown Source)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(Unknown Source)
at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(Unknown Source)
at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Unknown Source)
at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Unknown Source)
at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Unknown Source)
at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Unknown Source)
at com.sun.xml.internal.ws.client.Stub.process(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(Unknown Source)
at com.sun.proxy.$Proxy31.getLimits(Unknown Source)
at xxxxxxxxxxxxx.SOAPServerAdapter.connect(Unknown Source)
at xxxxxxxxxxxxxxxxxxxx(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
... 18 more
It turned out it was about IPv4 and IPv6. I don't have enough knowledge to give a perfect answer but I can post here what they told me. Maybe this helps other devs/users who have the same problem.
So some clients have unexpected connection resets and it's not about server load as it usually is.
If the ISP of the client tries to get away from IPv4 they will give each user a unique IPv6 address (note that the ISP might do this gradually). They do not really have an IPv4 address per client any more, other than the IPv4 used locally, since most still use something like 192.168.0.0/24 for their LAN.
Instead of classic IPv4 they use some transaction mechanism (e.g. Dual-Stack Lite). Those clients to not have direct access to the IPv4 internet. So if your server only supports IPv4 then they might experience similar problems you get when they use a proxy. They encapsulate IPv4 packets within IPv6 packets for some parts of the communication. From Wikipedia: "The original IPv4 packet is recovered and NAT is performed upon the IPv4 packet and is routed to the public IPv4 Internet."
I don't really know what's going wrong here. Maybe the NAT runs out of addresses / ports or something like that. Or the process takes too long you the connection is reset by some node that is involved in the communication.
So there are two things to do:
Inform the ISP about those problems. They probably will help you trace the exact problem and help their clients so they can use your service. For that you need to know the ISP of the users that have the "connection reset" problem. Send them to https://www.whoismyisp.org/ or similar site.
Upgrade to IPv6 as soon as possible. Your server can use both versions of the protocol at the same time.
Check the load on your server. Looks like server is closing connections because of load - exception on web-service call
I'm currently setting up a vm cluster to run Jenkins for use with a large project,
I have jenkins set up, all the VM's are running Windows 7 64-bit with plenty of ram, disk space and the slave agents deployed (running as windows service).I keep getting the following error after a few minutes.
Connection was broken
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at hudson.remoting.SocketChannelStream$1.read(SocketChannelStream.java:33)
at sun.nio.ch.ChannelInputStream.read(Unknown Source)
at sun.nio.ch.ChannelInputStream.read(Unknown Source)
at sun.nio.ch.ChannelInputStream.read(Unknown Source)
at java.io.InputStream.read(Unknown Source)
at sun.nio.ch.ChannelInputStream.read(Unknown Source)
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:70)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
The slaves disconnect for 30 seconds to 2 minutes and then reconnect,
the response time also varies wildly between 400ms and 5 seconds.
The cluster is on its own switch and a ping from any machine returns a < 1ms time.
Any help?
There are a lot of bugs with the new NIO stuff in the Jenkins slave. We are experiencing similar serious instability issues up until the latest release at this time of writing.
For example:
https://issues.jenkins-ci.org/browse/JENKINS-22758
This posts deserves an update (August 2017): we are running 2.46 now and the slaves are a lot more stable.
I have a Java/JSP web application served up by Tomcat that makes web service calls out to a partner web service to retrieve data. The technologies used in the partner service are unknown. The partner web service has frequent extended outages where it returns a SocketTimeoutException:
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source)
at sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
If the partner web service has a brief outage, then recovers quickly, my application handles everything nicely.
If the partner web service has an extended outage of over an hour, and my application has had hundreds of calls to the service that have all timed out, at some point my application reaches a state where it does not recover. The partner service comes back, but my application calls to that service still result in the same exact SocketTimeoutException error.
If I start and stop Tomcat at that point then everything works fine after.
I am not using http keep-alives. My code is anal about cleaning up all object instances regardless of whether exceptions occur or not. It seems like the Tomcat Java process is "using up" some resource (sockets?), throwing away one with each error, until there are no more that can be used. Has anyone seen this before, and is there a solution apparent? I have done much searching on the matter and not found anyone with an identical problem.
Thanks in advance!
John
I had a situation in the past where I was running out of slots in the TCP/IP stacks for connections that were in the TCP_WAIT state there are some hard limits complied into the operating systems that you can bump up against. The way to find out what the limit is is to use a tool like netstat if you are running on windows server you can use some of the tools from sysinternals.
The solution to your problem might be a design pattern called Circuit Braker which is explained in the book called http://pragprog.com/book/mnee/release-it
With the circuit breaker pattern what happens is that your calls to the remote web service flow through the circuit breaker which will open the breaker when too many calls to the remote service fail, when the breaker is in the open state calls to the remote service will fail right away in the breaker code, usually you can program the breaker to retry and see if it will open again. Anyway the book has a better explanation than the brief one I just gave you.
https://bitbucket.org/asaikali/circuitbreaker/ has an open source sample implementation of the CircuitBreaker pattern.
Testing an unmodified version of JCTerm (terminal emulator, can be used as an applet; I would like to use the applet functionality), everything seems to be working fine in Eclipse's AppletViewer, but testing the resulting jar file in an web page, all options display a message box with a flashing yellow warning symbol and do nothing. Some options display a "Establish the connection before this setting" error.
The option I'm interested in using is 'Open SHELL Session...' from the File menu. When trying 'Open SHELL Session...' option, the Java Console for the applet displays this stuff, which seems directly related. I don't have a clue as to what it means though.
network: Connecting http://xxx.xx.xx.xxx/crossdomain.xml with proxy=DIRECT
network: Connecting http://xxx.xx.xx.xxx:80/ with proxy=DIRECT
java.security.PrivilegedActionException: java.net.ConnectException: Connection refused: connect
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.deploy.net.CrossDomainXML.check(Unknown Source)
at com.sun.deploy.net.CrossDomainXML.check(Unknown Source)
at sun.plugin2.applet.Applet2SecurityManager.checkConnect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at com.jcraft.jsch.Util$1.run(Util.java:354)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.<init>(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at com.sun.deploy.net.CrossDomainXML$2.run(Unknown Source)
... 10 more
An example of this program working correctly can be found here, http://wiredx.net/jcterm/
I'm the author of jcterm.
The jar files at http://wiredx.net/jcterm/ have been digitally signed.
So, if you copy and intstall those files into your web server, it will work.
This looks like a security problem.
By default, an unsigned Java applet can only connect to the host if was loaded from. Additionally, if the host you want to connect to allows this with it's crossdomain.xml file, you can also connect to other hosts.
Judging from the stack trace, you want to connect to a host which does not have an HTTP server, and thus can't provide a crossdomain.xml file. For this reason, you get this exception here.
There are these ways out of this:
Put the applet on the same web server which you want to connect to with SSH later. (Every applet can connect to its own host.)
Let the SSH server have a minimal web server with a crossdomain.xml. (The crossdomain.xml must allow content from the applet's server to access this server.)
Sign the applet (and let the user trust it). (Signed and trusted applets are allowed to do everything.)
The official WiredX sample applet you linked uses the last method, this is why it works even when connecting to your server.
After a hard work, I have set up Java RMI tunnelling using apache as http
server. Everything is fine and Works like a charm at my office LAN.But when I installed at client's place, I am getting some exception.The RMI System works only on his server.
When I tried from other clients' pcs I get the following.
Can you guys help me solve this?
java.rmi.ConnectException: Connection refused to host: 172.xx.x.xxx;
nested exception is: java.net.ConnectException: Connection timed
out: connect at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown
Source) at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown
Source) at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown
Source) at sun.rmi.server.UnicastRef.invoke(Unknown Source) at
java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown
Source) at
java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source)
at $Proxy1.getUserID(Unknown Source) at
rmi.source.ServerImpl$JobScheduler.run(ServerImpl.java:265) at
java.util.TimerThread.mainLoop(Unknown Source) at
java.util.TimerThread.run(Unknown Source) Caused by:
java.net.ConnectException: Connection timed out: connect at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.PlainSocketImpl.doConnect(Unknown Source) at
java.net.PlainSocketImpl.connectToAddress(Unknown Source) at
java.net.PlainSocketImpl.connect(Unknown Source) at
java.net.SocksSocketImpl.connect(Unknown Source) at
java.net.Socket.connect(Unknown Source) at
java.net.Socket.connect(Unknown Source) at
java.net.Socket.<init>(Unknown Source) at
java.net.Socket.<init>(Unknown Source) at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(Unknown
Source) at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(Unknown
Source) ... 10 more
I am running files from roseindia rmi
there is mistake rmicregistry, one have to use rmiregistry:
I changed ip to localhost, how to run on special port number I don't know this moment
one has to use rmiregistry
not rmicregistry
so:
move to folder where does your class files take seat.
type : rmiregistry
then open 2 new command prompts move to same folder where classes seat and use one to run server and another to client.
RMI servers have the nasty habit to pass new port numbers to clients to handle the client communication to a particular Remote object. If this port number is blocked by the firewall, you would get an exception like that.
Make sure you register all Remote objects with the same port number. RMI will then multiplex the client calls for you.
after my research, the solution is:
System.setProperty("java.rmi.server.hostname", IP);
before you register the service on server side. following is an example, hope it might help you!
public static void main(String[] args) throws MalformedURLException, RemoteException, AlreadyBoundException {
RmiServer server=new RmiServer();
System.setProperty("java.rmi.server.hostname", 指定IP);
LocateRegistry.createRegistry(8808);
Naming.rebind("//10.10.116.74:8808/SAMPLE-SERVER", server);
}