java.net.ConnectException: Connection refused: connect under load - java

I'm running a loadtest on a brand new Windows 2008 64 bit machine.
The loader is a Java Applet which uses an HttpURLConnection to post requests to the server which is listening on a ServerSocket.accept(), both loader and server are running on the same machine.
On my old Windows 2003 server I was able to load over a 1000 users using this configuration.
However, with the new server, when loading around 400 sessions the loader starts throwing the following exception:
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:69)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:337)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
at sun.net.www.http.HttpClient.New(HttpClient.java:300)
at sun.net.www.http.HttpClient.New(HttpClient.java:316)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1087)
...
It looks as if the server or the machine is running out of some resource.
The ServerSocket backlog is set to 256 and it looks like it is not being exhausted.
The machine cpu utilization is less than 10% and the server has plenty of available memory.
Observing the client and server using Visual VM it looks like both are functioning properly at the time of the problem.
Any ideas ?

Maybe server is rejecting connections due to lack of some other resources? Each HTTP requests will require opening TCP connection, which under Linux uses 'file' (I'm not Linux expert, so please correct me if I'm wrong). So sometimes CPU and memory are low, but HTTP server opens hundreds of files which in the end causes failure and any further request will be rejected.
I'm not sure if this applies to Windows as well, but give it a shot.

There are some additional limitations imposed in 2008 that were not part of previous versions. I would start here... http://technet.microsoft.com/en-us/magazine/2007.12.network.aspx

Related

HTTP transport error: java.net.SocketException: Connection reset

A client application has been built using Jdeveloper 10.1.3.2 and it is running on OC4J server. This application is sending data to external server application. It is working for quite long time without any issue. Lately a connection issue occurred and the following stack trace is generated:
com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.SocketException: Connection reset
at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:133)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:153)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:93)
at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:629)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:588)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:573)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:470)
at com.sun.xml.ws.client.Stub.process(Stub.java:319)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:157)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:140)
at $Proxy44.sendRem(Unknown Source)
After goggling I found out a good discussion about the error sockets - What's causing my java.net.SocketException: Connection reset? .One answer in this link says that the issue mostly from the client side because if it is from the server side the exception will be (SocketException reset by peer).
What I did:
I tried out to increase the socket time out for the OC4J with the help of this form How to change OC4J HTTP Timeout. What I did is I changed the propriety oracle.j2ee.http.socket.timeout to be 5000 instead of 500 (10 times longer)
But the error still there. So, any suggestion to over come this issue?
Note: I able to use telnet command for external server IP and Port and it is working fine.
-------------------------------------------------------- Update 1 --------------------------------------------------------
I increase the server clock skew where the client application is running using the following command on server start up:
-Dweblogic.wsee.security.clock.skew=72000000
-Dweblogic.wsee.security.delay.max=72000000
But no luck, problem is not resolved.
-------------------------------------------------------- Update 2 --------------------------------------------------------
I realized that the problem is not from application at all; I test the external URL using SoapUI and I got the same error Connection rest. I think this new update clreay shows that there is nothing wrong with program code. But I need to know where to go or check now. Where is the starting point now to overcome the issue. Any clue will be helpful.
As you can see from Update 2 in the question, the problem was not from the client application because same error occurred from SoapUI.
The problem was that the machine where the client application was running have low bandwidth which was not enough for APIs communication. Using simple speed test , I found out that the upload bandwidth was low comparing to minimum requirements given by server application team.
I concluded this fact by monitoring the network resource using Resource Monitor in Windows while the client application was running and by using online speed check
To solve the issue, the machine bandwidth has to be increased where the client application is running.

Java ServerSocket Connection Refused after Certain number

In our application we have a standard TCP/IP Server socket programming to allow clients to connect to the Server.
What i have noticed is after certain number of connections the client is not able to open up new connection to the server. The number by which i encountered the issues is 946 client connection. After which I got connection refused error on the client side.
The server is running in Linux box (64 bit). The server is still up an running, no out-of-memory error.
I don't see an issue with backlog, since I am having only one client which is looping through and opening up the new connection and put it in the array.
I am performing the stress testing on maximum number of connections the client could open to the server and how to maximize the same.
Any thoughts?
Edit 1:
Server is not listening on the the port which I am trying to connect. On the server I got below error.
'08:10:41:055 [*] [TcpServer] - java.net.SocketException: Too many open files <stack trace follows>
08:10:41:055 [*] [TcpServer] - at java.net.PlainSocketImpl.socketAccept(Native Method)
08:10:41:055 [*] [TcpServer] - at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
08:10:41:055 [*] [TcpServer] - at java.net.ServerSocket.implAccept(ServerSocket.java:522)
08:10:41:055 [*] [TcpServer] - at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:317)
I increased the file limit as mentioned here. This solved the issue.

NoRouteToHostException on client or server?

I am getting
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:564)
at sun.reflect.GeneratedMethodAccessor638.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:130)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
javadoc says
Signals that an error occurred while attempting to connect a socket to
a remote address and port. Typically, the remote host cannot be
reached because of an intervening firewall, or if an intermediate
router is down.
Is this error on client side or remote side or it can be either of these?
Basically it says that your client cannot connect to the server due to the address is inaccessible from the client machine.
Please check that the address you are connecting to is accessible, either via ping command in your Command Prompt (Windows) or terminal (Unix-based):
ping <address>
or if it's a web server you can try to check it in your web browser.
The ping command is helpful for me in most cases, since I would know why exactly I can't connect to the address. It can be a mistyped address or like the javadoc suggests, problem with firewall.
Either. It could be a firewall on the client machine blocking outgoing calls or somewhere at the other end.
It may be possible the ping will provide responses, but the application may still fail to connect.
If that is the case, I would suggest using telnet to try and connect to the host using the desired port, telnet host.address port
If the connection is refused then the port on the host will need to be allowed. It that succeeds, but the application still won't connect:
Verify the address:port being used in your application are the same as those used in the telnet test.
It may be a local port on the client blocking the connection, in which case you would need to allow the port on the client
In my case it was solved like this:
1 - Clear the cache with 'yarn cache clean' or 'npm cache clean --force'
2 - after that reboot the pc.
If you clear the cache but do not reboot, the data will still be stored 'somewhere' on the network, so the port will not connect, it is still busy.
After restarting it connects smoothly.
I hope I was helpful!
I had the same issue, and resolve it by disabling the firewall on both sides :
systemctl disable firewalld
service firewalld stop
I had the same issue, I did run iptables flush on host server, this fixed the issue.
# iptables --flush

Network is unreachable?

I want to run a cyc (opencyc) Java class, but when I run this class I got an error mentioning that the network is unreachable. I am working on NetBeans 7.0 and opencyc 2.0 win 32 version. I am using localhost (127.0.0.1) to test. The error is below.
Network is unreachable: connect
java.net.SocketException: Network is unreachable: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at java.net.Socket.<init>(Socket.java:366)
at java.net.Socket.<init>(Socket.java:180)
at org.opencyc.api.CycConnection.initializeApiConnections(CycConnection.java:223)
at org.opencyc.api.CycConnection.<init>(CycConnection.java:199)
at org.opencyc.api.CycAccess.<init>(CycAccess.java:121)
at org.opencyc.api.CycAccess.<init>(CycAccess.java:61)
at org.opencyc.api.ApiDemo.<init>(ApiDemo.java:50)
at org.opencyc.api.ApiDemo.main(ApiDemo.java:147)
Exception in thread "main" java.lang.NullPointerException
at org.opencyc.api.ApiDemo.<init>(ApiDemo.java:56)
at org.opencyc.api.ApiDemo.main(ApiDemo.java:147)
Interruption while waiting Cyc connection establishment, closing sockets
Exception in thread "Thread-1" java.lang.NullPointerException
at org.opencyc.api.CycConnection$TaskProcessorBinaryResponseHandler.access$002(CycConnection.java:988)
at org.opencyc.api.CycConnection.close(CycConnection.java:268)
at org.opencyc.api.CycConnection$ConnectionTimer.run(CycConnection.java:1351)
Why is this happening? How can I connect to the cyc?
The problem is that your network settings do not allow you to connect to this machine. This is not a Java problem as such.
try
ping {hostname}
until this works, there is nothing you can do in java to work around this.
If you have the same version I have, it's trying to establish a connection to a Cyc image with the hostname and port defined by
CycConnection.DEFAULT_HOSTNAME
CycConnection.DEFAULT_BASE_PORT
You can find the proper values to use by evaluating the following in your Cyc image:
(get-machine-name)
*BASE-TCP-PORT*
Hope this works for you.
A dev on my team encountered this today, only their machine is seeing this issue out of many on the team.
Through much debugging we eventually discovered that localhost would not resolve for some reason. It would work in some JVM contexts and fail in others so it was completely inconsistent.
Replacing localhost with explicitly 127.0.0.1 fixed the issue. My hunch is that something was different with a local network and IPv6 perhaps, but we still don't really understand why this is the fix.

Can't connect to cassandra : NodeContext killing all pooled connections for session

When I try to connect to cassandra within my java application I receive the exception at the end of this message. I am using pelops library to access database. Database is working on linux and my development environment is on windos 7. Intresting line is
WARNING: 10.0.0.7 NodeContext killing all pooled connections for session 44
10.0.0.7 is the IP of the cassandra node. And I can connect this node with cassandra-cli. After a few hours of googling I decided to ask, what i am doing wrong?
Complete stack trace is below.
SEVERE: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.wyki.cassandra.pelops.ThriftPool$Connection.open(ThriftPool.java:329)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext.createConnection(ThriftPool.java:438)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext.access$5(ThriftPool.java:429)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext$1.run(ThriftPool.java:494)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 7 more
WARNING: 10.0.0.7 NodeContext killing all pooled connections for session 44
"Connection refused" means that Cassandra isn't listening on the interface you're connecting to. By default Cassandra listens on port 9160 on localhost. You will need to change that if you want to connect externally; read the comments for ThriftAddress in the configuration file.
Pelops isn't doing anything remotely tricky when it comes to making connections to Cassandra so it seems very unlikely it's Pelops specific.
I've just had a quick look at the Hector connection code and it looks pretty much the same:
https://github.com/rantav/hector/blob/master/core/src/main/java/me/prettyprint/cassandra/connection/HThriftClient.java
The cassandra-cli code looks pretty much the same as well:
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cli/CliMain.java#L57
p.s. Cassandra should not be configured to listen on 0.0.0.0. See the "Why can't I make Cassandra listen on 0.0.0.0 (all my addresses)?" on the Cassandra FAQ.
It seems that this is a problem with pelops, which is a library on top of cassandra's thrift client. It has some problems for connection to Cassandra working on linux and client on a windows machine.
I've got the same exception connecting with "any" client to cassandra remotely:
ConnectException: Connection refused
Looking through SO I couldn't find a quick solution for that, just reading carefully through cassandra docs and comments within cassandra.yaml - they are very handy!
So for me worked following:
change rpc_address value to the hostname or ipaddress you are using to connect to the cassandra instance with the client

Categories

Resources