My Thread pool is becoming full because, most of the threads are waiting for the socket connection. How to add timeout for jmx connect
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
- locked <0x05671ad0> (a java.net.SocksSocketImpl)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:548)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:351)
at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:71)
at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:105)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1871)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1841)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
This is a tough one. It depends on which part is timing out due to the environment you are in.
Most probably you are encountering a connect timeout at the TCP level due either firewall issues or a dead application with its socket still up.
The overall connecttimeout for the JVM is set to -1, so infinite by default. Hello thread leaks.
sun.net.client.defaultConnectTimeout (default: -1)
You can find the documentation for Java oracle network parameters here: https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html
The jmx/rmi specific documentation sadly doesn't have a connect timeout setting at that level. But they have a bunch of other controls on the read time-out: https://docs.oracle.com/javase/8/docs/technotes/guides/rmi/sunrmiproperties.html
You can test this while developing with something as simple as using netcat and pointing your app over to the localhost port:
# creat socket on port 3333
netcat -l 3333
# now point your app on the socket
# and you should be able to reproduce
This will stimulate a dead app or firewall block on the port.
Just add the Following System property - "-Dsun.rmi.transport.tcp.responseTimeout=60000"
Related
I am getting
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:564)
at sun.reflect.GeneratedMethodAccessor638.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:130)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
javadoc says
Signals that an error occurred while attempting to connect a socket to
a remote address and port. Typically, the remote host cannot be
reached because of an intervening firewall, or if an intermediate
router is down.
Is this error on client side or remote side or it can be either of these?
Basically it says that your client cannot connect to the server due to the address is inaccessible from the client machine.
Please check that the address you are connecting to is accessible, either via ping command in your Command Prompt (Windows) or terminal (Unix-based):
ping <address>
or if it's a web server you can try to check it in your web browser.
The ping command is helpful for me in most cases, since I would know why exactly I can't connect to the address. It can be a mistyped address or like the javadoc suggests, problem with firewall.
Either. It could be a firewall on the client machine blocking outgoing calls or somewhere at the other end.
It may be possible the ping will provide responses, but the application may still fail to connect.
If that is the case, I would suggest using telnet to try and connect to the host using the desired port, telnet host.address port
If the connection is refused then the port on the host will need to be allowed. It that succeeds, but the application still won't connect:
Verify the address:port being used in your application are the same as those used in the telnet test.
It may be a local port on the client blocking the connection, in which case you would need to allow the port on the client
In my case it was solved like this:
1 - Clear the cache with 'yarn cache clean' or 'npm cache clean --force'
2 - after that reboot the pc.
If you clear the cache but do not reboot, the data will still be stored 'somewhere' on the network, so the port will not connect, it is still busy.
After restarting it connects smoothly.
I hope I was helpful!
I had the same issue, and resolve it by disabling the firewall on both sides :
systemctl disable firewalld
service firewalld stop
I had the same issue, I did run iptables flush on host server, this fixed the issue.
# iptables --flush
I want to run a cyc (opencyc) Java class, but when I run this class I got an error mentioning that the network is unreachable. I am working on NetBeans 7.0 and opencyc 2.0 win 32 version. I am using localhost (127.0.0.1) to test. The error is below.
Network is unreachable: connect
java.net.SocketException: Network is unreachable: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at java.net.Socket.<init>(Socket.java:366)
at java.net.Socket.<init>(Socket.java:180)
at org.opencyc.api.CycConnection.initializeApiConnections(CycConnection.java:223)
at org.opencyc.api.CycConnection.<init>(CycConnection.java:199)
at org.opencyc.api.CycAccess.<init>(CycAccess.java:121)
at org.opencyc.api.CycAccess.<init>(CycAccess.java:61)
at org.opencyc.api.ApiDemo.<init>(ApiDemo.java:50)
at org.opencyc.api.ApiDemo.main(ApiDemo.java:147)
Exception in thread "main" java.lang.NullPointerException
at org.opencyc.api.ApiDemo.<init>(ApiDemo.java:56)
at org.opencyc.api.ApiDemo.main(ApiDemo.java:147)
Interruption while waiting Cyc connection establishment, closing sockets
Exception in thread "Thread-1" java.lang.NullPointerException
at org.opencyc.api.CycConnection$TaskProcessorBinaryResponseHandler.access$002(CycConnection.java:988)
at org.opencyc.api.CycConnection.close(CycConnection.java:268)
at org.opencyc.api.CycConnection$ConnectionTimer.run(CycConnection.java:1351)
Why is this happening? How can I connect to the cyc?
The problem is that your network settings do not allow you to connect to this machine. This is not a Java problem as such.
try
ping {hostname}
until this works, there is nothing you can do in java to work around this.
If you have the same version I have, it's trying to establish a connection to a Cyc image with the hostname and port defined by
CycConnection.DEFAULT_HOSTNAME
CycConnection.DEFAULT_BASE_PORT
You can find the proper values to use by evaluating the following in your Cyc image:
(get-machine-name)
*BASE-TCP-PORT*
Hope this works for you.
A dev on my team encountered this today, only their machine is seeing this issue out of many on the team.
Through much debugging we eventually discovered that localhost would not resolve for some reason. It would work in some JVM contexts and fail in others so it was completely inconsistent.
Replacing localhost with explicitly 127.0.0.1 fixed the issue. My hunch is that something was different with a local network and IPv6 perhaps, but we still don't really understand why this is the fix.
I'm running a loadtest on a brand new Windows 2008 64 bit machine.
The loader is a Java Applet which uses an HttpURLConnection to post requests to the server which is listening on a ServerSocket.accept(), both loader and server are running on the same machine.
On my old Windows 2003 server I was able to load over a 1000 users using this configuration.
However, with the new server, when loading around 400 sessions the loader starts throwing the following exception:
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:69)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:337)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
at sun.net.www.http.HttpClient.New(HttpClient.java:300)
at sun.net.www.http.HttpClient.New(HttpClient.java:316)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1087)
...
It looks as if the server or the machine is running out of some resource.
The ServerSocket backlog is set to 256 and it looks like it is not being exhausted.
The machine cpu utilization is less than 10% and the server has plenty of available memory.
Observing the client and server using Visual VM it looks like both are functioning properly at the time of the problem.
Any ideas ?
Maybe server is rejecting connections due to lack of some other resources? Each HTTP requests will require opening TCP connection, which under Linux uses 'file' (I'm not Linux expert, so please correct me if I'm wrong). So sometimes CPU and memory are low, but HTTP server opens hundreds of files which in the end causes failure and any further request will be rejected.
I'm not sure if this applies to Windows as well, but give it a shot.
There are some additional limitations imposed in 2008 that were not part of previous versions. I would start here... http://technet.microsoft.com/en-us/magazine/2007.12.network.aspx
My program works just fine when I do
Socket s = new Socket("127.0.0.1", 10000);
But when I replace localhost with my own external IP adress, it fails
What gives? It throws an UnknownHostException immediately, even though I can ping my external IP just fine.
Stack trace:
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at FileClient.main(FileClient.java:29)
What gives? It throws an
UnknownHostException immediately, even
though I can ping my external IP just
fine. Stack trace:
I don't see an UnknownHostException in your stack trace. The "remote" side actively refuses the connection. Could be because of a firewall, but unlikely, as firewalls usually silently drop packets instead of actively refusing. Most likely, the server application just isn't listening on the external interface. Most server software behave like that in the default configuration due to security reasons. Check what the output of the netstat -n -a command.
If that's your external IP you would need to open/port forward 100000
When I try to connect to cassandra within my java application I receive the exception at the end of this message. I am using pelops library to access database. Database is working on linux and my development environment is on windos 7. Intresting line is
WARNING: 10.0.0.7 NodeContext killing all pooled connections for session 44
10.0.0.7 is the IP of the cassandra node. And I can connect this node with cassandra-cli. After a few hours of googling I decided to ask, what i am doing wrong?
Complete stack trace is below.
SEVERE: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.wyki.cassandra.pelops.ThriftPool$Connection.open(ThriftPool.java:329)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext.createConnection(ThriftPool.java:438)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext.access$5(ThriftPool.java:429)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext$1.run(ThriftPool.java:494)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 7 more
WARNING: 10.0.0.7 NodeContext killing all pooled connections for session 44
"Connection refused" means that Cassandra isn't listening on the interface you're connecting to. By default Cassandra listens on port 9160 on localhost. You will need to change that if you want to connect externally; read the comments for ThriftAddress in the configuration file.
Pelops isn't doing anything remotely tricky when it comes to making connections to Cassandra so it seems very unlikely it's Pelops specific.
I've just had a quick look at the Hector connection code and it looks pretty much the same:
https://github.com/rantav/hector/blob/master/core/src/main/java/me/prettyprint/cassandra/connection/HThriftClient.java
The cassandra-cli code looks pretty much the same as well:
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cli/CliMain.java#L57
p.s. Cassandra should not be configured to listen on 0.0.0.0. See the "Why can't I make Cassandra listen on 0.0.0.0 (all my addresses)?" on the Cassandra FAQ.
It seems that this is a problem with pelops, which is a library on top of cassandra's thrift client. It has some problems for connection to Cassandra working on linux and client on a windows machine.
I've got the same exception connecting with "any" client to cassandra remotely:
ConnectException: Connection refused
Looking through SO I couldn't find a quick solution for that, just reading carefully through cassandra docs and comments within cassandra.yaml - they are very handy!
So for me worked following:
change rpc_address value to the hostname or ipaddress you are using to connect to the cassandra instance with the client