Related
Given this stack trace snippet
Caused by: java.net.SocketException:
Software caused connection abort:
socket write error at
java.net.SocketOutputStream.socketWrite0(Native
Method)
I tried to answer the following questions:
What code is throwing this exception? (JVM?/Tomcat?/My code?)
What causes this exception to be thrown?
Regarding #1:
Sun's JVM source doesn't contain this exact message, but I think the text Software caused connection abort: socket write error is from the native implementation of SocketOutputStream:
private native void socketWrite0(FileDescriptor fd, byte[] b, int off,
int len) throws IOException;
Regarding #2
My guess is that it is caused when the client has terminated the connection, before getting the full response (e.g. sent a request, but before getting the full response, it got closed / terminated / offline)
Questions:
Are the above assumptions correct (#1 and #2)?
Can this be diffrentiated from the situation: "could not write to the client, due to a network error on the server side"? or would that render the same error message?
And most important: Is there an official document (e.g from Sun) stating the above?
I need to have a proof that this stack trace is the socket client's "fault", and there is nothing that the server could have done to avoid it. (except catching the exception, or using a non Sun JVM SocketOutputStream, though both don't really avoid the fact the client has terminated)
This error can occur when the local network system aborts a
connection, such as when WinSock closes an established connection
after data retransmission fails (receiver never acknowledges data sent
on a datastream socket).
See this MSDN article. See also Some information about 'Software caused connection abort'.
The java.net.SocketException is thrown when there is an error creating or accessing a socket (such as TCP). This usually can be caused when the server has terminated the connection (without properly closing it), so before getting the full response. In most cases this can be caused either by the timeout issue (e.g. the response takes too much time or server is overloaded with the requests), or the client sent the SYN, but it didn't receive ACK (acknowledgment of the connection termination). For timeout issues, you can consider increasing the timeout value.
The Socket Exception usually comes with the specified detail message about the issue.
Example of detailed messages:
Software caused connection abort: recv failed.
The error indicates an attempt to send the message and the connection has been aborted by your server. If this happened while connecting to the database, this can be related to using not compatible Connector/J JDBC driver.
Possible solution: Make sure you've proper libraries/drivers in your CLASSPATH.
Software caused connection abort: connect.
This can happen when there is a problem to connect to the remote. For example due to virus-checker rejecting the remote mail requests.
Possible solution: Check Virus scan service whether it's blocking the port for the outgoing requests for connections.
Software caused connection abort: socket write error.
Possible solution: Make sure you're writing the correct length of bytes to the stream. So double check what you're sending. See this thread.
Connection reset by peer: socket write error / Connection aborted by peer: socket write error
The application did not check whether keep-alive connection had been timed out on the server side.
Possible solution: Ensure that the HttpClient is non-null before reading from the connection.E13222_01
Connection reset by peer.
The connection has been terminated by the peer (server).
Connection reset.
The connection has been either terminated by the client or closed by the server end of the connection due to request with the request.
See: What's causing my java.net.SocketException: Connection reset?
I have seen this most often when a corporate firewall on a workstation/laptop gets in the way, it kills the connection.
eg. I have a server process and a client process on the same machine. The server is listening on all interfaces (0.0.0.0) and the client attempts a connection to the public/home interface (note not the loopback interface 127.0.0.1).
If the machine is has its network disconnected (eg wifi turned off) then the connection is formed. If the machine is connected to the corporate network (directly or vpn) then the connection is formed.
However, if the machine is connected to a public wifi (or home network) then the firewall kicks in an kills the connection. In this situation connecting the client to the loopback interface works fine, just not to the home/public interface.
Hope this helps.
To prove which component fails I would monitor the TCP/IP communication using wireshark and look who is actaully closing the port, also timeouts could be relevant.
For anyone using simple Client Server programms and getting this error, it is a problem of unclosed (or closed to early) Input or Output Streams.
Have you checked the Tomcat source code and the JVM source ? That may give you more help.
I think your general thinking is good. I would expect a ConnectException in the scenario that you couldn't connect. The above looks very like it's client-driven.
I was facing the same issue.
Commonly This kind of error occurs due to client has closed its connection and server still trying to write on that client.
So make sure that your client has its connection open until server done with its outputstream.
And one more thing, Don`t forgot to close input and output stream.
Hope this helps.
And if still facing issue than brief your problem here in details.
Had an SSLPoke.bat (SSL troubleshooting script) window script that was getting this error despite importing the correct certificates into the cacerts trustore.
C:\Java\jdk1.8.0_111\jre\lib\security>SSLPoke.bat
C:\Java\jdk1.8.0_111\jre\lib\security>"C:\jdk1.8.0_101\jre\bin\java"
`SSLPoke tfs.corp.****.com 443`
java.net.SocketException: Software caused connection abort: recv failed
`at java.net.SocketInputStream.socketRead0(Native Method)`
`at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)`
`at java.net.SocketInputStream.read(SocketInputStream.java:170)`
`at java.net.SocketInputStream.read(SocketInputStream.java:141)`
`at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)`
`at sun.security.ssl.InputRecord.read(InputRecord.java:503)`
`at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)`
`at sun.security.ssl.SSLSocketImpl.performInitialHandshake
(SSLSocketImpl.java:1375)`
`at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)`
`at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)`
`at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:138)`
`at SSLPoke.main(SSLPoke.java:28)`
I then checked some old notes about some network changes at my job. We would
need in some cases to add the JVM parameter
-Djava.net.preferIPv4Stack=true to make connections to certain machines
in our network to avoid this error.
C:\Java\jdk1.8.0_111\jre\lib\security>"C:\Java\jdk1.8.0_111\bin\java"
**-Djava.net.preferIPv4Stack=true** SSLPoke tfs.corp.****.com 443
Successfully connected
The code for SSLPoke can be downloaded from here:
https://gist.github.com/4ndrej/4547029
This error happened to me while testing my soap service with SoapUI client, basically I was trying to get a very big message (>500kb) and SoapUI closed the connection by timeout.
On SoapUI go to:
File-->Preferences--Socket Timeout(ms)
...and put a large value, such as 180000 (3 minutes), this won't be the perfect fix for your issue because the file is in fact to large, but at least you will have a response.
Closed connection in another client
In my case, the error was:
java.net.SocketException: Software caused connection abort: recv failed
It was received in eclipse while debugging a java application accessing a H2 database. The source of the error was that I had initially opened the database with SQuirreL to check manually for integrity. I did use the flag to enable multiple connections to the same DB (i.e. AUTO_SERVER=TRUE), so there was no problem connecting to the DB from java.
The error appeared when, after a while --it is a long java process-- I decided to close SQuirreL to free resources. It appears as if SQuirreL were the one "owning" the DB server instance and that it was shut down with the SQuirreL connection.
Restarting the Java application did not yield the error again.
config
Windows 7
Eclipse Kepler
SQuirreL 3.6
org.h2.Driver ver 1.4.192
In the situation explained below, client side will throw such an exception:
The server is asked to authenticate client certificate, but the client provides a certificate which Extended Key Usage doesn't support client auth, so the server doesn't accept the client's certificate, and then it closes the connection.
My server was throwing this exception in the pass 2 days and I solved it by moving the disconnecting function with:
outputStream.close();
inputStream.close();
Client.close();
To the end of the listing thread.
if it will helped anyone.
In my case, I developped the client and the server side, and I have the exception :
Cause : error marshalling arguments; nested exception is:
java.net.SocketException: Software caused connection abort: socket
write error
when classes in client and server are different. I don't download server's classes (Interfaces) on the client, I juste add same files in the project.
But the path must be exactly the same.
For example, on the server project I have java\rmi\services packages with some serviceInterface and implementations, I have to create the same package on the client project. If I change it by java/rmi/server/services for example, I get the above exception.
Same exception if the interface version is different between client and server (even with an empty row added inadvertently ... I think rmi makes a sort of hash of classes to check version ... I don't know...
If it could help ...
I was facing the same problem with wireMock while mocking the rest API calls.
Earlier I was defining the server like this:
WireMockServer wireMockServer = null;
But it should be defined like as shown below:
#Rule
public WireMockRule wireMockRule = new WireMockRule(8089);
Given this stack trace snippet
Caused by: java.net.SocketException:
Software caused connection abort:
socket write error at
java.net.SocketOutputStream.socketWrite0(Native
Method)
I tried to answer the following questions:
What code is throwing this exception? (JVM?/Tomcat?/My code?)
What causes this exception to be thrown?
Regarding #1:
Sun's JVM source doesn't contain this exact message, but I think the text Software caused connection abort: socket write error is from the native implementation of SocketOutputStream:
private native void socketWrite0(FileDescriptor fd, byte[] b, int off,
int len) throws IOException;
Regarding #2
My guess is that it is caused when the client has terminated the connection, before getting the full response (e.g. sent a request, but before getting the full response, it got closed / terminated / offline)
Questions:
Are the above assumptions correct (#1 and #2)?
Can this be diffrentiated from the situation: "could not write to the client, due to a network error on the server side"? or would that render the same error message?
And most important: Is there an official document (e.g from Sun) stating the above?
I need to have a proof that this stack trace is the socket client's "fault", and there is nothing that the server could have done to avoid it. (except catching the exception, or using a non Sun JVM SocketOutputStream, though both don't really avoid the fact the client has terminated)
This error can occur when the local network system aborts a
connection, such as when WinSock closes an established connection
after data retransmission fails (receiver never acknowledges data sent
on a datastream socket).
See this MSDN article. See also Some information about 'Software caused connection abort'.
The java.net.SocketException is thrown when there is an error creating or accessing a socket (such as TCP). This usually can be caused when the server has terminated the connection (without properly closing it), so before getting the full response. In most cases this can be caused either by the timeout issue (e.g. the response takes too much time or server is overloaded with the requests), or the client sent the SYN, but it didn't receive ACK (acknowledgment of the connection termination). For timeout issues, you can consider increasing the timeout value.
The Socket Exception usually comes with the specified detail message about the issue.
Example of detailed messages:
Software caused connection abort: recv failed.
The error indicates an attempt to send the message and the connection has been aborted by your server. If this happened while connecting to the database, this can be related to using not compatible Connector/J JDBC driver.
Possible solution: Make sure you've proper libraries/drivers in your CLASSPATH.
Software caused connection abort: connect.
This can happen when there is a problem to connect to the remote. For example due to virus-checker rejecting the remote mail requests.
Possible solution: Check Virus scan service whether it's blocking the port for the outgoing requests for connections.
Software caused connection abort: socket write error.
Possible solution: Make sure you're writing the correct length of bytes to the stream. So double check what you're sending. See this thread.
Connection reset by peer: socket write error / Connection aborted by peer: socket write error
The application did not check whether keep-alive connection had been timed out on the server side.
Possible solution: Ensure that the HttpClient is non-null before reading from the connection.E13222_01
Connection reset by peer.
The connection has been terminated by the peer (server).
Connection reset.
The connection has been either terminated by the client or closed by the server end of the connection due to request with the request.
See: What's causing my java.net.SocketException: Connection reset?
I have seen this most often when a corporate firewall on a workstation/laptop gets in the way, it kills the connection.
eg. I have a server process and a client process on the same machine. The server is listening on all interfaces (0.0.0.0) and the client attempts a connection to the public/home interface (note not the loopback interface 127.0.0.1).
If the machine is has its network disconnected (eg wifi turned off) then the connection is formed. If the machine is connected to the corporate network (directly or vpn) then the connection is formed.
However, if the machine is connected to a public wifi (or home network) then the firewall kicks in an kills the connection. In this situation connecting the client to the loopback interface works fine, just not to the home/public interface.
Hope this helps.
To prove which component fails I would monitor the TCP/IP communication using wireshark and look who is actaully closing the port, also timeouts could be relevant.
For anyone using simple Client Server programms and getting this error, it is a problem of unclosed (or closed to early) Input or Output Streams.
Have you checked the Tomcat source code and the JVM source ? That may give you more help.
I think your general thinking is good. I would expect a ConnectException in the scenario that you couldn't connect. The above looks very like it's client-driven.
I was facing the same issue.
Commonly This kind of error occurs due to client has closed its connection and server still trying to write on that client.
So make sure that your client has its connection open until server done with its outputstream.
And one more thing, Don`t forgot to close input and output stream.
Hope this helps.
And if still facing issue than brief your problem here in details.
Had an SSLPoke.bat (SSL troubleshooting script) window script that was getting this error despite importing the correct certificates into the cacerts trustore.
C:\Java\jdk1.8.0_111\jre\lib\security>SSLPoke.bat
C:\Java\jdk1.8.0_111\jre\lib\security>"C:\jdk1.8.0_101\jre\bin\java"
`SSLPoke tfs.corp.****.com 443`
java.net.SocketException: Software caused connection abort: recv failed
`at java.net.SocketInputStream.socketRead0(Native Method)`
`at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)`
`at java.net.SocketInputStream.read(SocketInputStream.java:170)`
`at java.net.SocketInputStream.read(SocketInputStream.java:141)`
`at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)`
`at sun.security.ssl.InputRecord.read(InputRecord.java:503)`
`at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)`
`at sun.security.ssl.SSLSocketImpl.performInitialHandshake
(SSLSocketImpl.java:1375)`
`at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)`
`at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)`
`at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:138)`
`at SSLPoke.main(SSLPoke.java:28)`
I then checked some old notes about some network changes at my job. We would
need in some cases to add the JVM parameter
-Djava.net.preferIPv4Stack=true to make connections to certain machines
in our network to avoid this error.
C:\Java\jdk1.8.0_111\jre\lib\security>"C:\Java\jdk1.8.0_111\bin\java"
**-Djava.net.preferIPv4Stack=true** SSLPoke tfs.corp.****.com 443
Successfully connected
The code for SSLPoke can be downloaded from here:
https://gist.github.com/4ndrej/4547029
This error happened to me while testing my soap service with SoapUI client, basically I was trying to get a very big message (>500kb) and SoapUI closed the connection by timeout.
On SoapUI go to:
File-->Preferences--Socket Timeout(ms)
...and put a large value, such as 180000 (3 minutes), this won't be the perfect fix for your issue because the file is in fact to large, but at least you will have a response.
Closed connection in another client
In my case, the error was:
java.net.SocketException: Software caused connection abort: recv failed
It was received in eclipse while debugging a java application accessing a H2 database. The source of the error was that I had initially opened the database with SQuirreL to check manually for integrity. I did use the flag to enable multiple connections to the same DB (i.e. AUTO_SERVER=TRUE), so there was no problem connecting to the DB from java.
The error appeared when, after a while --it is a long java process-- I decided to close SQuirreL to free resources. It appears as if SQuirreL were the one "owning" the DB server instance and that it was shut down with the SQuirreL connection.
Restarting the Java application did not yield the error again.
config
Windows 7
Eclipse Kepler
SQuirreL 3.6
org.h2.Driver ver 1.4.192
In the situation explained below, client side will throw such an exception:
The server is asked to authenticate client certificate, but the client provides a certificate which Extended Key Usage doesn't support client auth, so the server doesn't accept the client's certificate, and then it closes the connection.
My server was throwing this exception in the pass 2 days and I solved it by moving the disconnecting function with:
outputStream.close();
inputStream.close();
Client.close();
To the end of the listing thread.
if it will helped anyone.
In my case, I developped the client and the server side, and I have the exception :
Cause : error marshalling arguments; nested exception is:
java.net.SocketException: Software caused connection abort: socket
write error
when classes in client and server are different. I don't download server's classes (Interfaces) on the client, I juste add same files in the project.
But the path must be exactly the same.
For example, on the server project I have java\rmi\services packages with some serviceInterface and implementations, I have to create the same package on the client project. If I change it by java/rmi/server/services for example, I get the above exception.
Same exception if the interface version is different between client and server (even with an empty row added inadvertently ... I think rmi makes a sort of hash of classes to check version ... I don't know...
If it could help ...
I was facing the same problem with wireMock while mocking the rest API calls.
Earlier I was defining the server like this:
WireMockServer wireMockServer = null;
But it should be defined like as shown below:
#Rule
public WireMockRule wireMockRule = new WireMockRule(8089);
I want to run a cyc (opencyc) Java class, but when I run this class I got an error mentioning that the network is unreachable. I am working on NetBeans 7.0 and opencyc 2.0 win 32 version. I am using localhost (127.0.0.1) to test. The error is below.
Network is unreachable: connect
java.net.SocketException: Network is unreachable: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at java.net.Socket.<init>(Socket.java:366)
at java.net.Socket.<init>(Socket.java:180)
at org.opencyc.api.CycConnection.initializeApiConnections(CycConnection.java:223)
at org.opencyc.api.CycConnection.<init>(CycConnection.java:199)
at org.opencyc.api.CycAccess.<init>(CycAccess.java:121)
at org.opencyc.api.CycAccess.<init>(CycAccess.java:61)
at org.opencyc.api.ApiDemo.<init>(ApiDemo.java:50)
at org.opencyc.api.ApiDemo.main(ApiDemo.java:147)
Exception in thread "main" java.lang.NullPointerException
at org.opencyc.api.ApiDemo.<init>(ApiDemo.java:56)
at org.opencyc.api.ApiDemo.main(ApiDemo.java:147)
Interruption while waiting Cyc connection establishment, closing sockets
Exception in thread "Thread-1" java.lang.NullPointerException
at org.opencyc.api.CycConnection$TaskProcessorBinaryResponseHandler.access$002(CycConnection.java:988)
at org.opencyc.api.CycConnection.close(CycConnection.java:268)
at org.opencyc.api.CycConnection$ConnectionTimer.run(CycConnection.java:1351)
Why is this happening? How can I connect to the cyc?
The problem is that your network settings do not allow you to connect to this machine. This is not a Java problem as such.
try
ping {hostname}
until this works, there is nothing you can do in java to work around this.
If you have the same version I have, it's trying to establish a connection to a Cyc image with the hostname and port defined by
CycConnection.DEFAULT_HOSTNAME
CycConnection.DEFAULT_BASE_PORT
You can find the proper values to use by evaluating the following in your Cyc image:
(get-machine-name)
*BASE-TCP-PORT*
Hope this works for you.
A dev on my team encountered this today, only their machine is seeing this issue out of many on the team.
Through much debugging we eventually discovered that localhost would not resolve for some reason. It would work in some JVM contexts and fail in others so it was completely inconsistent.
Replacing localhost with explicitly 127.0.0.1 fixed the issue. My hunch is that something was different with a local network and IPv6 perhaps, but we still don't really understand why this is the fix.
I have a few web services on a Weblogic 10 server. Each of these is part of a larger system. Running locally and on our qa environment the system works flawless, replies fast, and as expected. Everything looks to be okay.
Before going into production we're going to stress test the system, thus see how much load we can have before reply time becomes to large. When testing the web services (e.g. using front end or SOAPUI) we hit a certain load (e.g. to many replies per sec or something like that, I'm not sure what exactly triggers the system to fail) we get the error listed below. I haven't got the slightest clue as to why. Seconds later the system replies flawless again, so I'm guessing that it has something to do with the number of requests...
Any ideas or hints is much appreciated! I'm lost here, so please - anything will help.
We're running: Weblogic 10.3.2, Spring 2.5.6 (for architectural reasons we cannot upgrade), Spring-WS 1.5.9 (for architectural reasons we cannot upgrade) and Stripes 1.5.4
<11-11-2011 08:43:58 CET> <Error> <HTTP> <BEA-101017> <[ServletContext#11242741[app:salesoverview-ws-web module:salesoverview-ws-web path:/salesoverview-ws-web spec-version:2.5], request: weblogic.servlet.internal.ServletRequestImpl#1fbbfc5[POST /salesoverview-ws-web/services HTTP/1.1 Accept-Encoding: gzip,deflate Content-Type: text/xml;charset=UTF-8 SOAPAction: "" User-Agent: Jakarta Commons-HttpClient/3.1 Content-Length: 425]] Root cause of ServletException.
org.springframework.ws.soap.saaj.SaajSoapMessageException: Could not write message to OutputStream: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error; nested exception is javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:169)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:97)
at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:57)
at org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:230)
Truncated. see log file for complete stacktrace
Caused By: javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
at weblogic.xml.saaj.SOAPMessageImpl.SOAPPart_writeTo(SOAPMessageImpl.java:1011)
at weblogic.xml.saaj.SOAPMessageImpl.writeTo(SOAPMessageImpl.java:816)
at org.springframework.ws.soap.saaj.Saaj13Implementation.writeTo(Saaj13Implementation.java:292)
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:165)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
Truncated. see log file for complete stacktrace
>
By digging BEA-101017 I found a little info about the from the Weblogic error dok - although this doesn't help me:
Error: [context] Root cause of ServletException.
Description: [context] Root cause of ServletException, which the Web
application container caught while servicing the request.
Cause: The Web application container caught an unexpected exception.
Action: Check the exception for the exact error message.
Assuming that the web service from your example doesn't access other web services (and therefore the above trace corresponds to your web service sending the response):
It seems that your web service, via SAAJ, is trying to write to a disconnected (or otherwise unavailable) socket. An usual cause for this is that the client has disconnected while waiting for the server reply.
I'd suggest to:
Check if your client was waiting for too long before receiving the response, that could have caused it to disconnect.
Check if the operating system might be having issues allocating sockets. Use 'netstat' or other monitoring tool (like TCPView on Windows) to check how many sockets are open (most operating system impose limits on the number of sockets allowed per user or globally).
Ensure there are absolutely no network errors during your tests (shouldn't be the case if you are testing on localhost, but otherwise you need to ensure your network devices (routers, switches, other computers) are not dropping connections or packets. Perhaps this is happening when traffic load is high.
Make sure you have no threading conflicts that could cause your web service to use or close other requests' sockets (this would be a rare situation especially if you are using Spring).
Check this thread Official reasons for "Software caused connection abort: socket write error" and other possible causes of "Software caused connection abort" (note that the issue could be specific to your application server and operating system).
Hope that helps.
After debugging a lot I found out that the problem happened due to DB2 issues - we hit a corner of our database, which triggered an internal stack overflow, which then probagated to the Dao and onwards to the SOAP-part (only making it harder to detect due to Spring JDBC templates in the Dao).
A long story short and the issue was an uncaught exception, which by Spring-WS resulted in a "SaajSoapMessageException". The hint came from "Software caused connection abort: socket write error", but happened on the WS side (not client nor the communication between client/server).
Hint: Surround your database with try/catch and catch Exception thus being able to find the exact exception thrown. In my case it threw a DB2 exception ("SQLCode -1218") and this is normally used when you run out of resources (e.g. data source connections). I my case it was the SQL which DB2 didn't like - and really didn't like under load. I can't explain it, but it has to do with DB2s own internal resources - gah, go figure! :)
Thank you jjmontes, for hints and pointers, but it was not the problem in this case.
I start up my application which uses a Jetty server, using port 9000.
I then shut down my application with Ctrl-C
I check with "netstat -a" and see that the port 9000 is no longer being used.
I restart my application and get:
[ERROR,9/19 15:31:08] java.net.BindException: Only one usage of each socket address (protocol/network address/port) is normally permitted
[TRACE,9/19 15:31:08] java.net.BindException: Only one usage of each socket address (protocol/network address/port) is normally permitted
[TRACE,9/19 15:31:08] at java.net.PlainSocketImpl.convertSocketExceptionToIOException(PlainSocketImpl.java:75)
[TRACE,9/19 15:31:08] at sun.nio.ch.Net.bind(Net.java:101)
[TRACE,9/19 15:31:08] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
[TRACE,9/19 15:31:08] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
[TRACE,9/19 15:31:08] at org.mortbay.jetty.nio.BlockingChannelConnector.open(BlockingChannelConnector.java:73)
[TRACE,9/19 15:31:08] at org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:285)
[TRACE,9/19 15:31:08] at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
[TRACE,9/19 15:31:08] at org.mortbay.jetty.Server.doStart(Server.java:233)
[TRACE,9/19 15:31:08] at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
[TRACE,9/19 15:31:08] at ...
Is this a Java bug? Can I avoid it somehow before starting the Jetty server?
Edit #1 Here is our code for creating our BlockingChannelConnector, note the "setReuseAddress(true)":
connector.setReuseAddress( true );
connector.setPort( port );
connector.setStatsOn( true );
connector.setMaxIdleTime( 30000 );
connector.setLowResourceMaxIdleTime( 30000 );
connector.setAcceptQueueSize( maxRequests );
connector.setName( "Blocking-IO Connector, bound to host " + connector.getHost() );
Could it have something to do with the idle time?
Edit #2 Next piece of the puzzle that may or may not help: when running the application in Debug Mode (Eclipse) the server starts up without a problem!!! But the problem described above occurs reproducibly when running the application in Run Mode or as a built jar file. Whiskey Tango Foxtrot?
Edit #3 (4 days later) - still have the issue. Any thoughts?
During your first invocation of your program, did it accept at least one incoming connection? If so then what you are most likely seeing is the socket linger in effect.
For the best explanation dig up a copy of TCP/IP Illustrated by Stevens
(source: kohala.com)
But, as I understand it, because the application did not properly close the connection (that is BOTH client and server sent their FIN/ACK sequences) the socket you were listening on cannot be reused until the connection is considered dead, the so called 2MSL timeout. The value of 1 MSL can vary by operating system, but its usually a least a minute, and usually more like 5.
The best advice I have heard to avoid this condition (apart from always closing all sockets properly on exit) is to set the SO_LINGER tcp option to 0 on your server socket during the listen() phase. As freespace pointed out, in java this is the setReuseAddress(true) method.
You might want call setReuseAddress(true) before calling bind() on your socket object. This is caused by a TCP connection persisting even after the socket is closed.
I'm not sure about Jetty, but I have noticed that sometimes Tomcat will not shut down cleanly on some of our Linux servers. In cases like that, Tomcat will restart but not be able to use the port in question because the previous instance is still bound to it. In such cases, we have to find the rogue process and explicitly kill -9 it before we restart Tomcat. I'm not sure if this is a java bug or specific to Tomcat or the JVM we're using.
I must say I also thought that it's the usual issue solved by setReuseAddress(true). However, the error message in that case is usually something along the lines that the JVM can't bind to the port. I've never seen the posted error message before. Googling for it seems to suggest that another process is listening on one or more (but not all) network interfaces, and you request your process to bind to all interfaces, whereas it can bind to some (those that the other process isn't listening to) but not all of them. Just guessing here though...