Develop Reference

PDF File Transfer from server to client: "java.io.IOException: An existing connection was forcibly closed by the remote host" [duplicate]

I am working with a commercial application which is throwing a SocketException with the message,
An existing connection was forcibly closed by the remote host
This happens with a socket connection between client and server. The connection is alive and well, and heaps of data is being transferred, but it then becomes disconnected out of nowhere.
Has anybody seen this before? What could the causes be? I can kind of guess a few causes, but also is there any way to add more into this code to work out what the cause could be?
Any comments / ideas are welcome.
... The latest ...
I have some logging from some .NET tracing,
System.Net.Sockets Verbose: 0 : [8188] Socket#30180123::Send() DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Error: 0 : [8188] Exception in the Socket#30180123::Send - An existing connection was forcibly closed by the remote host DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Verbose: 0 : [8188] Exiting Socket#30180123::Send() -> 0#0
Based on other parts of the logging I have seen the fact that it says 0#0 means a packet of 0 bytes length is being sent. But what does that really mean?
One of two possibilities is occurring, and I am not sure which,
The connection is being closed, but data is then being written to the socket, thus creating the exception above. The 0#0 simply means that nothing was sent because the socket was already closed.
The connection is still open, and a packet of zero bytes is being sent (i.e. the code has a bug) and the 0#0 means that a packet of zero bytes is trying to be sent.
What do you reckon? It might be inconclusive I guess, but perhaps someone else has seen this kind of thing?

This generally means that the remote side closed the connection (usually by sending a TCP/IP RST packet). If you're working with a third-party application, the likely causes are:
You are sending malformed data to the application (which could include sending an HTTPS request to an HTTP server)
The network link between the client and server is going down for some reason
You have triggered a bug in the third-party application that caused it to crash
The third-party application has exhausted system resources
It's likely that the first case is what's happening.
You can fire up Wireshark to see exactly what is happening on the wire to narrow down the problem.
Without more specific information, it's unlikely that anyone here can really help you much.

Using TLS 1.2 solved this error.
You can force your application using TLS 1.2 with this (make sure to execute it before calling your service):
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Another solution :
Enable strong cryptography in your local machine or server in order to use TLS1.2 because by default it is disabled so only TLS1.0 is used.
To enable strong cryptography , execute these commande in PowerShell with admin privileges :
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
You need to reboot your computer for these changes to take effect.

This is not a bug in your code. It is coming from .Net's Socket implementation. If you use the overloaded implementation of EndReceive as below you will not get this exception.
SocketError errorCode;
int nBytesRec = socket.EndReceive(ar, out errorCode);
if (errorCode != SocketError.Success)
{
nBytesRec = 0;
}

Had the same bug. Actually worked in case the traffic was sent using some proxy (fiddler in my case). Updated .NET framework from 4.5.2 to >=4.6 and now everything works fine. The actual request was:
new WebClient().DownloadData("URL");
The exception was:
SocketException: An existing connection was forcibly closed by the
remote host

Simple solution for this common annoying issue:
Just go to your ".context.cs" file (located under ".context.tt" which located under your "*.edmx" file).
Then, add this line to your constructor:
public DBEntities()
: base("name=DBEntities")
{
this.Configuration.ProxyCreationEnabled = false; // ADD THIS LINE!
}

I've got this exception because of circular reference in entity.In entity that look like
public class Catalog
{
public int Id { get; set; }
public int ParentId { get; set; }
public Catalog Parent { get; set; }
public ICollection<Catalog> ChildCatalogs { get; set; }
}
I added [IgnoreDataMemberAttribute] to the Parent property. And that solved the problem.

If Running In A .Net 4.5.2 Service
For me the issue was compounded because the call was running in a .Net 4.5.2 service. I followed #willmaz suggestion but got a new error.
In running the service with logging turned on, I viewed the handshaking with the target site would initiate ok (and send the bearer token) but on the following step to process the Post call, it would seem to drop the auth token and the site would reply with Unauthorized.
Solution
It turned out that the service pool credentials did not have rights to change TLS (?) and when I put in my local admin account into the pool, it all worked.

I had the same issue and managed to resolve it eventually. In my case, the port that the client sends the request to did not have a SSL cert binding to it. So I fixed the issue by binding a SSL cert to the port on the server side. Once that was done, this exception went away.

For anyone getting this exception while reading data from the stream, this may help. I was getting this exception when reading the HttpResponseMessage in a loop like this:
using (var remoteStream = await response.Content.ReadAsStreamAsync())
using (var content = File.Create(DownloadPath))
{
var buffer = new byte[1024];
int read;
while ((read = await remoteStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
await content.WriteAsync(buffer, 0, read);
await content.FlushAsync();
}
}
After some time I found out the culprit was the buffer size, which was too small and didn't play well with my weak Azure instance. What helped was to change the code to:
using (Stream remoteStream = await response.Content.ReadAsStreamAsync())
using (FileStream content = File.Create(DownloadPath))
{
await remoteStream.CopyToAsync(content);
}
CopyTo() method has a default buffer size of 81920. The bigger buffer sped up the process and the errors stopped immediately, most likely because the overall download speeds increased. But why would download speed matter in preventing this error?
It is possible that you get disconnected from the server because the download speeds drop below minimum threshold the server is configured to allow. For example, in case the application you are downloading the file from is hosted on IIS, it can be a problem with http.sys configuration:
"Http.sys is the http protocol stack that IIS uses to perform http communication with clients. It has a timer called MinBytesPerSecond that is responsible for killing a connection if its transfer rate drops below some kb/sec threshold. By default, that threshold is set to 240 kb/sec."
The issue is described in this old blogpost from TFS development team and concerns IIS specifically, but may point you in a right direction. It also mentions an old bug related to this http.sys attribute: link
In case you are using Azure app services and increasing the buffer size does not eliminate the problem, try to scale up your machine as well. You will be allocated more resources including connection bandwidth.

I got the same issue while using .NET Framework 4.5. However, when I update the .NET version to 4.7.2 connection issue was resolved. Maybe this is due to SecurityProtocol support issue.

For me, it was because the app server I was trying to send email from was not added to our company's SMTP server's allowed list.
I just had to put in SMTP access request for that app server.
This is how it was added by the infrastructure team (I don't know how to do these steps myself but this is what they said they did):
1. Log into active L.B.
2. Select: Local Traffic > iRules > Data Group List
3. Select the appropriate Data Group
4. Enter the app server's IP address
5. Select: Add
6. Select: Update
7. Sync config changes

Yet another possibility for this error to occur is if you tried to connect to a third-party server with invalid credentials too many times and a system like Fail2ban is blocking your IP address.

I was trying to connect to the MQTT broker using the GO client,
broker address was given as address + port, or tcp://address:port
Example: ❌
mqtt://test.mosquitto.org
which indicates that you wish to establish an unencrypted connection.
To request MQTT over TLS use one of ssl, tls, mqtts, mqtt+ssl or tcps.
Example: ✅
mqtts://test.mosquitto.org

In my case, enable the IIS server & then restart and check again.

We are using a SpringBoot service. Our restTemplate code looks like below:
#Bean
public RestTemplate restTemplate(final RestTemplateBuilder builder) {
return builder.requestFactory(() -> {
final ConnectionPool okHttpConnectionPool =
new ConnectionPool(50, 30, TimeUnit.SECONDS);
final OkHttpClient okHttpClient =
new OkHttpClient.Builder().connectionPool(okHttpConnectionPool)
// .connectTimeout(30, TimeUnit.SECONDS)
.retryOnConnectionFailure(false).build();
return new OkHttp3ClientHttpRequestFactory(okHttpClient);
}).build();
}
All our call were failing after the ReadTimeout set for the restTemplate. We increased the time, and our issue was resolved.

This error occurred in my application with the CIP-protocol whenever I didn't Send or received data in less than 10s.
This was caused by the use of the forward open method. You can avoid this by working with an other method, or to install an update rate of less the 10s that maintain your forward-open-connection.

ZeroMQ blocked in a context.term() call. Why? How to prevent?

I have a java program that using ZeroMQ.
But I found the program blocked in context.term(); if receiving a message( recvMsg() ) time out!
ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket socket = context.socket(ZMQ.REQ);
socket.connect(mAddress);
ZMsg ZM = new ZMsg();
ZM.add(qString);
ZM.send(socket, true);
socket.setReceiveTimeOut(mTimeout);
ZMsg receivedZM = ZMsg.recvMsg(socket);
if(receivedZM != null) {
System.out.println(receivedZM.getFirst().toString());
}
socket.close();
context.term();
What is the reason cause it to blocked?
And how to solve this problem?

ZeroMQ is a system using many tricks behind the Context()-factory
I always advocate to automatically set .setsockopt( ZMQ_LINGER, 0 ) right upon a Socket-instantiation, right due to these types of behaviour, that otherwise remain outside of your local-code domain of control. A hanging Context-instance IO-thread(s) ( be it after a programmed .term() having been issued in spite of the not yet successful .close() of all socket-instances, instantiated under this Context-instance the .term() is to dismantle and release back all system resources from, or an unhandled exception case, when things just went straight wreck havoc ) is one of such never-more-s.
Feel free to follow schoolbook and online hacks/snippet examples, but a serious distributed system designer ought take all reasonable steps and measures so as to prevent her/his system code to fall into any deadlock-state ( the less into an un-salvageable one ).
What is the reason?
As documentation states - it is a designed-in feature of ZeroMQ:
attempting to terminate the socket's context with zmq_ctx_term() shall block until all pending messages have been sent to a peer.
Any case, where a .send()-dispatched ( just dispatched -- by no means meaning that it has already been sent-to-wire ) message is still inside the local-queue for any of the recognised ( and potentially disconnected or busy or ... ) peer-nodes, the just-default configured .term() cannot proceed and will block.
What is the solution:
Newer API versions started to say, a default LINGER value to stop being -1 == INFINITY, but as you never know, which version will your code interface with, an explicit ( manual ) call to a .setsockopt( ZMQ_LINGER, 0 ) method is a self-disciplining step and increases your team awareness on how to build reliable distributed-systems' code.
Using the try: / except: / finally: syntax-handlers is needless to be raised here. You simply always have to design with failures & collisions in mind, haven't you?

According to the API, http://api.zeromq.org/4-2:zmq-term, it will block when there's still messages to transmit. This suggests that you other machine or process, the one that will open the REP socket; isn't running.

Shutting down a netty TcpClient channel in Reactor 3.0.4

I just upgraded projectreactor.io from reactor OLD: [core: 3.0.1.RELEASE, netty: 0.5.2.RELEASE] to reactor NEW [core: 3.0.4.RELEASE, netty: 0.6.0.RELEASE].
I open a TcpClient connection and want to close it later.
In the OLD version I used
tcpClient.shutdown();
to disconnect my client from the server.
Is there an equivalent call in the NEW version? I could not find one!
I tried the following on both the NettyInbound and NettyOutbound that I get while creating my TcpClient with tcpClient.newHandler(...)
.context().dispose()
.context().channel().disconnect()
.context().channel().close()
TcpResources.reset()
None of them seem to do the job correctly.
I noticed that the respective .context().onClose(...)-callback is being called.
But after some additional waiting the server-side checks the connections.
Server-side is plain NIO2 not reactor/netty and while the client was upgraded, the server-side remained unchanged.
With the OLD client I got .isOpen() == false for every channel on server-side.
With the NEW client I get .isOpen() == true for every channel on server-side. Most of the time I can even write to the channel. And some channels switch to .isOpen() == false after writing few bytes.

This deserves an issue I think especially if channel().close() and reset() didn't work.
Otherwise it might be due to the default pooling and TcpClient.create(opts -> opts.disablePool()) might help, let us know and if you have a chance to post an issue on http://github.com/reactor/reactor-netty you would be a hero :D

Linked to this open issue https://github.com/reactor/reactor-netty/issues/15. We will review the dispose API.

The following code somehow destroys the channel but not completely.
ChannelFuture f = nettyContext.channel().close();
f.sync();
nettyContext.dispose();
The problem is that the channel still seems to be open on server-side.
For a NIO2-based server, the server should not test if the channel isOpen(). It's always true.
As a dirty workaround, the server must write to the channel twice. If it catches an ExecutionException on the second write then the channel was already closed by the Netty-TcpClient.
try {
channel.write(ByteBuffer.wrap("hello".getBytes())).get();
// No exception on first write but on second write.
channel.write(ByteBuffer.wrap("bye".getBytes())).get();
} catch (ExecutionException e) {
LOG.log(Level.SEVERE, "ExecutionException on writing from server into channel", e);
}

With reactor-core: 3.1.0.M3 and reactor-netty: 0.7.0.M1 the client-API was improved and works more reliable.
After blockingNettyContext.shutdown() I still need the following workaround on server-side to make sure the channel was closed:
I need to write into the channel and close it on exception:
// channel.isOpen == true
try {
channel.write(ByteBuffer.wrap("__test__".getBytes())).get();
} catch (ExecutionException e) {
channel.close();
}
// channel.isOpen == false

Client thread hang emulation blocks server from accepting any I/O for the time client is set to wait

As the topic suggests I have a server and some clients.
The server accepts I/O connections concurrently (no queueing in socket connections) but I have this troubling issue and I do not know how to bypass it!
If I force a client to throw an I/O Exception the server detects it and terminates the client thread correctly (verified from Task Manager (Windows) and System Monitor (Ubuntu) ). But If I emulate an I/O that is "hanging" like i.e. Thread.sleep(60*1000);or
private static Object lock = new Object();
synchronized(lock) {
while (true) {
try {
lock.wait();
} catch (InterruptedException e) {
/* Foo */
}
}
}
then all subsequent I/O operations (connection & data transfer) seem to block or wait until the "hanging" client is terminated. The applications makes use of the ExecutorService so if the "hanging" client does not complete the operations in the suggested time limit then the task will time out and the client is forced to exit. The subsequent "blocked" I/Os will resume but I wonder why the server doesn't accept any I/O connections or performs any I/O operations when a client "hangs"?
NOTE:The client threading takes place in the server main like this:
while (true) {
accept client connection;
submit client task;
||
\ /
\/
// ExecutorService here in the form
// spService.submit(new Callable<Tuple<String[], BigDecimal[]>>() {
// ... code ... }}).get(taskTimeout, taskTimeUnit);
check task result & perform cleanup if result is null;
otherwise continue;
}

The Problem :
This may very well indicate that your server ACCEPTS client connections concurrently, however, it only handles these connections synchronously. That means that even if a million clients connect, successfully, at any given time, if anyone of them takes a long time (or hangs), it will hold up the others.
The TEST:
To verify this : I would toggle the amount of time a client takes to connect by adding Thread.sleep statments(1000) in your clients.
Expected result :
I believe you will see that even adding a single Thread.sleep(1000) statement in your client delays all other connecting clients by 1000.

I think I have found the source of my problems!
I do use one thread-per-client model but I run my tests locally i.e. in the same machine which means all of them have the same IP! So each client is assigned the same IP with the server! I guess that this leaves server and clients to differ only in port number but since each client is mapped to a different localport for each server connection then the server shouldn't block. I have confirmed that each client and server use different I/Os (compared references) and I wrap their sockets using <Input/Output>Streams to BufferedReaders & PrintWriters but still when a client hangs all other clients hang too (so maybe the I/O channels are indeed the same???)!I will test this on another machine and check the results back with you! :)
EDIT: Confirmed the erratic behaviour. It seems that even with remote clients if one hangs the other clients seem to hang too! :/
Don't know but I am determined to fix this. It's just that it's pretty weird since I am pretty sure I use one thread-per-client (I/Os differ, client sockets differ, IPs seem to be not a problem, I even map each client in the server to a localport of my choice ...)
May be I'll switch to NIO if I don't find a solution soon enough.
SOLUTION: Solved the problem! It seemed that the ExecutorService had to be run in a seperate thread otherwise if an I/O in a client blocked, all I/Os would block! That's strange given the fact that I've tried both an Executors.newFixedThreadPool(<nThreads>); and Executors.newCachedThreadPool(); and the client actions (aka I/Os) should take place in a new Thread for each client.
In any case, I used a method and wrapped the calls so each client instace would use a final ExecutorService baseWorker = Executors.newSingleThreadExecutor(); and created a new Thread explicitly each time using <Thread instance>.start(); so each thread would run in the background :)

Issues receiving in RXTX

I've been using RXTX for about a year now, without too many problems. I just started a new program to interact with a new piece of hardware, so I reused the connect() method I've used on my other projects, but I have a weird problem I've never seen before.
The Problem
The device works fine, because when I connect with HyperTerminal, I send things and receive what I expect, and Serial Port Monitor(SPM) reflects this.
However, when I run the simple HyperTerminal-clone I wrote to diagnose the problem I'm having with my main app, bytes are sent, according to SPM, but nothing is received, and my SerialPortEventListener never fires. Even when I check for available data in the main loop, reader.ready() returns false. If I ignore this check, then I get an exception, details below.
Relevant section of connect() method
// Configure and open port
port = (SerialPort) CommPortIdentifier.getPortIdentifier(name)
.open(owner,1000)
port.setSerialPortParams(baud, databits, stopbits, parity);
port.setFlowControlMode(fc_mode);
final BufferedReader br = new BufferedReader(
new InputStreamReader(
port.getInputStream(),
"US-ASCII"));
// Add listener to print received characters to screen
port.addEventListener(new SerialPortEventListener(){
public void serialEvent(SerialPortEvent ev) {
try {
System.out.println("Received: "+br.readLine());
} catch (IOException e) { e.printStackTrace(); }
}
});
port.notifyOnDataAvailable();
Exception
java.io.IOException: Underlying input stream returned zero bytes
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:268)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read(BufferedReader.java:157)
at <my code>
The big question (again)
I think I've eliminated all possible hardware problems, so what could be wrong with my code, or the RXTX library?
Edit: something interesting
When I open HyperTerminal after sending a bunch of commands from java that should have gotten responses, all of the responses appear immediately, as if they had been put in the buffer somewhere, but unavailable.
Edit 2: Tried something new, same results
I ran the code example found here, with the same results. No data came in, but when I switched to a new program, it came all at once.
Edit 3
The hardware is fine, and even a different computer has the same problem. I am not using any sort of USB adapter.
I've started using PortMon, too, and it's giving me some interesting results. HyperTerminal and RXTX are not using the same settings, and RXTX always polls the port, unlike HyperTerminal, but I still can't see what settings would affect this. As soon as I can isolate the configuration from the constant polling, I'll post my PortMon logs.
Edit 4
Is it possible that some sort of Windows update in the last 3 months could have caused this? It has screwed up one of my MATLAB mex-based programs once.
Edit 5
I've also noticed some things that are different between HyperTerminal, RXTX, and a separate program I found that communicates with the device (but doesn't do what I want, which is why I'm rolling my own program)
HyperTerminal - set to no flow control, but Serial Port Monitor's RTS and DTR indicators are green
Other program - not sure what settings it thinks it's using, but only SPM's RTS indicator is green
RXTX - no matter what flow control I set, only SPM's CTS and DTR indicators are on.
From Serial Port Monitor's help files (paraphrased):
the indicators display the state of the serial control lines
RTS - Request To Send
CTS - Clear To Send
DTR - Data Terminal Ready

OK, sorry it's taken me so long to come back to this question. Here's how I got things working.
Note: This method will NOT work for everyone, please read below before copy/pasting into your own code
public void connect(CommPortIdentifier portId) throws Failure {
if (portId == null)
throw new Failure("No port set");
try { port = (SerialPort) portId.open(getClass().getName(), 10000); }
catch (PortInUseException e) {
throw new Failure("Port in use by " + e.currentOwner,e); }
try {
port.setSerialPortParams(9600, SerialPort.DATABITS_8,
SerialPort.STOPBITS_1, SerialPort.PARITY_NONE);
port.setFlowControlMode(SerialPort.FLOWCONTROL_RTSCTS_IN
| SerialPort.FLOWCONTROL_RTSCTS_OUT);
} catch (UnsupportedCommOperationException e) { throw new Failure(e); }
port.setRTS(true);
// More setup
}
So, in my case, the problem was that my particular device requires RTS flow control. Other devices may require different things (CTS, XON/XOFF), so check that device's manual. By default, RXTX disables all flow control mechanisms (unlike Hypertrm or other programs). Enabling each one is a two-step process.
Once you have a SerialPort object, call the setFlowControlMode() method, and bitwise-OR ('|') the necessary SerialPort.FLOWCONTROL_ constants
Set the appropriate flow control to true or false (like I did with port.setRTS(true))
For the others with similar problems, if this doesn't work, I suggest
Using a serial port monitoring program like Serial Port Monitor and/or PortMon (both Windows) to see what is actually going on.
Emailing the RXTX developers at rxtx#qbang.org (they are very helpful)

There is a simpler solution to this problem. This is what I did:
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String line;
while (keepRunning) {
try {
while ((br.ready()) && (line = br.readLine()) != null) {
....
}
If you check that the buffer "is ready" before you read it there should be no problem.

Ok, I do realize this thread is extremely old, but none of these solutions worked for me. I had the same problem and I tried everything to fix it, to no avail. Then I did some research on what causes the problem, and, when not dealing with Serial Communication, it happens at the end of a file. So, I figured I needed to add an ending to whatever is being received by the Java Application, specifically, a line return (\n). And sure enough, it fixed the problem for me! Hopefully this helps someone new, as I'm not expecting this to help anyone already on this thread...

(might be too simple, but might as well start somewhere...)
Is the port in use? Rather than:
port = (SerialPort) CommPortIdentifier.getPortIdentifier(name)
.open(owner,1000)
what about:
CommPortIdentifier portIdentifier;
try {
portIdentifier = CommPortIdentifier.getPortIdentifier(name);
} catch (NoSuchPortException nspe) {
// handle?
}
if (portIdentifier.isCurrentlyOwned()) {
// handle?
}
port = portIdentifier.open(owner, 1000);
if (!(port instanceof SerialPort)) {
// handle?
}
Are you swallowing any exceptions?

I tried RXTX a few months ago and ran into similar problems. I suggest two things:
Create a virtual comport using com0com. Enable trace logging. Compare the logs for when you use Hyperterminal versus when you run your own program. The difference will highlight what you are doing wrong.
In my humble opinion, RXTX's design is flawed and its implementation is quite buggy (take a look at its source-code, what a mess!). I've published an alternative library at http://kenai.com/projects/jperipheral with the following caveats: It's Windows-only and there are no pre-built binaries. Both of these will change in the near future. If you are interested in trying it out send me an email using http://desktopbeautifier.com/Main/contactus and I'll send you a pre-built version.

If anyone is still getting java.io.IOException: Underlying input stream returned zero bytes after you've read your characters using br.readline() for RXTX (even when you are checking first to see if br.readline() == null), just do this simple fix with a try/catch:
String line;
while (true){
try{
line = br.readLine();
}catch(IOException e){
System.out.println("No more characters received");
break;
}
//Print the line read
if (line.length() != 0)
System.out.println(line);
}
I've done some searching and it appears that this is the best/easiest way to get around this problem.
EDIT : I take that back. I tried this and still ended up having some problems. I'd recommend working with the raw InputStream directly, and implementing your own read/readLine method using InputStream.read(). That worked for me.