I have a problem. I am using WSO2 Micro Integration 4.1. I am sending request after "STATE_DESCRIPTION = Socket Timeout occurred after accepting the request headers and the request body, INTERNAL_STATE = REQUEST_DONE, DIRECTION = REQUEST, CAUSE_OF_ERROR = Connection between the client and the EI timeouts, HTTP_URL = /api/v1/test, HTTP_METHOD = POST, SOCKET_TIMEOUT = 180000, CLIENT_ADDRESS = /127.0.0.1:52694" is return my console and Error: socket hang up is return from postman. I tried http.connection.disable.keepalive=true in passthru-http.properties. I need help.
I tried http.connection.disable.keepalive=true in passthru-http.properties
As the description suggests this happened due to connection got time out. If you are having slow performing backend you can increase the timeout so that the server will wait until the response.
As an example following is the config to increase the synapse global timeout.
[synapse_properties]
'synapse.global_timeout_interval'=3000
Please note that you need to add the configurations on the deployment.toml file not in the passthru-http.properties file.
In that case if you want to make suggested change on the keep-alive settings, Do it as follows on the deployment.toml file.
[transport.http]
socket_timeout = 180000 # timeout in milliseconds
disable_connection_keepalive = true
connection_timeout = 90000 # in milliseconds
Apart from Check whether the 52694 port is accessible too as the server is localhost means that it is the same server as the MI is running.
Related
I have a certain piece of code that integrates with a third party using HTTP connection, which handles socket timeout and connection timeout differently.
I have been trying to simulate and test all the scenarios which could arise from the third party. was able to test connection timeout by connecting to a port which is blocked by the servers firewall e.g. port 81.
However I'm unable to simulate a socket timeout. If my understanding is not wrong socket timeout is associated with continuous packet flow, and the connection dropping in between. Is there a way I can simulate this?
So we are talking about to kinds of timeouts here, one is to connect to the server (connect timeout), the other timeout will happen when no data is send or received via the socket for a while (idle timeout).
Node sockets have a socket timeout, that can be used to synthesize both the connect and the idle timeout. This can be done by setting the socket timeout to the connect timeout and then when connected, setting it to the idle timeout.
example:
const request = http.request(url, {
timeout: connectTimeout,
});
request.setTimeout(idleTimeout);
This works because the timeout in the options is set immediately when creating the socket, the setTimeout function is run on the socket when connected!
Anyway, the question was about how to test the connect timeout. Ok so let's first park the idle timeout. We can simply test that by not sending any data for some time, that would cause the timeout. Check!
The connect timeout is a bit harder to test, the first thing that comes to mind is that we need a place to connect to that will not error, but also not connect. This would cause a timeout. But how the hell do we simulate that, in node?
If we think a little bit outside the box then we might figure out that this timeout is about the time it takes to connect. It does not matter why the connection takes as long as it does. We simply need to delay the time it takes to connect. This is not necessarily a server thing, we could also do it on the client. After all this is the part connecting, if we can delay it there, we can test the timeout.
So how could we delay the connection on the client side? Well, we can use the DNS lookup for that. Before the connection is made, a DNS lookup is done. If we simply delay that by 5 seconds or so we can test for the connect timeout very easily.
This is what the code could look like:
import * as dns from "dns";
import * as http from "http";
const url = new URL("http://localhost:8080");
const request = http.request(url, {
timeout: 3 * 1000, // connect timeout
lookup(hostname, options, callback) {
setTimeout(
() => dns.lookup(hostname, options, callback),
5 * 1000,
);
},
});
request.setTimeout(10 * 1000); // idle timeout
request.addListener("timeout", () => {
const message = !request.socket || request.socket.connecting ?
`connect timeout while connecting to ${url.href}` :
`idle timeout while connected to ${url.href}`;
request.destroy(new Error(message));
});
In my projects I usually use an agent that I inject. The agent then has the delayed lookup. Like this:
import * as dns from "dns";
import * as http from "http";
const url = new URL("http://localhost:8080");
const agent = new http.Agent({
lookup(hostname, options, callback) {
setTimeout(
() => dns.lookup(hostname, options, callback),
5 * 1000,
);
},
});
const request = http.request(url, {
timeout: 3 * 1000, // connect timeout
agent,
});
request.setTimeout(10 * 1000); // idle timeout
request.addListener("timeout", () => {
const message = !request.socket || request.socket.connecting ?
`connect timeout while connecting to ${url.href}` :
`idle timeout while connected to ${url.href}`;
request.destroy(new Error(message));
});
Happy coding!
"Connection timeout" determines how long it may take for a TCP connection to be established and this all happens before any HTTP related data is send over the line. By connecting to a blocked port, you have only partially tested the connection timeout since no connection was being made. Typically, a TCP connection on your local network is created (established) very fast. But when connecting to a server on the other side of the world, establishing a TCP connection can take seconds.
"Socket timeout" is a somewhat misleading name - it just determines how long you (the client) will wait for an answer (data) from the server. In other words, how long the Socket.read() function will block while waiting for data.
Properly testing these functions involves creating a server socket or a (HTTP) web-server that you can modify to be very slow. Describing how to create and use a server socket for connection timeout testing (if that is possible) is too much to answer here, but socket timeout testing is a common question - see for example here (I just googled "mock web server for testing timeouts") which leads to a tool like MockWebServer. "MockWebServer" might have an option for testing connection timeouts as well (I have not used "MockWebServer"), but if not, another tool might have.
On a final note: it is good you are testing your usage of the third-party HTTP library with respect to timeout settings, even if this takes some effort. The worst that can happen is that a socket timeout setting in your code is somehow not used by the library and the default socket timeout of "wait forever" is used. That can result in your application doing absolutely nothing ("hanging") for no apparent reason.
My Java application which resides in AWS private subnet connects to an http server via AWS Nat gateway. I am calling a POST request via HttpClient to the HTTP server. That request will take more than 10 minutes to complete. I have configured a socket time out and connection timeout of 1 hour as this this a background task . But the intermediate AWS NAT gateway will send back a RST packet after 300 secs [5 mins] and cause the connection to get resetted , there is no way i can increase the NAT gateway timeout. So i need to handle the problem from my application side.
My strategy is to use a TCP keep alive time which will send a packet say every 240 secs to keep the connection active. I have configured this
as below
CloseableHttpClient httpClient = HttpClients.createDefault()
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(params, 3600000); //connection Timeout
HttpConnectionParams.setSoTimeout(params, 3600000); // Socket Time out
HttpConnectionParams.setSoKeepalive(params, true); //Enable Socket level keep alive time
and then call the post request via execute method
HttpPost post = new HttpPost("http://url");
HttpResponse response = httpClient.execute(post);
Since I am using a Linux system I have configured the server with following sysctl values:
sysctl -w net.ipv4.tcp_keepalive_time=240
sysctl -w net.ipv4.tcp_keepalive_intvl=240
sysctl -w net.ipv4.tcp_keepalive_probes=10
But while executing the program the keep alive is not enabled and connection fails as previous.
I have checked this with netstat -o option and as shown below keep alive is off
tcp 0 0 192.168.1.141:43770 public_ip:80 ESTABLISHED 18134/java off (0.00/0/0)
Is there any way i can set TCP keep alive from java code using httpclient . Also I can see HttpConnectionParams are deprecated. But I couldn't find any new class which can set keep alive
I have found a solution to the problem . Curious case is there is no way i can use some builder class in httpclient to pass socket keep alive . One method as i specified in the question is using HttpConnectionParams as below but this is not working and this class is now deprecated.
HttpParams params = httpClient.getParams();
HttpConnectionParams.setSoKeepalive(params, true);
So while checking apache http docs I can see that now connection parameters are passed to httpclient via RequestConfig class . Builders of this class provide solution to set connection_time_out and socket_time_out. But checking the socurce code of this I couldnt see an option to enable SocketKeepAlive which is what we want. So the only solution is directly creating a Socket using SocketBuilder class and pass that to the HttpClientBuilder.
Following is the working code
SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive
RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();
CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).
setDefaultSocketConfig(socketConfig).build();
HttpPost post = new HttpPost(url.toString());
HttpResponse response = httpClient.execute(post);
While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel
tcp 0 0 localip:48314 public_ip:443 ESTABLISHED 14863/java keepalive (234.11/0/0)
If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.
Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.
Holding a connection open holds resources. From the perspective of the server (and network firewalls and routers) a client that opens a connection and begins a request (A POST in your case) but does not send any bytes for a long period is indistinguishable from a client that will never send any more data, because it is faulty or malicious (conducting a DOS attack). The server (and network hardware) is right to conclude that the right thing to do is to shutdown the connection and reclaim the resources used for it. You are trying to fight against correct behaviour that occurs for good reasons. Even if you manage to workaround the TCP shutdown you will find other problems, such as HTTP server timeouts and database timeouts.
You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider
Having the client wait until it has a complete upload to perform before starting the POST.
Splitting the uploads into smaller, more frequent uploads.
Use a protocol other than HTTP.
The approach above with Socket worked beautifully with a reset of tcp_keepalive_intvl value below the AWS Network Load Balancer timeout. Using both, reset the NLB tcp idle timeout that allowed java hour+ connections.
Sometimes, if the configuration is overwritten, the configuration does not take effect.My initial modification of setDefaultSocketConfig in buildClient didn't take effect.Because it is overwritten by getConnectionManager()
public CloseableHttpClient buildClient() throws Exception {
HttpClientBuilder builder = HttpClientBuilder.create()
.setDefaultSocketConfig(SocketConfig.custom().setSoKeepAlive(true).build()) // did not work
.setConnectionManager(getConnectionManager())
.setRetryHandler(getRequestRetryHandler())
.setConnectionReuseStrategy(getConnectionReuseStrategy())
.setDefaultConnectionConfig(getConnectionConfig())
.setDefaultRequestConfig(getRequestConfig())
.setDefaultHeaders(getDefaultHeaders())
.setDefaultCredentialsProvider(getDefaultCredentialsProvider())
.disableContentCompression() // gzip is not needed. Use lz4 when compress=1
.setDefaultCookieStore(cookieStoreProvider.getCookieStore(properties))
.disableRedirectHandling();
String clientName = properties != null ? properties.getClientName() : null;
if (!Utils.isNullOrEmptyString(clientName)) {
builder.setUserAgent(clientName);
}
return builder.build();
And then I move the config to getConnectionManager()，and it work.
private PoolingHttpClientConnectionManager getConnectionManager()
throws CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException, IOException {
RegistryBuilder<ConnectionSocketFactory> registry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.getSocketFactory());
if (properties.getSsl()) {
HostnameVerifier verifier = "strict".equals(properties.getSslMode()) ? SSLConnectionSocketFactory.getDefaultHostnameVerifier() : NoopHostnameVerifier.INSTANCE;
registry.register("https", new SSLConnectionSocketFactory(getSSLContext(), verifier));
}
//noinspection resource
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(
registry.build(),
null,
null,
new IpVersionPriorityResolver(),
properties.getTimeToLiveMillis(),
TimeUnit.MILLISECONDS
);
connectionManager.setDefaultMaxPerRoute(properties.getDefaultMaxPerRoute());
connectionManager.setMaxTotal(properties.getMaxTotal());
connectionManager.setDefaultConnectionConfig(getConnectionConfig());
connectionManager.setDefaultSocketConfig(SocketConfig.custom().setSoKeepAlive(true).build());
return connectionManager;
}
In my application.properties file I have...
server.port=8086
server.connection-timeout=15000
I know that the file is being loaded correctly because the server is running on port 8086.
In the application I have a RestController
#RestController
class TestController {
#GetMapping()
fun getValues(): ResponseEntity<*> {
return someLongRunningProcessPossiblyHanging()
}
}
When I call the endpoint, the request never times out, it just hangs indefinitely.
Am I missing something?
NOTE: I've also been informed that Tomcat uses this field in minutes, not milliseconds (rather unusual choice IMO). I've tried setting this to server.connection-timeout=1 denoting 1 minute, but this didn't work either.
NOTE: I don't want another HTTP request to cause the previous request to time out, I want each HTTP request to timeout of it's own accord, should too much time elapse to serve the request.
connection-timeout does not apply to long running requests. It does apply to the initial connection, when the server waits for the client to say something.
Tomcat docs (not Spring Boot) define it as The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented [...]
To test the setting server.connection-timeout=4000 I connect using netcat and I don't send any HTTP request/headers. I get:
$ time nc -vv localhost 1234
Connection to localhost 1234 port [tcp/*] succeeded!
real 0m4.015s
user 0m0.000s
sys 0m0.000s
Alternatives
1) Async
From brightinventions.pl - Spring MVC Thread Pool Timeouts:
In Spring MVC there is no way to configure a timeout unless you use async method. With async method one can use spring.mvc.async.request-timeout= to set amount of time (in milliseconds) before asynchronous request handling times out.
I've set spring.mvc.async.request-timeout=4000 and I get a timeout in the browser with this:
#GetMapping("/test-async")
public Callable<String> getFoobar() {
return () -> {
Thread.sleep(12000); //this will cause a timeout
return "foobar";
};
}
See Spring Boot REST API - request timeout?
2) Servlet filter
Another solution would be to use a servlet filter brightinventions.pl - Request timeouts in Spring MVC (Kotlin):
override fun doFilterInternal(request: HttpServletRequest, response: HttpServletResponse, filterChain: FilterChain) {
val completed = AtomicBoolean(false)
val requestHandlingThread = Thread.currentThread()
val timeout = timeoutsPool.schedule({
if (completed.compareAndSet(false, true)) {
requestHandlingThread.interrupt()
}
}, 5, TimeUnit.SECONDS)
try {
filterChain.doFilter(request, response)
timeout.cancel(false)
} finally {
completed.set(true)
}
}
3) Tomcat Stuck Thread Detection Valve?
Tomcat has a Stuck Thread Detection Valve but I don't know if this can be configured programmatically using Spring Boot.
From the official docs:
server.connection-timeout= # Time that connectors wait for another HTTP request before closing the connection. When not set, the connector's container-specific default is used. Use a value of -1 to indicate no (that is, an infinite) timeout.
Another ref, also mentions the same. It should work for you.
When I call the endpoint, the request never times out, it just hangs indefinitely.
server.connection-timeout isn't a request timeout. It is a timeout for idle connections, i.e. those that have already had a request/response pair and on which the server is now awaiting a second request. It is essentially a server-side read timeout.
The below lines of code
HttpRequestExecutor httpexecutor = new HttpRequestExecutor();
HttpResponse response2 = httpexecutor.execute(request2, conn, context);
log.info("Status Line "+response2.getStatusLine());
while executing the second statement my request/response is blocked by proxy. How to debug this.?
While the same statement executing from my local environment where no proxy is configured, am able to see
the log returns the below code
"Status Line HTTP/1.1 200 OK"
I think what you are looking for is to set a timeout on the request - basically telling the executor to give up if no connection can be established within a certain period of time.
This SO article covers this: Java HTTP Client Request with defined timeout
the accepted answer has important information about the fact that Apache's implementation has two timeouts - be sure to read that.
Note that it won't be possible for you to actually distinguish between a connection failure that was caused by some other problem - all you will know is that your attempt timed out. In your response to the user, you'll probably wind up saying "Unable to connect to host XXXXX. Possible causes are that your Internet connection is down, or that you have a proxy server that prevents outbound connections to that host."
I'm using the IA92 Java implementation for MQTT, which allows me to connect to a MQTT broker. In order to establish the connection, I'm doing something like this:
// Create connection spec
String mqttConnSpec = "tcp://the_server#the_port";
// Create the client and connect
mqttClient = MqttClient.createMqttClient(mqttConnSpec, null);
mqttClient.connect("the_id", true, 666);
The problem is that sometimes the server takes too much time to send a response, and it throws a timeout exception:
org.apache.harmony.luni.platform.OSNetworkSystem.connectStreamWithTimeoutSocket(OSNetworkSystem.java:130)
at org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.java:246)
at org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.java:533)
at java.net.Socket.connect(Socket.java:1055)
at com.ibm.mqtt.j2se.MqttJava14NetSocket.<init>((null):-1)
at com.ibm.mqtt.j2se.MqttJavaNetSocket.setConnection((null):-1)
at com.ibm.mqtt.Mqtt.tcpipConnect((null):-1)
at com.ibm.mqtt.MqttBaseClient.doConnect((null):-1)
at com.ibm.mqtt.MqttBaseClient.connect((null):-1)
at com.ibm.mqtt.MqttClient.connect((null):-1)
at com.ibm.mqtt.MqttClient.connect((null):-1)
What I need to do is setting a timeout manually, instead of letting the mqtt client decide that. The documentation says: There are also methods for setting attributes of the MQ Telemetry Transport connection, such as timeouts and retries.
But, honestly, I haven't found anything about it. I have taken a look at the whole javadoc reference and there's no evidence of timeout configuration. I can't see the source code since it's not open source.
So how can I set the timeout for the Mqtt connection?
If you have confusion you can go to MqttConnectionOptions for detail.
String userName="Ohelig";
String password="Pojke";
MqttClient client = new MqttClient("tcp://192.168.1.4:1883","Sending");
MqttConnectOptions authen = new MqttConnectOptions();
authen.setUserName(userName);
authen.setPassword(password.toCharArray());
authen.setKeepAliveInterval(30);
authen.setConnectionTimeout(300);
client.connect(authen);
I don't know anything about ia92, but I'd imagine that the 666 in the connect() call is what you're trying to set the timeout to?
The timeout the documentation is referring to is probably the keepalive timeout. This is the maximum number of seconds (chosen by the client) that can elapse without communication between the server and client. I think this is what you're most interested in.
Retries on the other hand are most likely to refer to the retrying of messages that seem to have gone astray when sending messages with QoS>0. This will be something handled by the client library code though, rather than the broker. This is something that comes into play only after you've connected though, so I very much doubt it's your problem.
To be sure that the keepalive timeout is being set correctly, I'd try pointing your client at a modified mosquitto broker. You can modify mqtt3_handle_connect() in src/read_handle_server.c to print out the keepalive value when you connect. This will ensure it's doing what you think, but won't help with the actual problem I'm afraid!
What broker do you use? Really Small Message Broker V1.1 Alpha, Mosquitto, the broker that comes with IBM WebSphere? You need to set this timeout value in your server configuration. Because the system works that way. You set a keep alive value in your broker and send a ping from the client before that interval expires, in order not for the broker to close the client-server connection, and the process restarts. Actually, even if that interval expires, server will still not close the connection until the 'grace period' ends. See http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/mqtt-v3r1.html#connect