Develop Reference

Republish message to same queue with updated headers after automatic nack in Spring AMQP

I am trying to configure my Spring AMQP ListenerContainer to allow for a certain type of retry flow that's backwards compatible with a custom rabbit client previously used in the project I'm working on.
The protocol works as follows:
A message is received on a channel.
If processing fails the message is nacked with the republish flag set to false
A copy of the message with additional/updated headers (a retry counter) is published to the same queue
The headers are used for filtering incoming messages, but that's not important here.
I would like the behaviour to happen on an opt-in basis, so that more standardised Spring retry flows can be used in cases where compatibility with the old client isn't a concern, and the listeners should be able to work without requiring manual acking.
I have implemented a working solution, which I'll get back to below. Where I'm struggling is to publish the new message after signalling to the container that it should nack the current message, because I can't really find any good hooks after the nack or before the next message.
Reading the documentation it feels like I'm looking for something analogous to the behaviour of RepublishMessageRecoverer used as the final step of a retry interceptor. The main difference in my case is that I need to republish immediately on failure, not as a final recovery step. I tried to look at the implementation of RepublishMessageRecoverer, but the many of layers of indirection made it hard for me to understand where the republishing is triggered, and if a nack goes before that.
My working implementation looks as follows. Note that I'm using an AfterThrowsAdvice, but I think an error handler could also be used with nearly identical logic.
/*
MyConfig.class, configuring the container factory
*/
#Configuration
public class MyConfig {
#Bean
// NB: bean name is important, overwrites autoconfigured bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(
ConnectionFactory connectionFactory,
Jackson2JsonMessageConverter messageConverter,
RabbitTemplate rabbitTemplate
) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(messageConverter);
// AOP
var a1 = new CustomHeaderInspectionAdvice();
var a2 = new MyThrowsAdvice(rabbitTemplate);
Advice[] adviceChain = {a1, a2};
factory.setAdviceChain(adviceChain);
return factory;
}
}
/*
MyThrowsAdvice.class, hooking into the exception flow from the listener
*/
public class MyThrowsAdvice implements ThrowsAdvice {
private static final Logger logger = LoggerFactory.getLogger(MyThrowsAdvice2.class);
private final AmqpTemplate amqpTemplate;
public MyThrowsAdvice2(AmqpTemplate amqpTemplate) {
this.amqpTemplate = amqpTemplate;
}
public void afterThrowing(Method method, Object[] args, Object target, ListenerExecutionFailedException ex) {
var message = message(args);
var cause = ex.getCause();
// opt-in to old protocol by throwing an instance of BusinessException in business logic
if (cause instanceof BusinessException) {
/*
NB: Since we want to trigger execution after the current method fails
with an exception we need to schedule it in another thread and delay
execution until the nack has happened.
*/
new Thread(() -> {
try {
Thread.sleep(1000L);
var messageProperties = message.getMessageProperties();
var count = getCount(messageProperties);
messageProperties.setHeader("xb-count", count + 1);
var routingKey = messageProperties.getReceivedRoutingKey();
var exchange = messageProperties.getReceivedExchange();
amqpTemplate.send(exchange, routingKey, message);
logger.info("Sent!");
} catch (InterruptedException e) {
logger.error("Sleep interrupted", e);
}
}).start();
// NB: Produce the desired nack.
throw new AmqpRejectAndDontRequeueException("Business logic exception, message will be re-queued with updated headers", cause);
}
}
private static long getCount(MessageProperties messageProperties) {
try {
Long c = messageProperties.getHeader("xb-count");
return c == null ? 0 : c;
} catch (Exception e) {
return 0;
}
}
private static Message message(Object[] args) {
try {
return (Message) args[1];
} catch (Exception e) {
logger.info("Bad cast parse", e);
throw new AmqpRejectAndDontRequeueException(e);
}
}
}
Now, as you can imagine, I'm not particularly pleased with the indeterminism of scheduling a new thread with a delay.
So my question is simply, is there any way I could produce a deterministic solution to my problem using the provided hooks of the ListenerContainer ?

Your current solution risks message loss; since you are publishing on a different thread after a delay. If the server crashes during that delay, the message is lost.
It would be better to publish immediately to another queue with a TTL and dead-letter configuration to republish the expired message back to the original queue.
Using the RepublishMessageRecoverer with retries set to maxattempts=1 should do what you need.

Channel ManagedChannelImpl was not shut down properly

If I run following these two tests I get the error.
1st test
#Rule
public GrpcCleanupRule grpcCleanup = new GrpcCleanupRule();
#Test
public void findAll() throws Exception {
// Generate a unique in-process server name.
String serverName = InProcessServerBuilder.generateName();
// Create a server, add service, start, and register for automatic graceful shutdown.
grpcCleanup.register(InProcessServerBuilder
.forName(serverName)
.directExecutor()
.addService(new Data(mockMongoDatabase))
.build()
.start());
// Create a client channel and register for automatic graceful shutdown.
RoleServiceGrpc.RoleServiceBlockingStub stub = RoleServiceGrpc.newBlockingStub(
grpcCleanup.register(InProcessChannelBuilder
.forName(serverName)
.directExecutor()
.build()));
RoleOuter.Response response = stub.findAll(Empty.getDefaultInstance());
assertNotNull(response);
}
2nd test
#Test
public void testFindAll() {
ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 8081)
.usePlaintext()
.build();
RoleServiceGrpc.RoleServiceBlockingStub stub = RoleServiceGrpc.newBlockingStub(channel);
RoleOuter.Response response = stub.findAll(Empty.newBuilder().build());
assertNotNull(response);
}
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
cleanQueue SEVERE: ~~~ Channel ManagedChannelImpl{logId=1,
target=localhost:8081} was not shutdown properly!!! ~~~
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:94)
If I comment out one of them, then no errors, unit tests pass though but the exception is thrown if both are ran together.
Edit
Based on the suggestion.
#Test
public void testFindAll() {
ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 8081)
.usePlaintext()
.build();
RoleServiceGrpc.RoleServiceBlockingStub stub = RoleServiceGrpc.newBlockingStub(channel);
RoleOuter.Response response = stub.findAll(Empty.newBuilder().build());
assertNotNull(response);
channel.shutdown();
}

Hey I just faced similar issue using Dialogflow V2 Java SDK where I received the error
Oct 19, 2019 4:12:23 PM io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=41, target=dialogflow.googleapis.com:443} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
Also, Having a huge customer base we started running into out of memory unable to create native thread error.
After performing a lot of Debugging operations and Using Visual VM Thread Monitoring I finally figured out that the problem was because of SessionsClient not closing. So I used the attached code block to solve that issue. Post testing that block I was finally able to free up all the used threads and also the error mentioned earlier was resolved.
SessionsClient sessionsClient = null;
QueryResult queryResult = null;
try {
SessionsSettings.Builder settingsBuilder = SessionsSettings.newBuilder();
SessionsSettings sessionsSettings = settingsBuilder
.setCredentialsProvider(FixedCredentialsProvider.create(credentials)).build();
sessionsClient = SessionsClient.create(sessionsSettings);
SessionName session = SessionName.of(projectId, senderId);
com.google.cloud.dialogflow.v2.TextInput.Builder textInput = TextInput.newBuilder().setText(message)
.setLanguageCode(languageCode);
QueryInput queryInput = QueryInput.newBuilder().setText(textInput).build();
DetectIntentResponse response = sessionsClient.detectIntent(session, queryInput);
queryResult = response.getQueryResult();
} catch (Exception e) {
e.printStackTrace();
}
finally {
sessionsClient.close();
}
The shorter values on the graph highlights the use of client.close(). Without that the threads were stuck in Parking State.

I had a similar issue recently using Google cloud task API.
Channel ManagedChannelImpl{logId=5, target=cloudtasks.googleapis.com:443} was
not shutdown properly!!! ~*~*~* (ManagedChannelOrphanWrapper.java:159)
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
In this case CloudTasksClient object implements AutoCloseable and we should call its .close() method after its done.
We can use a try block like this which would auto close when it's done.
try( CloudTasksClient client = CloudTasksClient.create()){
CloudTaskQueue taskQueue = new CloudTaskQueue(client);
}
or Add try/finally
CloudTasksClient client =null;
try{
client = CloudTasksClient.create() ;
CloudTaskQueue taskQueue = new CloudTaskQueue(client);
} catch (IOException e) {
e.printStackTrace();
} finally {
client.close();
}

In my case, I just shutdown the channel in try,finally block:
ManagedChannel channel = ManagedChannelBuilder.forAddress...
try{
...
}finally {
channel.shutdown();
}

how to create a connection pool/only once for a third party Application?

I'm using JAVA/Spring MVC and I need to make a Connection Pool for a Third Party Application integration in my application becouse when i try to connect it multiple time my application and server System utilize 100% RAM.
here i have to problem, when users start to hit a specific method (callGenerationService()) multiple time, my Heap memory(RAM space) increases and becomes 100% and application going to slow becouse of it connect third party application multiple times ? here i need to create a connection only once and get it multiple times. where my connection like,
public class ClickToCallServiceImpl implements ClickToCallServiceInterface {
Client client = null;
#Override
public ClickToCall callGenerationService(ClickToCall clickToCall) {
client = new Client();
client.connect("127.0.0.1", 8021 , "password", 10); //Every time Connection Connect.
client.setEventSubscriptions("plain", "all");
// client.sendSyncApiCommand("",""); //here i run command on every hit like.
client.sendSyncApiCommand(clickToCall.command1, clickToCall.command2);
client.close();
}
}
and here 'ClickToCall' is a #Component Bean/POJO Class with variables setters and getters.
Is there, how to we create a connection (either pool or only once connect) for above connection where i connect only once and hit clickToCall.Command1 and clickToCall.Command2 multiple times and utilize less RAM? Thanks in advance.

Please note that I'm not an expert of freeswitch esl so you must check the code properly. Anyway this is what I would do.
First I create a Factory for Client
public class FreeSwitchEslClientFactory extends BasePooledObjectFactory<Client> {
#Override
public Client create() throws Exception {
//Create and connect: NOTE I'M NOT AN EXPERT OF ESL FREESWITCH SO YOU MUST CHECK IT PROPERLY
Client client = new Client();
client.connect("127.0.0.1", 8021 , "password", 10);
client.setEventSubscriptions("plain", "all");
return client;
}
#Override
public PooledObject<Client> wrap(Client obj) {
return new DefaultPooledObject<Client>(obj);
}
}
Then I create a shareable GenericObjectPool:
#Configuration
#ComponentScan(basePackages= {"it.olgna.spring.pool"})
public class CommonPoolConfig {
#Bean("clientPool")
public GenericObjectPool<Client> clientPool(){
GenericObjectPool<Client> result = new GenericObjectPool<Client>(new FreeSwitchEslClientFactory());
//Pool config e.g. max pool dimension
result.setMaxTotal(20);
return result;
}
}
Finally I use the created pool in order to get the Client obj:
#Component
public class FreeSwitchEslCommandSender {
#Autowired
#Qualifier("clientPool")
private GenericObjectPool<Client> pool;
public void sendCommand(String command, String param) throws Exception{
Client client = null;
try {
client = pool.borrowObject();
client.sendSyncApiCommand(command, param);
} finally {
if( client != null ) {
client.close();
}
pool.returnObject(client);
}
}
}
I didn't test (also because I can't) it but it should work. In any case I pray you to properly check the configuration. I don't know if it's OK to always create a Client object and connect or if it's better to connect when you want to send command
I hope it can be useful
EDIT INFORMATION
Sorry I made an error early. You must return the client to the pool
I updated my FreeSwitchEslCommandSender class
Angelo

how to configure pooled connection idle timeout in reactor-netty

I am using reactor-netty http client (0.7.X series) with connection pooling and would like to configure pooled connection's idle timeout but don't know where.
More precisely, I need to configure reactor-netty http client connection pool in such a way that it will automatically close connections that did not see any activity within configurable timeout. These connections are open but no bytes were transferred in or out for some (configurable) amount of time.
How can I configure reactory-netty http client to close idle connections preemptively?

I managed to configure WebClient (via underlying TcpClient) to remove idle connections on timeout from connection pool in reactor-netty 0.8.9
My solution is partially based on the official documentation about IdleStateHandler extended with my research on how to properly apply it when creating an instance of HttpClient.
Here is how I did that:
public class IdleCleanupHandler extends ChannelDuplexHandler {
#Override
public void userEventTriggered(final ChannelHandlerContext ctx, final Object evt) throws Exception {
if (evt instanceof IdleStateEvent) {
final IdleState state = ((IdleStateEvent) evt).state();
if (state == IdleState.ALL_IDLE) { // or READER_IDLE / WRITER_IDLE
// close idling channel
ctx.close();
}
} else {
super.userEventTriggered(ctx, evt);
}
}
}
...
public static WebClient createWebClient(final String baseUrl, final int idleTimeoutSec) {
final TcpClient tcpClient = TcpClient.create(ConnectionProvider.fixed("fixed-pool"))
.bootstrap(bootstrap -> BootstrapHandlers.updateConfiguration(bootstrap, "idleTimeoutConfig",
(connectionObserver, channel) -> {
channel.pipeline()
.addLast("idleStateHandler", new IdleStateHandler(0, 0, idleTimeoutSec))
.addLast("idleCleanupHandler", new IdleCleanupHandler());
}));
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient)))
.baseUrl(baseUrl)
.build();
}
IMPORTANT UPDATE:
My further testing has indicated that adding handlers during bootstrap hook distructs the pool and sockets (channels) are not reused by Connection.
The right way to add the handlers is:
public static WebClient createWebClient(final String baseUrl, final int idleTimeoutSec) {
final TcpClient tcpClient = TcpClient.create(ConnectionProvider.fixed("fixed-pool"))
.doOnConnected(conn -> {
final ChannelPipeline pipeline = conn.channel().pipeline();
if (pipeline.context("idleStateHandler") == null) {
pipeline.addLast("idleStateHandler", new IdleStateHandler(0, 0, idleTimeoutSec))
.addLast("idleCleanupHandler", new IdleCleanupHandler());
}
});
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient)))
.baseUrl(baseUrl)
.build();
}
Note: in reactor-netty 0.9.x there will be a standard way to configure idle timeout for connections in the connection pool, see this commit: https://github.com/reactor/reactor-netty/pull/792

I was able to accomplish this on the 0.7.x branch by adding netty write and read time-out handlers to the channel pipeline. However, on 0.8.x, this approach no longer works.
HttpClient httpClient = HttpClient
.create((HttpClientOptions.Builder builder) -> builder
.host(endpointUrl.getHost())
.port(endpointUrl.getPort())
.poolResources(PoolResources.fixed(connectionPoolName, maxConnections, timeoutPool))
.afterChannelInit(channel -> {
channel.pipeline()
// The write and read timeouts are serving as generic socket idle state handlers.
.addFirst("write_timeout", new WriteTimeoutHandler(timeoutIdle, TimeUnit.MILLISECONDS))
.addFirst("read_timeout", new ReadTimeoutHandler(timeoutIdle, TimeUnit.MILLISECONDS));
})
.build());

The easiest way to do this in reactor-netty 0.9.x with TCP client is by using the below approach, I got this from the link referred by #Vladimir-L. Configure "maxIdleTime" for your question.
TcpClient timeoutClient = TcpClient.create(ConnectionProvider.fixed(onnectionPoolName, maxConnections, acquireTimeout,maxIdleTime));

I am currently at reactor-netty 0.8.2 because of spring-boot-starter-webflux and faced the same issue, the connection pool kept connections open for 60 seconds after they were finished.
With this approach you can't configure the timeout, but you can disable it:
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(
HttpClient.from(TcpClient.create()).keepAlive(false)))
.build()
.get()
.uri("someurl")
.retrieve()
.bodyToMono(String.class)

For Reactor Netty version 1 you need to create a reactor.netty.resources.ConnectionProvider which will contain the idle time configuration and then use that when creating the reactor.netty.http.client.HttpClient.
I'm using Spring so I then use that to create a Spring org.springframework.http.client.reactive.ClientHttpConnector as shown below.
ConnectionProvider connectionProvider = ConnectionProvider.builder("Name")
.maxIdleTime(Duration.ofSeconds(10))
.build();
HttpClient httpClient = HttpClient.create(connectionProvider)
.compress(true);
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.baseUrl(host);

Unable to configure "Keep Alive" in Camel HTTP component

I'm having some troubles with the right setup of the HTTP component. Currently a microservice pulls JSON Content from a provider, process it and send it to the next service for further processes. The main problem is that this microservice create a ton of CLOSE_WAIT socket connections. I understand that the whole concept of "KEEP-ALIVE" shall keep the connection open until I close it, but it's possible that the server will drop the connection for some reasons and creates this CLOSE_WAIT socket.
I've created a small service for debugging / testing purposes which make a GET Call to Google, but even this connection stays open until i close the program. I've tried many different solutions:
.setHeader("Connection", constant("Close"))
-Dhttp.keepAlive=false as VM argument
Switching from Camel-Http to Camel-Http4
httpClient.soTimeout=500 (Camel-HTTP), httpClient.socketTimeout=500 and connectionTimeToLive=500 (Camel-HTTP4)
.setHeader("Connection", simple("Keep-Alive")) and
.setHeader("Keep-Alive", simple("timeout=10")) (Camel-HTTP4)
Setting via debugging the response of DefaultConnectionKeepAliveStrategy from -1 (never ending) to a specific value in Camel-HTTP4 - that works but I was not able to inject my own strategy.
but i had no success. So maybe one of you can help me:
How can i tell the Camel-HTTP that it should close a connection when a specific time is passed? For example, the service pulls every hour from the content provider. After 3-4 hours the HttpComponent should close the connection after the pull and reopen it when the next pull is there. Currently every connection would be put back into the MultiThreadedHttpConnectionManager and the socket is still open.
If it's not possible to do that with Camel-HTTP: How can i inject a HttpClientBuilder into the Creation of my route? I know that it should be possible via httpClient option but I don't understand that specific part of the documentation.
Thank you all for your help

Unfortunately none of the proposed answers solved the CLOSE_WAIT connection status on my side until the application finally was closed.
I reproduced this problem with the following test case:
public class HttpInvokationTest extends CamelSpringTestSupport {
private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
#EndpointInject(uri = "mock:success")
private MockEndpoint successEndpoint;
#EndpointInject(uri = "mock:failure")
private MockEndpoint failureEndpoint;
#Override
protected AbstractApplicationContext createApplicationContext() {
return new AnnotationConfigApplicationContext(ContextConfig.class);
}
#Configuration
#Import(HttpClientSpringTestConfig.class)
public static class ContextConfig extends CamelConfiguration {
#Override
public List<RouteBuilder> routes() {
List<RouteBuilder> routes = new ArrayList<>(1);
routes.add(new RouteBuilder() {
#Override
public void configure() {
from("direct:start")
.log(LoggingLevel.INFO, LOG, CONFIDENTIAL, "Invoking external URL: ${header[ERPEL_URL]}")
.setHeader("Connection", constant("close"))
.recipientList(header("TEST_URL"))
.log(LoggingLevel.DEBUG, "HTTP response code: ${header["+Exchange.HTTP_RESPONSE_CODE+"]}")
.bean(CopyBodyToHeaders.class)
.choice()
.when(header(Exchange.HTTP_RESPONSE_CODE).isGreaterThanOrEqualTo(300))
.to("mock:failure")
.otherwise()
.to("mock:success");
}
});
return routes;
}
}
#Test
public void testHttpInvocation() throws Exception {
successEndpoint.expectedMessageCount(1);
failureEndpoint.expectedMessageCount(0);
ProducerTemplate template = context.createProducerTemplate();
template.sendBodyAndHeader("direct:start", null, "TEST_URL", "http4://meta.stackoverflow.com");
successEndpoint.assertIsSatisfied();
failureEndpoint.assertIsSatisfied();
Exchange exchange = successEndpoint.getExchanges().get(0);
Map<String, Object> headers = exchange.getIn().getHeaders();
String body = exchange.getIn().getBody(String.class);
for (String key : headers.keySet()) {
LOG.info("Header: {} -> {}", key, headers.get(key));
}
LOG.info("Body: {}", body);
Thread.sleep(120000);
}
}
and issuing netstat -ab -p tcp | grep 151.101.129.69 requests, where the IP is the one of meta.stackoverflow.com.
This gave responses like:
tcp4 0 0 192.168.0.10.52183 151.101.129.69.https ESTABLISHED 37562 2118
tcp4 0 0 192.168.0.10.52182 151.101.129.69.http ESTABLISHED 885 523
right after the invocation followeb by
tcp4 0 0 192.168.0.10.52183 151.101.129.69.https CLOSE_WAIT 37562 2118
tcp4 0 0 192.168.0.10.52182 151.101.129.69.http CLOSE_WAIT 885 523
responses until the application was closed due to the Connection: keep-alive header even with a configuration like the one below:
#Configuration
#EnableConfigurationProperties(HttpClientSettings.class)
public class HttpClientSpringTestConfig {
private final static Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
#Resource
private HttpClientSettings httpClientSettings;
#Resource
private CamelContext camelContext;
private SocketConfig httpClientSocketConfig() {
/*
socket timeout:
Monitors the time passed between two consecutive incoming messages over the connection and
raises a SocketTimeoutException if no message was received within the given timeout interval
*/
LOG.info("Creating a SocketConfig with a socket timeout of {} seconds", httpClientSettings.getSoTimeout());
return SocketConfig.custom()
.setSoTimeout(httpClientSettings.getSoTimeout() * 1000)
.setSoKeepAlive(false)
.setSoReuseAddress(false)
.build();
}
private RequestConfig httpClientRequestConfig() {
/*
connection timeout:
The time span the application will wait for a connection to get established. If the connection
is not established within the given amount of time a ConnectionTimeoutException will be raised.
*/
LOG.info("Creating a RequestConfig with a socket timeout of {} seconds and a connection timeout of {} seconds",
httpClientSettings.getSoTimeout(), httpClientSettings.getConTimeout());
return RequestConfig.custom()
.setConnectTimeout(httpClientSettings.getConTimeout() * 1000)
.setSocketTimeout(httpClientSettings.getSoTimeout() * 1000)
.build();
}
#Bean(name = "httpClientConfigurer")
public HttpClientConfigurer httpConfiguration() {
ConnectionKeepAliveStrategy myStrategy = new ConnectionKeepAliveStrategy() {
#Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
return 5 * 1000;
}
};
PoolingHttpClientConnectionManager conMgr =
new PoolingHttpClientConnectionManager();
conMgr.closeIdleConnections(5, TimeUnit.SECONDS);
return builder -> builder.setDefaultSocketConfig(httpClientSocketConfig())
.setDefaultRequestConfig(httpClientRequestConfig())
.setConnectionTimeToLive(5, TimeUnit.SECONDS)
.setKeepAliveStrategy(myStrategy)
.setConnectionManager(conMgr);
}
#PostConstruct
public void init() {
LOG.debug("Initializing HTTP clients");
HttpComponent httpComponent = camelContext.getComponent("http4", HttpComponent.class);
httpComponent.setHttpClientConfigurer(httpConfiguration());
HttpComponent httpsComponent = camelContext.getComponent("https4", HttpComponent.class);
httpsComponent.setHttpClientConfigurer(httpConfiguration());
}
}
or defining the settings directly on the respective HttpComponent.
On examining the respective proposed methods in the HttpClient code it gets obvious that these methods are single-shot operations and not configurations that HttpClient internally will check every few milliseconds itself.
PoolingHttpClientConnectionManager states further that:
The handling of stale connections was changed in version 4.4. Previously, the code would check every connection by default before re-using it. The code now only checks the connection if the elapsed time since the last use of the connection exceeds the timeout that has been set. The default timeout is set to 2000ms
which only occurs if an attempt is done on re-using a connection, which makes sense for a connection pool, especially if multiple messages are exchanged via the same connection. For single-shot invocations, that should more behave like a Connection: close there might not be a reuse of that connection for some time, leaving the connection open or half-closed as no further attempt is done to read from that connection and therefore recognizing itself that the connection could be closed.
I noticed that I already solved such an issue a while back with traditional HttpClients and started to port this solution to Camel, which worked out quite easily.
The solution basically consists of registering HttpClients with a service and then periodically (5 seconds in my case) call closeExpiredConnections() and closeIdleConnections(...).
This logic is kept in a singleton enum, as this is actually in a library that a couple of applications use, each running in their own JVM.
/**
* This singleton monitor will check every few seconds for idle and stale connections and perform
* a cleanup on the connections using the registered connection managers.
*/
public enum IdleConnectionMonitor {
INSTANCE;
private final static Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/** The execution service which runs the cleanup every 5 seconds **/
private ScheduledExecutorService executorService =
Executors.newScheduledThreadPool(1, new NamingThreadFactory());
/** The actual thread which performs the monitoring **/
private IdleConnectionMonitorThread monitorThread = new IdleConnectionMonitorThread();
IdleConnectionMonitor() {
// execute the thread every 5 seconds till the application is shutdown (or the shutdown method
// is invoked)
executorService.scheduleAtFixedRate(monitorThread, 5, 5, TimeUnit.SECONDS);
}
/**
* Registers a {#link HttpClientConnectionManager} to monitor for stale connections
*/
public void registerConnectionManager(HttpClientConnectionManager connMgr) {
monitorThread.registerConnectionManager(connMgr);
}
/**
* Request to stop the monitoring for stale HTTP connections.
*/
public void shutdown() {
executorService.shutdown();
try {
if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
LOG.warn("Connection monitor shutdown not finished after 3 seconds!");
}
} catch (InterruptedException iEx) {
LOG.warn("Execution service was interrupted while waiting for graceful shutdown");
}
}
/**
* Upon invocation, the list of registered connection managers will be iterated through and if a
* referenced object is still reachable {#link HttpClientConnectionManager#closeExpiredConnections()}
* and {#link HttpClientConnectionManager#closeIdleConnections(long, TimeUnit)} will be invoked
* in order to cleanup stale connections.
* <p/>
* This runnable implementation holds a weakly referable list of {#link
* HttpClientConnectionManager} objects. If a connection manager is only reachable by {#link
* WeakReference}s or {#link PhantomReference}s it gets eligible for garbage collection and thus
* may return null values. If this is the case, the connection manager will be removed from the
* internal list of registered connection managers to monitor.
*/
private static class IdleConnectionMonitorThread implements Runnable {
// we store only weak-references to connection managers in the list, as the lifetime of the
// thread may extend the lifespan of a connection manager and thus allowing the garbage
// collector to collect unused objects as soon as possible
private List<WeakReference<HttpClientConnectionManager>> registeredConnectionManagers =
Collections.synchronizedList(new ArrayList<>());
#Override
public void run() {
LOG.trace("Executing connection cleanup");
Iterator<WeakReference<HttpClientConnectionManager>> conMgrs =
registeredConnectionManagers.iterator();
while (conMgrs.hasNext()) {
WeakReference<HttpClientConnectionManager> weakConMgr = conMgrs.next();
HttpClientConnectionManager conMgr = weakConMgr.get();
if (conMgr != null) {
LOG.trace("Found connection manager: {}", conMgr);
conMgr.closeExpiredConnections();
conMgr.closeIdleConnections(30, TimeUnit.SECONDS);
} else {
conMgrs.remove();
}
}
}
void registerConnectionManager(HttpClientConnectionManager connMgr) {
registeredConnectionManagers.add(new WeakReference<>(connMgr));
}
}
private static class NamingThreadFactory implements ThreadFactory {
#Override
public Thread newThread(Runnable r) {
Thread t = new Thread(r);
t.setName("Connection Manager Monitor");
return t;
}
}
}
As mentioned, this singleton service spawns an own thread that invokes the two, above mentioned methods every 5 seconds. These invocations take care of closing connections that are either unused for a certain amount of time or that are IDLE for the stated amount of time.
In order to camelize this service EventNotifierSupport can be utilized in order to let Camel take care of shutting down the monitor thread once it is closing down.
/**
* This Camel service with take care of the lifecycle management of {#link IdleConnectionMonitor}
* and invoke {#link IdleConnectionMonitor#shutdown()} once Camel is closing down in order to stop
* listening for stale connetions.
*/
public class IdleConnectionMonitorService extends EventNotifierSupport {
private final static Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private IdleConnectionMonitor connectionMonitor;
#Override
public void notify(EventObject event) {
if (event instanceof CamelContextStartedEvent) {
LOG.info("Start listening for closable HTTP connections");
connectionMonitor = IdleConnectionMonitor.INSTANCE;
} else if (event instanceof CamelContextStoppingEvent){
LOG.info("Shutting down listener for open HTTP connections");
connectionMonitor.shutdown();
}
}
#Override
public boolean isEnabled(EventObject event) {
return event instanceof CamelContextStartedEvent || event instanceof CamelContextStoppingEvent;
}
public IdleConnectionMonitor getConnectionMonitor() {
return this.connectionMonitor;
}
}
In order to take advantage of that service, the connection manager that is used by the HttpClient Camel uses internally needs to be registered with the service, which is done in the code block below:
private void registerHttpClientConnectionManager(HttpClientConnectionManager conMgr) {
if (!getIdleConnectionMonitorService().isPresent()) {
// register the service with Camel so that on a shutdown the monitoring thread will be stopped
camelContext.getManagementStrategy().addEventNotifier(new IdleConnectionMonitorService());
}
IdleConnectionMonitor.INSTANCE.registerConnectionManager(conMgr);
}
private Optional<IdleConnectionMonitorService> getIdleConnectionMonitorService() {
for (EventNotifier eventNotifier : camelContext.getManagementStrategy().getEventNotifiers()) {
if (eventNotifier instanceof IdleConnectionMonitorService) {
return Optional.of((IdleConnectionMonitorService) eventNotifier);
}
}
return Optional.empty();
}
Last but not least the connection manager defined in httpConfiguration inside the HttpClientSpringTestConfig in my case needed to be past to the introduced register function
PoolingHttpClientConnectionManager conMgr = new PoolingHttpClientConnectionManager();
registerHttpClientConnectionManager(conMgr);
This might not be the prettiest solution, but it does close the half-closed connections on my machine.
#edit
I just learned that you can use a NoConnectionReuseStrategy which changes the connection state to TIME_WAIT rather than CLOSE_WAIT and therefore removes the connection after a short moment. Unfortunately, the request is still issued with a Connection: keep-alive header. This strategy will create a new connection per request, i.e. if you've got a 301 Moved Permanently redirect response the redirect would occur on a new connection.
The httpClientConfigurer bean would need to change to the following in order to make use of the above mentioned strategy:
#Bean(name = "httpClientConfigurer")
public HttpClientConfigurer httpConfiguration() {
return builder -> builder.setDefaultSocketConfig(socketConfig)
.setDefaultRequestConfig(requestConfig)
.setConnectionReuseStrategy(NoConnectionReuseStrategy.INSTANCE);
}

It can be done by closing idle connections if they are idle for configured time. You can achieve same by configuring idle connection timeout for Camel Http Component.
Camel Http provide interface to do so.
Cast org.apache.camel.component.http4.HttpComponent to PoolingHttpClientConnectionManager
PoolingHttpClientConnectionManager poolingClientConnectionManager = (PoolingHttpClientConnectionManager) httpComponent
.getClientConnectionManager();
poolingClientConnectionManager.closeIdleConnections(5000, TimeUnit.MILLISECONDS);
Visit Here [http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.html#closeIdleConnections(long, java.util.concurrent.TimeUnit)]

Firstly Roman Vottner, your answer and just your sheer dedication to finding the issue helped me a truckload. I have been struggling with the CLOSE_WAIT for 2 days now and your answer was what helped. Here is what I did. Added the following code in my CamelConfiguration class which essentially tampers with CamelContext at startup.
HttpComponent http4 = camelContext.getComponent("https4", HttpComponent.class);
http4.setHttpClientConfigurer(new HttpClientConfigurer() {
#Override
public void configureHttpClient(HttpClientBuilder builder) {
builder.setConnectionReuseStrategy(NoConnectionReuseStrategy.INSTANCE);
}
});
Worked like a charm.

You can provide your own clientConnectionManager to HTTP4. Generally you should use an instance of org.apache.http.impl.conn.PoolingHttpClientConnectionManager, which you'd configure with your own org.apache.http.config.SocketConfig by passing it to setDefaultSocketConfig method of the connection manager.
If you're using Spring with Java config, you would have a method:
#Bean
PoolingHttpClientConnectionManager connectionManager() {
SocketConfig socketConfig = SocketConfig.custom()
.setSoKeepAlive(false)
.setSoReuseAddress(true)
.build();
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setDefaultSocketConfig(socketConfig);
return connectionManager;
}
and then you'd just use it in your endpoint definition like so: clientConnectionManager=#connectionManager