Develop Reference

Spring boot - Connection reset by peer under heavy load

We are seeing some errors related to connection reset in Production under heavy load.
Basically the flow is
Microservice 1 -> Load Balancer() -> Microservice 2
Error message is
Stack trace:
|_ checkpoint ? Request to post
http://LoadbalancerURL/process [Default web client]
Error has been observed at the following site(s):
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
My question is how do we troubleshoot this issue.
Do we need to take tcpdump? if yes then what should I look for?
Possible workaround I could think of is use exponential retry. Any idea where I could insert retry logic in below code?
Below is the code snipnet on how we are calling the url from microservice 1
webclient
.create()
.post()
.uri(url)
.bodyValue(input)
.retrieve()
.toEntity(object.class)
.log()
.subscribe(
onSuccess -> {
//somelogic
},
onError -> {
//somelogic
},
()->{
})

Quarkus: The return value of "org.jboss.resteasy.reactive.server.mapping.RuntimeResource.getProduces()" is null

I created Reactive SQL client on Quarkus using this official guide, but while making GET request on http://localhost:8080/hello trying to query data from database, I get such an error:
2022-03-06 13:18:53,051 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-5) HTTP Request to /hello failed, error id: 2796a874-61d9-43ee-bd87-6f098cffa484-2: java.lang.NullPointerException: Cannot invoke "org.jboss.resteasy.reactive.common.util.ServerMediaType.getSortedMediaTypes()" because the return value of "org.jboss.resteasy.reactive.server.mapping.RuntimeResource.getProduces()" is null
at org.jboss.resteasy.reactive.server.handlers.PublisherResponseHandler.handle(PublisherResponseHandler.java:203)
at org.jboss.resteasy.reactive.server.handlers.PublisherResponseHandler.handle(PublisherResponseHandler.java:30)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:141)
at org.jboss.resteasy.reactive.server.handlers.RestInitialHandler.beginProcessing(RestInitialHandler.java:49)
at org.jboss.resteasy.reactive.server.vertx.ResteasyReactiveVertxHandler.handle(ResteasyReactiveVertxHandler.java:17)
at org.jboss.resteasy.reactive.server.vertx.ResteasyReactiveVertxHandler.handle(ResteasyReactiveVertxHandler.java:7)
at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:163)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:141)
at io.quarkus.vertx.http.runtime.StaticResourcesRecorder$2.handle(StaticResourcesRecorder.java:67)
at io.quarkus.vertx.http.runtime.StaticResourcesRecorder$2.handle(StaticResourcesRecorder.java:55)
at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1212)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:126)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:141)
at io.vertx.ext.web.handler.impl.StaticHandlerImpl.lambda$sendStatic$1(StaticHandlerImpl.java:274)
at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)

You need to add something like:
#Produces(MediaType.APPLICATION_JSON)
// Each element will be sent as JSON
#RestSseElementType(MediaType.APPLICATION_JSON)
to your JAX-RS method.
However, you most likely also need to include quarkus-resteasy-reactive-jackson.
Furthermore, you might want to revisit the use of Multi as a return type as you are likely looking for something like Uni<List<Address>>.

Spring #ExceptionHandler does not work with IllegalArgumentException

I got an API in a Kotlin Spring boot project and I'm customizing my error responses. I have a class for handling the exceptions to show a customized error response. I've added 2 handlers. They work perfectly:
#ControllerAdvice
class ControllerExceptionHandlers {
#ExceptionHandler(ConstraintViolationException::class)
fun constraintViolationException(e: ConstraintViolationException): ResponseEntity<Any> {
return ResponseEntity
.status(MyManError.BAD_PARAM_VALIDATION.status)
.body(
Error(
FaultDefinition(
MyManError.BAD_PARAM_VALIDATION.code,
MyManError.BAD_PARAM_VALIDATION.name,
"Parameter type mismatch field: '${e.message}'",
"Source Name: ${Security.getUserName()}",
null
)
)
)
}
#ExceptionHandler(MissingServletRequestPartException::class)
fun missingServletRequestPartException(e: MissingServletRequestPartException): ResponseEntity<Any> {
return ResponseEntity
.status(MyManError.BAD_PARAM_VALIDATION.status)
.body(
Error(
FaultDefinition(
MyManError.BAD_PARAM_VALIDATION.code,
"BAD_PARAM_${e.requestPartName.toUpperCase()}",
"Parameter validation failed for field $e.requestPartName",
"Source Name: ${Security.getUserName()}",
null
)
)
)
}
}
And I want to add 1 more error handler for IllegalArgumentException:
#ExceptionHandler(IllegalArgumentException::class)
fun illegalArgumentException(e: IllegalArgumentException): ResponseEntity<Any> {
return ResponseEntity
.status(MyManError.BAD_PARAM_VALIDATION.status)
.body(
Error(
FaultDefinition(
MyManError.BAD_PARAM_VALIDATION.code,
MyManError.BAD_PARAM_VALIDATION.name,
"'${e.message}'",
"Source Name: ${Security.getUserName()}",
null
)
)
)
}
I added server.max-http-header-size and assigned it to 10KB. So I tried to send 11KB request. But I got the error in log like that:
17:09:42.513 [http-nio-8080-exec-3] INFO o.a.coyote.http11.Http11Processor - Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Request header is too large
at org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:603)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:284)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
I'm exactly sure I'm importing like import java.lang.IllegalArgumentException but still it shows the log like that.
Why #ExceptionHandler is not working with IllegalArgumentException ? Other handlers are working well but this is not. Can you help me ?
EDIT: One strange thing is; if I add the handler for IllegalArgumentException, but if I do different thing, I can see my customized response like that (and that is as I wanted):
{
"fault": {
"code": 420018,
"message": "BAD_PARAM_VALIDATION",
"description": "'No topic found for name :'my-events.fifo''",
"source": "Source Name: Wicaledon"
}
}
I'm exactly sure that the new handler handled it because if I remove it, and if I check logs, I see same error as in my customized response:
java.lang.IllegalArgumentException: No topic found for name :'my-events.fifo'
at io.awspring.cloud.messaging.support.destination.DynamicTopicDestinationResolver.getTopicResourceName(DynamicTopicDestinationResolver.java:94)
at io.awspring.cloud.messaging.support.destination.DynamicTopicDestinationResolver.resolveDestination(DynamicTopicDestinationResolver.java:72)
at io.awspring.cloud.messaging.support.destination.DynamicTopicDestinationResolver.resolveDestination(DynamicTopicDestinationResolver.java:36)
at org.springframework.messaging.core.CachingDestinationResolverProxy.resolveDestination(CachingDestinationResolverProxy.java:92)
at io.awspring.cloud.messaging.core.support.AbstractMessageChannelMessagingSendingTemplate.resolveMessageChannelByLogicalName(AbstractMessageChannelMessagingSendingTemplate.java:94)
at io.awspring.cloud.messaging.core.support.AbstractMessageChannelMessagingSendingTemplate.convertAndSend(AbstractMessageChannelMessagingSendingTemplate.java:75)

Failed to obtain R2DBC Connection Java Spring

I am getting following exeption connecting to Mssql Server.
> org.springframework.dao.DataAccessResourceFailureException: Failed to obtain R2DBC Connection; nested exception is java.net.UnknownHostException: failed to resolve '' after 10 queries
at org.springframework.r2dbc.connection.ConnectionFactoryUtils.lambda$getConnection$0(ConnectionFactoryUtils.java:88) ~[spring-r2dbc-5.3.2.jar:5.3.2]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ Handler com.reactive.testreactive.controller.TestStreamController#findAll() [DispatcherHandler]
|_ checkpoint ⇢ HTTP GET "/test" [ExceptionHandlingWebHandler]
I connected to JDBC with the same configuration in properties but having an issue while trying to connect to R2DBC. Happens on rest and not on starting an app.
#Bean
public MssqlConnectionFactory connectionFactory() {
return new MssqlConnectionFactory(MssqlConnectionConfiguration.builder()
.host("host")
.port(1433)
.database("DataBase")
.username("username")
.password("password")
.build());
}

Do you use absolutely the same configuration?
Because if you are starting using R2DBC you should replace database URL: jdbc:postgresql://... -> r2dbc:postgresql://

If anyone still looking for solution, I'm adding solution below.
First, make sure your database host is visible. By looking at your error statement which says -- failed to resolve '' after 10 queries, means it is empty ''. The way that you have defined the host name is not correct. If you are injecting the host value from properties, you might want to cross check that and then follow the steps below.
If you are configuring through YMLs, you can use below configurations:
spring:
data:
r2dbc:
repositories:
enabled: true
r2dbc:
url: r2dbc:sqlserver://<just_host>:<port>
username: <db_username>
password: <db_password>
name: <db_name>
spring.data.r2dbc.repositories.enabled: true is optional here.
If you are creating custom bean, you can create bean like below. Please note that I'm returning ConnectionFactory rather than MssqlConnectionFactory.
#Bean
public ConnectionFactory connectionFactory() {
return new MssqlConnectionFactory(
MssqlConnectionConfiguration.builder()
.host("just_host")
.database("db_name")
.port(1433)
.username("db_username")
.password("db_password")
.build());
}

RestTemplate postForEntity experiencing java.io.IOException: Premature EOF error

Would appreciate any help regarding my issue on one of my maven projects.
Exception in thread "main" org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://test-services.domain.ph/campaign/": Premature EOF; nested exception is java.io.IOException: Premature EOF
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:407)
at homecredit.ph.CampaignConnector.call(CampaignConnector.java:46)
Caused by: java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:565)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
Origin:
ResponseEntity<ApiResponse> response = restTemplate.postForEntity(url, entity, ApiResponse.class);
Destination:
#RequestMapping(value="/campaign", method = RequestMethod.POST, consumes=MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<ApiResponse> insertCampaignRecord(
#Valid #RequestBody CampaignRecordInsertRequest campaignRecordInsertRequest){
logInfo("Incoming insert request. " + DescriptorUtility.convertToString(campaignRecordInsertRequest));
campaignDataService.insertNewRecord(CampaignRecordConverter.convertToCampaignRecord(campaignRecordInsertRequest));
return ResponseUtility.defaultResponse();
}
ResponseUtility
public static ResponseEntity<ApiResponse> defaultResponse(){
ApiResponse apiResponse = new ApiResponse();
apiResponse.setTimestamp(DateUtility.currentDateString());
apiResponse.setMessage(ResponseMessages.SUCCESS);
return new ResponseEntity<>(apiResponse, HttpStatus.OK);
}
CampaignData Service
#Async("AsyncExecutor")
public void insertNewRecord(CampaignRecord campaignRecord) {
try {
campaignRecordRepository.save(campaignRecord);
} catch (Exception e) {
logError(e);
}
}
Server Log
2017-09-11 11:11:11 INFO 18383 [http-nio-8773-exec-10] [CampaignRecordController] - Incoming insert request. {"dateCampaign":1504656000000,"cuid":...
2017-09-11 11:11:11 WARN 18383 [http-nio-8773-exec-10] [SqlExceptionHelper] - SQL Error: 1062, SQLState: 23000
2017-09-11 11:11:11 ERROR 18383 [http-nio-8773-exec-10] [SqlExceptionHelper] - Duplicate entry 'CMP_CLX##1208637#20170906' for key 'UNIQUE_KEY'
2017-09-11 11:11:11 ERROR 18383 [http-nio-8773-exec-10] [CampaignDataService] - could not execute statement; SQL [n/a]; constraint [null]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement
2017-09-11 11:11:11 ERROR 18383 [http-nio-8773-exec-10] [CampaignDataService] - could not execute statement
PS. Server logs is normal(return a successful response either record successfully saved or not)
Issue is intermittent. Occurs randomly when sending bulk requests.
Thanks in advance! :)

I had the same problem in Spring Boot 2.1. In my case I had 3 apps (call them A, B, and C) where B was really just a proxy between A and C:
A --> B --> C
The Premature EOF was occurring on the response from B to A. All indications were a successful HTTP response (200), but inspecting the body of the response using a debugger revealed it had a new line character in the middle of the serialized DTO data, instead of at the end where I expected it:
(Notice the return character after the id field, and lack of any content length; ignore the unreadable boxes at the end, they're part of the byte array that are not initialized/used)
In my case, Service B is both a server and a client. The code looked something like this:
public ResponseEntity<String> handle(String request, HttpHeaders headers) {
// Do some validation and then call Service C, and pass the response
// back to Service A
return restTemplate.postForEntity(
urlForServiceC,
new HttpEntity<>(request, headers),
String.class);
}
I didn't dive too far into the guts of RestTemplate or its message converters, but what tipped me off that there might be an issue with the response buffering is that I was using a Spring filter to log the responses of each service. This filter has to copy the response stream to avoid exceptions from other filters related to the body already being consumed.
What I noticed is that when I ran with this filter enabled, the Premature EOF exceptions went away. And when I disabled it, the exceptions came back. Something about copying the response stream had solved the Premature EOF errors.
This led me to try the following in Service B:
public ResponseEntity<String> handle(String request, HttpHeaders headers) {
// Do some validation and then call Service C, and pass the response
// back to Service A
String response = restTemplate.postForEntity(
urlForServiceC,
new HttpEntity<>(request, headers),
String.class).getBody();
return ResponseEntity.ok(response);
}
The subtle change is that I'm saving the response first to a local variable, which requires me to call ResponseEntity.getBody(). This forces the entire response body from Service C to be consumed before returning to Service A. After making this change my Premature EOF errors have not returned.

Based on the server logs seems like, the server is trying save some record and its failing (due to Unique key violation).
2017-09-11 11:11:11 WARN 18383 [http-nio-8773-exec-10] [SqlExceptionHelper] - SQL Error: 1062, SQLState: 23000
2017-09-11 11:11:11 ERROR 18383 [http-nio-8773-exec-10] [SqlExceptionHelper] - Duplicate entry 'CMP_CLX##1208637#20170906' for key 'UNIQUE_KEY'
2017-09-11 11:11:11 ERROR 18383 [http-nio-8773-exec-10] [CampaignDataService] - could not execute statement; SQL [n/a]; constraint [null]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement
2017-09-11 11:11:11 ERROR 18383 [http-nio-8773-exec-10] [CampaignDataService] - could not execute statement
Looks like the server is not able to handle the exception gracefully and the whole flow breaks, causing a HTTP-500 response code (probably) with an empty response.
Two actions you can take:
Handle the exception gracefully.
Verify why unique key is getting violated and fix that if possible.

For anyone who might be experiencing this.
Issue was caused by spring boot eureka.
Seems like there is a bug when it comes to passing ResponseEntity response in a massive scale (bulk processing) that causes the response status to be malformed.
Current workaround is to switch from ResponseEntity to the object body instead.

I have the same problem when i use RestTemplate with default http client factory.I found it missing 'Accept-Encoding:gzip' in the headers when I capture packets.Finally i get it by replace the default http client factory with apache's http client factory,like this:
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory(HttpClientBuilder.create().build());
RestTemplate restTemplate = new RestTemplate(clientHttpRequestFactory);