Develop Reference

Spring Boot multipart/form-data request file streaming to downstream service

I have a microservice architecture, where one service acts as a proxy, and must only forward the uploaded form data payload to the downstream service using restTemplate, preferably without loading anything from the request on disk or into memory.

I managed to resolve the issue taking the following steps.
Here I will describe the approaches, and the limitations used:
I have the following rest template configuration:
#Bean
public RestTemplate myRestTemplate() {
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate restTemplate = new RestTemplate(requestFactory);
restTemplate.setInterceptors(new ArrayList<>()); // to avoid interceptors loading data into memory
return restTemplate;
}
in my controller I am processing the HttpServletRequest directly using Apache Commons FileUpload Streaming Api with one asterix:
Special care on the multipart form data, so first the form fields are processed in the while loop, and then only one file was I able to process, since:
FileItemStream fileItemStream = uploadItemIterator.next();
return fileItemStream.openStream();
must be returned without invoking itemIterator.hasNext(), because that will result in FileItemStream.ItemSkippedException
which works wonderfully, no data is saved on disk
c:\Users\myuser\AppData\Local\Temp\tomcat.11416588345568217859.8077\
note: I have set the following property as stated in the documentation.
spring.application.servlet.multipart.enabled: false
From here, Using the streaming api I have an inputStream, which I will pass further down to create my HttpEntity as follows (simplified in example, full inspiration to include filename in request: here):
MultiValueMap<String, Object> multiPartBody = new LinkedMultiValueMap<>();
multiPartBody.add(FILE, inputStream);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(multiPartBody, myHeaders);
after this, I do make the call to my rest Template:
myRestTemplate.postForEntity(url, requestEntity, MyResponse.class);
this goes all the way via the following sequence:
RestTemplate.doExecute()
HttpAccessor.createRequest()
HttpComponentsClientHttpRequestFactory.createRequest() -> which will return a **HttpComponentsStreamingClientHttpRequest** <- this one is important
RestTemplate.doWithRequest(ClientHttpRequest httpRequest) -> calls: ((HttpMessageConverter<Object>) messageConverter).write(
requestBody, requestContentType, httpRequest);
FormHttpMessageConverter.write()
FormHttpMessageConverter.writeMultipart() -> where outputMessage instanceof StreamingHttpOutputMessage is true
HttpComponentsStreamingClientHttpRequest.executeInternal -> creates a new StreamingHttpEntity(...)
after which this goes down on InternalCLientExecution, and in execChain
sooner or later it will enter in the chain:
HttpComponentsStreamingClientHttpRequest.StreamingHttpEntity.writeTo(OutputStream outputStream) throws IOException {
this.body.writeTo(outputStream);
}
where body is a FormHttpMessageConverter.lambda from above:
if (outputMessage instanceof StreamingHttpOutputMessage streamingOutputMessage) {
streamingOutputMessage.setBody(outputStream -> {
writeParts(outputStream, parts, boundary);
writeEnd(outputStream, boundary);
});
}
so we get further down, and end up in:
FormHttpMessageConverter.writeParts()
FormHttpMessageConverter.writePart()
here a multipartMessage is composed and passed further down (or invoked the superclass AbstractHttpMessageConverter method)
multipartMessage = new MultipartHttpOutputMessage(os, charset);
...
((HttpMessageConverter<Object>) messageConverter).write(partBody, partContentType, multipartMessage);
from here we get into AbstractHttpMessageConverter.write where condition
if (outputMessage instanceof StreamingHttpOutputMessage)
evaluates to false because MultipartHttpOutputMessage is not an instance of StreamingHttpOutputMessage
But this seems not to affect anything, since the whole thing is invoked in the above mentioned lambda, sooner or later, we need to write the bytes from the inputStream into the outputStream.
one impediment:
if I configure the restTemplate as follows:
#Bean
#org.springframework.cloud.client.loadbalancer.LoadBalanced
public RestTemplate myRestTemplate() {
...
}
there is an interceptor/aspect overriding the RestTemplate HttpComponentsClientHttpRequestFactory with RibbonClientHttpRequestFactory (using spring netflix stack), which does not support setBufferRequestBody(false).
That is how I managed to solve the file streaming issue, hope it helps others too:
Limitations/Constraints:
You cannot use MultipartFile in your controllers since spring by default saves data into temp files on fileSystem (can't use resolve-lazily either: because), I was able to overcome this issue only with Apache Commons FileUpload
Using Apache Commons FileUpload I managed to process only one file, and the form data need to be processed before the file data
spring.application.servlet.multipart.enabled: false -> affects other endpoints too
composing downstream form data with correct Content-Disposition: form-data; name="file"; filename="my.txt" needs some strange embedded HttpEntity constructions
#LoadBalanced overrides the whole restTemplate requestFactory
Good luck everyone, and any feedback is welcome.

Spring - how to pass automatically http header between microservices

I'm having multiple microservices
1. MangerApp
2. ProcessApp
3. DoingStuffApp
4. .....
the "MangerApp Microservices" get an Http-Request
I'm looking for a way to transfer automatically some of the HTTP headers
in the call, while I don't want to go over each place and do - add Headers, my HTTP headers are stored as a thread-local Map.
since I call to other microservices, with RestTemplate I have many different calls some get/post/put/etc...
changing each one them and passing the header manually is not that efficient.
I'm looking for a way to manage it, other than extending the RestTemplate Class now.

You can use a ClientHttpRequestInterceptor to achieve what you need.
1) Create a HeaderInterceptor implementing ClientHttpRequestInterceptor. In this example it gets the Authorization and Accept headers from a ThreadLocal and propagates them:
public class HeaderInterceptor implements ClientHttpRequestInterceptor{
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException {
HttpHeaders headers = request.getHeaders();
List<String> authorization = HeaderThreadLocal.getAuthorization()
List<String> accept = HeaderThreadLocal.getAuthorization();
headers.addAll("Authorization", authorization);
headers.addAll("Accept", accept);
return execution.execute(request, body);
}
}
2) Configure your RestTemplate bean adding the header interceptor:
restTemplate.getInterceptors().add(new HeaderInterceptor());

SessionId lost when I make a request between backend of microservices

I am trying to make request between microservices in order to retrieve a list of users with the same roles. For this, first I make a request between FrontEnd and Backend inside the microservice 1. Following, I call an endpoint in the microservice 2 from Microservice 1 backend, but the session Id is lost in it, and I can retrieve the context.
I am using spring security and Redis for the session Control.
Manually, I retrieve the session Id from the microservice 1 and I add it as an attribute of the header of the second call, to the microservice 2. But it does not work.
String sessionID= RequestContextHolder.currentRequestAttributes().getSessionId();
RestTemplate rest = new RestTemplate();
HttpHeaders headers= new HttpHeaders();
headers.set("Session",sessionID);
HttpEntity<ResponseData> entity = new HttpEntity<ResponseData>(headers);
ResponseEntity<ResponseData> responseEntity =rest.exchange(targetApi, HttpMethod.GET, entity,ResponseData.class);

Finally, I resolved the problem adding an interceptor as a component:
#Component
public class SpringSessionClientHttpRequestInterceptor implements ClientHttpRequestInterceptor {
#Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution)
throws IOException {
request.getHeaders().add("Cookie", "SESSION=" + sessionId);
return execution.execute(request, body);
}
}
And I created a #Bean to configure the rest template:
#Bean
public RestTemplate restTemplate(){
RestTemplate rest = new RestTemplate();
ClientHttpRequestInterceptor interceptor= new SpringSessionClientHttpRequestInterceptor();
List<ClientHttpRequestInterceptor> interceptors = new ArrayList<ClientHttpRequestInterceptor>();
interceptors.add(interceptor);
rest.setInterceptors(interceptors);
return rest;
}

I don't know the exact answer to your question but in terms of your design I'd question if you really want to make your microservice1 depending on microsevice2. A microservice should be autonomous in the way it works and being able to be deployed on it's own (in theory anyway!). May be you could have an orchestrating microservice that receives your session information and then calls the 2 other microservices to pass that information on via 'standard' attributes.

headers.set("Session",sessionID);
I assume that the problem is that you are using the wrong identifier. As far as I know, it is JSESSIONID by default.
Another problem that I can see here is that JSESSIONID expected to be in cookies. Try to put it in cookies when sending a request to your 'microservice2'.

RestTemplate for local file testing

Since I want to use the RestTemplate from Spring I want to use the same class as well for Unit-Testing. The idea would be to download a JSON-File and save it locally for the purpose of testing. Therefore I would like to change the URI from a HTTP to a File address. When it as File-address I get an Excpetion
Exception in thread "main" java.lang.IllegalArgumentException: Object of class [sun.net.www.protocol.file.FileURLConnection] must be an instance of class java.net.HttpURLConnection
urlGETList = "http://api.geonames.org/countryInfoJSON?username=volodiaL";
RestTemplate restTemplate = new RestTemplate();
CountryInfoResponse results = restTemplate.getForObject(urlGETList, CountryInfoResponse.class);
Any ideas how I can use the same classes for Unit-Testing?

I think you could look into Wiremock.
Wiremock allows subbing of requests. The advantage is that you really test the complete stack and your tests make real requests against a server responding with mock responses. These mock response bodies can be files (there are other possibilities as well.)
In your unit test you set up wiremock server like this:
#Rule
public WireMockRule wireMockRule = new WireMockRule(port);
Then you can setup a stub with a file response like this:
public void givenResponse(int statusCode, MediaType contentType, String bodyPath) {
String responseBody;
try (InputStream data = new ClassPathResource(bodyPath).getInputStream()) {
responseBody = copyToString(data, UTF_8);
}
stubFor(any(urlPathEqualTo(getWireMockUri().getPath()))
.willReturn(aResponse()
.withStatus(statusCode)
.withHeader("Content-Type", contentType.toString())
.withHeader("Content-Length", String.valueOf(responseBody.length()))
.withBody(responseBody)
));
}
You could also put the complete stub into a stub file like described here
Afterwards you can check if a certain request has been made:
verify(postRequestedFor(urlEqualTo("/form"))
.withHeader("Content-Type", containing(MULTIPART_FORM_DATA_VALUE)));
You can find out more about verification here
You would use the RestTemplate to make requests. You just need to have host and port configurable so you can use localhost and the wiremock port in your tests.

The easiest way to proxy HttpServletRequest in Spring MVC controller

I'm building REST services using spring-mvc and what I'm looking for now is a way to proxy HTTP request to external REST service from inside Spring MVC controller.
I'm getting HttpServletRequest object and want to proxy it making as few changes as possible. What is essential for me is keeping all the headers and attributes of incoming request as they are.
#RequestMapping('/gateway/**')
def proxy(HttpServletRequest httpRequest) {
...
}
I was trying simply to send another HTTP request to external resource using RestTemplate but I failed to find a way to copy REQUEST ATTRIBUTES (which is very important in my case).
Thanks in advance!

I wrote this ProxyController method in Kotlin to forward all incoming requests to remote service (defined by host and port) as follows:
#RequestMapping("/**")
fun proxy(requestEntity: RequestEntity<Any>, #RequestParam params: HashMap<String, String>): ResponseEntity<Any> {
val remoteService = URI.create("http://remote.service")
val uri = requestEntity.url.run {
URI(scheme, userInfo, remoteService.host, remoteService.port, path, query, fragment)
}
val forward = RequestEntity(
requestEntity.body, requestEntity.headers,
requestEntity.method, uri
)
return restTemplate.exchange(forward)
}
Note that the API of the remote service should be exactly same as this service.

You can use the spring rest template method exchange to proxy the request to a third party service.
#RequestMapping("/proxy")
#ResponseBody
public String proxy(#RequestBody String body, HttpMethod method, HttpServletRequest request, HttpServletResponse response) throws URISyntaxException {
URI thirdPartyApi = new URI("http", null, "http://example.co", 8081, request.getRequestURI(), request.getQueryString(), null);
ResponseEntity<String> resp =
restTemplate.exchange(thirdPartyApi, method, new HttpEntity<String>(body), String.class);
return resp.getBody();
}
What is the restTemplate.exchange() method for?

if you think of applying the API gateway pattern for microservices,
have a look at Netflix zuul which is a good alternative in the spring boot ecosystem. A good example is provided here.