Develop Reference

Spring Boot multipart/form-data request file streaming to downstream service

I have a microservice architecture, where one service acts as a proxy, and must only forward the uploaded form data payload to the downstream service using restTemplate, preferably without loading anything from the request on disk or into memory.

I managed to resolve the issue taking the following steps.
Here I will describe the approaches, and the limitations used:
I have the following rest template configuration:
#Bean
public RestTemplate myRestTemplate() {
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate restTemplate = new RestTemplate(requestFactory);
restTemplate.setInterceptors(new ArrayList<>()); // to avoid interceptors loading data into memory
return restTemplate;
}
in my controller I am processing the HttpServletRequest directly using Apache Commons FileUpload Streaming Api with one asterix:
Special care on the multipart form data, so first the form fields are processed in the while loop, and then only one file was I able to process, since:
FileItemStream fileItemStream = uploadItemIterator.next();
return fileItemStream.openStream();
must be returned without invoking itemIterator.hasNext(), because that will result in FileItemStream.ItemSkippedException
which works wonderfully, no data is saved on disk
c:\Users\myuser\AppData\Local\Temp\tomcat.11416588345568217859.8077\
note: I have set the following property as stated in the documentation.
spring.application.servlet.multipart.enabled: false
From here, Using the streaming api I have an inputStream, which I will pass further down to create my HttpEntity as follows (simplified in example, full inspiration to include filename in request: here):
MultiValueMap<String, Object> multiPartBody = new LinkedMultiValueMap<>();
multiPartBody.add(FILE, inputStream);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(multiPartBody, myHeaders);
after this, I do make the call to my rest Template:
myRestTemplate.postForEntity(url, requestEntity, MyResponse.class);
this goes all the way via the following sequence:
RestTemplate.doExecute()
HttpAccessor.createRequest()
HttpComponentsClientHttpRequestFactory.createRequest() -> which will return a **HttpComponentsStreamingClientHttpRequest** <- this one is important
RestTemplate.doWithRequest(ClientHttpRequest httpRequest) -> calls: ((HttpMessageConverter<Object>) messageConverter).write(
requestBody, requestContentType, httpRequest);
FormHttpMessageConverter.write()
FormHttpMessageConverter.writeMultipart() -> where outputMessage instanceof StreamingHttpOutputMessage is true
HttpComponentsStreamingClientHttpRequest.executeInternal -> creates a new StreamingHttpEntity(...)
after which this goes down on InternalCLientExecution, and in execChain
sooner or later it will enter in the chain:
HttpComponentsStreamingClientHttpRequest.StreamingHttpEntity.writeTo(OutputStream outputStream) throws IOException {
this.body.writeTo(outputStream);
}
where body is a FormHttpMessageConverter.lambda from above:
if (outputMessage instanceof StreamingHttpOutputMessage streamingOutputMessage) {
streamingOutputMessage.setBody(outputStream -> {
writeParts(outputStream, parts, boundary);
writeEnd(outputStream, boundary);
});
}
so we get further down, and end up in:
FormHttpMessageConverter.writeParts()
FormHttpMessageConverter.writePart()
here a multipartMessage is composed and passed further down (or invoked the superclass AbstractHttpMessageConverter method)
multipartMessage = new MultipartHttpOutputMessage(os, charset);
...
((HttpMessageConverter<Object>) messageConverter).write(partBody, partContentType, multipartMessage);
from here we get into AbstractHttpMessageConverter.write where condition
if (outputMessage instanceof StreamingHttpOutputMessage)
evaluates to false because MultipartHttpOutputMessage is not an instance of StreamingHttpOutputMessage
But this seems not to affect anything, since the whole thing is invoked in the above mentioned lambda, sooner or later, we need to write the bytes from the inputStream into the outputStream.
one impediment:
if I configure the restTemplate as follows:
#Bean
#org.springframework.cloud.client.loadbalancer.LoadBalanced
public RestTemplate myRestTemplate() {
...
}
there is an interceptor/aspect overriding the RestTemplate HttpComponentsClientHttpRequestFactory with RibbonClientHttpRequestFactory (using spring netflix stack), which does not support setBufferRequestBody(false).
That is how I managed to solve the file streaming issue, hope it helps others too:
Limitations/Constraints:
You cannot use MultipartFile in your controllers since spring by default saves data into temp files on fileSystem (can't use resolve-lazily either: because), I was able to overcome this issue only with Apache Commons FileUpload
Using Apache Commons FileUpload I managed to process only one file, and the form data need to be processed before the file data
spring.application.servlet.multipart.enabled: false -> affects other endpoints too
composing downstream form data with correct Content-Disposition: form-data; name="file"; filename="my.txt" needs some strange embedded HttpEntity constructions
#LoadBalanced overrides the whole restTemplate requestFactory
Good luck everyone, and any feedback is welcome.

Calling multiple external APIs in Spring Boot

I am working on a project but it requires me to call multiple external APIs. I basically have to call an API to get a player id by giving a name. Then use that player id to get a list of match ids. Then make calls for each match id to get details on each match. its alot and doesnt seem optimal but its the only way to do it. I was going to use rest template to make a call to the following
https://americas.api.riotgames.com/lol/match/v5/matches/by-puuid/HDzjdaStxhHcceGGd8qJcc4Vw45FOlOQ1PNXKQ0h9_iqfwHP3oI0spl1bLUOw_7_J49vzaIKylv5Vg/ids?start=0&count=20
I have to pass in headers as well such as
riot token : token
"Origin": "https://developer.riotgames.com"
I was wondering how I can do this in Java Spring boot. I saw RestTemplate would be used but I couldnt figure out how to include the headers. Any guidance would be appreciated.

You can call RestTemplate.exchange() using either the method signature with RequestEntity or with HttpEntity.
// Using RequestEntity
RequestEntity<?> request= RequestEntity.get(url).header(headerName, headerValue).build();
ResponseEntity<String> response = restTemplate.exchange(request, String.class);
// Using HttpEntity
HttpHeaders headers = new HttpHeaders();
headers.set(headerName, headerValue);
HttpEntity httpEntity = new HttpEntity(/* this is nullable */ requestBody, headers);
ResponseEntity,String> response = restTemplate.exchange(url, HttpMethod.GET, httpEntity, String.class);
Would also recommend reading through the following:
https://howtodoinjava.com/spring-boot2/resttemplate/spring-restful-client-resttemplate-example/
https://www.baeldung.com/rest-template
How to set an "Accept:" header on Spring RestTemplate request?

Accepting multiple http methods?

Is there any way to accept multiple http mothods in Micronaut controller? For example, I would like to process GET, POST, and OPTIONS requests in a method. I tried annotated with three annotations #Get, #Post, and #Options, however, only first annotation works and others are ignored.
#Controller("/echo")
public class EchoController {
private ObjectMapper mapper = new ObjectMapper();
#Get
#Post
#Options
public HttpResponse<String> process(HttpRequest<?> request) throws JsonProcessingException {
Headers headers = request.getHeaders();
return ok(mapper.writeValueAsString(headers.asMap()));
}
}
Is adding separete methods for each HTTP method and annotate them with corresponding annotation the only way to do this?

There is an open issue to allow the HTTP annotations to be repeatable.
It looks like it is planned vor 1.2.

Testing MockRestServiceServer spring-test with multipart request

Recently I've started to use Spring's MockRestServiceServer to verify my RestTemplate based requests in tests.
When its used for simple get/post request - all good, however, I couldn't figure out how to use it with POST multipart request:
For example, my working code that I would like to test looks like this:
public ResponseEntity<String> doSomething(String someParam, MultipartFile
file, HttpHeaders headers) { //I add headers from request
MultiValueMap<String, Object> map = new LinkedMultiValueMap<>();
map.add("file", new ByteArrayResource(file.getBytes()) {
#Override
public String getFilename() {
return file.getOriginalFilename();
}
});
map.add("someParam", someParam);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new
HttpEntity<>(map, headers);
return this.restTemplate.exchange(
getDestinationURI(),
HttpMethod.POST,
requestEntity,
String.class);
}
So my question is How I can specify my expectations with org.springframework.test.web.client.MockRestServiceServer? Please notice, that I don't want to just mock the "exchange" method with mockito or something, but prefer to use MockRestServiceServer
I'm using spring-test-4.3.8.RELEASE version
A code snippet would be really appreciated :)
Thanks a lot in advance
Update:
As per James's request I'm adding non-working test snippet (Spock test):
MockRestServiceServer server = MockRestServiceServer.bindTo(restTemplate).build()
server.expect(once(), requestTo(getURI()))
.andExpect(method(HttpMethod.POST))
.andExpect(header(HttpHeaders.CONTENT_TYPE, startsWith("multipart/form-data;boundary=")))
.andExpect(content().formData(["someParam" : "SampleSomeParamValue", "file" : ???????] as MultiValueMap))
.andRespond(withSuccess("sample response", MediaType.APPLICATION_JSON))
multipartFile.getBytes() >> "samplefile".getBytes()
multipartFile.getOriginalFilename() >> "sample.txt"
I get exception while asserting the request content. The form data is different, because an actual form data is created internally with Content-Disposition, Content-Type, Content-Length per parameter and I don't know how to specify these expected values

Multipart request expectations have been added to MockRestServiceServer in Spring 5.3 - see:
pull request
final version
You can use
content().multipartData(MultiValueMap<String, ?> expectedMap)
Parse the body as multipart data and assert it contains exactly the values from the given MultiValueMap. Values may be of type:
String - form field
Resource - content from a file
byte[] - other raw content
content().multipartDataContains(Map<String,?> expectedMap)
Variant of multipartData(MultiValueMap) that does the same but only for a subset of the actual values.

I think this depends on how deeply you want to test the form data. One way, which is not 100% complete, but is a "good enough" for unit testing (usually) is to do something like:
server.expect(once(), requestTo(getURI()))
.andExpect(method(HttpMethod.POST))
.andExpect(content().string(StringContains.containsString('paramname=Value') ))....
This is ugly and incomplete, but is sometimes useful. Of course, you can also work to make the form setup it's own method and then use mocks to try to verify that the expected parameters are all in place.

How to properly setup a POST spring rest template by passing in parameters and return json?

I am a bit new to spring and I am having issues calling a rest service using post to a ur in spring by using spring templates
I am not sure how to properly pass in data into the rest template and how to get json data out currently i get 415 unsupport media type error.
So the RestservicesURL.signupurl = "abc.com/signup
the param would be = "name=john?email=john#doe.com?password=john" (which is acquired by #RequestParam Map by spring)
The response I need to get is a JSON object and I am not sure how to do this;
public void signUp(Map<String, String> param) {
try {
callService(RestServicesUrl.SIGNUP_URL, param);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Here is the rest template call
public static void callService(String url, Map<String, String> data) throws Exception {
List<MediaType> acceptableMediaTypes = new ArrayList<MediaType>();
acceptableMediaTypes.add(MediaType.ALL);
HttpHeaders headers = new HttpHeaders();
headers.setAccept(acceptableMediaTypes);
HttpEntity<String> entity = new HttpEntity<String>(headers);
RestTemplate rest = new RestTemplate();
rest.postForObject(url, entity, byte[].class);
}

Are you trying to post JSON as well as receive it?
Based on your "data" variable it looks like you might be trying to send JSON. I see you have "ALL" set on the acceptable media types, if you're posting JSON and expect to receive it back, you should put the acceptable media type and Content-Type headers to "application/json". I believe Spring has constants for these. This could be the cause of your 415 unsupported mediatype error.
As for creating your JSON, you should strongly consider a serialization framework to convert your Java objects into JSON for you and vice versa - Jackson is a very popular option and is tied closely into Spring's functionality for this, see details here: RestTemplate + Jackson
If that's not an option, you'll need to figure out a way to get your data object converted into the appropriate HTTP request with an HTTPMessageConverter.