I have a microservice architecture, where one service acts as a proxy, and must only forward the uploaded form data payload to the downstream service using restTemplate, preferably without loading anything from the request on disk or into memory.
I managed to resolve the issue taking the following steps.
Here I will describe the approaches, and the limitations used:
I have the following rest template configuration:
#Bean
public RestTemplate myRestTemplate() {
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate restTemplate = new RestTemplate(requestFactory);
restTemplate.setInterceptors(new ArrayList<>()); // to avoid interceptors loading data into memory
return restTemplate;
}
in my controller I am processing the HttpServletRequest directly using Apache Commons FileUpload Streaming Api with one asterix:
Special care on the multipart form data, so first the form fields are processed in the while loop, and then only one file was I able to process, since:
FileItemStream fileItemStream = uploadItemIterator.next();
return fileItemStream.openStream();
must be returned without invoking itemIterator.hasNext(), because that will result in FileItemStream.ItemSkippedException
which works wonderfully, no data is saved on disk
c:\Users\myuser\AppData\Local\Temp\tomcat.11416588345568217859.8077\
note: I have set the following property as stated in the documentation.
spring.application.servlet.multipart.enabled: false
From here, Using the streaming api I have an inputStream, which I will pass further down to create my HttpEntity as follows (simplified in example, full inspiration to include filename in request: here):
MultiValueMap<String, Object> multiPartBody = new LinkedMultiValueMap<>();
multiPartBody.add(FILE, inputStream);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(multiPartBody, myHeaders);
after this, I do make the call to my rest Template:
myRestTemplate.postForEntity(url, requestEntity, MyResponse.class);
this goes all the way via the following sequence:
RestTemplate.doExecute()
HttpAccessor.createRequest()
HttpComponentsClientHttpRequestFactory.createRequest() -> which will return a **HttpComponentsStreamingClientHttpRequest** <- this one is important
RestTemplate.doWithRequest(ClientHttpRequest httpRequest) -> calls: ((HttpMessageConverter<Object>) messageConverter).write(
requestBody, requestContentType, httpRequest);
FormHttpMessageConverter.write()
FormHttpMessageConverter.writeMultipart() -> where outputMessage instanceof StreamingHttpOutputMessage is true
HttpComponentsStreamingClientHttpRequest.executeInternal -> creates a new StreamingHttpEntity(...)
after which this goes down on InternalCLientExecution, and in execChain
sooner or later it will enter in the chain:
HttpComponentsStreamingClientHttpRequest.StreamingHttpEntity.writeTo(OutputStream outputStream) throws IOException {
this.body.writeTo(outputStream);
}
where body is a FormHttpMessageConverter.lambda from above:
if (outputMessage instanceof StreamingHttpOutputMessage streamingOutputMessage) {
streamingOutputMessage.setBody(outputStream -> {
writeParts(outputStream, parts, boundary);
writeEnd(outputStream, boundary);
});
}
so we get further down, and end up in:
FormHttpMessageConverter.writeParts()
FormHttpMessageConverter.writePart()
here a multipartMessage is composed and passed further down (or invoked the superclass AbstractHttpMessageConverter method)
multipartMessage = new MultipartHttpOutputMessage(os, charset);
...
((HttpMessageConverter<Object>) messageConverter).write(partBody, partContentType, multipartMessage);
from here we get into AbstractHttpMessageConverter.write where condition
if (outputMessage instanceof StreamingHttpOutputMessage)
evaluates to false because MultipartHttpOutputMessage is not an instance of StreamingHttpOutputMessage
But this seems not to affect anything, since the whole thing is invoked in the above mentioned lambda, sooner or later, we need to write the bytes from the inputStream into the outputStream.
one impediment:
if I configure the restTemplate as follows:
#Bean
#org.springframework.cloud.client.loadbalancer.LoadBalanced
public RestTemplate myRestTemplate() {
...
}
there is an interceptor/aspect overriding the RestTemplate HttpComponentsClientHttpRequestFactory with RibbonClientHttpRequestFactory (using spring netflix stack), which does not support setBufferRequestBody(false).
That is how I managed to solve the file streaming issue, hope it helps others too:
Limitations/Constraints:
You cannot use MultipartFile in your controllers since spring by default saves data into temp files on fileSystem (can't use resolve-lazily either: because), I was able to overcome this issue only with Apache Commons FileUpload
Using Apache Commons FileUpload I managed to process only one file, and the form data need to be processed before the file data
spring.application.servlet.multipart.enabled: false -> affects other endpoints too
composing downstream form data with correct Content-Disposition: form-data; name="file"; filename="my.txt" needs some strange embedded HttpEntity constructions
#LoadBalanced overrides the whole restTemplate requestFactory
Good luck everyone, and any feedback is welcome.
Related
In Spring Boot 1.5.x, I could use interceptors with AsyncRestTemplate to grab headers from an incoming request to a RestController endpoint and put them in any exchange requests made via the AsyncRestTemplate.
I don't see how this can work with the WebClient. It looks like if you build a WebClient that all its headers, etc are set and unchangeable:
WebClient client = WebClient.builder()
.baseUrl( "http://blah.com" )
.defaultHeader( "Authorization", "Bearer ey..." )
.build();
While I can change these using client.mutate(), that instantiates a completely new WebClient object. I'd prefer not to have to create a new one on every request. Is there no way to keep a WebClient and have per-request headers and other parameters?
It seems like a big waste and poor performance to force creating a new object every time.
What you're using here are the default headers that should be sent for all requests sent by this WebClient instance. So this is useful for general purpose headers.
You can of course change the request headers on a per-request basis like this:
Mono<String> result = this.webClient.get()
.uri("/greeting")
.header("Something", "value")
.retrieve().bodyToMono(String.class);
If you wish to have an interceptor-like mechanism to mutate the request before sending it, you can configure the WebClient instance with a filter:
WebClient
.builder()
.filter((request, next) -> {
// you can mutate the request before sending it
ClientRequest newRequest = ClientRequest.from(request)
.header("Something", "value").build();
return next.exchange(newRequest);
})
Please check out the Spring Framework documentation about WebClient.
What is the most convenient way to influence the priority of the message converters Spring applies when POSTing with RestTemplate?
Use case: I want to ensure a given entity is POSTed as JSON rather than e.g. XML when I do restTemplate.postForEntity(url, entity, Void.class).
Default
By default the entity is converted to XML because the MappingJackson2XmlHttpMessageConverter takes precedence over the MappingJackson2HttpMessageConverter. The default list of converters for my app appears to be (Spring scans the classpath to see what's available):
Option 1
You can configure the message converters explicitly for a given RestTemplate instance like so restTemplate.setMessageConverters(Lists.newArrayList(new MappingJackson2HttpMessageConverter())). This is inconvenient if the instance is shared (as a Spring bean for example) as you might need converter X in one case and converter Y in a different one.
Option 2
You can set Accept and Content-Type HTTP headers explicitly in which case Spring will use a matching message converter. The downside is that you have to resort to RestTemplate.exchange instead of RestTemplate.postForEntity which means: extra code, less convenience.
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setContentType(MediaType.APPLICATION_JSON);
requestHeaders.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
HttpEntity requestEntity = new HttpEntity(entity, requestHeaders);
restTemplate.exchange(url, HttpMethod.POST, requestEntity, Void.class);
Option 3
This might be what I'm looking for :)
This issue is answered in detail here.
Basically, when you add the below-mentioned library, it adds MappingJackson2XmlHttpMessageConverter before MappingJackson2HttpMessageConverter. As a result, Spring boot assumes every request accepts application/XML.
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId>
</dependency>
To avoid this behaviour, you might want to swap the two message converters.
Example:
#Bean
RestTemplate restTemplate() {
RestTemplate restTemplate = new RestTemplate();
// move XML converter to the end of list
List<HttpMessageConverter<?>> messageConverters = restTemplate.getMessageConverters();
for (int i = 0; i < messageConverters.size() -1 ; i++) {
if (messageConverters.get(i) instanceof MappingJackson2XmlHttpMessageConverter) {
Collections.swap(messageConverters, i,messageConverters.size() - 1);
}
}
restTemplate.setMessageConverters(messageConverters);
// add interceptors if necessary
restTemplate.setInterceptors(Collections.singletonList(catalogInterceptior()));
return restTemplate;
}
According to the Spring javadoc (https://docs.spring.io/spring-framework/docs/current/javadoc-api/index.html?org/springframework/web/client/RestTemplate.html) you can still use postForEntity,
public <T> ResponseEntity<T> postForEntity(java.lang.String url,
#Nullable
java.lang.Object request,
java.lang.Class<T> responseType,
java.util.Map<java.lang.String,?> uriVariables)
throws RestClientException
....
The request parameter can be a HttpEntity in order to add additional HTTP headers to the request.
I am trying to make request between microservices in order to retrieve a list of users with the same roles. For this, first I make a request between FrontEnd and Backend inside the microservice 1. Following, I call an endpoint in the microservice 2 from Microservice 1 backend, but the session Id is lost in it, and I can retrieve the context.
I am using spring security and Redis for the session Control.
Manually, I retrieve the session Id from the microservice 1 and I add it as an attribute of the header of the second call, to the microservice 2. But it does not work.
String sessionID= RequestContextHolder.currentRequestAttributes().getSessionId();
RestTemplate rest = new RestTemplate();
HttpHeaders headers= new HttpHeaders();
headers.set("Session",sessionID);
HttpEntity<ResponseData> entity = new HttpEntity<ResponseData>(headers);
ResponseEntity<ResponseData> responseEntity =rest.exchange(targetApi, HttpMethod.GET, entity,ResponseData.class);
Finally, I resolved the problem adding an interceptor as a component:
#Component
public class SpringSessionClientHttpRequestInterceptor implements ClientHttpRequestInterceptor {
#Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution)
throws IOException {
request.getHeaders().add("Cookie", "SESSION=" + sessionId);
return execution.execute(request, body);
}
}
And I created a #Bean to configure the rest template:
#Bean
public RestTemplate restTemplate(){
RestTemplate rest = new RestTemplate();
ClientHttpRequestInterceptor interceptor= new SpringSessionClientHttpRequestInterceptor();
List<ClientHttpRequestInterceptor> interceptors = new ArrayList<ClientHttpRequestInterceptor>();
interceptors.add(interceptor);
rest.setInterceptors(interceptors);
return rest;
}
I don't know the exact answer to your question but in terms of your design I'd question if you really want to make your microservice1 depending on microsevice2. A microservice should be autonomous in the way it works and being able to be deployed on it's own (in theory anyway!). May be you could have an orchestrating microservice that receives your session information and then calls the 2 other microservices to pass that information on via 'standard' attributes.
headers.set("Session",sessionID);
I assume that the problem is that you are using the wrong identifier. As far as I know, it is JSESSIONID by default.
Another problem that I can see here is that JSESSIONID expected to be in cookies. Try to put it in cookies when sending a request to your 'microservice2'.
Recently I've started to use Spring's MockRestServiceServer to verify my RestTemplate based requests in tests.
When its used for simple get/post request - all good, however, I couldn't figure out how to use it with POST multipart request:
For example, my working code that I would like to test looks like this:
public ResponseEntity<String> doSomething(String someParam, MultipartFile
file, HttpHeaders headers) { //I add headers from request
MultiValueMap<String, Object> map = new LinkedMultiValueMap<>();
map.add("file", new ByteArrayResource(file.getBytes()) {
#Override
public String getFilename() {
return file.getOriginalFilename();
}
});
map.add("someParam", someParam);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new
HttpEntity<>(map, headers);
return this.restTemplate.exchange(
getDestinationURI(),
HttpMethod.POST,
requestEntity,
String.class);
}
So my question is How I can specify my expectations with org.springframework.test.web.client.MockRestServiceServer? Please notice, that I don't want to just mock the "exchange" method with mockito or something, but prefer to use MockRestServiceServer
I'm using spring-test-4.3.8.RELEASE version
A code snippet would be really appreciated :)
Thanks a lot in advance
Update:
As per James's request I'm adding non-working test snippet (Spock test):
MockRestServiceServer server = MockRestServiceServer.bindTo(restTemplate).build()
server.expect(once(), requestTo(getURI()))
.andExpect(method(HttpMethod.POST))
.andExpect(header(HttpHeaders.CONTENT_TYPE, startsWith("multipart/form-data;boundary=")))
.andExpect(content().formData(["someParam" : "SampleSomeParamValue", "file" : ???????] as MultiValueMap))
.andRespond(withSuccess("sample response", MediaType.APPLICATION_JSON))
multipartFile.getBytes() >> "samplefile".getBytes()
multipartFile.getOriginalFilename() >> "sample.txt"
I get exception while asserting the request content. The form data is different, because an actual form data is created internally with Content-Disposition, Content-Type, Content-Length per parameter and I don't know how to specify these expected values
Multipart request expectations have been added to MockRestServiceServer in Spring 5.3 - see:
pull request
final version
You can use
content().multipartData(MultiValueMap<String, ?> expectedMap)
Parse the body as multipart data and assert it contains exactly the values from the given MultiValueMap. Values may be of type:
String - form field
Resource - content from a file
byte[] - other raw content
content().multipartDataContains(Map<String,?> expectedMap)
Variant of multipartData(MultiValueMap) that does the same but only for a subset of the actual values.
I think this depends on how deeply you want to test the form data. One way, which is not 100% complete, but is a "good enough" for unit testing (usually) is to do something like:
server.expect(once(), requestTo(getURI()))
.andExpect(method(HttpMethod.POST))
.andExpect(content().string(StringContains.containsString('paramname=Value') ))....
This is ugly and incomplete, but is sometimes useful. Of course, you can also work to make the form setup it's own method and then use mocks to try to verify that the expected parameters are all in place.
RestTemplate restTemplate = new RestTemplate();
final MappingJackson2XmlHttpMessageConverter converter = new MappingJackson2XmlHttpMessageConverter();
final List<MediaType> supportedMediaTypes = new LinkedList<MediaType>(converter.getSupportedMediaTypes());
supportedMediaTypes.add(MediaType.ALL);
converter.setSupportedMediaTypes(supportedMediaTypes);
restTemplate.getMessageConverters().add(converter);
ResponseEntity<MyDTO[]> response = restTemplate.getForEntity(urlBase, MyDTO[].class);
HttpHeaders headers = response.getHeaders();
URI location = headers.getLocation(); // Has my redirect URI
response.getBody(); //Always null
I was under the impression that a 302 would automatically be followed. Am I incorrect in this assumption? I now need to pick off this location and re-request?
Using the default ClientHttpRequestFactory implementation - which is the SimpleClientHttpRequestFactory - the default behaviour is to follow the URL of the location header (for responses with status codes 3xx) - but only if the initial request was a GETrequest.
Details can be found in this class - searching for the following method:
protected void prepareConnection(HttpURLConnection connection, String httpMethod) throws IOException {
...
if ("GET".equals(httpMethod)) {
connection.setInstanceFollowRedirects(true);
}
Here the relevant doc comment of HttpURLConnection.setInstanceFollowRedirects method:
Sets whether HTTP redirects (requests with response code 3xx) should
be automatically followed by this {#code HttpURLConnection}
instance.
The default value comes from followRedirects, which defaults to true.
Are you trying to redirect from one protocol to another, e.g. from http to https or vise versa? If so the automatic redirect won't work. See this comment: URLConnection Doesn't Follow Redirect
After discussion among Java Networking engineers, it is felt that we
shouldn't automatically follow redirect from one protocol to another,
for instance, from http to https and vise versa, doing so may have
serious security consequences
from https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4620571
Otherwise if you debug the RestTemplate code you will see that by default HttpURLConnection is set properly with getInstanceFollowRedirects() == true.
When using the CommonsClientHttpRequestFactory (which uses HttpClient v3 underneath) you can override the postProcessCommonsHttpMethod method and set to follow redirects.
public class FollowRedirectsCommonsClientHttpRequestFactory extends CommonsClientHttpRequestFactory {
#Override
protected void postProcessCommonsHttpMethod(HttpMethodBase httpMethod) {
httpMethod.setFollowRedirects(true);
}
}
You can then use it like this (with optional, possibly preconfigured, HttpClient instance) and requests will follow the location headers in response:
RestTemplate restTemplate = new RestTemplate(
new FollowRedirectsCommonsClientHttpRequestFactory());
Try create RestTemplate like this (Spring config):
#Bean("smartRestTemplate")
public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder) {
return restTemplateBuilder
.setConnectTimeout(Duration.ofSeconds(..))
.setReadTimeout(Duration.ofSeconds(..))
.build();
}