I am trying to stream the result of a file download directly into another post using spring's RestTemplate
My current approach is the following:
ResponseEntity<InputStreamResource> downloadResponse = restTemplate.getForEntity(fileToDownloadUri, InputStreamResource.class);
InputStreamResource imageInputStreamResource = downloadResponse.getBody();
ResponseEntity<String> response = restTemplate.exchange(storageUri, POST, new HttpEntity<>(imageInputStreamResource), String.class);
However, I get the following exception running the code above:
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://host:port/path/some.jpg": stream is closed; nested exception is java.io.IOException: stream is closed
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:6
...
Caused by: java.io.IOException: stream is closed
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.ensureOpen(HttpURLConnection.java:3348)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3373)
It seems that the response is always closed as the final step of processing. With the response, the HttpURLConnection is closed, and the stream is no longer processable.
I would like to be able to implement this scenario without having to hold the file completely in memory or writing it to a file (as described here).
Any hints are highly appreciated.
If you want to forward the response directly without ever holding it in memory, you have to directly write to the response:
#RequestMapping(value = "/yourEndPoint")
public void processRequest(HttpServletResponse response) {
RestTemplate restTemplate = new RestTemplate();
response.setStatus(HttpStatus.OK.value());
restTemplate.execute(
fileToDownloadUri,
HttpMethod.GET,
(ClientHttpRequest requestCallback) -> {},
responseExtractor -> {
IOUtils.copy(responseExtractor.getBody(), response.getOutputStream());
return null;
});
}
Since you tell RestTemplate to expect InputStreamResource it will try and use an appropriate converter to convert your message to a InputStreamResource. ( I'm guessing there is none that handles this as you want )
You should be able to let it expect a Resource from where you can get an input stream and read that.
import org.springframework.core.io.Resource;
ResponseEntity<Resource> exchange = RestTemplate.exchange(url, HttpMethod.GET, new HttpEntity(httpHeaders), Resource.class);
InputStream inputStream = exchange.getBody().getInputStream();
using this you can write the response to somewhere else. Files.write(inputStream, new File("./test.json")); wrote the file for me, so I assume the inputstream can also be used somewhere else. ( I used Spring 4.3.5 )
edit:
As the OP states, this will still load the file in memory. Behind the scene the InputStream is a ByteArrayInputStream.
The default RestTemplate and MessageConverters are not made for streaming content at all.
You could write your own implementation of a org.springframework.web.client.ResponseExtractor and maybe a MessageConverter. In ResponseExtractor you have access to the org.springframework.http.client.ClientHttpResponse
imho for your use case, you might be better of using Apache Httpcomponents HttpClient where you find HttpEntity#writeTo(OutputStream).
Related
I am using WebClient of the Spring webflux module for the first time.
The idea is to query a certain URL which gives me a CSV in shift-jis encoding.
String content = webClient
.method(HttpMethod.GET)
.acceptCharset(Charset.forName("shift-jis"))
.uri(uri)
.accept(MediaType.TEXT_PLAIN, new MediaType("text", "csv"))
.retrieve()
.toEntity(String.class)
.mapNotNull(HttpEntity::getBody)
.block();
This works somewhat, but the content String looks like it is in a wrong encoding - probably UTF-8.
Alternatively, an Apache HttpClient version seems to work:
HttpClient client = HttpClientBuilder.create().build();
HttpGet get = new HttpGet(uri);
String content = IOUtils.toString(
client.execute(get).getEntity().getContent(),
Charset.forName("shift-jis"));
So I am wondering how one could interfere with the conversion happening in toEntity. I tried adding this to the webClient call:
.codecs(configurer -> {
StringDecoder decoder = StringDecoder.textPlainOnly();
decoder.setDefaultCharset(Charset.forName("shift-jis"));
configurer.customCodecs().registerWithDefaultConfig(decoder);
})
Alas, this does not seem to be picked up. Setting a breakpoint in StringDecoder#decode and manually overwriting the defaultCharset field during debugging, however, does help.
Ah... Just after posting I realized: Maybe my custom text/csv mime type is not really interpreted as textPlainOnly.
And, indeed:
StringDecoder decoder = StringDecoder.allMimeTypes();
apparently does the trick.
Leaving the question open in case my approach is awfully hacky and there are better/correct ways to tackle this issue.
I am downloading a large file from a third party URL using Spring RestTemplate . Im using the below method to download the file so as not to load the entirety into memory.
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor
This works fine, however we intermittently get a Premature end of Content-Length delimited message body
I read from another stack overflow post that this is a server side issue. How can I make sure it's a server side issue? How can I know if it's a memory issue on the third party side or incorrect content length in the response header and such? I need to make sure before I approach the third party for a resolution.
Depending how easy it is to trigger the error, you can use a tool like curl to call the URL (just for testing, outside your application). If you still see the failure, then it is definitely not because of the client, because those well-known third-party tools are well tested and do not have such bugs.
I'm attempting to "pass through" an upload from a controller to another server as part of a multipart request using WebClient. I've finally got the file upload working, but am having trouble getting the file name in a streaming fashion from the FilePart. I'm not sure this is even possible?
This is the piece of code that is causing the issue:
private MultiValueMap<String, HttpEntity<?>> getMultipartPayload(Flux<FilePart> fileUploadFlux) {
MultipartBodyBuilder builder = new MultipartBodyBuilder();
Flux<DataBuffer> dataBufferFlux = fileUploadFlux.flatMap(Part::content);
// spring doesn't have an encoder for FilePart, so we must extract the DataBuffer
builder.asyncPart("file", dataBufferFlux, DataBuffer.class).headers(h -> {
// need to set content-disposition header
// but - this doesn't work as the filename is still a flux
h.setContentDispositionFormData("file", fileUploadFlux.map(FilePart::filename));
});
return builder.build();
}
Note that I've had to extract the Flux<DataBuffer> from the Flux<FilePart> as Spring was unable to find a writer for FilePart (No suitable writer found for part: file).
Is it possible to set the content-disposition header in this fashion?
I'm working on unit tests (backed by mocks) for my dropwizard resources. I recently added some response headers and different status codes, and want to test those.
Unfortunately I can't find an example in their documentation where they get both the entity and examine the response.
The method I'm testing streams back a file. The method itself is defined as:
#GET
#Path("/{assetId}")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
#PermitAll
public Response download(#Auth User user,
#PathParam("assetId") String assetId,
#HeaderParam("Range") String rangeHeader) {
To verify the download, I was initially making a request from my test like this:
final InputStream responseStream = resource.client()
.target("/assets/123")
.request()
.get(InputStream.class);
I could then write the stream to a file and validate that it was the file being returned from the resource. Worked fine. (Here, "resource" is dropwizard's ResourceTestRule.)
However, as above, I now want to get a handle on the response itself. There are examples floating around which use Jersey's ClientResponse, but that doesn't work (yet). I tried:
final ClientResponse response = resource.client()
.target("/assets/123")
.request()
.get(ClientResponse.class);
However, this ends up with an exception:
javax.ws.rs.client.ResponseProcessingException: org.glassfish.jersey.message.internal.MessageBodyProviderNotFoundException: MessageBodyReader not found for media type=application/octet-stream, type=class org.glassfish.jersey.client.ClientResponse, genericType=class org.glassfish.jersey.client.ClientResponse.
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:811)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
...which I'm not sure what to do with. Any help?
Thanks
Ah, this was simpler than I thought.
final Response response = resource.client()
.target("/assets/123")
.request()
.get();
InputStream responseStream = (InputStream) response.getEntity();
I am somewhat new to HTTP REST operations on Android, and a server I am working with uses PUT commands to process updates. I am having a difficult time using Spring (for Android) with Jackson2. The server doesn't seem to work with application/json put requests (though it will reply with them), and only seem to work with application/x-www-form-urlencoded versions (tested with python and curl. On python, if I set the header type to application/json, it fails.
I am using the latest versions of Spring and Jackson2, and I know everything is setup properly because my get request on the same URL gets me all the correct information.
I am using Robospice, but I don't really think that is relevant. Here is my request code.
#Override
public GPIO loadDataFromNetwork() throws Exception {
String url = String.format("http://%s/api/control/gpio", ip);
RestTemplate rt = getRestTemplate();
DefaultHttpClient client = new DefaultHttpClient();
Credentials defaultcreds = new UsernamePasswordCredentials("admin",
password);
client.getCredentialsProvider().setCredentials(
new AuthScope(routerip, 80, AuthScope.ANY_REALM), defaultcreds);
// Makes authentication work.
rt.setRequestFactory(new HttpComponentsClientHttpRequestFactory(client));
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
HttpEntity request = new HttpEntity(data, requestHeaders);
List<HttpMessageConverter<?>> messageConverters = new ArrayList<HttpMessageConverter<?>>();
MappingJackson2HttpMessageConverter map = new MappingJackson2HttpMessageConverter();
messageConverters.add(map);
messageConverters.add(new FormHttpMessageConverter());
rt.setMessageConverters(messageConverters);
ResponseEntity<GPIO> r = rt.exchange(url, HttpMethod.PUT, request, GPIO.class);
return r.getBody();
}
I am getting the exception stating it cannot find a way to convert:
02-01 10:59:29.466: E//DefaultRequestRunner.java:138(30086):
10:59:29.474 Thread-11651 An exception occurred during request network
execution :Could not write request: no suitable HttpMessageConverter
found for request type [com.xxxxx.control.gpio.GPIO] and content
type [application/x-www-form-urlencoded]
GPIO is my POJO object. I want to 'put' that to the server, as in serialize and put it.
I have looked at the following question that seems fairly relevant:
Deserializing Nested objects using RestTemplate
However, I need the result of my put command, and that requires me to use exchange() because Spring's put() returns nothing.
I have tried several different items (such as removing GPIO references, setting specific headres...) and none seem to work. I have a feeling this is probably an easy solution that I don't know how to fix. If anyone can help me that would be great.
TLDR: I'm using Spring for Android with Jackson2. I want to serialize my object (in this example, GPIO) so I can do a PUT request with the content type application/x-www-form-urlencoded. However I cannot get jackson to convert to that type, only to application/json, which does not work for me. I am not sure how to fix this, and I have run out of ideas. If I can't find a solution I'll probably have to dumb robospice. (or jackson, not sure which yet.)
Solution
Spring for Android doesn't seem to simplify things, so I dumped it and used the apache client directly in my loadDataFromNetwork() method. Robospice handles it pretty well and I can get the responses I need. If you are new to HTTP like I was take the time and learn the apache client, it's far easier in my opinion. Tweaking the ObjectMapper (like making a JsonTree and parsing that) made it much easier to get the data I needed without having to do as much work with POJO objects.
If you can format the data you want to send into MultiValueMap<String, String>, then a possible way around this is to use FormHttpMessageConverter.
FormHttpMessageConverter: (you can see examples in the link)
An HttpMessageConverter implementation that can read and write form
data from the HTTP request and response. By default, this converter
reads and writes the media type application/x-www-form-urlencoded.
Form data is read from and written into a MultiValueMap<String,String>.
After re-reading, here's a shot at a real answer - you are explicitly using the x-www-form-urlencoded content type by using this RequestHeader:
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
You should be using APPLICATION_JSON - Here's the Javadoc.
You should also consider specifying the charset and datatype in the headers. Jaxson is very specific about this, and if you don't have access to the server code you don't know what headers they expect.
dude i am using Loopj's AsyncHttpClient for rest and json dataset.
Here is the link below Here
Very simple and easy to understand. U can try this thing.