How to gzip HTTP request, created by org.springframework.web.client.RestTemplate?
I am using Spring 4.2.6 with Spring Boot 1.3.5 (Java SE, not Android or Javascript in the web browser).
I am making some really big POST requests, and I want request body to be compressed.
I propose two solutions, one simpler without streaming and one that supports streaming.
If you don't require streaming, use a custom ClientHttpRequestInterceptor, a Spring feature.
RestTemplate rt = new RestTemplate();
rt.setInterceptors(Collections.singletonList(interceptor));
Where interceptor could be:
ClientHttpRequestInterceptor interceptor = new ClientHttpRequestInterceptor() {
#Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution)
throws IOException {
request.getHeaders().add("Content-Encoding", "gzip");
byte[] gzipped = getGzip(body);
return execution.execute(request, gzipped);
}
}
getGzip I copied
private byte[] getGzip(byte[] body) throws IOException {
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
try {
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream);
try {
zipStream.write(body);
} finally {
zipStream.close();
}
} finally {
byteStream.close();
}
byte[] compressedData = byteStream.toByteArray();
return compressedData;
}
After configuring the interceptor all requests will be zipped.
The disadvantage of this approach is that it does not support streaming as the ClientHttpRequestInterceptor receives the content as a byte[]
If you require streaming create a custom ClientHttpRequestFactory, say GZipClientHttpRequestFactory, and use it like this:
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
ClientHttpRequestFactory gzipRequestFactory = new GZipClientHttpRequestFactory(requestFactory);
RestTemplate rt = new RestTemplate(gzipRequestFactory);
Where GZipClientHttpRequestFactory is:
public class GZipClientHttpRequestFactory extends AbstractClientHttpRequestFactoryWrapper {
public GZipClientHttpRequestFactory(ClientHttpRequestFactory requestFactory) {
super(requestFactory);
}
#Override
protected ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod, ClientHttpRequestFactory requestFactory)
throws IOException {
ClientHttpRequest delegate = requestFactory.createRequest(uri, httpMethod);
return new ZippedClientHttpRequest(delegate);
}
}
And ZippedClientHttpRequest is:
public class ZippedClientHttpRequest extends WrapperClientHttpRequest
{
private GZIPOutputStream zip;
public ZippedClientHttpRequest(ClientHttpRequest delegate) {
super(delegate);
delegate.getHeaders().add("Content-Encoding", "gzip");
// here or in getBody could add content-length to avoid chunking
// but is it available ?
// delegate.getHeaders().add("Content-Length", "39");
}
#Override
public OutputStream getBody() throws IOException {
final OutputStream body = super.getBody();
zip = new GZIPOutputStream(body);
return zip;
}
#Override
public ClientHttpResponse execute() throws IOException {
if (zip!=null) zip.close();
return super.execute();
}
}
And finally WrapperClientHttpRequest is:
public class WrapperClientHttpRequest implements ClientHttpRequest {
private final ClientHttpRequest delegate;
protected WrapperClientHttpRequest(ClientHttpRequest delegate) {
super();
if (delegate==null)
throw new IllegalArgumentException("null delegate");
this.delegate = delegate;
}
protected final ClientHttpRequest getDelegate() {
return delegate;
}
#Override
public OutputStream getBody() throws IOException {
return delegate.getBody();
}
#Override
public HttpHeaders getHeaders() {
return delegate.getHeaders();
}
#Override
public URI getURI() {
return delegate.getURI();
}
#Override
public HttpMethod getMethod() {
return delegate.getMethod();
}
#Override
public ClientHttpResponse execute() throws IOException {
return delegate.execute();
}
}
This approach creates a request with chunked transfer encoding, this can be changed setting the content length header, if size is known.
The advantage of the ClientHttpRequestInterceptor and/or custom ClientHttpRequestFactory approach is that it works with any method of RestTemplate. An alternate approach, passing a RequestCallback is possible only with execute methods, this because the other methods of RestTemplate internally create their own RequestCallback(s) that produce the content.
BTW it seems that there is little support to decompress gzip request on the server. Also related: Sending gzipped data in WebRequest? that points to the Zip Bomb issue. I think you will have to write some code for it.
Further to the above answer from #TestoTestini, if we take advantage of Java 7+'s 'try-with-resources' syntax (since both ByteArrayOutputStream and GZIPOutputStream implement closeable() ) then we can shrink the getGzip function into the following:
private byte[] getGzip(byte[] body) throws IOException {
try (ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream)) {
zipStream.write(body);
byte[] compressedData = byteStream.toByteArray();
return compressedData;
}
}
(I couldn't find a way of commenting on #TestoTestini's original answer and retaining the above code format, hence this Answer).
Since I cannot comment on #roj 's post I'm writing an answer here.
#roj snippet although is neat it actually does not do the same job as #Testo Testini 's snippet.
Testo is closing the streams before:
byteStream.toByteArray();
where in #rog answer, this occurs before the stream.close(), since streams are in the try/resource block.
If you need to use try-with-resources, zipStream should be closed before the byteStream.toByteArray();
The complete snippet should be:
private byte[] getGzip(byte[] body) throws IOException {
try (ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream)) {
zipStream.write(body);
zipStream.close();
byte[] compressedData = byteStream.toByteArray();
return compressedData;
}
}
The was getting an error ("Compressed file ended before the end-of-stream marker was reached") and the above fixed the error in my case and I thought that I should share this.
Related
We have a RestController with the below endpoint
#PostMapping(path = "/downloadFile", produces = MediaType.MULTIPART_FORM_DATA_VALUE,
consumes = MediaType.APPLICATION_JSON_VALUE)
public FileDownloadResponse downloadFile(#RequestBody FileDownloadRequest request) {
FileDownloadResponse downloadResponse = new FileDownloadResponse();
File file = new File("c:/fileLocation/"+request.getFileName());
try (InputStream stream = new FileInputStream(file)) {
byte[] bytes = IOUtil.toByteArray(stream);
downloadResponse.setFileName(file.getName());
downloadResponse.setCheckSum(calculateCheckSum(bytes));
downloadResponse.setFileContents(new FileSystemResource(bytes, file.getName()));
} catch (Exception e) {
e.printStackTrace();
}
return downloadResponse;
}
public class FileDownloadResponse {
private String fileName;
private Long checkSum;
private Resource fileContents;
}
public static class FileSystemResource extends ByteArrayResource {
private String fileName;
public FileSystemResource(byte[] byteArray , String filename) {
super(byteArray);
this.fileName = filename;
}
public String getFilename() { return fileName; }
public void setFilename(String fileName) { this.fileName= fileName; }
}
And on the Client Side we have the below code,
public class FileDownloadResponseClient {
private String fileName;
private Long checkSum;
private MultipartFile fileContents;
}
public FileDownloadResponseClient download(FileDownloadRequest request) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
headers.setAccept(Collections.singletonList(Mediatype.ALL));
HttpEntity<FileDownloadRequest> requestEntity = new HttpEntity<>(request, headers);
return restTemplate.postForEntity(downloadUrl, requestEntity, FileDownloadResponseClient.class);
}
When we run the Rest Client above, we are getting the below error,
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : [no body]
Is it possible to download a multipartfile along with other additional fields? If yes, what is that we are missing here, please let us know.
Thanks in Advance!
org.springframework.web.multipart has a method boolean isEmpty() to find if the file has no content. Best put that check there and redirect to a message about such a file multipart form.
Of [no body] i have found that message on test requests to http server but in entirety generally means there is nothing in the form or no extra information needed for the server to complete the request.
For now i presume the spring framework handles all the url decoding and boundary marker stripping (on both ends) of uploaded files.
Background
here is the method defined in #RestController, it reads file from disk then stream back.
#RequestMapping(value = "/bill", method = RequestMethod.GET)
public ResponseEntity<Object> getbill(){
...
InputStream in = new FileInputStream(file);
InputStreamResource inputStreamResource = new InputStreamResource(in);
httpHeaders.setContentLength(file.Length());
return new ResponseEntity(inputStreamResource, httpHeaders, HttpStatus.OK);
}
Issue
I would like to delete the file after request is served, but unable to find a good place.
I would assume it should be after inputStream gets closed (https://github.com/spring-projects/spring-framework/blob/v4.3.9.RELEASE/spring-web/src/main/java/org/springframework/http/converter/ResourceHttpMessageConverter.java#L117) . it can not be done in above method since file is opened by Inputstream.
Answer Summary
Thank you all for helping with this.
The accepted answer requires least change and working well.
Aside from the fact that it is bad practice in a RESTfull service to perform destructive operations on GET requests this can not be done by the default Java libraries. The more widely accepted implementation would be a GET that streams the file followed by a DELETE call to remove the file.
But you can do it by implementing your own InputStream, see an earlier thread in Stackoverflow on deleting files on closing a InputStream.
Assuming that you are creating the file in the same controller.
You can use:
try (BufferedWriter out = Files
.newBufferedWriter(newFilePath, Charset.defaultCharset(),
StandardOpenOption.DELETE_ON_CLOSE)) {
InputStream in = new FileInputStream(newFilePath.toFile());
InputStreamResource inputStreamResource = new InputStreamResource(in);
httpHeaders.setContentLength(file.Length());
return new ResponseEntity(inputStreamResource, httpHeaders, HttpStatus.OK);
} catch (Exception e) {
}
As the BufferedWriter will close on return, the file will get deleted.
Based on #phlogratos's answer, you can try like this.
#GetMapping("/download")
public ResponseEntity<InputStreamResource> download() throws Exception {
... codes ...
InputStreamResource isr = new InputStreamResource(new FileInputStream(file) {
#Override
public void close() throws IOException {
super.close();
boolean isDeleted = file.delete();
logger.info("export:'{}':" + (isDeleted ? "deleted" : "preserved"), filename);
}
});
return new ResponseEntity<>(isr, respHeaders, HttpStatus.OK);
}
Extend FileInputStream with your own implementation and then overwrite close. When the input stream is closed, your file gets deleted as well.
public class MyFileInputStream extends FileInputStream {
private final File myFile;
public MyFileInputStream(File file) throws FileNotFoundException {
super(file);
myFile = file;
}
#Override
public void close() throws IOException {
super.close();
myFile.delete();
}
}
I have implemented filter and I have called getEntityStream of ContainerRequestContext and set the exact value back by using setEntitystream. If i use this filter then #FormParameter data becomes null and if i don't use filter then everything will be fine (as I am not calling getEntityStream) and i have to use filter to capture request data.
Note: I am getting form params from MultivaluedMap formParams but not from #FormParameter.
Environment :- Rest Easy API with Jboss Wildfly 8 server.
#Provider
#Priority(Priorities.LOGGING)
public class CustomLoggingFilter implements ContainerRequestFilter, ContainerResponseFilter{
final static Logger log = Logger.getLogger(CustomLoggingFilter.class);
#Context
private ResourceInfo resourceInfo;
#Override
public void filter(ContainerRequestContext requestContext)
throws IOException {
MDC.put("start-time", String.valueOf(System.currentTimeMillis()));
String entityParameter = readEntityStream(requestContext);
log.info("Entity Parameter :"+entityParameter);
}
private String readEntityStream(ContainerRequestContext requestContext){
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
final InputStream inputStream = requestContext.getEntityStream();
final StringBuilder builder = new StringBuilder();
int read=0;
final byte[] data = new byte[4096];
try {
while ((read = inputStream.read(data)) != -1) {
outStream.write(data, 0, read);
}
} catch (IOException e) {
e.printStackTrace();
}
byte[] requestEntity = outStream.toByteArray();
if (requestEntity.length == 0) {
builder.append("");
} else {
builder.append(new String(requestEntity));
}
requestContext.setEntityStream(new ByteArrayInputStream(requestEntity) );
return builder.toString();
}
return null;
}
}
class customResource
{
//// This code is not working
#POST
#Path("voiceCallBack")
#ApiOperation(value = "Voice call back from Twilio")
public void voiceCallback(#FormParam("param") String param)
{
log.info("param:" + param);
}
// This code is working
#POST
#Path("voiceCallBackMap")
#ApiOperation(value = "Voice call back from Twilio")
public void voiceCallbackMap(final MultivaluedMap<String, String> formParams)
{
String param = formParams.getFirst("param");
}
}
please suggest me solution & Thanks in Advance.
I found during run time that instance of the entity stream (from http request) is of type org.apache.catalina.connector.CoyoteInputStream (I am using jboss-as-7.1.1.Final). But we are setting entity stream with the instance of java.io.ByteArrayInputStream. So Resteasy is unable to bind individual formparmeters.
There are two solutions for this you can use any one of them :
Use this approach How to read JBoss Resteasy's servlet request twice while maintaing #FormParam binding?
Get form parameters like this:
#POST
#Path("voiceCallBackMap")
#ApiOperation(value = "Voice call back from Twilio")
public void voiceCallbackMap(final MultivaluedMap<String, String> formParams)
{
String param = formParams.getFirst("param");
}
As discussed in How to use Jersey interceptors to get request body, I am modifying the EntityInputStream in a ContainerRequestFilter.
public filter(ContainerRequest request){
ByteArrayOutputStream out = new ByteArrayOutputStream();
InputStream in = request.getEntityInputStream();
try{
Readerwriter.writeTo(in, out);
byte[] requestEntity = out.toByteArray();
// DO SOMETHING WITH BYTES HERE
request.setEntityInputStream(new ByteArrayInputStream(requestEntity));
}/// error handling code here
}
However, later on I can't figure out how to access the modified InputStream. I can get the ServletContext in the resource, but I can't figure out how to get ahold of the object I actually modified in the filter, the ContainerRequest.
Can I do something like this? Jersey can't start up out when I try this:
#Post
#Path("/test")
public Response test(#Context ContainerRequest cr){
// blah blah
return....
}
Jersey error:
Missing dependecy for method public javax.ws.rs.core.Response example.TestController.test(com.sun.jersey.spi.container.ContainerRequest), annotated with POST of resource, class example.TestController, is not recognized as a valid resource method.
I am stuck on an old version of jersey, 1.8, so I'm not sure if that's part of the problem.
All you need to do is accept an InputStream as the entity body in your resource method. If you want the ByteArrayInputStream just cast it.
#POST
public Response post(InputStream in) {
ByteArrayInputStream bin = (ByteArrayInputStream)in;
}
If you don't already know, how Jersey converts the request stream (for the request body) into Java types (for instance JSON to POJO) is through MessageBodyReaders. You can read more about them at JAX-RS Entity Providers.
Jersey already comes with some standard readers for easily convertible types, for instance String. Most content-types can be converted to String. Likewise, it has a reader to handle InputStream. This is probably the easiest conversion, as the request is already coming in as an InputStream, so really all the reader would need to do is return the original stream, and that's what would get passed to our method.
If we look at the implementation InputStreamProvider, we can see that that's what actually happens. The original stream is simply returned. And since the filter happens before the readers, the reader simply returns the stream that we set.
Here is a complete example using Jersey Test Framework
public class StreamFilterTest extends JerseyTest {
public static class InputStreamFilter implements ContainerRequestFilter {
#Override
public ContainerRequest filter(ContainerRequest request) {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
InputStream in = request.getEntityInputStream();
ReaderWriter.writeTo(in, out);
byte[] requestBytes = out.toByteArray();
byte[] worldBytes = " World".getBytes(StandardCharsets.UTF_8);
byte[] newBytes = new byte[requestBytes.length + worldBytes.length];
System.arraycopy(requestBytes, 0, newBytes, 0, requestBytes.length);
System.arraycopy(worldBytes, 0, newBytes, requestBytes.length, worldBytes.length);
request.setEntityInputStream(new ByteArrayInputStream(newBytes));
} catch (IOException ex) {
Logger.getLogger(InputStreamFilter.class.getName()).log(Level.SEVERE, null, ex);
throw new RuntimeException(ex);
}
return request;
}
}
#Path("stream")
public static class StreamResource {
#POST
public String post(InputStream in) throws Exception {
ByteArrayInputStream bin = (ByteArrayInputStream) in;
StringWriter writer = new StringWriter();
ReaderWriter.writeTo(new InputStreamReader(bin), writer);
return writer.toString();
}
}
public static class AppConfig extends DefaultResourceConfig {
public AppConfig() {
super(StreamResource.class);
getContainerRequestFilters().add(new InputStreamFilter());
}
}
#Override
public WebAppDescriptor configure() {
return new WebAppDescriptor.Builder()
.initParam(WebComponent.RESOURCE_CONFIG_CLASS,
AppConfig.class.getName())
.build();
}
#Test
public void should_return_hello_world() {
String response = resource().path("stream").post(String.class, "Hello");
assertEquals("Hello World", response);
}
}
Here's the test dependency
<dependency>
<groupId>com.sun.jersey.jersey-test-framework</groupId>
<artifactId>jersey-test-framework-grizzly2</artifactId>
<version>1.17.1</version>
<scope>test</scope>
</dependency>
How do I get the actual body of request I am about to do?
Invocation i = webTarget.path("somepath")
.request(MediaType.APPLICATION_JSON)
.buildPut(Entity.entity(account, MediaType.APPLICATION_JSON));
log.debug(i.... ); // I want to log the request
You could try to wrap the Outputstream for the Entity. First, by using a javax.ws.rs.client.ClientRequestFilter to add a custom Outputstream to the ClientRequestContext.
Client client = ClientBuilder.newClient().register(MyLoggingFilter.class);
public class MyLoggingOutputStreamWrapper extends OutputStream{
static final Logger logger = Logger.getLogger(...);
ByteArrayOutputStream myBuffer = new ...
private OutputStream target;
public MyLoggingOutputStreamWrapper(OutputStream target){ ...
// will be smarter to implement write(byte [], int, int) and call it from here
public void write(byte [] data){
myBuffer.write(data);
target.write(data);
}
... // other methods to delegate to target, especially the other write method
public void close(){
// not sure, if converting the buffer to a string is enough. may be in a different encoding than the platform default
logger.log(myBuffer.toString());
target.close();
}
}
#Provider
public class MyLoggingFilter implements ClientRequestFilter{
// implement the ClientRequestFilter.filter method
#Override
public void filter(ClientRequestContext requestContext) throws IOException {
requestContext.setEntityOutputstream(new MyLoggingOutputStreamWrapper(requestContext.getEntityOutputstream()));
}
I'm not sure at which point the outputstream is used to serialize the data. It could be at the moment you invoke buildPut(), but more likely it will be on the fly at access of the webclient.
Another approach would be getting the underlying HttpClient and registering some listener there to get the body.
I had a similar problem. I couldn't use the Jersey LoggingFilter (and the new LoggingFeature in 2.23) because I needed to customize the output. For using the other options you can see this post: Jersey: Print the actual request
I've simplified what I did for brevity. It is pretty similar to the original answer, but I adapted the Jersey LoggingStream (it is an internal class you can't access) and took out the ability to log up to a maximum size.
You have a class that extends the OutputStream so you can capture the entity in it. It will write to your OutputStream as well as the original.
public class MyLoggingStream extends FilterOutputStream
{
private final ByteArrayOutputStream baos = new ByteArrayOutputStream();
public MyLoggingStream(final OutputStream inner)
{
super(inner);
}
public String getString(final Charset charset)
{
final byte[] entity = baos.toByteArray();
return new String(entity, charset);
}
#Override
public void write(final int i) throws IOException
{
baos.write(i);
out.write(i);
}
}
Then you have a filter class. It was important for my use case that I was able to grab the entity and log it separately (I've put it as println below for simplicity). In Jersey's LoggingFilter and LoggingFeature the entity gets logged by the Interceptor, so you can't capture it.
#Provider
public class MyLoggingClientFilter implements ClientRequestFilter, ClientResponseFilter, WriterInterceptor
{
protected static String HTTPCLIENT_START_TIME = "my-http-starttime";
protected static String HTTPCLIENT_LOG_STREAM = "my-http-logging-stream";
#Context
private ResourceInfo resourceInfo;
public void filter(final ClientRequestContext requestContext) throws IOException
{
requestContext.setProperty(HTTPCLIENT_START_TIME, System.nanoTime());
final OutputStream stream = new MyLoggingStream(requestContext.getEntityStream());
requestContext.setEntityStream(stream);
requestContext.setProperty(HTTPCLIENT_LOG_STREAM, stream);
}
public void filter(final ClientRequestContext requestContext, final ClientResponseContext responseContext)
{
StringBuilder builder = new StringBuilder("--------------------------").append(System.lineSeparator());
long startTime = (long)requestContext.getProperty(HTTPCLIENT_START_TIME);
final double duration = (System.nanoTime() - startTime) / 1_000_000.0;
builder.append("Response Time: ").append(duration);
if(requestContext.hasEntity())
{
final MyLoggingStream stream = (MyLoggingStream)requestContext.getProperty(HTTPCLIENT_LOG_STREAM);
String body = stream.getString(MessageUtils.getCharset(requestContext.getMediaType()));
builder.append(System.lineSeparator()).append("Entity: ").append(body);
}
builder.append(System.lineSeparator()).append("--------------------------");
System.out.println(builder.toString());
requestContext.removeProperty(HTTPCLIENT_START_TIME);
requestContext.removeProperty(HTTPCLIENT_LOG_STREAM);
}
#Override
public void aroundWriteTo(WriterInterceptorContext context) throws IOException, WebApplicationException
{
// This forces the data to be written to the output stream
context.proceed();
}
}