Develop Reference

How to handle compressed (gzip) HTTP requests (NOT response) in JAVA servlet - Simple Example?

I struggled with this problem for quite some time; and after finding a simple solution... wanted to ask a question & answer!!
The question has been asked in different ways multiple times on stack overflow, and the accepted solutions are either partially correct and complex or talk about response compression.
Aggregating some old Q&A on this topic:
wrong accepted ans/partially correct & complex
subsequent ans.
HTTP request compression
How to decode Gzip compressed request body in Spring MVC: (Almost correct 2nd ans, but complex overkill)
The accepted ans is wrong. It's about RESPONSE compression and not REQUEST.
Can I compress HTTP Requests using GZIP?
Compress an HTTP Request in java
Similar Questions - terminated at "NO request compression"
Does Java HTTP Client handle compression
A specific question & ans for Spring RestTemplate framework: How to zip- compress HTTP request with Spring RestTemplate?

A simple solution is by using a filter. (See servlet-filter tutorial)
Create a Servlet Filter:
Make sure that the filter is called either first/before any filters which use request body.
I. Register filter in web.xml:
<filter>
<filter-name>GzipRequestFilter</filter-name>
<filter-class>com...pkg...GzipRequestFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>GzipRequestFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
II. Code for filter class:
public class GzipRequestFilter implements Filter {
// Optional but recommended.
private static final Set<String> METHODS_TO_IGNORE = ImmutableSet.of("GET", "OPTIONS", "HEAD");
#Override
public void doFilter(
final ServletRequest request,
final ServletResponse response,
final FilterChain chain) throws IOException, ServletException {
HttpServletRequest httpServletRequest = (HttpServletRequest) request;
String method = httpServletRequest.getMethod().toUpperCase();
String encoding = Strings.nullToEmpty(
httpServletRequest.getHeader(HttpHeaders.CONTENT_ENCODING));
if (METHODS_TO_IGNORE.contains(method) || !encoding.contains("application/gzip")) {
chain.doFilter(request, response); // pass through
return;
}
HttpServletRequestWrapper requestInflated = new GzippedInputStreamWrapper(httpServletRequest);
chain.doFilter(requestInflated, response);
}
#Override
public void init(final FilterConfig filterConfig) throws ServletException {}
#Override
public void destroy() {}
}
III. Followed by code for GzipInputStream wrapper:
// Simple Wrapper class to inflate body of a gzipped HttpServletRequest.
final class GzippedInputStreamWrapper extends HttpServletRequestWrapper {
private GZIPInputStream inputStream;
GzippedInputStreamWrapper(final HttpServletRequest request) throws IOException {
super(request);
inputStream = new GZIPInputStream(request.getInputStream());
}
#Override
public ServletInputStream getInputStream() throws IOException {
return new ServletInputStream() {
// NOTE: Later versions of javax.servlet library may require more overrides.
public int read() throws IOException {
return inputStream.read();
}
public void close() throws IOException {
super.close();
inputStream.close();
}
};
}
#Override
public BufferedReader getReader() throws IOException {
return new BufferedReader(new InputStreamReader(inputStream));
}
}
Now what remains is how to send a compressed request?
Postman does not yet support sending compressed HttpRequest bodies. You can still make it work by using the binary option and use a gzipped file containing the properly encoded request body.
One way is using a nodejs script with pako compression library. For a multipart/form-data request see form-data library
const pako = require('pako')
const axios = require('axios')
var params = qs.stringify({
'num': 42,
'str': 'A string param',
});
data = pako.gzip(Buffer.from(params));
var config = {
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Encoding': 'application/gzip';
},
}
await axios.post(
'http://url-for-post-api-accepting-urlencoded',
data,
config,
).then((res) => {
console.log(`status: ${res.status} | data: ${res.data}`)
}).catch((error) => {
console.error(error)
})
NOTES:
We are using Content-Encoding: application/gzip header to specify that a request is compressed. Yes this is standard.
Do not use Content-Type as it will not work with multipart/form-data.
The HTTP protocol has been running under the assumption that size of HttpRequests are dwarfed by HttpResponses.
Further, due to assumed limited computing power in browser/client side, the norm has been to compress response and not requests. Browsers cannot natively compress but can do decompression natively.
But, unfortunately after years of many developers pushing code; some HTTP APIs evolve to consume large strings/data!!
It's a piece of cake to allow java servlets to have the option of working with compressed requests.

Passing parameters between JAX-RS WriterInterceptor and ReaderInterceptor

I'm using JAX-RS and within the WriterInterceptor, I need to access some information contained in the original request.
As an example, consider the below request body.
{
"ClientId": "MY_CLIENT_ID",
"UserId": "MY_USER_ID",
"AccountId": "MY_ACCOUNT_ID",
"Scope" : "MY_SCOPES",
}
Within my WriteInterceptor, I need to read Client ID and User ID from the request and add those values to the response.
I am currently working on a ReadInterceptor implementation for this. I initially assumed there is a way to put parameters to ReaderInterceptorContext and then read it somehow from the WriterInterceptorContext. But It seems there is no way to do that. ( Please correct me if I'm wrong).
So, now I'm trying to use a concurrent hashmap to store these parameters in the ReaderInterceptor and retrieve it in the WriteInterceptor. I need a unique key to create the correlation between request and response. Is it ok to use the thread ID for this?
Please point me if there is a better approach to resolve this problem

I resolved this problem by adding a container response filter which can add a header to the response. Read interceptor reads required parameters from the request and set those as context properties.
#Override
public Object aroundReadFrom(ReaderInterceptorContext readerInterceptorContext)
throws IOException, WebApplicationException {
InputStream is = readerInterceptorContext.getInputStream();
String requestBody = new Scanner(is, StandardCharsets.UTF_8.name()).useDelimiter("\\A").next();
JSONObject request = new JSONObject(requestBody);
//Adding the stream back to the context object
readerInterceptorContext.setInputStream(new ByteArrayInputStream(requestBody.getBytes()));
//Adding properties to read in filter
readerInterceptorContext.setProperty("ClientId", request.get("ClientId"));
readerInterceptorContext.setProperty("UserId","UserId"));
return readerInterceptorContext.proceed();
}
These properties are then read inside the container response filter and added as a response header.
#Override
public void filter(ContainerRequestContext containerReqContext, ContainerResponseContext containerResponseContext) {
//Adding temporary headers to read in WriterInterceptor
containerResponseContext.getHeaders().add(
"ClientId", containerReqContext.getProperty("ClientId"));
containerResponseContext.getHeaders().add(
"UserId", containerReqContext.getProperty("UserId"));
}
Existing writer interceptor read these headers, add those to JWT and then remove as header values. I did a POC for this and it is working as expected
#Override
public void aroundWriteTo(WriterInterceptorContext writerInterceptorContext) throws IOException {
OutputStream outputStream = writerInterceptorContext.getOutputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
writerInterceptorContext.setOutputStream(baos);
String clientId = writerInterceptorContext.getHeaders().getFirst("ClientId").toString();
String user = writerInterceptorContext.getHeaders().getFirst("UserId").toString();
}

Spring Security: deserialize request body twice (oauth2 processing)

This question is a result of some work I'm doing with the Spring Security Oauth2 library. I've set up an oauth2 authorization server and an oauth2 resource server, the latter of which is meant to authorize based on access tokens.
The problem is that normally access tokens are passed in a header, but the big client we're setting this up for wants to pass the access token in a JSON request body. There's an interface you can use to set up custom access token extraction, but it looks like this:
public interface TokenExtractor {
/**
* Extract a token value from an incoming request without authentication.
*
* #param request the current ServletRequest
* #return an authentication token whose principal is an access token (or null if there is none)
*/
Authentication extract(HttpServletRequest request);
}
So, as best I can tell, all I have access to is the raw HTTPServletRequest, from which I need to deserialize the request and extract the access token.
Further complicating things, though, is the fact that the request body also contains other parameters needed for processing, so I want to deserialize it to a DTO class that I pass into my controller, something like so:
#RequestMapping("/oauth/someresource")
#Transactional
public Map<String, String> resource(#AuthenticationPrincipal UserDetails userDetails,
#RequestBody ClientRequestDto clientRequestDto) {
// Do some processing based on the request dto
}
I tried manually deserializing the request in the token extractor, but then I get an error "java.lang.IllegalStateException: getReader() has already been called for this request".
I was brainstorming a few possible solutions that I could research, and so far I've come up with:
find a way to reset the input stream
deserialize the object in the Token Extractor, attach it to the raw request object, and just access the raw request object in my controller instead of using #RequestBody
like 2, but find a way to add a custom deserializer that fetches the object attached to the raw request instead of processing the request's input stream.
Anyways, those are just some thoughts, if anyone has any ideas in terms of an elegant way of solving this, I'd greatly appreciate it.
EDIT: I did find this question which is similar: Spring reading request body twice, and the last answer did have one possible solution (creating a decorator request class that allows multiple input stream reads and creating a filter early on in the filter chain that wraps the HttpServletRequest). It seems workable, but a little heavy duty, so I'll leave this up to see if anyone has any other ideas as well.

So I ended up finding yet another question that addressed this issue that I didn't see before posting (How can I read request body multiple times in Spring 'HandlerMethodArgumentResolver'?). That one also suggested creating a decorator around the HttpServletRequest, so I adapted the info from http://www.myjavarecipes.com/how-to-read-post-request-data-twice-in-spring/, adding a protection against large requests.
Here's what I came up with, in case anyone has any feedback:
public class MultiReadHttpServletRequest extends HttpServletRequestWrapper {
// We include a max byte size to protect against malicious requests, since this all has to be read into memory
public static final Integer MAX_BYTE_SIZE = 1_048_576; // 1 MB
private String _body;
public MultiReadHttpServletRequest(HttpServletRequest request) throws IOException {
super(request);
_body = "";
InputStream bounded = new BoundedInputStream(request.getInputStream(), MAX_BYTE_SIZE);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(bounded));
String line;
while ((line = bufferedReader.readLine()) != null){
_body += line;
}
}
#Override
public ServletInputStream getInputStream() throws IOException {
final ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(_body.getBytes());
return new ServletInputStream() {
public int read() throws IOException {
return byteArrayInputStream.read();
}
#Override
public boolean isFinished() {
return byteArrayInputStream.available() == 0;
}
#Override
public boolean isReady() {
return true;
}
#Override
public void setReadListener(ReadListener readListener) {
}
};
}
#Override
public BufferedReader getReader() throws IOException {
return new BufferedReader(new InputStreamReader(this.getInputStream()));
}
}
I used the following configuration:
#Bean
FilterRegistrationBean multiReadFilter() {
FilterRegistrationBean registrationBean = new FilterRegistrationBean();
MultiReadRequestFilter multiReadRequestFilter = new MultiReadRequestFilter();
registrationBean.setFilter(multiReadRequestFilter);
registrationBean.setOrder(SecurityProperties.DEFAULT_FILTER_ORDER - 2);
registrationBean.setUrlPatterns(Sets.newHashSet("/path/here"));
return registrationBean;
}

Send Spring response stream immediately rather than wait until method return

I have a request mapping that looks like this:
private final static byte[] byteArray = ...;
#RequestMapping(value=Array("/foobar"))
void sendByteArray(#RequestBody Request request, OutputStream os) {
os.write(byteArray);
os.flush();
doLengthyCleanup();
}
I'm finding that the request client does not actually receive the response body until after the service has completed doLengthyCleanup().
Since the cleanup doesn't affect the response itself, I'd like to improve my response time by performing the cleanup after sending the response. How can I do this?

#RequestMapping(value=Array("/foobar"))
void sendByteArray(#RequestBody Request request, OutputStream os) {
os.write(byteArray);
os.flush(); // not sure
doLengthyCleanup();
}
#Async
void doLengthyCleanup() {
// this will be executed asynchronously
}
Update: taken from this question
If you are calling the #Async method from another method in the same class, unless you enable AspectJ proxy mode for the #EnableAsync (and provide a weaver of course) that won't work (google "proxy self-invocation"). The easiest fix is to put the #Async method in another #Bean.

As shown in this answer, you need to indicate to Spring that you are handling the response yourself by accepting the response directly and setting the status code yourself:
void sendByteArray(#RequestBody Request request, HttpServletResponse response) {
response.setStatus(HttpStatus.SC_OK);
OutputStream os = response.getOutputStream();
os.write(byteArray);
os.flush();
os.close();
doLengthyCleanup();
}

Need help with creating custom HttpServletResponse

Suddenly stuck on generating custom servlet response. I want to replace servlet response with predefined one:
public class MyCustomResponse extends HttpServletResponseWrapper {
private String customOutput;
public MyCustomResponse(String customOutput, HttpServletResponse response) {
super(response);
// PrintWriter and Outputstream should stream this variable as output
this.customOutput = customOutput;
}
//
// Below I need to override something
//
}
and filter code snipped as follows:
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
//
//
MyCustomResponse customResponse = new MyCustomResponse("Hello world!", (HttpServletResponse) response);
chain.doFilter(request, customResponse);
}
Shame on me, but i'm really stuck on coding this simple task :(
Any help would be appreciated.
UPDATE:
All I want is to implement custom response wrapper which, once it's put into filter chain, would always respond with some predefined text. I know how to write custom data from within doFilter() method, but I want MyCustomResponse to be responsible for that - just instantiate and put in chain. Any well-reasoned responses "You cant do that because..." are also welcome.

As quoted in one of your comments :
"I want my custom response to return a
string in response to getWriter or
getOutputStream method invocation"
For that, you have to provide your own implementation for getWriter() & getOutputStream() by overriding them.
//---
private PrintWriter printWriter = null;
private ServletOutputStream outputStream = null;
public PrintWriter getWriter( ) throws IOException {
if (this.outputStream != null) {
throw new IllegalStateException(
"Cannot call getWriter( ) after getOutputStream( )");
}
if (this.printWriter == null) {
// initialize printWriter
}
return this.printWriter;
}
public ServletOutputStream getOutputStream( ) throws IOException {
if (this.printWriter != null) {
throw new IllegalStateException(
"Cannot call getOutputStream( ) after getWriter( )");
}
if (this.outputStream == null) {
// initialize outputStream
}
return this.outputStream;
}
//---

I am sorry, but
it is not clear what is your
problem. You code is written, so? It
does not work? what exactly does not work?
Why do you want to do this? The "right" solution is to pass information as session attribute.
I do not believe this can work. Really, you do not call directly the next filter in chain. You are kindly asking the app. server to do this. And you are not expected to replace the servlet request/response by your own. Use method explained above (#2)

Your response wrapper is useless as is, since it only stores a string in the Java object used to model the actual HTTP response.
The actual HTTP response that the client receives is the stream of bytes (resp. characters) sent via the output stream (resp. writer) of the HttpServletResponse object (and the headers, cookies, etc. stored in the HttpServletResponse object).
If you want to send a custom output string to the client, just use response.getWriter().print("Hello worlds!").
Passing the response to the rest of the filter chain is questionable, since the rest of the chain will probably want to add its own data to the response stream.
If you want to hard-code the response to send to the client to your custom output, but be able to still pass the response to the chain and ignore whatever the rest of the chain puts in the response, you could try to add the following to your wrapper :
private ServletOutputStream fakeOutputStream =
new ServletOutputStream() {
#Override
public void write(int b) throws IOException {
// do nothing. Everything written to this stream is ignored
}
}
private PrintWriter fakeWriter = new PrintWriter(fakeOutputStream);
public MyCustomResponse(String customOutput, HttpServletResponse response) {
super(response);
response.getWriter().print(customOutput);
}
#Override
public ServletOutputStream getOutputStream() {
return fakeOutputStream;
}
#Override
public PrintWriter getWriter() {
return fakeWriter;
}

I don't see the reason of what you want to do, but if you want to use your wrapper, my suggestion would be:
Create your own servlet that uses your wrapper and register it in web.xml, in something like this:
Extend javax.servlet.GenericServlet and override the service(ServletRequest, ServletResponse) method. Then you use the Template Method pattern to create a service(ServletRequest, ServletResponseWrapper). OR
Extend javax.servlet.HttpServlet and override service(HttpServletRequest, HttpServletResponse) method. Use the Template Method pattern to create a service(HttpServletRequest, HttpServletResponseWrapper). This will require that you don't use the doGet, doPost, doPut, doTrace methods already provided by HttpServlet but, instead create your own that uses your wrapper.
Hope this helps.