Develop Reference

How to handle compressed (gzip) HTTP requests (NOT response) in JAVA servlet - Simple Example?

I struggled with this problem for quite some time; and after finding a simple solution... wanted to ask a question & answer!!
The question has been asked in different ways multiple times on stack overflow, and the accepted solutions are either partially correct and complex or talk about response compression.
Aggregating some old Q&A on this topic:
wrong accepted ans/partially correct & complex
subsequent ans.
HTTP request compression
How to decode Gzip compressed request body in Spring MVC: (Almost correct 2nd ans, but complex overkill)
The accepted ans is wrong. It's about RESPONSE compression and not REQUEST.
Can I compress HTTP Requests using GZIP?
Compress an HTTP Request in java
Similar Questions - terminated at "NO request compression"
Does Java HTTP Client handle compression
A specific question & ans for Spring RestTemplate framework: How to zip- compress HTTP request with Spring RestTemplate?

A simple solution is by using a filter. (See servlet-filter tutorial)
Create a Servlet Filter:
Make sure that the filter is called either first/before any filters which use request body.
I. Register filter in web.xml:
<filter>
<filter-name>GzipRequestFilter</filter-name>
<filter-class>com...pkg...GzipRequestFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>GzipRequestFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
II. Code for filter class:
public class GzipRequestFilter implements Filter {
// Optional but recommended.
private static final Set<String> METHODS_TO_IGNORE = ImmutableSet.of("GET", "OPTIONS", "HEAD");
#Override
public void doFilter(
final ServletRequest request,
final ServletResponse response,
final FilterChain chain) throws IOException, ServletException {
HttpServletRequest httpServletRequest = (HttpServletRequest) request;
String method = httpServletRequest.getMethod().toUpperCase();
String encoding = Strings.nullToEmpty(
httpServletRequest.getHeader(HttpHeaders.CONTENT_ENCODING));
if (METHODS_TO_IGNORE.contains(method) || !encoding.contains("application/gzip")) {
chain.doFilter(request, response); // pass through
return;
}
HttpServletRequestWrapper requestInflated = new GzippedInputStreamWrapper(httpServletRequest);
chain.doFilter(requestInflated, response);
}
#Override
public void init(final FilterConfig filterConfig) throws ServletException {}
#Override
public void destroy() {}
}
III. Followed by code for GzipInputStream wrapper:
// Simple Wrapper class to inflate body of a gzipped HttpServletRequest.
final class GzippedInputStreamWrapper extends HttpServletRequestWrapper {
private GZIPInputStream inputStream;
GzippedInputStreamWrapper(final HttpServletRequest request) throws IOException {
super(request);
inputStream = new GZIPInputStream(request.getInputStream());
}
#Override
public ServletInputStream getInputStream() throws IOException {
return new ServletInputStream() {
// NOTE: Later versions of javax.servlet library may require more overrides.
public int read() throws IOException {
return inputStream.read();
}
public void close() throws IOException {
super.close();
inputStream.close();
}
};
}
#Override
public BufferedReader getReader() throws IOException {
return new BufferedReader(new InputStreamReader(inputStream));
}
}
Now what remains is how to send a compressed request?
Postman does not yet support sending compressed HttpRequest bodies. You can still make it work by using the binary option and use a gzipped file containing the properly encoded request body.
One way is using a nodejs script with pako compression library. For a multipart/form-data request see form-data library
const pako = require('pako')
const axios = require('axios')
var params = qs.stringify({
'num': 42,
'str': 'A string param',
});
data = pako.gzip(Buffer.from(params));
var config = {
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Encoding': 'application/gzip';
},
}
await axios.post(
'http://url-for-post-api-accepting-urlencoded',
data,
config,
).then((res) => {
console.log(`status: ${res.status} | data: ${res.data}`)
}).catch((error) => {
console.error(error)
})
NOTES:
We are using Content-Encoding: application/gzip header to specify that a request is compressed. Yes this is standard.
Do not use Content-Type as it will not work with multipart/form-data.
The HTTP protocol has been running under the assumption that size of HttpRequests are dwarfed by HttpResponses.
Further, due to assumed limited computing power in browser/client side, the norm has been to compress response and not requests. Browsers cannot natively compress but can do decompression natively.
But, unfortunately after years of many developers pushing code; some HTTP APIs evolve to consume large strings/data!!
It's a piece of cake to allow java servlets to have the option of working with compressed requests.

Spring data rest sorting fields with underscores

We are using a very simple setup of #RepositoryRestResource on top of a PagingAndSortingRepository connected to a postgres database. Also we have configured spring.jackson.property-naming-strategy=SNAKE_CASE to return pretty json. It was all fine and dandy until we started sorting. As we have discovered - sorting requires us to provide the actual class field names (which we of course have in camel case):
get("/thing?sort=dateCreated,desc")
And when we try to do javascript friendly
get("/thing?sort=date_created,desc")
it fails miserably because jpa tries to split the parameter by the underscore.
Is there a simple way to have the path params the same format as we have them in the json that we are returning?

There is a bug for this - DATAREST-883. It was fixed and released. But then, due to regressions (DATAREST-909) this has been dropped in the very next release. I asked them on Github if they plan to have this again as this has bitten me in the past as well. We'll see what they have to say about this.
For now you can:
leave it be
go with the camel case property names
work around this (e.g. go with Alan Haye's answer) - this seems fragile IMHO, but probably will do in a short term.
The status of the feature in the spring-boot versions I've tested with:
1.4.0 (spring-data-rest 2.5.2): not yet implemented
1.4.1 (spring-data-rest 2.5.3): works -> code
1.4.2 (spring-data-rest 2.5.5): dropped

It is unclear whether you can do this in some Spring Data Rest specific way however you should be able to handle it by means of a standard Servlet filter which would look something like the below:
public class SortParameterConversionFilter extends GenericFilterBean {
// as we are extending Spring's GenericFilterBean
// you can then *possibly* inject the RepositoryRestConfiguration
// and use RepositoryRestConfiguration#getSortParamName
// to avoid hard coding
private static final String SORT_PARAM_KEY = "sort";
#Override
public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest request = (HttpServletRequest) req;
if (shouldApply(request)) {
chain.doFilter(new CollectionResourceRequestWrapper(request), res);
} else {
chain.doFilter(req, res);
}
}
/**
*
* #param request
* #return True if this filter should be applied for this request, otherwise
* false.
*/
protected boolean shouldApply(HttpServletRequest request) {
return request.getServletPath().matches("some-pattern");
}
/**
* HttpServletRequestWrapper implementation which allows us to wrap and
* modify the incoming request.
*
*/
public class CollectionResourceRequestWrapper extends HttpServletRequestWrapper {
public ResourceRequestWrapper(HttpServletRequest request) {
super(request);
}
#Override
public String getParameter(final String name) {
if (name.equals(SORT_PARAM_KEY)) {
String [] parts = super.getParameter(SORT_PARAM_KEY).split(",");
StringBuilder builder = new StringBuilder();
int index = 0;
for (String part : parts) {
// using some mechanism of you choosing
// convert from underscore to camelCase
// Using the Guava library for example
String convertedPart = CaseFormat.LOWER_UNDERSCORE.to(
CaseFormat.LOWER_CAMEL, part);
++index;
builder.append(convertedPart).append(index < parts.length ? "," : "");
}
return builder.toString();
}
return super.getParameter(name);
}
}
}

How to parse a URL and run a method with Spring MVC 'reflectively'?

I have a Spring Boot application that uses Spring MVC in the usual manner, with a bunch of #RequestMapping methods, Freemarker definitions, and the like. This is all tied together with a WebMvcConfigurerAdapter class.
I'd like to provide a service where the user submits a list of valid URLs, and the webapp would work out which controller would be called, passes in the parameters, and returns a combined result for every URL — all in one request.
This would save the user from having to make hundreds of HTTP calls, but would still allow them to make one-off requests if need be. Ideally, I'd just inject an auto-configured Spring bean, so I don't have to repeat the URL resolving and adapting and handling that Spring does internally, and the controller's list of other controllers would never go out of sync with the real list of controllers.
I expected to write something like this (simplified to only deal with one URL, which is pointless but easier to understand):
#Autowired BeanThatSolvesAllMyProblems allMappings;
#PostMapping(path = "/encode", consumes = MediaType.TEXT_PLAIN_VALUE)
#ResponseBody
public String encode(#RequestBody String inputPath) {
if (allMappings.hasMappingForPath(inputPath)) {
return allMappings.getMapping(inputPath).execute();
} else {
return "URL didn't match, sorry";
}
}
Instead, I've had to define Spring beans I don't know what they do and have been repeating some of what Spring is meant to do for me, which I'm worried won't work quite the same as it would if the user just made the call themselves:
// these two are #Beans, with just their default constructor called.
#Autowired RequestMappingHandlerMapping handlers;
#Autowired RequestMappingHandlerAdapter adapter;
#PostMapping(path = "/encode", consumes = MediaType.TEXT_PLAIN_VALUE)
#ResponseBody
public String encode(#RequestBody String inputText) {
final HttpServletRequest mockRequest = new MockHttpServletRequest(null, inputText);
final StringBuilder result = new StringBuilder();
this.handlers.getHandlerMethods().forEach((requestMappingInfo, handlerMethod) -> {
if (requestMappingInfo.getPatternsCondition().getMatchingCondition(mockRequest) != null) {
try {
final MockHttpServletResponse mockResponse = new MockHttpServletResponse();
result.append("Result: ").append(adapter.handle(mockRequest, mockResponse, handlerMethod));
result.append(", ").append(mockResponse.getContentAsString());
result.append("\n");
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
});
return result.toString();
}
I thought I was doing quite well going down this path, but it's failing with Missing URI template variable errors, and not only do I have no idea how to put the request parameters in (another thing which Spring could be able to handle itself), but I'm not even sure that this is the right way to go about doing this. So how do I simulate a Spring MVC request "reflectively", from within the webapp itself?

JSON API spec. solves this problem by allowing sending multiple operations per request. There even exists a quite mature implementation that supports this feature which is called Elide. But I guess this is might not fully meet your requirements.
Anyway, here's what you can do.
You have to take into consideration that DispatcherServlet holds handlerMappings list that is used to detect appropriate request handler and handlerAdaptors. The selection strategy for both lists is configurable (see DispatcherServlet#initHandlerMappings and #initHandlerAdapters).
You should work out a way you would prefer to retrieve this lists of handlerMappings/initHandlerAdapters and stay in sync with DispatcherServlet.
After that you can implement your own HandlerMapping/HandlerAdaptor (or present a Controller method as in your example) that would handle the request to /encode path.
Btw, HandlerMapping as javadoc says is
Interface to be implemented by objects that define a mapping between
requests and handler objects
or simply saying if we take DefaultAnnotationHandlerMapping that would map our HttpServletRequests to #Controller methods annotated with #RequestMapping. Having this mapping HandlerAdapter prepares incoming request to consuming controller method, f.ex. extracting request params, body and using them to call controller's method.
Having this, you can extract URLs from main request, create a list of stub HttpRequests holding the information needed for further processing and loop through them calling this:
HandlerExecutionChain getHandler(HttpServletRequest request) throws Exception {
for (HandlerMapping hm : this.handlerMappings) {
if (logger.isTraceEnabled()) {
logger.trace(
"Testing handler map [" + hm + "] in DispatcherServlet with name '" + getServletName() + "'");
}
HandlerExecutionChain handler = hm.getHandler(request);
if (handler != null) {
return handler;
}
}
return null;
}
having a handlerMapping you call
HandlerAdapter getHandlerAdapter(Object handler) throws ServletException {
for (HandlerAdapter ha : this.handlerAdapters) {
if (logger.isTraceEnabled()) {
logger.trace("Testing handler adapter [" + ha + "]");
}
if (ha.supports(handler)) {
return ha;
}
}
and then you can finally call
ha.handle(processedRequest, response, mappedHandler.getHandler());
which in turn would execute the controller's method with params.
But having all this, I would not recommend to following this approach, instead, think about usage of JSON API spec or any other.

How about using Springs RestTemplate as client for this? You could call your controllers within the spring controller as if it would be an external resource:
#ResponseBody
public List<String> encode(#RequestBody List inputPaths) {
List<String> response = new ArrayList<>(inputPaths.size());
for (Object inputPathObj : inputPaths) {
String inputPath = (String) inputPathObj;
try {
RequestEntity.BodyBuilder requestBodyBuilder = RequestEntity.method(HttpMethod.GET, new URI(inputPath)); // change to appropriate HttpMethod, maybe some mapping?
// add headers and stuff....
final RequestEntity<Void> requestEntity = requestBodyBuilder.build(); // when you have a request body change Void to e.g. String
ResponseEntity<String> responseEntity = null;
try {
responseEntity = restTemplate.exchange(requestEntity, String.class);
} catch (final HttpClientErrorException ex) {
// add your exception handling here, e.g.
responseEntity = new ResponseEntity<>(ex.getResponseHeaders(), ex.getStatusCode());
throw ex;
} finally {
response.add(responseEntity.getBody());
}
} catch (URISyntaxException e) {
// exception handling here
}
}
return response;
}
Note that generic do not work for the #RequestBody inputPaths.
See alse http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/client/RestTemplate.html and https://spring.io/guides/gs/consuming-rest/ .

I agree with the other answers that you should consider this feature outside of your project, instead of having it in the code. It is a question of design and you can choose the approach you want. Based on your comment that these are GET requests, you can achieve what you want with a request dispatcher to trigger your requests within your special Controller service method for each URL and capture the response with a HttpServletResponseWrapper instance.
In the following code sample, the "consolidate" method takes comma separated URLs like this ("http://localhost:8080/index/index1,index2", here "index1,index2" is the URL list), consolidates their text output into a single payload and returns it. For this example URL, the consolidated outputs of http://localhost:8080/index1 and http://localhost:8080/index2 will be returned. You might want to extend/modify this with added parameters, validation, etc for the URLs. I tested this code with Spring Boot 1.2.x.
#Controller
public class MyController {
#RequestMapping("/index/{urls}")
#ResponseBody
String consolidate(#PathVariable String[] urls, HttpServletRequest request, HttpServletResponse response) {
StringBuilder responseBody = new StringBuilder();
//iterate for each URL provided
for (String url : urls) {
RequestDispatcher dispatcher = request.getServletContext().getRequestDispatcher("/" + url);
HttpServletResponseWrapper wrapper = new HttpServletResponseWrapper((HttpServletResponse) response) {
private CharArrayWriter output = new CharArrayWriter();
#Override
public PrintWriter getWriter() {
return new PrintWriter(output);
}
#Override
public String toString() {
return output.toString();
}
};
try {
dispatcher.include(request, wrapper);
//append the response text
responseBody.append(wrapper.toString());
} catch (ServletException | IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
//This holds the consolidated output
return responseBody.toString();
}
#RequestMapping("/index1")
String index1() {
return "index1";
}
#RequestMapping("/index2")
String index2() {
return "index2";
}
}

encodeRedirectURL changes some characters

I have an Interceptor on Struts2, and I want for some pages to redirect to the ssl version of them.
Example: http://localhost/xhtml/path.do?ossesionid=value1 to https://localhost/xhtml/path.do?ossesionid=value1
For doing this I created a Interceptor that does this:
public String intercept(ActionInvocation invocation) throws Exception {
// initialize request and response
final ActionContext context = invocation.getInvocationContext();
final HttpServletRequest request = (HttpServletRequest) context
.get(StrutsStatics.HTTP_REQUEST);
final HttpServletResponse response = (HttpServletResponse) context
.get(StrutsStatics.HTTP_RESPONSE);
// check scheme
String scheme = request.getScheme().toLowerCase();
// check method
String method = request.getMethod().toUpperCase();
// If the action class uses the SSLProtected marker annotation, then see
// if we need to
// redirect to the SSL protected version of this page
if (invocation.getAction() instanceof SSLProtected) {
if (HTTP_GET.equals(method) && SCHEME_HTTP.equals(scheme)) {
// initialize https port
String httpsPortParam = request.getSession().getServletContext().getInitParameter(HTTP_PORT_PARAM);
int httpsPort = httpsPortParam == null ? HTTPS_PORT : Integer.parseInt(httpsPortParam);
response.setCharacterEncoding("UTF-8");
URI uri = new URI(SCHEME_HTTPS, null, request.getServerName(), httpsPort, response.encodeRedirectURL(request.getRequestURI()), request.getQueryString(), null);
log.debug("Going to SSL mode, redirecting to " + uri.toString());
response.sendRedirect(uri.toString());
return null;
}
}
My problem is that I expect this
https://localhost/xhtml/path.do?ossesionid=value1
and got
https://localhost/xhtml/path.do;jsessionid=value1?osessionid=value1
And I'm Completly lost! help anyone?

i strongly suggest you to use S2-SSL plugin which is more flexible and provides a much better support to handle switch from SSL to non-SSL and vice-versa.
regarding generation of Jsessionid,JSESSIONID cookie is created/sent when session is created. Session is created when your code calls request.getSession() or request.getSession(true) for the first time. If you just want get session.You have ways to disable the creation of Jsessionid
There are number of way you can disable the creation of this id, please refer to this discussion thread.
I am still not sure what is the problem you are facing with this session-id as it is a very common case in web applications
is-it-possible-to-disable-jsessionid-in-tomcat-servlet

Use GZIP, JSON responses and JQuery

However, I want to compress my responses with GZIP wheren possible. I tried using the Compression filter code available for free download in the headfirst site. It works great for html, images, css and javascript.
I post the filter next. It checks if GZIP is an accepted encoding and it adds gzip as Content-Encoding. See: wrappedResp.setHeader("Content-Encoding", "gzip");
public class CompressionFilter implements Filter {
private ServletContext ctx;
private FilterConfig cfg;
/**
* The init method saves the config object and a quick reference to the
* servlet context object (for logging purposes).
*/
public void init(FilterConfig cfg)
throws ServletException {
this.cfg = cfg;
ctx = cfg.getServletContext();
//ctx.log(cfg.getFilterName() + " initialized.");
}
/**
* The heart of this filter wraps the response object with a Decorator
* that wraps the output stream with a compression I/O stream.
* Compression of the output stream is only performed if and only if
* the client includes an Accept-Encoding header (specifically, for gzip).
*/
public void doFilter(ServletRequest req,
ServletResponse resp,
FilterChain fc)
throws IOException, ServletException {
HttpServletRequest request = (HttpServletRequest) req;
HttpServletResponse response = (HttpServletResponse) resp;
// Dose the client accept GZIP compression?
String valid_encodings = request.getHeader("Accept-Encoding");
if ( (valid_encodings != null) && (valid_encodings.indexOf("gzip") > -1) ) {
// Then wrap the response object with a compression wrapper
// We'll look at this class in a minute.
CompressionResponseWrapper wrappedResp = new CompressionResponseWrapper(response);
// Declare that the response content is being GZIP encoded.
wrappedResp.setHeader("Content-Encoding", "gzip");
// Chain to the next component (thus processing the request)
fc.doFilter(request, wrappedResp);
// A GZIP compression stream must be "finished" which also
// flushes the GZIP stream buffer which sends all of its
// data to the original response stream.
GZIPOutputStream gzos = wrappedResp.getGZIPOutputStream();
gzos.finish();
// The container handles the rest of the work.
//ctx.log(cfg.getFilterName() + ": finished the request.");
} else {
fc.doFilter(request, response);
//ctx.log(cfg.getFilterName() + ": no encoding performed.");
}
}
public void destroy() {
// nulling out my instance variables
cfg = null;
ctx = null;
}
}
I was using the next code to send JSON responses in Struts web application.
public ActionForward get(ActionMapping mapping,
ActionForm form,
HttpServletRequest request,
HttpServletResponse response) {
JSONObject json = // Do some logic here
RequestUtils.populateWithJSON(response, json);
return null;
}
public static void populateWithJSON(HttpServletResponse response,JSONObject json) {
if(json!=null) {
response.setContentType("text/x-json;charset=UTF-8");
response.setHeader("Cache-Control", "no-cache");
try {
response.getWriter().write(json.toString());
} catch (IOException e) {
throw new ApplicationException("IOException in populateWithJSON", e);
}
}
}
It works fine without compression but if I compress JSON responses, I can not see my JSON objects anymore. I handle JSON Ajax calls with JQuery with code snippets as follows:
$.post(url,parameters, function(json) {
// Do some DOM manipulation with the data contained in the JSON Object
}, "json");
If I see the response with Firebug it is empty.
Should I refractor my compression filter to skip compression in JSON responses? or there is a workaround to this?
For me, it looks like JQuery does not recognize the response as JSON because I am adding the Gzip compression.

If I see the response with Firebug it
is empty.
There's your clue - it's not a JQuery problem, it's server-side. (I'm afraid I can't help you with that, other than to suggest you stop looking at the client-side)
There's no problem gzipping ajax responses - if you can't see the response in Firebug, then JQuery can't see it either.

you have to add one more header "content-encoding: gzip" if you are compressing it.

Have you tried with an explicit java-based client to ensure it's a problem with jQuery or browser? If java client fails, something is wrong with server response.
But I am guessing that whereas browser can deal with uncompression with direct requests, this is perhaps not applied to Ajax calls.
It's an interesting question, I hope we'll get a more definitive answer. :)