Develop Reference

How to selectively GZIP encode POST and PUT requests

I'm using Jersey on both the server and client of a web application. On the server I have Interceptors as noted in https://jersey.java.net/documentation/latest/filters-and-interceptors.html to handle GZIP compression going out and coming in. From the server side, it's easy enough to select which resource methods are compressed using the #Compress annotation. However, if I also want to selectively compress entities from the Client to the Server, what's the best way to do that?
I had started adding a Content-Encoding: x-gzip header to the request, but my client side Interceptor does not see that header (presumably because it's not an official client side header).
Before you point to section 10.6 of the Jersey documentation, note that this works for the Server side. Although I could do something similar on the Client, I don't want to restrict it by URL. I'd rather control the compression flag as close to the request as possible (i.e. Header?).
Here's what I have so far, but it does not work since my header is removed:
class GzipWriterClientInterceptor implements WriterInterceptor {
private static final Set<String> supportedEncodings = new GZipEncoder().getSupportedEncodings(); //support gzip and x-gzip
#Override
public void aroundWriteTo(WriterInterceptorContext context)
throws IOException, WebApplicationException {
if (supportedEncodings.contains(context.getHeaders().getFirst(HttpHeaderConstants.CONTENT_ENCODING_HEADER))) {
System.out.println("ZIPPING DATA");
final OutputStream outputStream = context.getOutputStream();
context.setOutputStream(new GZIPOutputStream(outputStream));
} else {
context.headers.remove(HttpHeaderConstants.CONTENT_ENCODING_HEADER) //remove it since we won't actually be compressing the data
}
context.proceed();
}
}
Sample Request:
Response response = getBaseTarget().path(getBasePath()).path(graphic.uuid.toString())
.request(DEFAULT_MEDIA_TYPE)
.header(HttpHeaderConstants.CONTENT_ENCODING_HEADER, MediaTypeConstants.ENCODING_GZIP)
.put( Entity.entity(graphic, DEFAULT_MEDIA_TYPE))
I also have a logging filter as well that shows all the request headers. I've simplified the above, but all other headers I add are logged.

How to send a large input stream to a Spring REST service?

Have a Spring Rest application that run inside an embedded Jetty container.
On Client I use RestTemplate(try to).
Use case :
Having an InputStream (I don't have the File), I want to send it to the REST service.
The InputStream can be quite large (no byte[] !).
What I've tried so far :
Added StandardServletMultipartResolver to the Dispatcher context;
On servlet registration executed :
ServletRegistration.Dynamic dispatcher = ...
MultipartConfigElement multipartConfigElement = new MultipartConfigElement("D:/temp");
dispatcher.setMultipartConfig(multipartConfigElement);
On client :
restTemplate.getMessageConverters().add(new FormHttpMessageConverter());
MultiValueMap<String, Object> parts = new LinkedMultiValueMap<String, Object>();
parts.add("attachmentData", new InputStreamResource(data) {
// hacks ...
#Override
public String getFilename() {
//avoid null file name
return "attachment.zip";
}
#Override
public long contentLength() throws IOException {
// avoid calling getInputStream() twice
return -1L;
}
}
ResponseEntity<Att> saved = restTemplate.postForEntity(url, parts, Att.class)
On server :
#RequestMapping("/attachment")
public ResponseEntity<Att> saveAttachment(#RequestParam("attachmentData") javax.servlet.http.Part part) {
try {
InputStream is = part.getInputStream();
// consume is
is.close();
part.delete();
return new ResponseEntity<Att>(att, HttpStatus.CREATED);
}
}
What is happening :
The uploaded InputStream is stored successfully in the configured temp folder (MultiPart1970755229517315824), the Part part parameter is correctly Injected in the handler method.
The delete() method does not delete the file (smth still has opened handles on it).
Anyway it looks very ugly.
Is there a smoother solution ?

You want to use HTTP's Chunked Transfer Coding. You can enable that by setting SimpleClientHttpRequestFactory.setBufferRequestBody(false). See SPR-7909.

You should rather use byte[], and write a wrapper around the webservice to actually send the "large string" in chunks. Add a parameter in the webservice which will indicate the "contentID" of the content, so that the other side knows this part belongs to which half-filled "bucket". Another parameter "chunkID" would help in sequencing of the chunks on the other side. Finally, third parameter, "isFinalChunk" would be set if whatever you are sending is the final thing. This is pretty non-fancy functionality achievable in less than 100 lines of code.
The only issue with this is that you end up making "n" calls to the webservice rather than just one call, which would aggregate the connect delays etc. For realtime stuff, some more network QoS is required, but otherwise you should be fine.
I think this is much simpler, and once you have your own class wrapper to do this simple chopping and gluing, it is scalable to a great extent if your server can handle multiple webservice calls.

Why is HttpServletRequest inputstream empty?

I have this code where I read the input from a request input stream and use a JacksonMapper to convert into a POJO. Its running in a jetty 7 container with guice support.
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
try {
RequestType requestType = mapper.readValue(req.getInputStream(), RequestType.class);
} Catch(Exception ex) {
....
}
}
However, sometimes under load the following exception is thrown. I have checked my client and I am sure its sending a valid json string. What is going wrong? Is it expected behavior for Jetty 7 under load?
java.io.EOFException: No content to map to Object due to end of input
at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2433)
at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2385)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1637)
at com.ea.wsop.user.LoginServlet.processRequest(LoginServlet.java:69)
at com.ea.wsop.user.LoginServlet.doPost(LoginServlet.java:63)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.CGLIB$doPost$0(<generated>)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd$$FastClassByGuice$$c6f479ee.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.ea.monitor.MethodExecutionTimer.invoke(MethodExecutionTimer.java:130)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.doPost(<generated>)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.CGLIB$service$8(<generated>)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd$$FastClassByGuice$$c6f479ee.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.ea.monitor.MethodExecutionTimer.invoke(MethodExecutionTimer.java:130)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.service(<generated>)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.CGLIB$service$9(<generated>)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd$$FastClassByGuice$$c6f479ee.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.ea.monitor.MethodExecutionTimer.invoke(MethodExecutionTimer.java:130)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.service(<generated>)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)

I had a similar problem running a Spring Boot application. My Spring Boot app is a simple Dispatcher servlet that reads the request body and processes it.
In my case, the client (curl) sets a content-type header of application/x-www-form-urlencoded if the curl command line uses -d {some-data} and does not set an specific content-type header via -Hcontent-type=some-other-media-type.
Inside the Apache Catalina servlet engine that Spring Boot runs, the Request class makes the following test in parseParameters()
if (!("application/x-www-form-urlencoded".equals(contentType))) {
success = true;
return;
}
For other content-type values, Request returns here, done.
However, if the content type matches application/x-www-form-urlencoded, Request continues:
try {
if (readPostBody(formData, len) != len) {
parameters.setParseFailedReason(FailReason.REQUEST_BODY_INCOMPLETE);
return;
}
} catch (....)
which will consume the body. So in my case, even though my servlet does nothing other than call request.getInputStream() and try to read() from it, it is already too late - the runtime Request already reads the input and does not buffer or unread it. The only workaround is to set a different Content-Type.
The culprit is
OrderedHiddenHttpMethodFilter(HiddenHttpMethodFilter).doFilterInternal(HttpServletRequest, HttpServletResponse, FilterChain) line 70
which is looking for the "_method" query parameter.
I was able to disable the filter by adding
#Bean
public FilterRegistrationBean registration(HiddenHttpMethodFilter filter) {
FilterRegistrationBean registration = new FilterRegistrationBean(filter);
registration.setEnabled(false);
return registration;
}
(which was used to solve another problem)

It will be empty if it's already consumed beforehand. This will be implicitly done whenever you call getParameter(), getParameterValues(), getParameterMap(), getReader(), etc on the HttpServletRequest. Make sure that you don't call any of those kind of methods which by themselves need to gather information from the request body before calling getInputStream(). If your servlet isn't doing that, then start checking the servlet filters which are mapped on the same URL pattern.
Update: this seems to be GAE 1.5 specific. See also
http://code.google.com/p/googleappengine/issues/detail?id=5161
http://code.google.com/p/googleappengine/issues/detail?id=5898
I'm afraid that there's no solution/workaround until they get it fixed. You could try to check if it's available inside a Filter and if so, then copy and store it as request attribute. But this might affect further processing by some GAE servlet.

I had the problem that my request InputStream was always empty with Jetty 6.1.15, and found out that it was caused by a missing or wrong "Content-Type" header.
I generate the requests in another Java program with HttpUrlConnection. When I did not set the Content-Type header explicitly, the InputStream returned by request.getInputStream() in the receiving program was always empty. When I set the content type to "binary/octet-stream", the InputStream of the request contained the correct data.
The only method that is called on the request object before getInputStream() is getContentLength().

I was using mod_jk 1.2.39 which had a bug that caused this issue. After updating to 1.2.40 it started working.

I've had this problem with a post. I solved it by FIRST reading the inputstream and putting it in a cache, before reading the parameters. That seemed to do the trick

Systematic approach is:
Get source code for your container, or at least it's web part (can be hard to find), import in your IDE.
Make break point in your code where before HttpServletRequest->getInputStream() is called.
Step into HttpServletRequest->getInputStream() method, now you are in some ...Impl class.
Set a new break point in that getInputStream() implemmentation, or even in its read() method.
Repeat test call and see what is consuming your data.

I ended up with the problem when enabling debug logging for org.springframework in a Spring Boot 2.2.1 project, and thus using spring-webmvc 5.2.1.
This is caused by the request logging of the parameter-map, which reads the input stream if the Content-Type is application/x-www-form-urlencoded. I believe this spring issue is related to it.
See the following code which causes the problem.
private void logRequest(HttpServletRequest request) {
LogFormatUtils.traceDebug(logger, traceOn -> {
String params;
if (isEnableLoggingRequestDetails()) {
params = request.getParameterMap().entrySet().stream()
.map(entry -> entry.getKey() + ":" + Arrays.toString(entry.getValue()))
.collect(Collectors.joining(", "));
}
else {
params = (request.getParameterMap().isEmpty() ? "" : "masked");
}
...
source
I ended up reporting an issue and and changing the content-type in the request instead.

how to write a file object on server response and without saving file on server?

I am using Spring with DWR . I want to return a file object as response , however I save the file (to be sent) at server temporary location and then send its location as href for anchor tag on client side , however I wonder if there could be a way to throw the file directly to browser on response object without saving it temporarily on server.
I expected if there could be a way to send file as a response via DWR.

public ModelAndView writeFileContentInResponse(HttpServletRequest request, HttpServletResponse response) throws IOException {
FileInputStream inputStream = new FileInputStream("FileInputStreamDemo.java"); //read the file
response.setHeader("Content-Disposition","attachment; filename=test.txt");
try {
int c;
while ((c = inputStream.read()) != -1) {
response.getWriter().write(c);
}
} finally {
if (inputStream != null)
inputStream.close();
response.getWriter().close();
}
}

It has been years since I've used Spring, and I'm unfamiliar with DWR, but the essence of your question is basic to the web.
The answer is yes, you can. In effect, you need to set the HTTP header Content-Disposition: attachment, then stream down the contents. All of this will be in the response to the original request (as opposed to sending back a link).
The actual code to achieve this will depend on your circumstances, but this should get you started.

you call the method from Java Script, right? I didn't really understand how Spring is related in this flow, but as far as I know DWR allows you to produce Java Script Stubs and call the Java methods of the exposed bean directly on server right from your java script client code.
You can read the file byte-by-byte and return it from your java method as long as it really returns a byte array.
However what would you do with this byte array on client?
I just think in this specific flow you shouldn't use the DWR but rather issue an ordinar AJAX request (if DWR can wrap it somehow for convenience - great). This request shouldn't come to DWRServlet, but rather be proceeded by a regular servlet/some web-based framework, like Spring MVC :)
Once the request comes to the servlet, use
response.setHeader("Content-Disposition","attachment; filename=test.txt");
as was already stated.
Hope this helps,
Good luck!
Mark

An example which return a excel to download from client:
//Java side:
public FileTransfer getExcel(Parametros param){
byte[] result = <here get data>;
InputStream myInputStream = new ByteArrayInputStream(result);
String excelFormat = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
FileTransfer dwrExcelFile = new FileTransfer("excel.xlsx", excelFormat, myInputStream);
return dwrExcelFile;
}
//Javascript side:
function downloadExcelFile() {
dwr.engine.setTimeout(59000);
var params = <params_to_send>;
<Java_class>.getExcel(params, {callback:function(dataFromServer) {
downloadExcelCallback(dataFromServer);
}});
}
function downloadExcelCallback(data) {
dwr.engine.openInDownload(data);
}

RequestDispatcher for remote server?

I am trying to create a HttpServlet that forwards all incoming requests as is, to another serlvet running on a different domain.
How can this be accomplished? The RequestDispatcher's forward() only operates on the same server.
Edit: I can't introduce any dependencies.

You can't when it doesn't run in the same ServletContext or same/clustered webserver wherein the webapps are configured to share the ServletContext (in case of Tomcat, check crossContext option).
You have to send a redirect by HttpServletResponse.sendRedirect(). If your actual concern is reusing the query parameters on the new URL, just resend them along.
response.sendRedirect(newURL + "?" + request.getQueryString());
Or when it's a POST, send a HTTP 307 redirect, the client will reapply the same POST query parameters on the new URL.
response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT);
response.setHeader("Location", newURL);
Update as per the comments, that's apparently not an option as well since you want to hide the URL. In that case, you have to let the servlet play for proxy. You can do this with a HTTP client, e.g. the Java SE provided java.net.URLConnection (mini tutorial here) or the more convenienced Apache Commons HttpClient.
If it's GET, just do:
InputStream input = new URL(newURL + "?" + request.getQueryString()).openStream();
OutputStream output = response.getOutputStream();
// Copy.
Or if it's POST:
URLConnection connection = new URL(newURL).openConnection();
connection.setDoOutput(true);
// Set and/or copy request headers here based on current request?
InputStream input1 = request.getInputStream();
OutputStream output1 = connection.getOutputStream();
// Copy.
InputStream input2 = connection.getInputStream();
OutputStream output2 = response.getOutputStream();
// Copy.
Note that you possibly need to capture/replace/update the relative links in the HTML response, if any. Jsoup may be extremely helpful in this.

As others have pointed out, what you want is a proxy. Your options:
Find an open-source Java library that does this. There are a few out there, but I haven't used any of them, so I can't recommend any.
Write it yourself. Shouldn't be too hard, just remember to deal with stuff like passing along all headers and response codes.
Use the proxy module in Apache 2.2. This is the one I'd pick, because I already know that it works reliably.

Jetty has a sample ProxyServlet implementation that uses URL.openConnection() under the hood. Feel free to use as-is or to use as inspiration for your own implementation. ;-)
Or you can use Apache HttpClient, see the tutorial.