Have a Spring Rest application that run inside an embedded Jetty container.
On Client I use RestTemplate(try to).
Use case :
Having an InputStream (I don't have the File), I want to send it to the REST service.
The InputStream can be quite large (no byte[] !).
What I've tried so far :
Added StandardServletMultipartResolver to the Dispatcher context;
On servlet registration executed :
ServletRegistration.Dynamic dispatcher = ...
MultipartConfigElement multipartConfigElement = new MultipartConfigElement("D:/temp");
dispatcher.setMultipartConfig(multipartConfigElement);
On client :
restTemplate.getMessageConverters().add(new FormHttpMessageConverter());
MultiValueMap<String, Object> parts = new LinkedMultiValueMap<String, Object>();
parts.add("attachmentData", new InputStreamResource(data) {
// hacks ...
#Override
public String getFilename() {
//avoid null file name
return "attachment.zip";
}
#Override
public long contentLength() throws IOException {
// avoid calling getInputStream() twice
return -1L;
}
}
ResponseEntity<Att> saved = restTemplate.postForEntity(url, parts, Att.class)
On server :
#RequestMapping("/attachment")
public ResponseEntity<Att> saveAttachment(#RequestParam("attachmentData") javax.servlet.http.Part part) {
try {
InputStream is = part.getInputStream();
// consume is
is.close();
part.delete();
return new ResponseEntity<Att>(att, HttpStatus.CREATED);
}
}
What is happening :
The uploaded InputStream is stored successfully in the configured temp folder (MultiPart1970755229517315824), the Part part parameter is correctly Injected in the handler method.
The delete() method does not delete the file (smth still has opened handles on it).
Anyway it looks very ugly.
Is there a smoother solution ?
You want to use HTTP's Chunked Transfer Coding. You can enable that by setting SimpleClientHttpRequestFactory.setBufferRequestBody(false). See SPR-7909.
You should rather use byte[], and write a wrapper around the webservice to actually send the "large string" in chunks. Add a parameter in the webservice which will indicate the "contentID" of the content, so that the other side knows this part belongs to which half-filled "bucket". Another parameter "chunkID" would help in sequencing of the chunks on the other side. Finally, third parameter, "isFinalChunk" would be set if whatever you are sending is the final thing. This is pretty non-fancy functionality achievable in less than 100 lines of code.
The only issue with this is that you end up making "n" calls to the webservice rather than just one call, which would aggregate the connect delays etc. For realtime stuff, some more network QoS is required, but otherwise you should be fine.
I think this is much simpler, and once you have your own class wrapper to do this simple chopping and gluing, it is scalable to a great extent if your server can handle multiple webservice calls.
Related
I'm trying to implement a REST service (or even a single Servlet) that will allow me to get a quick "preview" of incoming file (multipart request with a single file in it).
The idea is to parse and store in DB few first lines of potentially huge file coming into the service.
Problem I face is that Spring #RestController (on Tomcat) is run after the whole request is received by a server and MultipartFile is already stored on filesystem.
I managed to implement something that is working with pure Servlet implementation (reading multipart request directly from HttpServletRequest) but... then I need to do all the multipart parsing stuff manually. I tried with commons fileupload (http://commons.apache.org/proper/commons-fileupload/) but it also caches files on file system, so when I call:
List<FileItem> uploads = new ServletFileUpload(new DiskFileItemFactory()).parseRequest(request);
all freezes util whole file is uploaded on server.
Any suggestions how to deal with this - preferably on Tomcat and with #RestController if it is at all possible.
I believe you were on the right track with commons-fileupload. You just need to use the "streaming" API. Something similar should work:
#RestController
public class ARestController {
#RequestMapping(value = Routes.A_ROUTE, method = RequestMethod.POST)
public ResponseEntity<?> processMultiPart(HttpServletRequest request) {
try {
ServletFileUpload upload = new ServletFileUpload();
final FileItemIterator itemIterator = upload.getItemIterator(request);
while (itemIterator.hasNext()) {
final FileItemStream fileItemStream = itemIterator.next();
if (!fileItemStream.isFormField()) {
try (InputStream inputStream = fileItemStream.openStream()) {
// process the stream the way you want
}
}
}
} catch (Exception e) {
// ...
}
}
}
This way you could read the parts as stream, use whatever you need from them, and discard the rest. There won't be anything stored on the FS.
I want to start a time consuming process after sending a response in java spring boot application. I am not able to get an exact example of implementation.
Example code is given below: I want to simulate similar
#POST
#Path("upload")
#Produces(MediaType.APPLICATION_JSON)
#Consumes(MediaType.MULTIPART_FORM_DATA)
public PutObjectResult uploadFile(#FormDataParam("files") InputStream file, #FormDataParam("files") FormDataContentDisposition fileDetail) {
PutObjectResult putObjectResult = this.coreModule.save(file, fileDetail);
if (putObjectResult != null) {
return putObjectResult;
}
return new PutObjectResult();
//i want this time consuming process to not block my response. basically i am doing it for high user experience.
TimeConsumingProcess timeConsumingProcess = new TimeConsumingProcess();
timeConsumingProcess.start();
}
I know that execution after return is not correct. But i want to simulate a very similar action. Probably i think performing a completable future might be a good idea so that is not blocking my return statement, but i am thinking if there is an elegant way of handling a situation like this!!!!
I'm using Jersey on both the server and client of a web application. On the server I have Interceptors as noted in https://jersey.java.net/documentation/latest/filters-and-interceptors.html to handle GZIP compression going out and coming in. From the server side, it's easy enough to select which resource methods are compressed using the #Compress annotation. However, if I also want to selectively compress entities from the Client to the Server, what's the best way to do that?
I had started adding a Content-Encoding: x-gzip header to the request, but my client side Interceptor does not see that header (presumably because it's not an official client side header).
Before you point to section 10.6 of the Jersey documentation, note that this works for the Server side. Although I could do something similar on the Client, I don't want to restrict it by URL. I'd rather control the compression flag as close to the request as possible (i.e. Header?).
Here's what I have so far, but it does not work since my header is removed:
class GzipWriterClientInterceptor implements WriterInterceptor {
private static final Set<String> supportedEncodings = new GZipEncoder().getSupportedEncodings(); //support gzip and x-gzip
#Override
public void aroundWriteTo(WriterInterceptorContext context)
throws IOException, WebApplicationException {
if (supportedEncodings.contains(context.getHeaders().getFirst(HttpHeaderConstants.CONTENT_ENCODING_HEADER))) {
System.out.println("ZIPPING DATA");
final OutputStream outputStream = context.getOutputStream();
context.setOutputStream(new GZIPOutputStream(outputStream));
} else {
context.headers.remove(HttpHeaderConstants.CONTENT_ENCODING_HEADER) //remove it since we won't actually be compressing the data
}
context.proceed();
}
}
Sample Request:
Response response = getBaseTarget().path(getBasePath()).path(graphic.uuid.toString())
.request(DEFAULT_MEDIA_TYPE)
.header(HttpHeaderConstants.CONTENT_ENCODING_HEADER, MediaTypeConstants.ENCODING_GZIP)
.put( Entity.entity(graphic, DEFAULT_MEDIA_TYPE))
I also have a logging filter as well that shows all the request headers. I've simplified the above, but all other headers I add are logged.
I have this code where I read the input from a request input stream and use a JacksonMapper to convert into a POJO. Its running in a jetty 7 container with guice support.
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
try {
RequestType requestType = mapper.readValue(req.getInputStream(), RequestType.class);
} Catch(Exception ex) {
....
}
}
However, sometimes under load the following exception is thrown. I have checked my client and I am sure its sending a valid json string. What is going wrong? Is it expected behavior for Jetty 7 under load?
java.io.EOFException: No content to map to Object due to end of input
at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2433)
at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2385)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1637)
at com.ea.wsop.user.LoginServlet.processRequest(LoginServlet.java:69)
at com.ea.wsop.user.LoginServlet.doPost(LoginServlet.java:63)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.CGLIB$doPost$0(<generated>)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd$$FastClassByGuice$$c6f479ee.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.ea.monitor.MethodExecutionTimer.invoke(MethodExecutionTimer.java:130)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.doPost(<generated>)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.CGLIB$service$8(<generated>)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd$$FastClassByGuice$$c6f479ee.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.ea.monitor.MethodExecutionTimer.invoke(MethodExecutionTimer.java:130)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.service(<generated>)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.CGLIB$service$9(<generated>)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd$$FastClassByGuice$$c6f479ee.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.ea.monitor.MethodExecutionTimer.invoke(MethodExecutionTimer.java:130)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at com.ea.wsop.user.LoginServlet$$EnhancerByGuice$$a91c2ebd.service(<generated>)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
I had a similar problem running a Spring Boot application. My Spring Boot app is a simple Dispatcher servlet that reads the request body and processes it.
In my case, the client (curl) sets a content-type header of application/x-www-form-urlencoded if the curl command line uses -d {some-data} and does not set an specific content-type header via -Hcontent-type=some-other-media-type.
Inside the Apache Catalina servlet engine that Spring Boot runs, the Request class makes the following test in parseParameters()
if (!("application/x-www-form-urlencoded".equals(contentType))) {
success = true;
return;
}
For other content-type values, Request returns here, done.
However, if the content type matches application/x-www-form-urlencoded, Request continues:
try {
if (readPostBody(formData, len) != len) {
parameters.setParseFailedReason(FailReason.REQUEST_BODY_INCOMPLETE);
return;
}
} catch (....)
which will consume the body. So in my case, even though my servlet does nothing other than call request.getInputStream() and try to read() from it, it is already too late - the runtime Request already reads the input and does not buffer or unread it. The only workaround is to set a different Content-Type.
The culprit is
OrderedHiddenHttpMethodFilter(HiddenHttpMethodFilter).doFilterInternal(HttpServletRequest, HttpServletResponse, FilterChain) line 70
which is looking for the "_method" query parameter.
I was able to disable the filter by adding
#Bean
public FilterRegistrationBean registration(HiddenHttpMethodFilter filter) {
FilterRegistrationBean registration = new FilterRegistrationBean(filter);
registration.setEnabled(false);
return registration;
}
(which was used to solve another problem)
It will be empty if it's already consumed beforehand. This will be implicitly done whenever you call getParameter(), getParameterValues(), getParameterMap(), getReader(), etc on the HttpServletRequest. Make sure that you don't call any of those kind of methods which by themselves need to gather information from the request body before calling getInputStream(). If your servlet isn't doing that, then start checking the servlet filters which are mapped on the same URL pattern.
Update: this seems to be GAE 1.5 specific. See also
http://code.google.com/p/googleappengine/issues/detail?id=5161
http://code.google.com/p/googleappengine/issues/detail?id=5898
I'm afraid that there's no solution/workaround until they get it fixed. You could try to check if it's available inside a Filter and if so, then copy and store it as request attribute. But this might affect further processing by some GAE servlet.
I had the problem that my request InputStream was always empty with Jetty 6.1.15, and found out that it was caused by a missing or wrong "Content-Type" header.
I generate the requests in another Java program with HttpUrlConnection. When I did not set the Content-Type header explicitly, the InputStream returned by request.getInputStream() in the receiving program was always empty. When I set the content type to "binary/octet-stream", the InputStream of the request contained the correct data.
The only method that is called on the request object before getInputStream() is getContentLength().
I was using mod_jk 1.2.39 which had a bug that caused this issue. After updating to 1.2.40 it started working.
I've had this problem with a post. I solved it by FIRST reading the inputstream and putting it in a cache, before reading the parameters. That seemed to do the trick
Systematic approach is:
Get source code for your container, or at least it's web part (can be hard to find), import in your IDE.
Make break point in your code where before HttpServletRequest->getInputStream() is called.
Step into HttpServletRequest->getInputStream() method, now you are in some ...Impl class.
Set a new break point in that getInputStream() implemmentation, or even in its read() method.
Repeat test call and see what is consuming your data.
I ended up with the problem when enabling debug logging for org.springframework in a Spring Boot 2.2.1 project, and thus using spring-webmvc 5.2.1.
This is caused by the request logging of the parameter-map, which reads the input stream if the Content-Type is application/x-www-form-urlencoded. I believe this spring issue is related to it.
See the following code which causes the problem.
private void logRequest(HttpServletRequest request) {
LogFormatUtils.traceDebug(logger, traceOn -> {
String params;
if (isEnableLoggingRequestDetails()) {
params = request.getParameterMap().entrySet().stream()
.map(entry -> entry.getKey() + ":" + Arrays.toString(entry.getValue()))
.collect(Collectors.joining(", "));
}
else {
params = (request.getParameterMap().isEmpty() ? "" : "masked");
}
...
source
I ended up reporting an issue and and changing the content-type in the request instead.
I am using Spring with DWR . I want to return a file object as response , however I save the file (to be sent) at server temporary location and then send its location as href for anchor tag on client side , however I wonder if there could be a way to throw the file directly to browser on response object without saving it temporarily on server.
I expected if there could be a way to send file as a response via DWR.
public ModelAndView writeFileContentInResponse(HttpServletRequest request, HttpServletResponse response) throws IOException {
FileInputStream inputStream = new FileInputStream("FileInputStreamDemo.java"); //read the file
response.setHeader("Content-Disposition","attachment; filename=test.txt");
try {
int c;
while ((c = inputStream.read()) != -1) {
response.getWriter().write(c);
}
} finally {
if (inputStream != null)
inputStream.close();
response.getWriter().close();
}
}
It has been years since I've used Spring, and I'm unfamiliar with DWR, but the essence of your question is basic to the web.
The answer is yes, you can. In effect, you need to set the HTTP header Content-Disposition: attachment, then stream down the contents. All of this will be in the response to the original request (as opposed to sending back a link).
The actual code to achieve this will depend on your circumstances, but this should get you started.
you call the method from Java Script, right? I didn't really understand how Spring is related in this flow, but as far as I know DWR allows you to produce Java Script Stubs and call the Java methods of the exposed bean directly on server right from your java script client code.
You can read the file byte-by-byte and return it from your java method as long as it really returns a byte array.
However what would you do with this byte array on client?
I just think in this specific flow you shouldn't use the DWR but rather issue an ordinar AJAX request (if DWR can wrap it somehow for convenience - great). This request shouldn't come to DWRServlet, but rather be proceeded by a regular servlet/some web-based framework, like Spring MVC :)
Once the request comes to the servlet, use
response.setHeader("Content-Disposition","attachment; filename=test.txt");
as was already stated.
Hope this helps,
Good luck!
Mark
An example which return a excel to download from client:
//Java side:
public FileTransfer getExcel(Parametros param){
byte[] result = <here get data>;
InputStream myInputStream = new ByteArrayInputStream(result);
String excelFormat = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
FileTransfer dwrExcelFile = new FileTransfer("excel.xlsx", excelFormat, myInputStream);
return dwrExcelFile;
}
//Javascript side:
function downloadExcelFile() {
dwr.engine.setTimeout(59000);
var params = <params_to_send>;
<Java_class>.getExcel(params, {callback:function(dataFromServer) {
downloadExcelCallback(dataFromServer);
}});
}
function downloadExcelCallback(data) {
dwr.engine.openInDownload(data);
}