I'm trying to implement a REST service (or even a single Servlet) that will allow me to get a quick "preview" of incoming file (multipart request with a single file in it).
The idea is to parse and store in DB few first lines of potentially huge file coming into the service.
Problem I face is that Spring #RestController (on Tomcat) is run after the whole request is received by a server and MultipartFile is already stored on filesystem.
I managed to implement something that is working with pure Servlet implementation (reading multipart request directly from HttpServletRequest) but... then I need to do all the multipart parsing stuff manually. I tried with commons fileupload (http://commons.apache.org/proper/commons-fileupload/) but it also caches files on file system, so when I call:
List<FileItem> uploads = new ServletFileUpload(new DiskFileItemFactory()).parseRequest(request);
all freezes util whole file is uploaded on server.
Any suggestions how to deal with this - preferably on Tomcat and with #RestController if it is at all possible.
I believe you were on the right track with commons-fileupload. You just need to use the "streaming" API. Something similar should work:
#RestController
public class ARestController {
#RequestMapping(value = Routes.A_ROUTE, method = RequestMethod.POST)
public ResponseEntity<?> processMultiPart(HttpServletRequest request) {
try {
ServletFileUpload upload = new ServletFileUpload();
final FileItemIterator itemIterator = upload.getItemIterator(request);
while (itemIterator.hasNext()) {
final FileItemStream fileItemStream = itemIterator.next();
if (!fileItemStream.isFormField()) {
try (InputStream inputStream = fileItemStream.openStream()) {
// process the stream the way you want
}
}
}
} catch (Exception e) {
// ...
}
}
}
This way you could read the parts as stream, use whatever you need from them, and discard the rest. There won't be anything stored on the FS.
Related
I am sending a POST request to my localhost Tomcat 8.0 Servlet 3.1 web application but request.getParameter("") returns me null.
This is how I make my request.
I am using PostMan to perform my POST request. And on the java side, I am calling request.getParameter("code") and this gives me null. Same goes for qwe field. I am not using any framework. It's raw servlets. This is only a back-end so it's all about handling the data and responding to client.
If I use "x-www-form-urlencoded", I am able to retrieve the parameters through getParameter() call but I still want to know why I am not able to get form-data.
Thanks.
Thanks to #cjstehno,
When he said "form-data" was actually a multipart data, then I attempted to read it as multipart data but taking isFormField() method in mind to distinguish between file and parameter. So from a raw servlet, one can read form-data through the code below. From the performance view, I am pretty sure that this might be improved.
try {
ServletFileUpload upload = new ServletFileUpload();
FileItemIterator iter = upload.getItemIterator(request);
while (iter.hasNext()) {
FileItemStream item = iter.next();
String name = item.getFieldName();
if (item.isFormField()) {
String value = Streams.asString(item.openStream());
}
}
} catch (Exception ex) {}
On my company's site we have some tables that we need to export to a csv file.
There are some varying parameters, so the csv file needs to be dynamically created on request.
My problem is that after clicking to download, the response hangs, and waits for the whole file to be created (which can take some time) and only then downloads the entire file in one instant.
I'm using AngularJS, so I'm using window.location = <url_for_file_download> In order to make the browser download the file.
On the server side I'm using Java Spring and I've followed all the instructions I could find on the web in order to create a file download controller.
My controller code is something like this:
#RequestMapping(value = "http://yada.yada.yada/csv/myFile.csv", method = RequestMethod.GET)
public #ResponseBody
void getCustomers(HttpServletResponse response,
#RequestParam(required = false) String someParameters)
throws NotAuthorizedException, IOException {
// set headers
setHeaders(response);
// generate writer
CSVWriter write = generateWriter(response);
// get data
List<String[]> data = getData();
// write and flush and all that
.
.
.
}
My code for setting the response headers are:
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition","attachment; filename=\"" + fileName + ".csv\"");
I've also tried adding the following headers:
response.setHeader("Transfer-Encoding", "Chunked");
response.setHeader("Content-Description", "File Transfer");
and I've also tried setting the Content-type to "application/octet-stream".
Notice that I don't add a Content-length header, since the file doesn't exist yet, and is being written on the fly.
For writing the csv file I'm using OpenCSV and my code is as follows:
OutputStream resOs = response.getOutputStream();
OutputStream buffOs = new BufferedOutputStream(resOs);
OutputStreamWriter outputWriter = new OutputStreamWriter(buffOs,"UTF-8");
CSVWriter writer = new CSVWriter(outputWriter);
I iterate over the data and write it like so:
for (String[] row: data) {
writer.writeNext(line);
}
(It's not exactly the code - but this is more or else what happens in the code)
And at the end I flush and close:
writer.flush();
writer.close();
I also tried flushing after each line I write.
So why isn't the file being transferred before it has all been written?
Why is my browser (Google chrome) downloading the file in one instant after waiting a long time? And how can I fix this.
I hope I've added enough code, if there's something missing just please tell me and I'll try to add it here.
Thank you so much in advance.
Can you try returning a null value in your java
return null ;
Or you can try below code also
1. Jquery code upon clicking the submit button
$(document).ready( function() {
$('#buttonName').click(function(e){
$("#formName").submit();
//alert("The file ready to be downloaded");
});
});
Your controller code
#RequestMapping(value="/name",method=RequestMethod.POST)
public ModelAndView downloadCSV(ModelMap model,HttpSession session,#ModelAttribute(value="Pojo") Pojo pojo
,HttpServletRequest request,HttpServletResponse response){
----------------some code----------------
response.setContentType("application/csv");
("application/unknown");
response.setHeader("content-disposition","attachment;filename =filename.csv");
ServletOutputStream writer = response.getOutputStream();
logger.info("downloading contents to csv");
writer.print("A");
writer.print(',');
writer.println("B");
for(int i=0;i<limit;i++){
writer.print(""+pojo.get(i).getA());
writer.print(',');
writer.print(pojo.get(i).getB());
writer.println();
}
writer.flush();
writer.close();
---------------some code-----------
return null;
}
Hope this helps
The Controller will wait for the response to be written before the response is send back to the client.
Here is a nice post with multiple approaches / options outlined
Downloading a file from spring controllers
This post talks about flushing the output periodically to help fasten the download.
how to download large files without memory issues in java
If all you are trying to do is let the user know that the file download is in progress and due soon, I think an Ajax progress status indicaor might be your solution.
Trigger the ajax call to the back-end to generate the file
Show progress indicator to the user while file is being generated server side
once response is available, file is presented to the user.
I think something similar is being explored here download file with ajax() POST Request via Spring MVC
Hope this helps!
Thanks,
Paul
I faced the same issue. The code that didn't work for me was
#RequestMapping(value = "/test")
public void test(HttpServletResponse response)
throws IOException, InterruptedException {
response.getOutputStream().println("Hello");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("How");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("are");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("you");
response.getOutputStream().flush();
}
The culprit was ShallowEtagHeaderFilter. When this filter is enabled the response is sent in one chunk. When this filter is diabled the response is send in multiple chunks.
From this thread Tomcat does not flush the response buffer it looks like another possible culprit can be GzipFilter
Have a Spring Rest application that run inside an embedded Jetty container.
On Client I use RestTemplate(try to).
Use case :
Having an InputStream (I don't have the File), I want to send it to the REST service.
The InputStream can be quite large (no byte[] !).
What I've tried so far :
Added StandardServletMultipartResolver to the Dispatcher context;
On servlet registration executed :
ServletRegistration.Dynamic dispatcher = ...
MultipartConfigElement multipartConfigElement = new MultipartConfigElement("D:/temp");
dispatcher.setMultipartConfig(multipartConfigElement);
On client :
restTemplate.getMessageConverters().add(new FormHttpMessageConverter());
MultiValueMap<String, Object> parts = new LinkedMultiValueMap<String, Object>();
parts.add("attachmentData", new InputStreamResource(data) {
// hacks ...
#Override
public String getFilename() {
//avoid null file name
return "attachment.zip";
}
#Override
public long contentLength() throws IOException {
// avoid calling getInputStream() twice
return -1L;
}
}
ResponseEntity<Att> saved = restTemplate.postForEntity(url, parts, Att.class)
On server :
#RequestMapping("/attachment")
public ResponseEntity<Att> saveAttachment(#RequestParam("attachmentData") javax.servlet.http.Part part) {
try {
InputStream is = part.getInputStream();
// consume is
is.close();
part.delete();
return new ResponseEntity<Att>(att, HttpStatus.CREATED);
}
}
What is happening :
The uploaded InputStream is stored successfully in the configured temp folder (MultiPart1970755229517315824), the Part part parameter is correctly Injected in the handler method.
The delete() method does not delete the file (smth still has opened handles on it).
Anyway it looks very ugly.
Is there a smoother solution ?
You want to use HTTP's Chunked Transfer Coding. You can enable that by setting SimpleClientHttpRequestFactory.setBufferRequestBody(false). See SPR-7909.
You should rather use byte[], and write a wrapper around the webservice to actually send the "large string" in chunks. Add a parameter in the webservice which will indicate the "contentID" of the content, so that the other side knows this part belongs to which half-filled "bucket". Another parameter "chunkID" would help in sequencing of the chunks on the other side. Finally, third parameter, "isFinalChunk" would be set if whatever you are sending is the final thing. This is pretty non-fancy functionality achievable in less than 100 lines of code.
The only issue with this is that you end up making "n" calls to the webservice rather than just one call, which would aggregate the connect delays etc. For realtime stuff, some more network QoS is required, but otherwise you should be fine.
I think this is much simpler, and once you have your own class wrapper to do this simple chopping and gluing, it is scalable to a great extent if your server can handle multiple webservice calls.
I am using Spring with DWR . I want to return a file object as response , however I save the file (to be sent) at server temporary location and then send its location as href for anchor tag on client side , however I wonder if there could be a way to throw the file directly to browser on response object without saving it temporarily on server.
I expected if there could be a way to send file as a response via DWR.
public ModelAndView writeFileContentInResponse(HttpServletRequest request, HttpServletResponse response) throws IOException {
FileInputStream inputStream = new FileInputStream("FileInputStreamDemo.java"); //read the file
response.setHeader("Content-Disposition","attachment; filename=test.txt");
try {
int c;
while ((c = inputStream.read()) != -1) {
response.getWriter().write(c);
}
} finally {
if (inputStream != null)
inputStream.close();
response.getWriter().close();
}
}
It has been years since I've used Spring, and I'm unfamiliar with DWR, but the essence of your question is basic to the web.
The answer is yes, you can. In effect, you need to set the HTTP header Content-Disposition: attachment, then stream down the contents. All of this will be in the response to the original request (as opposed to sending back a link).
The actual code to achieve this will depend on your circumstances, but this should get you started.
you call the method from Java Script, right? I didn't really understand how Spring is related in this flow, but as far as I know DWR allows you to produce Java Script Stubs and call the Java methods of the exposed bean directly on server right from your java script client code.
You can read the file byte-by-byte and return it from your java method as long as it really returns a byte array.
However what would you do with this byte array on client?
I just think in this specific flow you shouldn't use the DWR but rather issue an ordinar AJAX request (if DWR can wrap it somehow for convenience - great). This request shouldn't come to DWRServlet, but rather be proceeded by a regular servlet/some web-based framework, like Spring MVC :)
Once the request comes to the servlet, use
response.setHeader("Content-Disposition","attachment; filename=test.txt");
as was already stated.
Hope this helps,
Good luck!
Mark
An example which return a excel to download from client:
//Java side:
public FileTransfer getExcel(Parametros param){
byte[] result = <here get data>;
InputStream myInputStream = new ByteArrayInputStream(result);
String excelFormat = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
FileTransfer dwrExcelFile = new FileTransfer("excel.xlsx", excelFormat, myInputStream);
return dwrExcelFile;
}
//Javascript side:
function downloadExcelFile() {
dwr.engine.setTimeout(59000);
var params = <params_to_send>;
<Java_class>.getExcel(params, {callback:function(dataFromServer) {
downloadExcelCallback(dataFromServer);
}});
}
function downloadExcelCallback(data) {
dwr.engine.openInDownload(data);
}
I need to read a file located on a server but I see that in GWT is not possible use some java library.
what I have to do?
try requestBuilder!! this code can help?
RequestBuilder requestBuilder = new RequestBuilder( RequestBuilder.GET, "yourfile.txt" );
try {
requestBuilder.sendRequest( null, new RequestCallback(){
public void onError(Request request, Throwable exception) {
GWT.log( "failed file reading", exception );
}
public void onResponseReceived(Request request, Response response) {
String result=response.getText();
}} );
} catch (RequestException e) {
GWT.log( "failed file reading", e );
}
The Rule: JavaScript cannot read data from a URL that doesn’t have a host name and port that matches those of the page the JavaScript is running in.
In other words: If it is on a different site — you can’t read it directly with JS and therefore GWT, which is nothing more than Javascript once compiled.
It applies to data from XMLHttpRequest, frames, and anything else you care to name.
This may change in the future, but for now the rule stands.
With this in mind there are a couple of workarounds.
1) Call your server with RPC or whatever mechanism and have your server do the request and then send it back to the client. Here is a sample.
2) There are several hacks on allowing JavaScript to access cross-domain sites just do a google search on how to get this. Some browsers will flag this as being dangerous.
3) If you are using Firefox and Firefox only it looks like Firefox has the ability to do this, but you will need to enable this manually.
Simply write first a servlet that sends the file located on the server to the user.
Then when the user clicks on a button for instance you call the servlet with the proper parameter.
Here is an excerpt from our servlet implementation
response.reset();
response.setContentType("application/octet-stream");
response.setContentLength(contentLength);
response.setHeader("Content-disposition", "attachment;
filename=\"" + filename + "\"");
output = new
BufferedOutputStream(response.getOutputStream());
int data = input.read();
while (data != -1)
{
output.write(data);
data = input.read();
}
output.flush();