Develop Reference

REST call to save large ResultSet to downloadable CSV

I am creating a REST service that loads data from Oracle database table using JDBC, and saves the resultset in CSV.
Since the table is very large, it's expected that the process will take about one hour.
How can I download CSV while saving data to it
(so we get a cycle like this: load chunk (some amount of rows), save to CSV, and flush that part (download) )?
this is intended to:
prevent holding the whole file in server memory, by flushing it periodically to the client.(Assuming JDBC driver does not fetch all table immediately)
show (almost) immediate progress to the user (so user won't wait until CSV is complete)

something like this is good enough:
#GET
#Produces("text/csv")
#Path("/get")
public Response getData(#Context HttpServletRequest request,
#Context HttpServletResponse response) throws IOException {
response.setHeader("Content-Disposition", "attachment; filename=data_file.csv");
ServletOutputStream outputStream = response.getOutputStream();
// here you read from ResultSet.
while (resultSet.next()) {
byte b = (byte) (resultSet.getByte("columnA"));
outputStream.write(b);
}
outputStream.flush();
outputStream.close();
return Response.ok().build();
}

Java Spring - dynamically generated csv file download response is hanging

On my company's site we have some tables that we need to export to a csv file.
There are some varying parameters, so the csv file needs to be dynamically created on request.
My problem is that after clicking to download, the response hangs, and waits for the whole file to be created (which can take some time) and only then downloads the entire file in one instant.
I'm using AngularJS, so I'm using window.location = <url_for_file_download> In order to make the browser download the file.
On the server side I'm using Java Spring and I've followed all the instructions I could find on the web in order to create a file download controller.
My controller code is something like this:
#RequestMapping(value = "http://yada.yada.yada/csv/myFile.csv", method = RequestMethod.GET)
public #ResponseBody
void getCustomers(HttpServletResponse response,
#RequestParam(required = false) String someParameters)
throws NotAuthorizedException, IOException {
// set headers
setHeaders(response);
// generate writer
CSVWriter write = generateWriter(response);
// get data
List<String[]> data = getData();
// write and flush and all that
.
.
.
}
My code for setting the response headers are:
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition","attachment; filename=\"" + fileName + ".csv\"");
I've also tried adding the following headers:
response.setHeader("Transfer-Encoding", "Chunked");
response.setHeader("Content-Description", "File Transfer");
and I've also tried setting the Content-type to "application/octet-stream".
Notice that I don't add a Content-length header, since the file doesn't exist yet, and is being written on the fly.
For writing the csv file I'm using OpenCSV and my code is as follows:
OutputStream resOs = response.getOutputStream();
OutputStream buffOs = new BufferedOutputStream(resOs);
OutputStreamWriter outputWriter = new OutputStreamWriter(buffOs,"UTF-8");
CSVWriter writer = new CSVWriter(outputWriter);
I iterate over the data and write it like so:
for (String[] row: data) {
writer.writeNext(line);
}
(It's not exactly the code - but this is more or else what happens in the code)
And at the end I flush and close:
writer.flush();
writer.close();
I also tried flushing after each line I write.
So why isn't the file being transferred before it has all been written?
Why is my browser (Google chrome) downloading the file in one instant after waiting a long time? And how can I fix this.
I hope I've added enough code, if there's something missing just please tell me and I'll try to add it here.
Thank you so much in advance.

Can you try returning a null value in your java
return null ;
Or you can try below code also
1. Jquery code upon clicking the submit button
$(document).ready( function() {
$('#buttonName').click(function(e){
$("#formName").submit();
//alert("The file ready to be downloaded");
});
});
Your controller code
#RequestMapping(value="/name",method=RequestMethod.POST)
public ModelAndView downloadCSV(ModelMap model,HttpSession session,#ModelAttribute(value="Pojo") Pojo pojo
,HttpServletRequest request,HttpServletResponse response){
----------------some code----------------
response.setContentType("application/csv");
("application/unknown");
response.setHeader("content-disposition","attachment;filename =filename.csv");
ServletOutputStream writer = response.getOutputStream();
logger.info("downloading contents to csv");
writer.print("A");
writer.print(',');
writer.println("B");
for(int i=0;i<limit;i++){
writer.print(""+pojo.get(i).getA());
writer.print(',');
writer.print(pojo.get(i).getB());
writer.println();
}
writer.flush();
writer.close();
---------------some code-----------
return null;
}
Hope this helps

The Controller will wait for the response to be written before the response is send back to the client.
Here is a nice post with multiple approaches / options outlined
Downloading a file from spring controllers
This post talks about flushing the output periodically to help fasten the download.
how to download large files without memory issues in java
If all you are trying to do is let the user know that the file download is in progress and due soon, I think an Ajax progress status indicaor might be your solution.
Trigger the ajax call to the back-end to generate the file
Show progress indicator to the user while file is being generated server side
once response is available, file is presented to the user.
I think something similar is being explored here download file with ajax() POST Request via Spring MVC
Hope this helps!
Thanks,
Paul

I faced the same issue. The code that didn't work for me was
#RequestMapping(value = "/test")
public void test(HttpServletResponse response)
throws IOException, InterruptedException {
response.getOutputStream().println("Hello");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("How");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("are");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("you");
response.getOutputStream().flush();
}
The culprit was ShallowEtagHeaderFilter. When this filter is enabled the response is sent in one chunk. When this filter is diabled the response is send in multiple chunks.
From this thread Tomcat does not flush the response buffer it looks like another possible culprit can be GzipFilter

java - Download and then write image as servlet response

How to download an image from a server and then write it as a response in my servlet.
What is the best way to do it keeping good performance?
Here's my code:
JSONObject imageJson;
... //getting my JSON
String imgUrl = imageJson.get("img");

if you don't need to hide your image source and if server is accessible from the client as well, I'd just point your response to remote server (as you already have the url) => you don't need to do a download to your server first, but possibly client could access it directly => you don't waste your resources.
However if you still need to download it to your server first, following post might help: Writing image to servlet response with best performance

It's important to avoid intermediate buffering of image in servlet. Instead just stream whatever was received to the servlet response:
InputStream is = new URL(imgUrl).openStream();
OutputStream os = servletResponse.getOutputStream();
IOUtils.copy(is, os);
is.close();
I'm using IOUtils from Apache Commons (not necessary, but useful).

The complete solution : download a map and save to file.
String imgUrl = "http://maps.googleapis.com/maps/api/staticmap?center=-15.800513,-47.91378&zoom=11&size=200x200&sensor=false";
InputStream is = new URL(imgUrl).openStream();
File archivo = new File("c://temp//mapa.png");
archivo.setWritable(true);
OutputStream output = new FileOutputStream(archivo);
IOUtils.copy(is, output);
IOUtils.closeQuietly(output);
is.close();

how to write a file object on server response and without saving file on server?

I am using Spring with DWR . I want to return a file object as response , however I save the file (to be sent) at server temporary location and then send its location as href for anchor tag on client side , however I wonder if there could be a way to throw the file directly to browser on response object without saving it temporarily on server.
I expected if there could be a way to send file as a response via DWR.

public ModelAndView writeFileContentInResponse(HttpServletRequest request, HttpServletResponse response) throws IOException {
FileInputStream inputStream = new FileInputStream("FileInputStreamDemo.java"); //read the file
response.setHeader("Content-Disposition","attachment; filename=test.txt");
try {
int c;
while ((c = inputStream.read()) != -1) {
response.getWriter().write(c);
}
} finally {
if (inputStream != null)
inputStream.close();
response.getWriter().close();
}
}

It has been years since I've used Spring, and I'm unfamiliar with DWR, but the essence of your question is basic to the web.
The answer is yes, you can. In effect, you need to set the HTTP header Content-Disposition: attachment, then stream down the contents. All of this will be in the response to the original request (as opposed to sending back a link).
The actual code to achieve this will depend on your circumstances, but this should get you started.

you call the method from Java Script, right? I didn't really understand how Spring is related in this flow, but as far as I know DWR allows you to produce Java Script Stubs and call the Java methods of the exposed bean directly on server right from your java script client code.
You can read the file byte-by-byte and return it from your java method as long as it really returns a byte array.
However what would you do with this byte array on client?
I just think in this specific flow you shouldn't use the DWR but rather issue an ordinar AJAX request (if DWR can wrap it somehow for convenience - great). This request shouldn't come to DWRServlet, but rather be proceeded by a regular servlet/some web-based framework, like Spring MVC :)
Once the request comes to the servlet, use
response.setHeader("Content-Disposition","attachment; filename=test.txt");
as was already stated.
Hope this helps,
Good luck!
Mark

An example which return a excel to download from client:
//Java side:
public FileTransfer getExcel(Parametros param){
byte[] result = <here get data>;
InputStream myInputStream = new ByteArrayInputStream(result);
String excelFormat = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
FileTransfer dwrExcelFile = new FileTransfer("excel.xlsx", excelFormat, myInputStream);
return dwrExcelFile;
}
//Javascript side:
function downloadExcelFile() {
dwr.engine.setTimeout(59000);
var params = <params_to_send>;
<Java_class>.getExcel(params, {callback:function(dataFromServer) {
downloadExcelCallback(dataFromServer);
}});
}
function downloadExcelCallback(data) {
dwr.engine.openInDownload(data);
}

how read a remote file with GWT

I need to read a file located on a server but I see that in GWT is not possible use some java library.
what I have to do?

try requestBuilder!! this code can help?
RequestBuilder requestBuilder = new RequestBuilder( RequestBuilder.GET, "yourfile.txt" );
try {
requestBuilder.sendRequest( null, new RequestCallback(){
public void onError(Request request, Throwable exception) {
GWT.log( "failed file reading", exception );
}
public void onResponseReceived(Request request, Response response) {
String result=response.getText();
}} );
} catch (RequestException e) {
GWT.log( "failed file reading", e );
}

The Rule: JavaScript cannot read data from a URL that doesn’t have a host name and port that matches those of the page the JavaScript is running in.
In other words: If it is on a different site — you can’t read it directly with JS and therefore GWT, which is nothing more than Javascript once compiled.
It applies to data from XMLHttpRequest, frames, and anything else you care to name.
This may change in the future, but for now the rule stands.
With this in mind there are a couple of workarounds.
1) Call your server with RPC or whatever mechanism and have your server do the request and then send it back to the client. Here is a sample.
2) There are several hacks on allowing JavaScript to access cross-domain sites just do a google search on how to get this. Some browsers will flag this as being dangerous.
3) If you are using Firefox and Firefox only it looks like Firefox has the ability to do this, but you will need to enable this manually.

Simply write first a servlet that sends the file located on the server to the user.
Then when the user clicks on a button for instance you call the servlet with the proper parameter.
Here is an excerpt from our servlet implementation
response.reset();
response.setContentType("application/octet-stream");
response.setContentLength(contentLength);
response.setHeader("Content-disposition", "attachment;
filename=\"" + filename + "\"");
output = new
BufferedOutputStream(response.getOutputStream());
int data = input.read();
while (data != -1)
{
output.write(data);
data = input.read();
}
output.flush();