I am creating a REST service that loads data from Oracle database table using JDBC, and saves the resultset in CSV.
Since the table is very large, it's expected that the process will take about one hour.
How can I download CSV while saving data to it
(so we get a cycle like this: load chunk (some amount of rows), save to CSV, and flush that part (download) )?
this is intended to:
prevent holding the whole file in server memory, by flushing it periodically to the client.(Assuming JDBC driver does not fetch all table immediately)
show (almost) immediate progress to the user (so user won't wait until CSV is complete)
something like this is good enough:
#GET
#Produces("text/csv")
#Path("/get")
public Response getData(#Context HttpServletRequest request,
#Context HttpServletResponse response) throws IOException {
response.setHeader("Content-Disposition", "attachment; filename=data_file.csv");
ServletOutputStream outputStream = response.getOutputStream();
// here you read from ResultSet.
while (resultSet.next()) {
byte b = (byte) (resultSet.getByte("columnA"));
outputStream.write(b);
}
outputStream.flush();
outputStream.close();
return Response.ok().build();
}
Related
There is an API that downloads Excel files. Whenever we click on the 'Download excel report' button, it calls this API which gets data from the database, transfers it to an Excel file, and downloads it.
excelName = key + ".xlsx";
response.setHeader("Content-Disposition", "attachment; filename=" + excelName);
ByteArrayInputStream in = getDownloadBusinessAnalysisKey(key, customerScopeId, response, getBusinessKeyResult, sheetName);
response.setHeader("Content-Length", String.valueOf(in.available()));
try {
FileCopyUtils.copy(in, response.getOutputStream());
} catch (IOException e) {
logger.error("Error in report " + e.getLocalizedMessage());
}
It downloads fine when there are about 10,000 records (or close to it).
However, when there are about 100,000 records it just doesn't download. I get no response from the endpoint. It works when trying to download the same file via Postman in local database.
Is there some limit on download size? Would this be a browser, Tomcat server, or HTTP header issue?
When you're using a ByteArrayInputSteam you're loading the whole data into memory. What needs to be done is to make getDownloadBusinessAnalysisKey store to a place you can stream, usually to a file. Then use FileInputStream to read the file and write it to the response output stream.
Based on your statement that you're using an API to download the Excel file, I am assuming it is not done inside your current JVM (otherwise I'd recommend you look up SXSSF which would require a bit of code change and rethinking to get it to work.
From your code sample it looks like you're using Servlet API as well based on the setHeader. So here's a bit of code that would make a get connection and more or less proxy it up. There's no need for a temporary file if it is just straight through, there's also no need to buffer unless you can confirm that the servlet engine does not provide you with buffered data.
protect void doGet(
HttpServletRequest req,
HttpServletResponse response)
throws ServletException, IOException {
var url = new URL("http://myapi");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
int responseCode = con.getResponseCode();
int len = con.getHeaderFieldInt("Content-Length",-1);
int contentType = con.getHeaderField("Content-Type", "application/octet-stream");
assert responseCode == 200
response.setIntHeader("Content-Length", len);
response.setHeader("Content-Type", contentType);
try (
InputStream is = con.getInputStream();
OutputStream os = response.getOutputStreeam();
) {
int c = is.read();
while (c != -1) {
os.write(c);
c = is.read();
}
}
}
This can be optimized by using Async Servlet APIs to reduce blocking I/O which would be needed if you are dealing with a lot of connections.
I think that the process to get data from db and fetch into excel taking too long time and it is longer than max response time of server. You should tuning the process getting data from db and fetching into excel, it will solve your issue.
final StreamingOutput stream = new StreamingOutput() {
#Override
public void write(final OutputStream out) {
dao.getData(
query,
new SdmxObserver(writerFactory.getDataWriter(sdmxFormat, out, properties), request
.getRemoteAddr(), request.getHeader("User-Agent"), query.toString(), sdmxFormat
.toString(), query.getlist()));
}
};
res = Response.ok(stream, MediaType.valueOf("application/vnd.sdmx.genericdata+xml;version=2.1"))
.cacheControl(cc).lastModified(lastModified).header("Vary", "Accept,Accept-Encoding").build();
return res;
The database call to retrieve the data is taking long as the data is huge and for that before downloading the data in the browser it is taking long time and user thought the server is not reachable. How could we send the chunk of data by using multiple database query in the same web service response, so that the download starts quickly and keep adding the response when next query fetch the data from database.
Paginate the response content into pages up to X records. This will allow the API to retrieve a single page of results at a time, one by one.
http://localhost/api/v1/resource?page=1&size=20
http://localhost/api/v1/resource?page=2&size=20
http://localhost/api/v1/resource?page=3&size=20
...
You can take a look at PageRequest and Pageable defined in Spring Data when defining your API.
On my company's site we have some tables that we need to export to a csv file.
There are some varying parameters, so the csv file needs to be dynamically created on request.
My problem is that after clicking to download, the response hangs, and waits for the whole file to be created (which can take some time) and only then downloads the entire file in one instant.
I'm using AngularJS, so I'm using window.location = <url_for_file_download> In order to make the browser download the file.
On the server side I'm using Java Spring and I've followed all the instructions I could find on the web in order to create a file download controller.
My controller code is something like this:
#RequestMapping(value = "http://yada.yada.yada/csv/myFile.csv", method = RequestMethod.GET)
public #ResponseBody
void getCustomers(HttpServletResponse response,
#RequestParam(required = false) String someParameters)
throws NotAuthorizedException, IOException {
// set headers
setHeaders(response);
// generate writer
CSVWriter write = generateWriter(response);
// get data
List<String[]> data = getData();
// write and flush and all that
.
.
.
}
My code for setting the response headers are:
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition","attachment; filename=\"" + fileName + ".csv\"");
I've also tried adding the following headers:
response.setHeader("Transfer-Encoding", "Chunked");
response.setHeader("Content-Description", "File Transfer");
and I've also tried setting the Content-type to "application/octet-stream".
Notice that I don't add a Content-length header, since the file doesn't exist yet, and is being written on the fly.
For writing the csv file I'm using OpenCSV and my code is as follows:
OutputStream resOs = response.getOutputStream();
OutputStream buffOs = new BufferedOutputStream(resOs);
OutputStreamWriter outputWriter = new OutputStreamWriter(buffOs,"UTF-8");
CSVWriter writer = new CSVWriter(outputWriter);
I iterate over the data and write it like so:
for (String[] row: data) {
writer.writeNext(line);
}
(It's not exactly the code - but this is more or else what happens in the code)
And at the end I flush and close:
writer.flush();
writer.close();
I also tried flushing after each line I write.
So why isn't the file being transferred before it has all been written?
Why is my browser (Google chrome) downloading the file in one instant after waiting a long time? And how can I fix this.
I hope I've added enough code, if there's something missing just please tell me and I'll try to add it here.
Thank you so much in advance.
Can you try returning a null value in your java
return null ;
Or you can try below code also
1. Jquery code upon clicking the submit button
$(document).ready( function() {
$('#buttonName').click(function(e){
$("#formName").submit();
//alert("The file ready to be downloaded");
});
});
Your controller code
#RequestMapping(value="/name",method=RequestMethod.POST)
public ModelAndView downloadCSV(ModelMap model,HttpSession session,#ModelAttribute(value="Pojo") Pojo pojo
,HttpServletRequest request,HttpServletResponse response){
----------------some code----------------
response.setContentType("application/csv");
("application/unknown");
response.setHeader("content-disposition","attachment;filename =filename.csv");
ServletOutputStream writer = response.getOutputStream();
logger.info("downloading contents to csv");
writer.print("A");
writer.print(',');
writer.println("B");
for(int i=0;i<limit;i++){
writer.print(""+pojo.get(i).getA());
writer.print(',');
writer.print(pojo.get(i).getB());
writer.println();
}
writer.flush();
writer.close();
---------------some code-----------
return null;
}
Hope this helps
The Controller will wait for the response to be written before the response is send back to the client.
Here is a nice post with multiple approaches / options outlined
Downloading a file from spring controllers
This post talks about flushing the output periodically to help fasten the download.
how to download large files without memory issues in java
If all you are trying to do is let the user know that the file download is in progress and due soon, I think an Ajax progress status indicaor might be your solution.
Trigger the ajax call to the back-end to generate the file
Show progress indicator to the user while file is being generated server side
once response is available, file is presented to the user.
I think something similar is being explored here download file with ajax() POST Request via Spring MVC
Hope this helps!
Thanks,
Paul
I faced the same issue. The code that didn't work for me was
#RequestMapping(value = "/test")
public void test(HttpServletResponse response)
throws IOException, InterruptedException {
response.getOutputStream().println("Hello");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("How");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("are");
response.getOutputStream().flush();
Thread.sleep(2000);
response.getOutputStream().println("you");
response.getOutputStream().flush();
}
The culprit was ShallowEtagHeaderFilter. When this filter is enabled the response is sent in one chunk. When this filter is diabled the response is send in multiple chunks.
From this thread Tomcat does not flush the response buffer it looks like another possible culprit can be GzipFilter
I am using Apache POI for generating Excel file in Java Servlets.
getExcel() function returns HSSFWorkbook, which I want to send to the client.
HSSFWorkbook wb = getExcel();
This is what I have tried so far.
//block1
ByteArrayOutputStream outByteStream = new ByteArrayOutputStream();
wb.write(outByteStream);
byte [] outArray = outByteStream.toByteArray();
response.setContentType("application/ms-excel");
response.setContentLength(outArray.length);
response.setHeader("Expires:", "0");
response.setHeader("Content-Disposition", "attachment; filename=Demo1.xls");
OutputStream outStream = response.getOutputStream();
outStream.write(outArray);
outStream.flush();
//block2
request.setAttribute("Message", str1);
request.setAttribute("MessageDetails", str2);
request.getRequestDispatcher("/MyFile.jsp").forward(request, response);
Above code sends excel file to the client, but gives me exception:
java.lang.IllegalStateException: Cannot forward after response has been committed
If I remove the block1 or block2 from above code then it will not give error, but I want to send client Excel file and two attributes which I have added to request object.
So can send Excel file to client using request.getRequestDispatcher ? Or is there any better way for doing this?
Any suggestion will be appreciated.
Edit1
I know why I am getting the IllegalStateException, but then my question is how should I send ExcelFile and Request Attributes both to the client?
Edit2
The Reason why I want to send both Excel file and Attributes to the client is that MyFile.jsp has a <div> which will show message send from servlet.
<div style="background-color: aliceblue">
<h3>${Message}</h3>
</div>
Edit3
The Reason why I want to send message to client is that I am sending this Excel file as an response to Import Excel operation in which client will provide excel file for inserting data in database, and then I am highlighting the excel rows which cannot be inserted due to duplication or any other reasons. So I want to show Import statistics in the Message to client and give him copy of excel file with error rows highlighted.
You are flushing your response and then trying to forward. Container has already sent the response back to the client and now is in a dilemma as to how to forward the request to another JSP, hence it aborts the operation mid way throwing an Exception. HTTP is a request-response model . Once you request , you get back a response . But once the response is already committed the whole transaction is over.
outStream.write(outArray);
// you already committed the response here by flushing the output stream
outStream.flush();
//block2
request.setAttribute("Message", str1);
request.setAttribute("MessageDetails", str2);
// this is illegal after you have already flushed the response
request.getRequestDispatcher("/MyFile.jsp").forward(request, response);
As per the Javadoc:
IllegalStateException - if the response was already committed.
After EDIT1:
No you cannot do both . You need to decide what you want. Write the bytes to the response setting proper HEADERS and MIME-TYPE. You cannot get the browser download something as well as show a JSP page from the same response.
In action method (JSF) i have something like below:
public String getFile() {
byte[] pdfData = ...
// how to return byte[] as file to web browser user ?
}
How to send byte[] as pdf to browser ?
In the action method you can obtain the HTTP servlet response from under the JSF hoods by ExternalContext#getResponse(). Then you need to set at least the HTTP Content-Type header to application/pdf and the HTTP Content-Disposition header to attachment (when you want to pop a Save As dialogue) or to inline (when you want to let the webbrowser handle the display itself). Finally, you need to ensure that you call FacesContext#responseComplete() afterwards to avoid IllegalStateExceptions flying around.
Kickoff example:
public void download() throws IOException {
// Prepare.
byte[] pdfData = getItSomehow();
FacesContext facesContext = FacesContext.getCurrentInstance();
ExternalContext externalContext = facesContext.getExternalContext();
HttpServletResponse response = (HttpServletResponse) externalContext.getResponse();
// Initialize response.
response.reset(); // Some JSF component library or some Filter might have set some headers in the buffer beforehand. We want to get rid of them, else it may collide.
response.setContentType("application/pdf"); // Check http://www.iana.org/assignments/media-types for all types. Use if necessary ServletContext#getMimeType() for auto-detection based on filename.
response.setHeader("Content-disposition", "attachment; filename=\"name.pdf\""); // The Save As popup magic is done here. You can give it any filename you want, this only won't work in MSIE, it will use current request URL as filename instead.
// Write file to response.
OutputStream output = response.getOutputStream();
output.write(pdfData);
output.close();
// Inform JSF to not take the response in hands.
facesContext.responseComplete(); // Important! Else JSF will attempt to render the response which obviously will fail since it's already written with a file and closed.
}
That said, if you have the possibility to get the PDF content as an InputStream rather than a byte[], I would recommend to use that instead to save the webapp from memory hogs. You then just write it in the well-known InputStream-OutputStream loop the usual Java IO way.
You just have to set the mime type to application/x-pdf into your response. You can use the setContentType(String contentType) method to do this in the servlet case.
In JSF/JSP you could use this, before writing your response:
<%# page contentType="application/x-pdf" %>
and response.write(yourPDFDataAsBytes()); to write your data.
But I really advise you to use servlets in this case. JSF is used to render HTML views, not PDF or binary files.
With servlets you can use this :
public MyPdfServlet extends HttpServlet {
protected doGet(HttpServletRequest req, HttpServletResponse resp){
OutputStream os = resp.getOutputStream();
resp.setContentType("Application/x-pdf");
os.write(yourMethodToGetPdfAsByteArray());
}
}
Resources :
mimeapplication.net - pdf
Javadoc - ServletResponse
Javadoc - HttpServlet
When sending raw data to the browser using JSF, you need to extract the HttpServletResponse from the FacesContext.
Using the HttpServletResponse, you can send raw data to the browser using the standard IO API.
Here is a code sample:
public String getFile() {
byte[] pdfData = ...
FacesContext context = FacesContext.getCurrentInstance();
HttpServletResponse response = (HttpServletResponse) context.getExternalContext().getResponse();
OutputStream out = response.getOutputStream();
// Send data to out (ie, out.write(pdfData)).
}
Also, here are some other things you might want to consider:
Set the content type in the HttpServletResponse to inform the browser you're sending PDF data:
response.setContentType("application/pdf");
Inform the FacesContext that you sent data directly to the user, using the context.responseComplete() method. This prevents JSF from performing additional processing that is unnecessary.