I am writing an application to use JAVA POI to read over 65536 records from oracle table and write to EXCEL .XLSX via JAVA POI library XSSF and SXSSF.
However, the error "out of memory" pops up. We have tried the following ways but the problem still occurs.
Load 66000 records into memory and write to .XLSX file
a. Larger heap size “Java.exe –Xmx4096m –Xmx4096m –XX:MaxPermSize=256m” is applied
Result : It takes about an hour to run and the file cannot be created.
Process 66000 records by region
a. load each region records’ of a season from oracle table into memory
b. append each region records from memory to a single .XLSX file
c. Larger heap size Java.exe –Xmx4096m –Xmx4096m –XX:MaxPermSize=256m is applied
Result : When it processes about 30000 records, the java memory error still pops up.
Except rewriting the program by using XML, is there any chance of fixing the memory error issue of JAVA POI XSSF and SXSSF ?
Please kindly advise
Related
I want to process an excel file with java spring. I am using apche poi to process the file. The excel file is auto generated and keeps growing. Example: Excel file has 20 lines on day 1. On day 2 the excel file has 35 lines. The first 20 lines are the same, but there are 15 new lines. It is unknown how many lines are added or when the excel will be uploaded.
The data from the excel is mapped to POJOs and saved to the database.
Is there a fast and reliable way to identify which new lines were added and only proccess those lines?
edit: I realised that this might not be an excel processing problem but (also) a database optimisation problem.
You can use the newer API of Apache POI, SXSSF, which an API-compatible streaming extension of XSSF to be used when very large spreadsheets have to be produced, and heap space is limited. It consumes less memory. Check this link.
Excel allows around 65K records in a .xls file.
For more data, I am using .xlsx file .
I used HSSF previously and now using XSSF for reading this xlsx in java.
Now my question is, when I save data in .xls (<65K ) and run with HSSF I am able to
and when storing it in .xlsx and reading with XSSF(<65K still) my program throws out of heap memory.
Want to know if it's even possible because I verified it multiple times.
What I also observed the same data as xls has file size 19MB and as xlsx as 16MB
still reading xls doesn't throw memory issue but xlsx does.
Any idea how to solve it
ans what might be the reason causing this behaviour.
I have tried changing multiple sizes from 1024M to 4096M
It either fails to create a VM with given size or throws out of heap space.
Why is it specifically for xlsx read and not when xls read.
In the existing project user can download report and it has 10 million records, this process gets data from database and writes to csv by using super csv java api then sends an email to user by attaching, it takes huge heap space to hold 10 million java objects and writing these records to csv files, because of this server is crashing and going down as application has many reports like this. is there any better way to handle this.? I red sxssfworkbook documentation and it says specified records count can keep in memory and remaining records will be pushed to hard disk but this is using to create excel files. is there any similar api to create csv files or sxssfworkbook can be used to create csv files.?
There are few Java libraries for reading and writing CSV files. They typically support "streaming", so they do not have the problem of needing to hold the source data or the generated CSV in memory.
The Apache Commons CSV library would be a good place to start. Here is the User Guide. It supports various of flavors of CSV file, including the CSV formats generated by Microsoft Excel.
However, I would suggest that sending a CVS file containing 10 million records (say 1GB uncompressed data) is not going to make you popular with the people who run your users' email servers! Files that size should be made available via an web or file transfer service.
When I create xlsx file with Apache POI sometimes (when the file is big) it creates such a file that can't be opened by this same Apache POI while MS Excel or LibreOffice Calc open it without problems.
When I try to open this workbook with Apache POI it says that
Zip bomb detected
I can open it only if I call ZipSecureFile.setMinInflateRatio(0) or resave it in LibreOffice (MS Excel doesn't help here).
How to fix this? Why POI creates file which it can't open?
Simply do as the error message suggests and set the limits differently via
ZipSecureFile.setMinInflateRatio(0)
You seem to have a rather special use-case which produces a file that is similar to some files that malicious users could use to make your server crash, use up CPU or go out of memory. To avoid this, Apache POI has this limit, but allows to set it differently if needed. So if the file is not coming from untrusted users, you can easily adjust these limits to avoid the error message.
Excel or LibreOffice might optimize the file-content more than Apache POI does and thus produce a file that does not reach these limits.
i've been searching around for a solution for this problem and haven't found anything good =(
so the problem is i need to create an excel file up to 50.000 registers and when i do this show me this error at 50.000 app register:
java heap space
as far as i'm looking on, one way to solve this is increasing the heap memory, but quite don't like me (first aids) because the problem persist if the file gets to bigger and the server freezes and comsume a lot of memory trying to do this and gets slow
i think a solution would be writte out the excel file instead of leaving the data in the heap memory, but i don't get it work
any ideas ?? or another framework for excel 2007 files ???????
Call the write() method on your HSSFWorkbook to persist your Excel workbook to a file:
FileOutputStream fileOut = new FileOutputStream("workbook.xls");
wb.write(fileOut);
fileOut.close();
But as this thread indicates there's no way to do streaming writes to file in POI. Alternatives to POI are jXLS (open source) or Aspose.Cells for Java (proprietary)