I have a requirement where i need to generate reports of around millions of records and later export it to Excel. am using jasper reports for the purpose. Am able to preview the report. But when i try to export the report to Excel, getting Heap Memory error. But am able to get the entire dataset in the ByteArrayOutputStream object. After a long Googling came to a conclusion that it is common if the data is so huge. So i decided to write the data to excel chunk by chunk. But how to achieve that? How to divide the data and how to write data to excel part by part so that the final output file is a consolidated one. Please suggest.. Thanks in advance.
Have you tried to use "poi" instead of jasper reports?
Poi is a library for Microsoft Documents by Apache. Link: http://poi.apache.org/
In this library there is a class called "SXSSFWorkbook" that uses streaming technology. Read the javadoc for more information: http://poi.apache.org/apidocs/index.html
Related
In the existing project user can download report and it has 10 million records, this process gets data from database and writes to csv by using super csv java api then sends an email to user by attaching, it takes huge heap space to hold 10 million java objects and writing these records to csv files, because of this server is crashing and going down as application has many reports like this. is there any better way to handle this.? I red sxssfworkbook documentation and it says specified records count can keep in memory and remaining records will be pushed to hard disk but this is using to create excel files. is there any similar api to create csv files or sxssfworkbook can be used to create csv files.?
There are few Java libraries for reading and writing CSV files. They typically support "streaming", so they do not have the problem of needing to hold the source data or the generated CSV in memory.
The Apache Commons CSV library would be a good place to start. Here is the User Guide. It supports various of flavors of CSV file, including the CSV formats generated by Microsoft Excel.
However, I would suggest that sending a CVS file containing 10 million records (say 1GB uncompressed data) is not going to make you popular with the people who run your users' email servers! Files that size should be made available via an web or file transfer service.
My application requires a reporting facility in excel/csv format. In case of large report, the generated CSV is corrupt. Though i am able to e-mail the generated CSV using smtp.
I tried changing the following with no lead, your help on this is appreciated
Change the library to POI
Changed the library to JXL
Monitored if there is a memory leakage
This is a web based application and the code is written in JSP.
POI is mainly for MS office formats like xls, xlsx, doc. JXL is also for xls files. You should use a framework which is for CSV like OpenCSV.
I has a excel file with 4 excel sheets in it. Now i want to read or write to required excel sheets using java without using any third party lib.
I know i can read and write data using FileInputStream and FileOutputStream respecitvely. But i can handle the work sheets??
No, you can'not, There is numerous way in Java for reading/writing files, but there is no built-in support for MS Office/Excel spreadsheets. http://poi.apache.org/ - is a key to victory.
If your goal is to interface with data from an excel sheet from your Java application, I'd suggest to use the solutions suggested by other posters, it will save you a lot of work.
If, however, you want to be able to read excel files from Java (or any other programming language for that matter) 'just because you can' then you could take a look at this file and read the instructions on this web-page. I would warn you that it would take considerably more time to implement your own API if you base it only on the file-specs that are publicly available. You might want to check out the work done by the people from the Apache POI project to get an idea of how to approach it. Or (even better) contribute to the project. Here you can find out how to go about doing that
I'm trying to read excel file and pass all the data to DB. I found a few code examples but all of them required external jars. How can I read excel files using only the standard library?
IF you don't want to use a library then you will have to download the Excel file format specs from MS and write an Excel parser yourself (which is extremely complicated and takes > 10 years for one developer). For the OpenXML format spec see here and here.
Thus I really recommend using a library for that...
Try Apache POI - a free Java library for dealing with MS Office documents..
You can save as the excel file *.cvs and sperated ";". Then, you can read file line by line and get the columns which is getting from each token.
Microsoft excel uses a binary way to save its data, so manually reading excel files might be a hassle. If you could convert the excel (xls) to a comma seperated values (csv) file, then you can just read the file and split your input on the comma's.
This is a difficult problem. First off, it is not as simple as "adding a third party library". There are no existing EXCEL reading libraries that do not cost money and the one that I know that does work is very expensive AND has bugs in it.
One strategy is to create an Excel add in that reads the data and transfers it to your application by OLE or the clipboard or by a TCP/IP port or saves it to a temporary file. If you look in the source code for OPeNDAP.org's ODC project you can find an Excel add in and TCP capability to do this.
You can try referring to the reader in OpenOffice which is open source code, however, in my opinion that code is not easily refactorable into a private project for various reasons.
Microsoft has components and tools to open Excel files and expose them via COM objects.
You can also learn the BIFF format and write your own parser. You probably would want to write a parser for BIFF5, but be forewarned, this is a BIG project, even if you only parse a limited number of data types.
I am very new to java. I am trying to fetch some data from a database and the result set is displayed in excel. I am able to inetract with database. But how should I go ahead for inserting data into excel sheet.Its simple Java program and in future I would like to generate files in other format say PDF, doc etc.
I am looking for an approach with lesser load on CPU, faster.
thanks in advance for help.
Just spit out a CSV file. It's lightweight and portable. You can grab a csv writer from Apache commons I think but spending the 10 min it would take to write one might be a good learning exercise as well.
If you want a real solution where you want different outputs (eg excel, pdf, rich text etc) then use a reporting tool. There are plenty of opensource tools like ireport which will let you create a template then write a simple java app that renders that to pdf, excel etc. Otherwise you will end up doing it by hand. It's a bit heavyweight but anything more than trival tabular output will be easier.
Apache POI is for you in this case, but you will find it a little bit overwhelmed if you just need to write/read data from an excel file.
Try jExcel instead, the API is simple and straightforward, you can also manipulate sheets within an excel workbook.
The easiest and standard way of doing this is to use POI library:
http://poi.apache.org/spreadsheet/quick-guide.html
The new xlsx format is based on Open-XML and would provide a method of generating these files without any dependency on Microsoft-office COM libraries -- the same could done for docx and pptx formats later, as well as other open-Xml formats like EPUB.
The Apache POI project looks like it might provide one possible solution. There's also an article on the MSDN interop blog that discusses this in some detail.
The key words you should google for are OLE and DDE.
Though, Java is not the best language for interface Microsoft's software.
for generating excel i think you should try simreport in jsimreport.com. in my opinion, it's quite simple to make an excel report, it uses the excel sheet to generate report so easy to config and visually