I am developing a website that stores data in a mysql database. One of the user of the website wishes to record data for the website in an excel spreadsheet. I intend to develop a solution that consist of three parts:
A controller class that saves the uploaded excel spreadsheet into a directory in the server.
A service layer that read the saved excel spreadsheet from the directory using apache POI. 3. JPA to record data in the excel spreadsheet into the mysql database.
Is this a good pattern for the intended solution ? Has anyone implemented a similar solution ?
Sounds reasonable.
See if the user would upload a csv file instead of xls file, as this will be easier to handle.
try to stream upload directly to disk. If you store it in memory, then large uploads will cause out of memory error for jvm.
Related
I'm writing a portlet web app to allow user upload a excel file, and then the backend will do the validation to the excel file and convert the content into sql statement
The back-end is using java with library aspose.cells to handle the excel access.
The problem is, how can I pass the excel to back end to access.
Now I've think of after user upload the excel file, the back end first will download the excel file to local (maybe C:\temp), and then it can directly read the excel file from C:\temp rather than server file.
But it's possible to do it?
You seem to be looking for some sample code where you may upload an Excel file and then using Aspose.Cells for Java, process this uploaded file. You may have a look at the following link where a tutorial is there for uploading files using portlets.
http://www.codeyouneed.com/liferay-portlet-file-upload-tutorial/
Once the file is uploaded, use com.aspose.cells.Workbook class object for loading this stream directly into Workbook and then process it as per your requirement.
Following is the link to sample code for loading Excel file (in stream) into Workbook object:
https://docs.aspose.com/display/cellsjava/Opening+Files+with+Different+Formats#OpeningFileswithDifferentFormats-OpeningthroughStream
Note: I am working as Support developer/ Evangelist at Aspose.
In the existing project user can download report and it has 10 million records, this process gets data from database and writes to csv by using super csv java api then sends an email to user by attaching, it takes huge heap space to hold 10 million java objects and writing these records to csv files, because of this server is crashing and going down as application has many reports like this. is there any better way to handle this.? I red sxssfworkbook documentation and it says specified records count can keep in memory and remaining records will be pushed to hard disk but this is using to create excel files. is there any similar api to create csv files or sxssfworkbook can be used to create csv files.?
There are few Java libraries for reading and writing CSV files. They typically support "streaming", so they do not have the problem of needing to hold the source data or the generated CSV in memory.
The Apache Commons CSV library would be a good place to start. Here is the User Guide. It supports various of flavors of CSV file, including the CSV formats generated by Microsoft Excel.
However, I would suggest that sending a CVS file containing 10 million records (say 1GB uncompressed data) is not going to make you popular with the people who run your users' email servers! Files that size should be made available via an web or file transfer service.
Is there a way to read large xls files?
I have used Apache POI to read files but only till some limit.
I have a database which has some data and now I want to upload a file of large size (like i told). This file contains little more data in one of its sheets as compare to the data that database has. now how should i update that data in oracle database without setting up the xls file.
For large xls files you should use the streaming extension of XSSF, called SXSSF. It should be able to handle your requirements without memory problems.
As for the database problem: I'd suggest you read the xls files using XSSF and create temporary tables for each file in the database (depending on your needs). Once all the rows are stored in temp tables you can easily merge them with your existing data.
Here's the problem:
Run a java client as a batch job on a unix box which will connect to Oracle and fetch some data.
Update an excel sheet (on a Windows machine) with the data fetched.
Create a chart/graph from the data in excel sheet.
Send the excel (with data & graph) in an email.
All the above steps must be performed without any manual intervention.
Could there be a better option to excel based solution? The main intent is to have a history of data fetched and have a chart created from that data.
I know there are many open source libraries available for creating charts like JFreeChart, but is there anything in the JDK that allows you to create charts? Could JavaFX be used for this problem?
In short, is it possible to do this with just jdk (without using any open source libraries)?
Any help/suggestion will be appreciated.
You could try google docs spread sheet api to workaround MS-Excel
I think all that is pretty doable from Google docs, and yet, you still have the option of downloading the spread sheet.
From the link:
Spreadsheets Data API
The Spreadsheets Data API lets you access worksheet data within your own application or website. You can view and modify data, create and delete worksheets, issue structured queries, and more.
Spreadsheets Gadgets & Visualization API
Spreadsheets Gadgets take advantage of the Google Visualization API to embed graphical comparisons of of structured data within a spreadsheet.
Sounds like what you need.
Pretty straightforward, just use Hibernate or even jdbc.
You should be able to update the Excel sheet with Apache POI. You could also try Java Excel Designer or http://www.moyosoft.com/jec/
At least one of the above should be able to create charts.
Just use Java Mail
You could wrap the above in ANT tasks as appropriate.
I am having a database in .dbf (FoxPro) format.
How to retrieve data from FoxPro using Java?
If the data can be migrated to MySQL, How to do the conversion?
Taking the data to intermediate formats seems flawed as there are limitation with memo fields and CSV or Excel files.
If you are interested in a more direct approach you could consider something like "VFP2MySQL Data Upload program" or "Stru2MySQL_2", both written by Visual FoxPro developers. Search for them on this download page:
http://leafe.com/dls/vfp
DB-Convert (http://dbconvert.com/convert-foxpro-to-mysql-sync.php) is a commercial product that you might find helpful.
Rick Schummer, VFP MVP
You can use XBaseJ to access (and even modify write) data from FoxPro databases directly from Java with simple API.
This would allow you to have the two applications (the old FoxPro and the new Java one) side by side by constantly synchronizing the data until the new application is ready to replace the old one (e.g. many times the customers still hang on and trust more their old application for a while).
Do you have a copy of FoxPro? You can save the database as an HTML file, if you want. Then, from HTML, you can save to any format you want. I recently did this to save a FoxPro table as an Excel spreadsheet (not that I'd suggest using that for your Java code).
If you plan on using Java, once you have access to the data, why not use one of Java's native storage types?
I worked on the same project once long back where the project had be done with FoxPro and then we migrated that project to Java with MySQL.
We had the data in Excel sheets or .txt files, so we created tables as exact replica of the FoxPro data and transferred the data from the Excel/CSV /txt to MySQL using the Import data feature.
Once we did this, I think further you can take care from MySQL Data.
But remember work will take some time, and we need to be patient.
I suppose doing a CSV export of your FoxPro data and then writing a little Java programme that takes the CSV as input is your best bet. Writing a programme that both connects to FoxPro and MySQL in Java is needlessly complicated, you are doing a one time migration.
By the way PHP could do an excellent job at inserting the data into MySQL too. The main thing is that you get your data in the MySQL schema, so you can use it with your new application (which I assume is in Java.)
Two steps: DBF => CSV and the CSV => MySQL.
To convert DBF(Foxpro tables) to CSV the below link helps a lot
http://1stopit.blogspot.com/2009/06/dbf-to-mysql-conversion-on-windows.html
CSV => MySQL
MySQL itself supports CSV import option (or) to read csv file this link helps
http://www.csvreader.com/java_csv.php
I read the CSV file using Java CsvReader and inserted the records through program. For that i used PreparedSatement with Batch the below link gives samples for that
http://www.codeweblog.com/switch-to-jdbc-oracle-many-data-insert/