Develop Reference

Java Apache POI and JDBC: Smart way to fetch huge amount of data into Excel [duplicate]

This question already has answers here:
Writing a large resultset to an Excel file using POI
(7 answers)
Closed 6 years ago.
I want to fetch a huge amount of data (over 100.000 rows with 50 columns) from my Oracle database into an Excel File using Java Apache POI and Oracle JDBC. That is working well for small amount of data but what is the smartest way to handle tasks like this? Maybe using multithreading? I get errors like "java.lang.OutOfMemoryError: Java heap space" although I already increased the memory to 1024MB.
This is my current code:
public void fetchDataIntoExcelFile(String sqlStmt){
// Check whether jdbc driver exists or not
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
} catch (ClassNotFoundException e) {
System.out.println("Please install the Oracle JDBC Driver!");
e.printStackTrace();
return;
}
// Try to establish a connection to the db
try {
conn = DriverManager.getConnection("jdbc:oracle:thin:#myserver", "user", "password");
} catch (SQLException e) {
e.printStackTrace();
return;
}
XSSFWorkbook wbx = new XSSFWorkbook();
XSSFSheet sheet = wbx.createSheet();
FileOutputStream fos = null;
try {
fos = new FileOutputStream("filename.xlsx");
wbx.write(fos);
} catch (IOException e) {
e.printStackTrace();
System.out.println("Error occured while creating file.");
}
// Execute SQL Statement if connection was successful
if (conn != null) {
try {
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sqlStmt);
ResultSetMetaData columns = rs.getMetaData();
XSSFRow row = null;
XSSFCell cell = null;
// Start writing results into the excel file
int i = 0;
while (rs.next()){
row = sheet.createRow(i++);
for (int j = 1; j <= columns.getColumnCount(); j++){
cell = row.createCell(j);
if (i == 1){
cell.setCellValue(columns.getColumnLabel(j));
}else{
cell.setCellValue(rs.getString(columns.getColumnLabel(j)));
}
}
}
try {
wbx.write(fos);
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
} catch (SQLException e) {
System.out.println("Connection Failed!");
e.printStackTrace();
return;
}
} else {
System.out.println("Failed to make connection!");
}
}

It helped to use the SXSSF API as suggested. Here is the link:
https://poi.apache.org/spreadsheet/how-to.html#sxssf

cannot convert blob to string, SQLite+java

I used following code for retrieve data from SQLite database. The 4th column of database stored string as BLOB. Now i need to get that string values back. But Blob blob = exe.getBlob("col_4"); gave me an exception not implemented by SQLite JDBC driver. How can i solve this ?
Connection c = null;
Statement stm = null;
String db = "C:\\Users\\Asus™\\Desktop\\New folder (2)\\tempo.db";
try {
Class.forName("org.sqlite.JDBC");
c = DriverManager.getConnection("jdbc:sqlite:" + db);
c.setAutoCommit(false);
System.out.println("DB opened successfully !");
stm = c.createStatement();
int rc = 0;
try (ResultSet exe = stm.executeQuery("SELECT * FROM tblTest;")) {
while (exe.next()) {
Blob blob = exe.getBlob("col_4");
String password = new String(blob.getBytes(1, (int) blob.length()));
System.out.println(password
+ exe.getString("col_1") + "\t"
+ exe.getString("col_2") + "\t"
+ exe.getString("col_3") + "\t"
+ password
);
rc++;
}
System.out.println(rc++);
}
stm.close();
} catch (ClassNotFoundException ex) {
Logger.getLogger(SQLiteBrowser.class.getName()).log(Level.SEVERE, null, ex);
} catch (SQLException ex) {
Logger.getLogger(SQLiteBrowser.class.getName()).log(Level.SEVERE, null, ex);
}
Exception :
java.sql.SQLException: not implemented by SQLite JDBC driver
at org.sqlite.jdbc4.JDBC4ResultSet.unused(JDBC4ResultSet.java:320)
at org.sqlite.jdbc4.JDBC4ResultSet.getBlob(JDBC4ResultSet.java:345)
at sqlite.SQLiteBrowser.jButton1ActionPerformed(SQLiteBrowser.java:256)
at sqlite.SQLiteBrowser.access$000(SQLiteBrowser.java:29)
at sqlite.SQLiteBrowser$1.actionPerformed(SQLiteBrowser.java:76)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2346)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.Component.processMouseEvent(Component.java:6525)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3321)

It seems, that ResultSet#getBlob method is not supported by this JDBC library implementation (due to sources, JDBC4ResultSet doesn't implement it). You may try to use ResultSet#getBytes instead:
byte[] bytes = exe.getBytes("col_4");
String password = new String(bytes);

getBlob() method is not supportedby SQLite ... but blobs are supported ... you need to use getBytes :
byte[] bytes = exe.getBytes("col_4");

JavaFX,SQLite : How to Set BLOB Value in update Query?

I'am getting a problem whene i execute upadate query , the problem is whene i wante to update the blob column im my sqlite table, i notice that only the path this blob file are stored. but whene is use insert query with binding value like that :pst.setBinaryStream(File); it works fine!
I get my blob file with this code :
File image=null;
URL photoURL = null;
try {
photoURL = new URL(addadh_adherent_photo_label.getText());
} catch (MalformedURLException ex) {
Logger.getLogger(ParametresController.class.getName()).log(Level.SEVERE, null, ex);
}
try {
image= new File( URLDecoder.decode( photoURL.getFile(), "UTF-8" ) );
} catch (UnsupportedEncodingException ex) {
Logger.getLogger(ParametresController.class.getName()).log(Level.SEVERE, null, ex);
}
FileInputStream fis = null;
try {
fis = new FileInputStream(image);
} catch (FileNotFoundException ex) {
Logger.getLogger(ParametresController.class.getName()).log(Level.SEVERE, null, ex);
}
and here is my update query :
int AderentId =100;
String updateAderent=" UPDATE gss_aderent SET"
+ "ad_photo='"+image+"'
+ "WHERE ad_id='"+AderentId+"' ";
stmt.executeUpdate(updateAderent);
My problem is : The updated value isn't a blob value but just his path:( my question is how to store(update) blob value in update query ? does exist a method to bind BinaryStream parametre for update query ? can i use preparedStatement in update query ?

Your application looks like it's open to SQL Injection attacks
Use a PreparedStatement and setBlob(int x, InputStream in)

Reading to text file from MySQL

I'm trying to store a text file in a MySQL database, and when needed, save it to a file.
To save the file, I do:
public void saveFile_InDB(File file)
{
try {
String sql = "INSERT INTO sent_emails (fileName, time, clientName) values (?, ?, ?)";
PreparedStatement statement = conn.prepareStatement(sql);
statement.setString(1, new Date().toString());
statement.setString(2, new Date().toString());
InputStream inputStream = new FileInputStream(file);
statement.setBinaryStream(3, inputStream);
int row = statement.executeUpdate();
if (row > 0) {
System.out.println("File saved sucessfully.");
}
conn.close();
} catch (SQLException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
}
And to retreive and save the file:
public void retrieveFile_fromDB()
{
try {
Statement stmt = (Statement) conn.createStatement();
ResultSet res = stmt.executeQuery("SELECT * FROM sent_emails WHERE clientName='sally'");
FileOutputStream fos = new FileOutputStream("file.txt");
if (res.next()) {
Blob File = (Blob) res.getBlob("fileName");
InputStream is = File.getBinaryStream();
int b = 0;
while ((b = is.read()) != -1) {
fos.write(b);
}
fos.flush();
}
} catch (IOException e) {
e.getMessage (); e.printStackTrace();
System.out.println(e);
} catch (SQLException e) {
e.getMessage (); e.printStackTrace();
System.out.println(e);
}
}
Storing the file works, but when I try to retrieve and save it, nothing is stored in the output file?

if you want read file from db Mysql
change this part in your code
Blob File = (Blob) res.getBlob("fileName");
InputStream is = File.getBinaryStream();
int b = 0;
while ((b = is.read()) != -1) {
fos.write(b);
}
fos.flush();
use this code read array of bytes
byte [] bs=res.getBytes("fileName");
fos.write(bs);
it will work
if you return multiple files from db you must declare
FileOutputStream fos = new FileOutputStream("file.txt");
inside while loop and change name of file to avoid overriding

You do not seem to put into the database the things that the column names describe?
fileName and time are for example both set to a timestamp, and clientName is set to the contents of the file. When you later try to select based on clientName, you are actually selecting based on the contents of the file.
Furthermore, when reading the data, you are reading the blob data from the column fileName, but this is wrong because:
fileName contains new Date().toString(), not the contents of the file
fileName should surely contain the file's name, not its contents?

Writing Video Files in Java

I'm trying to find a way to save a video file. Initially I'd put the video files as blob data into a database, and now, I'm trying to get the blob data back, convert it into bytes, and then write it to a new file. I've been successful in doing this, but the problem is, I can't make the resulting files to run. I tried storing, retrieving, and writing .flv and .mp4 files, but neither work :/ Can anyone help me? Much appreciated! :)
Here is my code: :)
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
con = DriverManager.getConnection(connectionURL, "root","password");
Statement st1 = (Statement) con.createStatement();
PreparedStatement pstmt = null;
ResultSet rs = null;
pstmt = con.prepareStatement("SELECT video_file from video where video_id = " + video_id);
rs = pstmt.executeQuery();
Blob blob = null;
byte[] blyte = null;
if(rs.next()) {
blob = rs.getBlob("video_file");
InputStream is = blob.getBinaryStream();
FileOutputStream fos = new FileOutputStream("C:\\Downloads\\file2.mp4");
int b = 0;
while(b != -1){
fos.write(b);
b = bis.read();
}
}
//exceptions beyond this point
} catch (InstantiationException e) {
} catch (IllegalAccessException e) {
} catch (ClassNotFoundException e) {
} catch (SQLException e) {
} catch (FileNotFoundException e) {
} catch (IOException e) {
}

There is a '\0' byte written to fos in first iteration of the while-loop which does not come from bis.read().