I am trying to output my table as a .csv file. I am using the sqlite-jdbc java library to execute my query. Here are the statements:
.mode csv
.output test.csv
select * from geninfo;
.output stdout
and this is my java code:
try {
try {
// create a database connection
Class.forName("org.sqlite.JDBC");
} catch (ClassNotFoundException ex) {
Logger.getLogger(SQLite.class.getName()).log(Level.SEVERE, null, ex);
}
connection = DriverManager.getConnection("jdbc:sqlite:702Data.db");
Statement statement = connection.createStatement();
statement.setQueryTimeout(30); // set timeout to 30 sec.
statement.executeUpdate("create table if not exists geninfo (id TEXT, DeviceID TEXT)");
statement.executeUpdate(".mode csv");
statement.executeUpdate(".output test.csv");
statement.executeUpdate("select * from geninfo;");
statement.executeUpdate(".output stdout");
ResultSet rs = statement.executeQuery("select * from geninfo");
while (rs.next()) {
// read the result set
System.out.println("DeviceID = " + rs.getString("DeviceID"));
System.out.println("id = " + rs.getInt("id"));
}
} catch (SQLException e) {
// if the error message is "out of memory",
// it probably means no database file is found
System.err.println(e.getMessage());
} finally {
try {
if (connection != null) {
connection.close();
}
} catch (SQLException e) {
// connection close failed.
System.err.println(e);
}
}
}
I have tried adding parenthesis around the period and also removing the period and both cause the same syntax error.
Those commands are SQLite console commands, not pure SQL. JDBC can only handle SQL.
If you want to do this from java, try executing the shell command to open the console using ProcessBuilder and feed it the commands via its InputStream.
However, if calling from java is not necessary, you may be better served writing a shell script of some sort and using that directly, or calling the script from java.
If all you are using SQLite for is to parse the CSV file, consider not using SQLite at all and instead load the data directly from the file into your code using one of the several open source CSV parsing libraries out there. This approach will be order of magnitude faster and simpler.
Related
I am having an java package, which connects with a database and fetches some data. At some rare case, I am getting heap memory exception, since the fetched query data size is exceeding the java heap space. Increasing the java heap space is not something the business can think for now.
Other option is to catch the exception and continue the flow with stopping the execution. ( I know catching OOME is not a good idea but here only me local variables are getting affected). My code is below:
private boolean stepCollectCustomerData() {
try {
boolean biResult = generateMetricCSV();
} catch (OutOfMemoryError e) {
log.error("OutOfMemoryError while collecting data ");
log.error(e.getMessage());
return false;
}
return true;
}
private boolean generateMetricCSV(){
// Executing the PAC & BI cluster SQL queries.
try (Connection connection = DriverManager.getConnection("connectionURL", "username", "password")) {
connection.setAutoCommit(false);
for (RedshiftQueryDefinition redshiftQueryDefinition: redshiftQueryDefinitions){
File csvFile = new File(dsarConfig.getDsarHomeDirectory() + dsarEntryId, redshiftQueryDefinition.getCsvFileName());
log.info("Running the query for metric: " + redshiftQueryDefinition.getMetricName());
try( PreparedStatement preparedStatement = createPreparedStatement(connection,
redshiftQueryDefinition.getSqlQuery(), redshiftQueryDefinition.getArgumentsList());
ResultSet resultSet = preparedStatement.executeQuery();
CSVWriter writer = new CSVWriter(new FileWriter(csvFile));) {
if (resultSet.next()) {
resultSet.beforeFirst();
log.info("Writing the data to CSV file.");
writer.writeAll(resultSet, true);
log.info("Metric written to csv file: " + csvFile.getAbsolutePath());
filesToZip.put(redshiftQueryDefinition.getCsvFileName(), csvFile);
} else {
log.info("There is no data for the metric " + redshiftQueryDefinition.getCsvFileName());
}
} catch (SQLException | IOException e) {
log.error("Exception while generating the CSV file: " + e);
e.printStackTrace();
return false;
}
}
} catch (SQLException e){
log.error("Exception while creating connection to the Redshift cluster: " + e);
return false;
}
return true;
}
We are getting exception in the line "ResultSet resultSet = preparedStatement.executeQuery()" in the later method and i am catching this exception in the parent method. Now, i need to make sure when the exception is caught in the former method, is the GC already triggered and cleared the local variables memory? (such as connection and result set variable) If not, when that will be happen?
I am worried about the java heap space because, this is continuous flow and I need to keep on fetching the data for another users.
The code that i have provided is only to explain the underlying issue and flow and kindly ignore syntax, etc.., I am using JDK8
Thanks in advance.
I'm developing a desktop app to organize different events, thus creating a DB for each event. So far, I've managed to create a DB with whatever name the user wants, using a simple GUI.
However, I can't create tables nor columns for said database, even though it's exactly the same syntax I use in SQL Server Manager.
My code so far:
public static void creDB(String db_name, String table_name){
Connection conn = null;
Statement stmt = null;
ResultSet rs = null;
try {
Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
conn = DriverManager.getConnection(connectionUrl);
String SQL = "CREATE DATABASE " + db_name;
stmt = conn.createStatement();
int result = stmt.executeUpdate(SQL);
String SQL3 = "USE " + db_name;
boolean ree = stmt.execute(SQL3);
String SQL4 = "GO";
boolean rr = stmt.execute(SQL4);
if (result == 0){
System.out.println("Se insertó :D!");
String SQL2 = "CREATE TABLE Pepe(Name_emp INT NOT NULL PRIMARY KEY)";
int res = stmt.executeUpdate(SQL2);
if (res == 0)
System.out.println("GRACIAS DIOS");
}else
System.out.println("Raios shico");
}catch (Exception e) {e.printStackTrace();}
finally {
if (rs != null) try {rs.close();} catch (Exception e) {e.printStackTrace();}
if (stmt != null) try {stmt.close();} catch (Exception e) {e.printStackTrace();}
if (conn != null) try {conn.close();} catch (Exception e) {e.printStackTrace();}
}
}
The error I get is when I try to actually use the DB, using the use [DB name] go; I tried already using that same syntax in one single SQL statement, however it didn't work, so I tried doing it separately and got this error:
com.microsoft.sqlserver.jdbc.SQLServerException: Could not find stored procedure 'GO'.
I know the code above looks like a mess, and it is, but it's just for testing purposes since I'm new to doing DB-related projects with Java; I mixed-matched some of the concepts of this site, which were successful up until the creation of the tables.
I know there's a better way of managing several databases, but as I said, I'm just starting so any advice would be greatly appreciated.
You should not use statements like USE <dbname> when using JDBC, it may lead to unexpected behavior because parts of the driver may still use metadata for the original connected database. You should either use setCatalog on the current connection to switch databases or create an entirely new connection to the new database.
In short, after creating the database, you should use:
conn.setCatalog(db_name);
That's it.
Also, go is not part of the SQL Server syntax, it is only used by tools like the Management Studio, see What is the use of GO in SQL Server Management Studio & Transact SQL? The equivalent in JDBC is to simply execute the statement.
I am trying to create a new database file named test in the folder D:\sqlite, using JDBC, as follows:
import java.sql.*;
public class Class1 {
private static void createNewDataBase(){
String url = "jdbc:sqlite:D:sqlite/";
Connection conn = null;
Statement statement = null;
try{
conn = DriverManager.getConnection(url);
System.out.println("Connection Established");
statement = conn.createStatement();
} catch (SQLException e) {
System.out.println(e.getMessage());
}
try {
statement.execute("CREATE DATABASE test");
} catch (SQLException e) {
System.out.println(e.getMessage());
} finally{
try{
if(statement != null)
statement.close();
}catch(SQLException e){
System.out.println(e.getMessage());
}
try{
if(conn != null)
conn.close();
}catch(SQLException e){
System.out.println(e.getMessage());
}
}
}
public static void main(String[] args){
createNewDataBase();
}
}
When I run the project, I get the following output:
Connection Established
[SQLITE_ERROR] SQL error or missing database (near "DATABASE": syntax error)
Process finished with exit code 0
It says the syntax is wrong but I can't find the error. I've looked for answers for similar questions, but none solved my problem. Can anyone tell me what's the problem? Thanks in advance!
As already stated by #Andreas, there is no CREATE DATABASE SQL statement for SQLite. To create a new SQLite database you need to merely make a connection to one and it is automatically created (you do however need to ensure that the D:/sqlite/ path already exists within the local file system).
The following code should create an empty (no tables) SQLite Database named MyNewDatabase.sqlite within the folder sqlite located in the root of drive D of your local file system:
String dbPath = "D:/sqlite/MyNewDatabase.sqlite";
Connection conn = null;
try {
Class.forName("org.sqlite.JDBC");
conn = DriverManager.getConnection("jdbc:sqlite:" + dbPath;);
}
catch (ClassNotFoundException ex) {
ex.printStackTrace();
}
catch (SQLException ex) {
ex.printStackTrace();
}
finally {
try {
if (conn != null) {
conn.close();
}
}
catch (SQLException e) { e.printStackTrace(); }
}
Now you need to create one or more Tables for your new database to make it useful. SQLite does accept the CREATE TABLE SQL statement and you could do this through the same connection if desired.
The database exists, it has been connected i.e the database is the connection, which is basically the file. Hence there is no SQL for CREATE DATABASE.
Inside the database you would typically create tables (and perhaps other components).
e.g. CREATE TABLE mytable (mycolumn TEXT, myothercolumn INTEGER) which would create a table named mytable with 2 columns mycolumn and myothercolumn (the former being defined with a column type of TEXT, the latter with a column type of INTEGER).
As such, if you were to change :-
statement.execute("CREATE DATABASE test");
to :-
statement.execute("CREATE TABLE mytable (mycolumn TEXT, myothercolumn INTEGER)");
Note you'd only want to do this once, otherwise it would fail as the table already exists, so you could use CREATE TABLE IF NOT EXISTS mytable (mycolumn TEXT, myothercolumn INTEGER), of cousre it depends upon your requirements.
You may find that it will work. Obviously you will need to do other things such as add some data as the table will be empty.
Perhaps then try
statement.execute("INSERT INTO mytable VALUES('fred',100)");
Every time this is run a new ROW will be added to the table name mytable.
I just started with sqlite and I stuck with a strange (maybe just for me) phenomenon. When I connect the testDB.db file in java, and make one or some query, the data and the table itself is disappearing. The consol said that SQL error or missing database, and when I check the database file in cmd, the situation is really that; there is no data in the file. Could anybody help me out with this basic problem? (I suppose that this is just because of the lack of my knowledge in this topic, but I'm open to new information)
public class jdbcTest{
public static void main(String[] strg) {
Connection connection = null;
try {
connection = DriverManager.getConnection("jdbc:sqlite:C:\\Users\\Username\\Documents\\sqlite\\testDB");
Statement statement = connection.createStatement();
statement.setQueryTimeout(30);
//statement.executeUpdate("drop table if exists person");
ResultSet rs = statement.executeQuery("select * from company");
while (rs.next()){
System.out.print("id = "+rs.getInt("id")+" ");
System.out.println("name = "+rs.getString("name"));
}
}
catch (SQLException e) {
System.err.println(e.getMessage());
}
finally {
try {
if (connection!=null){
connection.close();
}
}
catch (SQLException e){
System.err.println(e);
}
}
}
}
When opening a nonexistent database file, SQLite will happily create a new, empty one.
The file name you're giving to getConnection does not actually point to the database you saved.
Input set: thousands(>10000) of csv files, each containing >50000 entries.
output: Store those data in mysql db.
Approach taken:
Read each file and store the data into database. Below is the code snippet for the same. Please suggest if this approach is ok or not.
PreparedStatement pstmt2 = null;
try
{
pstmt1 = con.prepareStatement(sqlQuery);
result = pstmt1.executeUpdate();
con.setAutoCommit(false);
sqlQuery = "insert into "
+ tableName
+ " (x,y,z,a,b,c) values(?,?,?,?,?,?)";
pstmt2 = con.prepareStatement(sqlQuery);
Path file = Paths.get(filename);
lines = Files.lines(file, StandardCharsets.UTF_8);
final int batchsz = 5000;
for (String line : (Iterable<String>) lines::iterator) {
pstmt2.setString(1, "somevalue");
pstmt2.setString(2, "somevalue");
pstmt2.setString(3, "somevalue");
pstmt2.setString(4, "somevalue");
pstmt2.setString(5, "somevalue");
pstmt2.setString(6, "somevalue");
pstmt2.addBatch();
if (++linecnt % batchsz == 0) {
pstmt2.executeBatch();
}
}
int batchResult[] = pstmt2.executeBatch();
pstmt2.close();
con.commit();
} catch (BatchUpdateException e) {
log.error(Utility.dumpExceptionMessage(e));
} catch (IOException ioe) {
log.error(Utility.dumpExceptionMessage(ioe));
} catch (SQLException e) {
log.error(Utility.dumpExceptionMessage(e));
} finally {
lines.close();
try {
pstmt1.close();
pstmt2.close();
} catch (SQLException e) {
Utility.dumpExceptionMessage(e);
}
}
I've used LOAD DATA INFILE ins situations like this in the past.
The LOAD DATA INFILE statement reads rows from a text file into a
table at a very high speed. LOAD DATA INFILE is the complement of
SELECT ... INTO OUTFILE. (See Section 14.2.9.1, “SELECT ... INTO
Syntax”.) To write data from a table to a file, use SELECT ... INTO
OUTFILE. To read the file back into a table, use LOAD DATA INFILE. The
syntax of the FIELDS and LINES clauses is the same for both
statements. Both clauses are optional, but FIELDS must precede LINES
if both are specified.
The IGNORE number LINES option can be used to ignore lines at the start of the file. For example, you can use IGNORE 1 LINES to skip over an initial header line containing column names:
LOAD DATA INFILE '/tmp/test.txt' INTO TABLE test IGNORE 1 LINES;
http://dev.mysql.com/doc/refman/5.7/en/load-data.html
As #Ridrigo has already pointed out, LOAD DATA INFILE is the way to go. Java is not really needed at all.
If the format of your CSV is not something that can directly be inserted into the database, your Java code can renter the picture. Use it to reorganize/transform the CSV and save it as another CSV file instead of writing it into the database.
You can also use the Java code to iterate through the folder that contains the CSV, and then execute the system command for the
Runtime r = Runtime.getRuntime();
Process p = r.exec("mysql -p password -u user database -e 'LOAD DATA INFILE ....");
you will find that this is much much faster than running individual sql queries for each row of the CSV file.