Usually a SQLite db is read from a file. Each operation can require a file I/O operation if the db is large enough, and this can be slow.
However SQLite provides a way to load the db to memory - where it can be accessed using a JDBC url which looks something likejdbc:memory...
For example, this question describes how to achieve this using python: How to load existing db file to memory in Python sqlite3?
However I can't figure out how to achieve the same thing in Java or Scala using JDBC.
Any ideas?
I just tried with Xerial's sqlite-jdbc-3.27.2.1.jar and it appears that they let us restore from a native (binary) SQLite database file into a :memory: database like so:
try (Connection conn = DriverManager.getConnection("jdbc:sqlite::memory:")) {
try (Statement st = conn.createStatement()) {
st.execute("restore from C:\\__tmp\\thing.sqlite");
// let's see if we can read one of the tables
ResultSet rs = st.executeQuery("SELECT * FROM Table1");
while (rs.next()) {
System.out.println(rs.getString(1));
}
}
} catch (Throwable e) {
e.printStackTrace(System.err);
}
Related
So, for a school project, I am building a discord bot. One of the features that I have built in is that he can retrieve gif links from a MySQL database, and send them in a message. Now, my issue is that I am only able to retrieve one record from my database, and no other records. If I put the query that I use into MySQL workbench and run it, it will retrieve those records.
This is the method for retrieving the gifs
public static ArrayList<Gif> GetGifsFromDB(String msg){
ArrayList<Gif> gifs = new ArrayList<>();
try(Connection conn = (Connection)DriverManager.getConnection(url, userName, password)){
Class.forName("com.mysql.jdbc.Driver").newInstance();
Statement stmnt = conn.createStatement();
String sql = "Select * from gif WHERE Type = '" + msg + "'";
stmnt.execute(sql);
try(ResultSet rs = stmnt.getResultSet()){
while(rs.next()){
Gif g = new Gif();
g.setID(rs.getInt("GifID"));
g.setURL(rs.getString("GifURL"));
System.out.println(g.getID() + g.getURL());
gifs.add(g);
}
rs.close();
conn.close();
}
}
catch(SQLException ex){
System.err.println(ex.getMessage());
}
catch(Exception ex){
System.err.println(ex.getMessage());
}
return gifs;
}
The "Type" in the database it just a category. With the test data I have in there, the 3 types are no, surprised and lonely. Only no returns a gif.
Remove closing ResultSet and Connection lines:
rs.close();
conn.close();
You are already closing it using try-with-resources
Issue ended up being with MySql not committing records to the database. Once workbench was refreshed, the added records disappeared. Rather strange that even though the records weren't in the database, they could be retrieve.
Most likely your msg is not exactly matching with any of the values for the database Type column.
Test by running
SELECT COUNT(*) FROM gif WHERE Type = '... put msg content here ...'
Do this manually directly on the database.
You can also try to put following line of code at the end:
System.out.println("Number of Selected Gifs: "+gifs.size());
If either of those results zero, then it means that msg was not exactly matched with Type. Maybe uppercase/lowercase issue?
Also to avoid SQL Injection, and other issues, please strongly consider using bind variables using a PreparedStatement.
I have a blacklist stored in a table that contains approximately 1.5 billion records. My goal is to load the records into a HashSet so my program can later check if domain names are blacklisted (this is not the entire functionality of the program, just a piece). I currently have the following code to load the records:
HashSet<String> list = new HashSet<String>();
try {
Statement stmt = conn.createStatement();
stmt.setFetchSize(100000000);
try {
ResultSet rs = stmt.executeQuery("SELECT DNname FROM " + table);
try {
while (rs.next()) {
list.add(rs.getString(1));
}
} finally {
rs.close();
}
} finally {
stmt.close();
}
} catch (SQLException e) {
System.out.println("Error loading blacklist from DB");
e.printStackTrace();
}
However, this takes incredibly long to complete. Is there a more efficient way to accomplish my goal?
"this takes incredibly long to complete" - what takes long to complete? How long does the query take if you run it from a DB client console?
You've not provided any information on what indexes are present in the table, so that's something you'll want to look into.
setFetchSize is a hint to the JDBC driver - you really want a WHERE clause where you can limit the records fetched. Oracle DB can do that using ROWNUM; for other DBs, you'll have to look for yourself.
That said, I concur with #ScaryWombat; keeping 1.5 billion records in memory is not scalable design. An alternative would to be build up a client cache by caching the database look up for each domain; in all practicality, the cache will have fewer than 1.5 billion entries.
Is it possible to find the sizes in GB of the hive tables using Java/jdbc? I don't want to depend on the hive warehouse folder in HDFS (as described in link) as different tables may have different locations
If you are mention 'totalSize' from 'tblproperties' then it is possible with similar approach:
String driverName = "org.apache.hive.jdbc.HiveDriver";
String connectionURL = "jdbc:hive2://HOSTNAME:PORT/default";
try {
Class.forName(driverName);
Connection connection = DriverManager.getConnection( connectionURL, "", "");
} catch (ClassNotFoundException e) {
e.printStackTrace();
System.exit(1);
}
Statement stmt = connection.createStatement("show tblproperties TABLENAME");
ResultSet rs = stmt.executeQuery(stmt);
while(rs.next()){
//doWhatYouWant
}
Currently not be possible but it can be done .
To get file size you have to run on file system( HDFS) commands .
In case of RDMS data bases ie sql server have encapsulated file system commands in SYS views and SYS functions (DMF and DMVs) .
If some one write or develop such UDF it will be possible but Internally the UDF will be calling same comand.
I have to design an application for accessing excel application as database. My problem is I have to create multiple connection for each transaction and if I miss any of closing it, the excel is not being update.
I want to design pattern where i am able to access the excel. Any one help me in designing a common pattern through which i wont be having problem. I want something like this, but we are not able to use it to access excel.
Thanks in advance!
i have this method in utility class
static ResultSet getExcelData(String filePath,String sqlQuery){
ResultSet rs=null;
try{
conn = DriverManager.getConnection("jdbc:odbc:Driver={Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)};DBQ="+filePath+";READONLY=false");
stmt= conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
rs=stmt.executeQuery( sqlQuery );
}catch (Exception e) {
e.printStackTrace();
return null;
// TODO: handle exception
}finally{
}
return rs;
}
and i am calling it this way
ResultSet rs=JdbcUtil.getExcelData("D:\\AB_demo\\AB_demo\\test.xls", "Select max(int(ID)) from [MAIN$] where HEADER_IND is not Null AND int(ID)<"+excelId);
int databaseId = 0;
if(rs.next())
{
databaseId=rs.getInt(1);
}
ResultSet rs1=JdbcUtil.getExcelData("D:\\AB_demo\\AB_demo\\test.xls", "SELECT * from [MAIN$] where id= '"+databaseId+"'or id='"+excelId+"'");
i am calling this method twice after which im updating the excel file by using
stmt.executeUpdate(sql);
its returning the integer 1 but its not reflecting in excel.when i use process explorer the file is still in use.i need a design pattern or come code to overcome his kind of problem.
I think more right way is generate Excel file from database. Otherwise you must create server side for ensure transactions and connections control.
Main problem of your task - Excel is
not database
not network database
Other words you must use other tools, other approach for your tasks.
I am new to using Derby and databses in eclipse, and I have become a tad lost and in need to a bit of help. I have established a database connection, created a new database, and a new schema, within which I have some tables containing some test data. I don't have any problem with the sql queries to select the relevant data. The problem I have is getting to a point where I can use queries. I am trying to create a class which connects to the database, and for testing purposes, uses a simple query to select some data. This is what I have so far:
public void getExerciseInfo() {
try {
Class.forName("org.apache.derby.jdbc.ClientDriver").newInstance();
connect = DriverManager.getConnection("jdbc:derby://localhost/c:/TestDatabase");
PreparedStatement statement = connect.prepareStatement("SELECT * from TESTSCHEMA.TESTTABLE");
resultSet = statement.executeQuery();
while (resultSet.next()) {
String name= resultSet.getString("NAME");
String type = resultSet.getString("TYPE");
System.out.println(name);
System.out.println(type);
}
} catch (Exception e) {
} finally {
close();
}
}
All I am trying to do is output the data in the table to the console, but I cant even do this simple task :( Im guessing my connection url is invalid, is it supposed to be the file path to the database folder in my eclipse workspace?
Anyhow, I am very lost, and any help would be greatly appreciated.
Did you take a look over: http://db.apache.org/derby/integrate/plugin_help/derby_app.html ? You seem to be using the network server but your db URL is wrong.
If you are not running the Derby server, you can establish an embedded database connection or use an EmbeddedDataSource, shown here.