I am fairly new in SQL(now working on SQLITE application) and it is a section in my app when i try this piece of Code:
public void addSong(LibrarySong song){
for(int i=0; i<intoPanel.getComponentCount(); i++) //Check if doublicates exist
if(song.getName().equals(intoPanel.getComponent(i).getName()))
return;
intoPanel.add(song);
//Add the song to the database table
try{
Container.DataBase.connection.prepareStatement("INSERT INTO '"
+ Container.libWindow.libInfoWindow.currentLib.getName()+ "'" //Table Name
+ " (PATH,STARS,DATE,HOUR) VALUES ('"
+ song.getName() + "'," + song.stars + ",'"
+ song.dateCreated + "','" + song.hourCreated + "')").executeUpdate();
}catch(SQLException sql){ sql.printStackTrace(); };
}
The Problem:
The above method just add the song to a Jtable and then to database table.The problem is that the performance is too bad for the database.Why might this happen? i use the statement somewhere wrong or i have to to the update with different way?Thanks for reply.
The most expensive part of accessing a database is not the execution of the statement itself, but all the synchronizations done for transactions.
By default, each SQL command is put into an automatic transaction, so you get the overhead for all of them.
If you have multiple updates, you should group them into a single transaction:
Container.DataBase.connection.setAutoCommit(false);
...
for (...)
addSong(...);
Container.DataBase.connection.commit();
Basically the problem boils down to that every write to disc is done by sqlite itself in default mode
you could enable
PRAGMA journal_mode = WAL
PRAGMA synchronous = NORMAL
To make use of the operating system disc buffer.
Just remember to flush/commit everything regularly or at break points after inserts. Only risk is if there is sudden powerloss or reboot, your database might end up corrupted.
https://www.sqlite.org/atomiccommit.html
I think, that the component usage is suboptimal: the first for-loop.
Use a
Set<String> songNames = new HashSet<>();
if (songNames.contains(song.getName())) { return; }
songNames.add(song.getName());
Or maybe a Map for other uses.
And use the prepared statement with placeholders ?. This escapes single quotes too. And is safer.
Related
I want to create testdata and have written a function for storing the products, my product generators generate in my database.
The plan is to create about 10,000,000 products or more for testing purposes.
I want to check every time before I insert a product, if the same product name exists.
If it does, the product isn't stored in the database.
I know that the performance issue is the checking if the products exist, which takes longer and longer the more products are in the database. But there is no other way, I know, how I can improve this issue.
I may use indexes, but I don't know how to in this scenario.
If you have other ideas how to improve performance please feel free to comment your ideas.
tldr: I want to create testdata but it does take too long because it is checking if the products already exist. Want to improve performance.
Here is my code:
public String insertProdukt(String name, Double preis, Integer kat_id) throws SQLException, ClassNotFoundException {
Connection connection = ConnectionUtils.createNewConnection();
// does the product exist?
Statement statement = connection.createStatement();
ResultSet resultSet = statement.executeQuery("select * from pro_produkte where pro_name=\"" + name + "\" AND pro_preis=\"" + preis + "\" AND pro_kat_id=\"" + kat_id + "\"");
if (resultSet.next()) {
//it does exist
System.out.println("Produkt: " + resultSet.getString("pro_name") + " existiert bereits");
} else {
//it dosen't -> insert into database
String sql = "Insert INTO pro_produkte (pro_name, pro_preis, pro_kat_id)"
+ "VALUES (\"" + name + "\", \"" + preis + "\", \"" + kat_id + "\")";
statement.executeUpdate(sql);
System.out.println("Produkt: " + name + " erstellt");
}
resultSet.close();
statement.close();
connection.close();
return null;
}
Thanks!
Instead of simply INSERT ..., use
INSERT IGNORE ...
And have a UNIQUE (or PRIMARY) that will catch the "duplicate".
INSERTing one row at a time is about 10 times as slow as inserting 100 rows at a time. So, if you are generating them by code, do
INSERT IGNORE INTO t
(col1, col2, ...)
VALUES
(1,2,...),
(22,55,...),
... ;
Or
LOAD DATA LOCAL INFILE '...' IGNORE ...
if reading from a file.
First thing - do not open a connection for every insert, unless you are using a connection pool.
Second thing - use PreparedStatement. Not only will this save you from SQL injection, it will also make it faster because it will avoid repetitive parsing.
Third thing - use PreparedStatement.addBatch() and commit a batch every 5000 rows (or something like that). This implies you use the same Connection and PreparedStatement for all inserts.
Fourth thing - if you are only filling the database with test data and you know that your test data is unique, create index AFTER you insert all the records. It will be significantly faster.
Fifth thing - if you are using InnoDB, make sure you have enough buffer space to keep entire index in memory, and put the database on SSD (~30x faster than HDD).
If you can do it outside Java, you can use database's proprietary features for bulk loading, restoring from backups or snapshots. Check what features your database provides.
I have developed a mobile application using the CodeName One plugin for Java in the Netbeans IDE.
CodeName One uses the Database API. https://www.codenameone.com/javadoc/com/codename1/db/Database.html
I'm running some tests (there are around 10 values I would like to upload, however, just testing the connection ect by uploading ID, Fname and Lname values.
Database db = null;
Cursor cur = null;
String Fname = findTxtFirstn(c).getText();
String Lname = findTxtLastn(c).getText();
try{
Database ARdb = Display.getInstance().openOrCreate("RecordsDB.db");
System.out.println("Connection secured to database.");
ARdb.beginTransaction();
String createTable = "CREATE TABLE IF NOT EXISTS RecordsTable ("
+ "ID integer PRIMARY KEY,"
+ "First_Name text NOT NULL,"
+ "Last_Name text NOT NULL)";
String query = "insert into RecordsTable (ID,First_Name,Last_Name) values (3,'Test','Testerton')";
ARdb.execute(createTable);
ARdb.execute(query);
ARdb.commitTransaction();
} catch(Exception e){
System.out.println("Error! Connection Failed to DB" +e.getMessage());
} finally {
Util.cleanup (db);
Util.cleanup(cur);
}
I get no errors and everything runs, however, the values are not in the database when I check it. Am I missing something here? I have followed tutorials and looked over the Codename One API. I can't find the solution.
Edit: I need to change the value of the primary number each run (else I get an error: number needs to be unique), This tells me the values are being stored on the database, unfortunately, when I check the database in question there are no records on it, so where it the data going?
I used DB Browser for SQLite.
Have you tried performing your modifying change in a transaction? In databases with transactions all modifying operations (CREATE, INSERT, ...) need to be performed in a transaction.
ARdb.beginTransaction();
// your create code
ARdb.commitTransaction();
I have written the following code:
Statement stmt=con.createStatement();
String query = "UPDATE " + table + " SET " + field + "="
+ ((DataElement)getModel().getSelectedItem()).intValue + where;
System.out.println(query);
stmt.executeUpdate(query);
stmt.close();
firePropertyChange(table+"."+field,false,true);
when I try to react to the propertyChange, my database will either respond with the value before the change or fire an exception: database locked.
How can I make sure, the database commits the query before proceeding. I have also tried to turn off auto-commit and add a con.commit(); after the stmt.close, but that doesn't change anything.
There is one transaction and lock per connection.
To see a consistent view of the database, the code that responds to the notification must use the same connection.
I have a spring application which reads data from the Database and sends it to system 'X'. I am using task executors to spin up threads, so there are like 5 threads which are reading the database for rows at the same time. For each thread I need to make sure that unique records are selected.
To achieve this I am using JdbcTemplate and "select for update"
I have written the code but in the logs I am able to see 2 threads picking up the same rows. I am not able to figure out the root cause of this issue.
Does anyone has a suggestion
try {
List<Map<String, Object>> rows = getJdbcTemplate().queryForList(
SELECT_FOR_UPDATE,
new Object[] {a,b,c,d});
for (Map<String,Object> row : rows) {
Header a = new Header();
a.setMailID(((BigDecimal)row.get("mailID")).intValue());
a.setVersion(((BigDecimal)row.get("version")).intValue());
// some other parameters to get
getJdbcTemplate().update(UPDATE_MSG_STATE_VERSION_N_ORIG_MSG_STAT,
x,
a.getVersion()+1,
y),
a.getMailID(),
a.getVersion());
headers.add(a);
}
}
UPDATE_MSG_STATE_VERSION_N_ORIG_MSG_STAT = update MESSAGE set MSG_STAT_CD = ?, VERSION_NBR = ?, ORIG_MSG_STAT_CD=?, LAST_UPD_TS=SYSTIMESTAMP where MESSAGE.MAIL_ID = ? and VERSION_NBR = ?
String SELECT_FOR_UPDATE = "select m.MAIL_ID mailID, m.VERSION_NBR version, m.MSG_STAT_CD state,"
+ "from message m "
+ "and m.MSG_STAT_CD in ('Nerwerw')"
+ " and m.create_ts > (sysdate - ?)"
+ " and mod(mail_id,?) = ?"
+ " and ROWNUM <= ?"
+ " order by mt.MSG_PRIORITY FOR UPDATE";
You need to annotate your class with #Repostitory tag and the #Transactional tag to make sure that all the actions in the same call are handled in one transaction.
If they are not handled in the same transaction then each SELECT_FOR_UPDATE will happen on a different transaction and thus your threads queries will not be syncronized and your select_for_update does not matter.
Have you had transaction control properly set up?
If not, the transaction will only happen for the duration of the update statement, and will be committed automatically (You are using Oracle I believe, base on your syntax).
That means, although you acquired the lock of those records, they are released right-away.
Do you have access to modify the database? If I understand your question correctly I recently had a similar problem and implemented a scheme like this:
Add a new column to your database like "thread_number" or something like that. Set it to some default value like 0. Give each thread a unique identifier. Then you "claim" a record in the database by updating its "thread_number" to the identifier of the thread processing it. Then the other threads will not find it when querying if you include "where thread_number = 0" in the SQL.
I know it's kind of broad, but I hope it helps.
just kinda struggling with identifying the id during the piece update. I pass the id as a var and am trying to use it in conjunction with the WHERE statement and cant seem to figure out the correct syntax
String update = "UPDATE LawnMowers"+ " SET LMPrice = '"+returnedPrice+"' " + " WHERE LMID = '"+returnedID+'";
the return id is the issue, and the error is "invalid character constant". I believe the issue is normally the statement would be:
"WHERE LMID = int/double.ect"; rather then using " WHERE LMID = '"+varr+'"; errors and adding the additional " or )" ect options I have tried dont work either. Just wondering if any one had some insight
my DB is on Godaddy
thanks for reading
I think your statement should end
+ returnedID + "'";
Yours currently ends
+returnedID+'";
Also please read about prepared statements, these are much easier to use and leave you at far less risk of SQLi security vulnerabilities. It would look something like this:
String update = "UPDATE LawnMowers SET LMPrice = ? WHERE LMID = ?";
updateStatement = con.prepareStatement(update);
updateStatement.setInt(1, returnedPrice);
updateStatement.setInt(2, returnedId);
updateStatement.executeUpdate();