I've written a class method that will take "batches" of data (each row that makes a "value" to be inserted, via SQL, to the database comes from a two-dimensional array labeled "data_values").
However, there will be instances when my program will be getting redundant data, i.e. data that might already be in the database. Because there's a primary key in the database, the program will break if it cannot upload the data because of a duplicate entry.
Is there a way to use a try/catch so that the program will continue uploading data, effectively "skipping" the duplicates? If so, how can I implement it?
Thank you in advance. If I could clarify my question, please let me know.
My current code is here:
public void insertData(ArrayList<String> data_types, String[][] data_values) {
try{
c.setAutoCommit(false);
// creates insert statement
String insertDataScript = "INSERT INTO "+tableName+" VALUES (";
for(int q = 0; q < data_types.size()-1; q++) {
insertDataScript += "?, ";
}
insertDataScript += "?)";
PreparedStatement stmt = c.prepareStatement(insertDataScript);
for (int i = 0; i < data_values.length; i++) {
for(int j = 1; j < data_types.size()+1; j++) {
if(data_types.get(j-1).toLowerCase().equals("double")) {
stmt.setDouble(j, Double.valueOf(data_values[i][j-1]));
}
else if(data_types.get(j-1).toLowerCase().equals("string")) {
stmt.setString(j, data_values[i][j-1]);
}
else {
System.out.println("Error");
}
}
stmt.addBatch();
}
stmt.executeBatch();
c.commit();
c.setAutoCommit(true);
stmt.close();
}
catch ( Exception e ) {
System.err.println( e.getClass().getName() + ": " + e.getMessage() );
System.exit(0);
}
}
My first suggestion would be to deduplicate the data before inserting it into the db. (Edit: totally missed the "already in the db" part, so this probably won't work unless you want to do a query before every insert. Maybe you can use an INSERT IGNORE?)
If you cannot do this because you do not have control over the primary key or there is no way to ignore duplicates in the insert, then there are ways to catch specific exception types and continue the program instead of calling System.exit. In order to do that you would probably need to have smaller prepared statements and put the try/catch inside the for loop over 'data_values`.
Here is a post talking about catching this type of exception: Catch duplicate key insert exception.
INSERT OR IGNORE
Simply change (albeit it not really exception handling, but rather exception bypassing)
String insertDataScript = "INSERT INTO "+tableName+" VALUES (";
to
String insertDataScript = "INSERT OR IGNORE INTO "+tableName+" VALUES (";
Consider the following demo (equivalent to suggested and then what you currently have) :-
rowid has been used for convenience as it's basically a build in primary key.
the only reason why the columns have been specified i.e.(rowid,othercolumn,mydatecolumn) is that rowid is normally hidden. In your case just VALUES (without the preceding columns) will expect values for all columns and thus include the defined primary key column(s).
shown/actioned in reverse order as both can then run together
:-
INSERT OR IGNORE INTO mytable (rowid,othercolumn,mydatecolumn) -- rowid is a PRIMARY KEY as such
VALUES
(10,'x','x'),
(11,'x','x'),
(12,'x','x'),
(13,'x','x'),
(14,'x','x'),
(10,'x','x')
;
INSERT INTO mytable (rowid,othercolumn,mydatecolumn) -- rowid is a PRIMARY KEY as such
VALUES
(20,'x','x'),
(21,'x','x'),
(22,'x','x'),
(23,'x','x'),
(24,'x','x'),
(20,'x','x')
;
results in :-
INSERT OR IGNORE INTO mytable (rowid,othercolumn,mydatecolumn) -- rowid is a PRIMARY KEY as such
VALUES
(10,'x','x'),
(11,'x','x'),
(12,'x','x'),
(13,'x','x'),
(14,'x','x'),
(10,'x','x')
> Affected rows: 5
> Time: 0.208s
i.e. 5 of the 6 were added the 6th a duplicate (according to the primary key) was skipped.
INSERT INTO mytable (rowid,othercolumn,mydatecolumn) -- rowid is a PRIMARY KEY as such
VALUES
(20,'x','x'),
(21,'x','x'),
(22,'x','x'),
(23,'x','x'),
(24,'x','x'),
(20,'x','x')
> UNIQUE constraint failed: mytable.rowid
> Time: 0.006s
i.e. none are inserted due to 1 duplicate.
INSERT OR REPLACE (may be useful)
If you wanted the data from the duplicates to be applied then instead of INSERT OR IGNORE, you could use INSERT OR REPLACE.
e.g. the following (run after the above i.e. all are duplicates bit with different data):-
INSERT OR REPLACE INTO mytable (rowid,othercolumn,mydatecolumn) -- rowid is a PRIMARY KEY as such
VALUES
(10,'xx','x'),
(11,'x','xx'),
(12,'aa','x'),
(13,'x','aa'),
(14,'x','bb'),
(10,'cc','x')
;
then you get :-
INSERT OR REPLACE INTO mytable (rowid,othercolumn,mydatecolumn) -- rowid is a PRIMARY KEY as such
VALUES
(10,'xx','x'),
(11,'x','xx'),
(12,'aa','x'),
(13,'x','aa'),
(14,'x','bb'),
(10,'cc','x')
> Affected rows: 6
> Time: 0.543s
i.e. now all 6 INSERTs are actioned (5 rows updated as the 1st and last update the same row twice).
I am having some trouble figuring out a query that will update values in a column in one of my tables. Below is my function:
public void increasePrice(String [] str) {
PreparedStatement ps = null;
try {
ps = connection.prepareStatement("Update Journey Set price+=? where distance <?",PreparedStatement.RETURN_GENERATED_KEYS);
ps.setDouble(1,Double.parseDouble(str[1]));
ps.setDouble(2, Double.parseDouble(str[0]));
ps.executeUpdate();
ps.close();
System.out.println("1 rows updated.");
} catch (SQLException ex) {
Logger.getLogger(Jdbc.class.getName()).log(Level.SEVERE, null, ex);
}
}
To illustrate, the array passed in contains a value for distance and price and I am wanting to update the prices in the 'Journey' table based on their distance. For example, if a record in the table has a distance (type double) that is less than a given distance (the value of str[0]), I want to increase the price (also a double) of that record by the value 'str[1]' and do this for all records in the table.
The above code doesn't give any errors however, the records in the database never get updated. I could really use some help with this as I've searched around for a while now to try and find a solution and have not yet succeeded.
I do not know what database you are using but my guess is that this line:
ps = connection.prepareStatement("Update Journey Set price+=? where distance <?",PreparedStatement.RETURN_GENERATED_KEYS);
should be written like this:
ps = connection.prepareStatement("Update Journey Set price=price+? where distance <?",PreparedStatement.RETURN_GENERATED_KEYS);
And not related to your question but the line
System.out.println("1 rows updated.");
may make you waste hours of debugging in the future because 0 or more rows can be actually updated.
try {
Statement s = conn.createStatement();
ResultSet result2 = s.executeQuery("Select Distinct * From Poem p,Recording r Where r.PoemTitle = p.PoemTitle AND r.poemTitle = 'poem1'");
System.out.print("Result (Select with Join): ");
while (result2.next()) {
System.out.println(result2.getString(1)+ " " + result2.getString(2)+ result2.getString(3));
}
} catch(Exception e) {
System.out.print(e.getMessage());
}
I am trying to output the poemtitle and the date it was recorded. When this runs it outputs the poemtitle and then gives the date the poem was created instead of recorded? Is this because of the relationship?
Most likely it's because of the * in the SELECT list.
Specify the columns that you want returned, in the order you want them returned.
We're just guessing at the name of the column that contains "date recorded" and which table it's in:
SELECT p.PoemTitle
, r.dateRecorded
, r.readBy
FROM Poem p
JOIN Recording r
ON r.PoemTitle = p.PoemTitle
WHERE r.poemTitle = 'poem1'
GROUP
BY p.PoemTitle
, r.dateRecorded
, r.readBy
ORDER
BY p.PoemTitle
, r.dateRecorded DESC
, r.readBy
Notes:
Ditch the old-school comma syntax for the join operation and use the JOIN keyword instead, and relocate the join predicates from the WHERE clause to an ON clause.
Avoid using * in the SELECT list. Explicitly list the columns/expressions to be returned. When we read the code, and that SQL statement, we don't know how many columns are being returned, what order the columns are in, or what the datatypes are. (We'd have to go look at the table definitions.)
Explicitly listing the columns/expressions being returned only takes a little bit of work. If code was only ever written, then it would be fine, save the time writing. But code is READ ten times more than it is written. (And the SQL statement with the * makes the SQL statement virtually indecipherable in terms of which column is being referenced by getString(1).
Listing the columns columns can also make it more efficient on the database, to prepare a resultset with a few columns vs a resultset of dozens of columns, and we also transfer a smaller resultset from the database to the client. With a subset of columns, its more likely we can use a covering index for the query.
I've had a look around on the web but can't seem to find a definite answer to my question.
Basically, I have a database and table that are successfully working. Now I want to read each line from my table one by one and store the result into a array and I am trying to use a for loop to be more professional rather then using repetition.
I have this code
for (int i=1; i<=8; i++)
{
String query = "Select * FROM Table1 WHERE ID = i";
Rs = St.executeQuery(query);
COL1Title[i] = Rs.getString("CO1Name");
COL2Age[i] = Rs.getString("CO2Rating");
}
The for loop is in a try catch statement and it's complaining with the error "Unknown column 'i' in 'where clause'"
Im guessing there's a certain way for how variable i is to be inserted in the the query.
I should point out ID is a column that has the auto increment feature added on and is primary key if that helps
Could anyone help me out here?
First, we can simplify the task be executing a single query. Note the addition of the range limit and the ORDER BY - without an ORDER BY the results have an unspecified order!
PreparedStatement stmt = "Select ID, CO1Name, CO2Rating"
+ " FROM Table1"
+ " WHERE ID >= ? AND ID <= ?"
+ " ORDER BY ID";
And bind in placeholders (unless there is good reason otherwise, always use placeholders when injecting data into a query). The values could have been hard-coded above in this case, just as they are hard-coded in the for-loop, but the binding is shown here for future reference:
stmt.setInt(1, 1);
stmt.setInt(2, 8);
Then execute the query:
ResultSet rs = stmt.executeQuery();
And iterate the results. Note that rs.next() must be invoke once before any column is read (the cursor starts before any records) and, in this case, it makes it easy to handle a bunch of results.
while (rs.next()) {
int id = rs.getInt("ID");
String title = rs.getString("CO1Name");
String name = rs.getString("CO2Rating");
// do stuff with this record
}
Note that even though the ORDER BY guarantees that the results are iterated in order of ID, assuming a database cardinality rule ensures each result has a unique ID, there may be 0 to 8 records returned - that is, non-existent records may need to be detected/handled separately.
Also (but not shown), make sure to cleanup (close) the ResultSet when done: use a try/finally or try-with-resources construct.
You need to pass i in string as integer, Replace line by:
String query = String.format("Select * FROM Table1 WHERE ID = %d",i);
I`m building a decision support system in java. Below is a part of code, that scans the search word that is typed in by user, then compares it with database searchlog.searchcolumn values and if the word is not there creates a new entry. BUT in the if statement i want it to check for the entry, and if it IS already in searchlog.searchcolumn column, then I want it NOT to create a new duplicate entry, but to add +1 value to searchlog.counter column for the specific word.
for example if search word is "UMBRELLA" and there is already one entry for umbrella in database, i want it to add +1 to counter column in UMBRELLA row.
the purpose of this, is to store all searchwords and keep a track of the most popular ones.
Thank you for your time
String CheckSearch = "SELECTsearchcolumn FROMsearchlog";
String InsertColumn = "INSERT INTO `mydb`.`searchlog` (`searchcolumn`) VALUES ('"+ InputScanner + "');
//
if (InputScanner.equals(CheckSearch))
System.out.println("value ealready exist, counter well be updated");
else
stmt.executeUpdate(InsertColumn);
EDIT
Thank you for advice of using PreparedStatement, but this is my first more or less serious challenge and for this time, let`s just ignore vulnerability of my code. Thanks
What database are you using? If you are using MySQL, then you should look into INSERT ... ON DUPLICATE KEY UPDATE statement. (Other SQL databases have MERGE, which I'm less familiar with.) Here is the MySQL Documentation.
You will need make your searchcolumn a UNIQUE or PRIMARY column, then something along the lines of: INSERT INTO searchlog (searchcolumn, count) VALUES ("my search query", 0) ON DUPLICATE KEY UPDATE count = count + 1; will accomplish what you want.
your query should be :
String InsertColumn = "INSERT INTO `mydb`.`searchlog` (`searchcolumn`) VALUES ('"+ InputScanner + "'");
Values clause should also be wrapped around brackets.
and always use equals() to check if two strings are meaningfully equal. In case of objects == checks if two reference variables refer to the same object.
if (InputScanner == CheckSearch) {
should be:
if (InputScanner.equals(CheckSearch)) {
Then, your if statement would return true if InputScanner is same as checkSearch.
ADVICE:
I strongly recommend you to use PreparedStatement rather than simple Statement to prevent SQL Injection.
PreparedStatement st = conn.preparedStatement(InsertColumn);
st.setString(1, val1);
1) Values also need to be in braces
Example:
VALUES ('"+ InputScanner + "');
2) Assuming InputScanner and CheckSearch are Strings/comparable objects, you need to do .equals() instead of ==
Example:
if (InputScanner.equals(CheckSearch)) {
Note: Your SQL statement is prone to SQL Injection attack. Better to use PreparedStatement.
There is an MySQL statement for your task:
Requierement: searchcolumn is a unique key.
String sql = "INSERT INTO mydb.searchlog(searchcolumn, counter) VALUES(?, 1) "
+ " ON DUPLICATE KEY UPDATE counter = counter + 1";
PreparedStatement stmt = connection.createPreparedStatement(sql);
stmt.setParameter(1, checkSearch.toUpperCase());
int updateCount = stmt.executeUpdate();
I do not know whether updateCount distinghuishes whether an INSERT or UPDATE happened though. You could use getGeneratedKeys for that purpose, if there is a unmentioned AUTOINCR primary key.