This is using MySQL 5.7.28 as a Docker image, in case that's relevant. The client is JDBC 5.1.48, running on GraalVM 11.20 on Mac.
When I run the following query in the information_schema schema, I get 3,105 rows:
PreparedStatement ps = conn.prepareStatement("select * from COLUMNS");
ResultSet rs = ps.executeQuery();
int numRows = 0;
while (rs.next()) {
numRows++;
}
But if I do this repeatedly in 100 parallel threads, each thread using its own connection, I occasionally get a slightly different result set. About 0.5% of the time, the number of rows is slightly lower, like 3,098. It's inconsistent, but easily reproducible.
Notice that this is not select count(*) -- this is select * and then iterate over all the rows.
This only happens against tables in the information_schema catalog -- for instance the TABLES table.
When doing the same thing against tables in the mysql catalog, or any other catalog actually, this does not happen, so my best guess is that this is due to the fact that the information_schema schema is special, like maybe it's cached in the JDBC driver, or the server, or something like that.
Can anyone shed any light on this?
Here is the complete code, which should run against any MySQL once you change the URL, user name and password:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.List;
import java.util.ArrayList;
import java.util.concurrent.atomic.AtomicInteger;
public class ManyQueryThreads {
private static final List<Thread> threads = new ArrayList<>();
private static final List<QueryThread> queryThreads = new ArrayList<>();
private static final int NUM_THREADS = 100;
private static final int NUM_ITERS = 20;
/**
* This gets set by the first thread to complete the query. All subsequent queries are expected
* to return the same number of rows.
*/
private static final AtomicInteger expectedNumRows = new AtomicInteger(-1);
private static class QueryThread implements Runnable {
public boolean done = false;
#Override
public void run() {
Connection conn;
try {
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/information_schema?useSSL=false",
"root", "Password1");
}
catch(Exception ex) {
throw new RuntimeException(ex);
}
for (int i = 0; i < NUM_ITERS; i++) {
try {
PreparedStatement ps = conn.prepareStatement("select * from COLUMNS");
ResultSet rs = ps.executeQuery();
int numRows = 0;
while (rs.next()) {
numRows++;
}
expectedNumRows.compareAndExchange(-1, numRows);
if (numRows != expectedNumRows.get()) {
System.out.println("Incorrect number of rows, expected " +
expectedNumRows.get() + ", but got " + numRows + ", thread " + Thread.currentThread().getId() +
", iteration " + i);
}
rs.close();
ps.close();
}
catch (Exception ex) {
throw new RuntimeException(ex);
}
}
try {
conn.close();
}
catch(Exception ex) {
throw new RuntimeException(ex);
}
done = true;
}
}
public static void main(String[] args) throws Exception {
System.out.println("Creating " + NUM_THREADS + " threads...");
long startTime = System.currentTimeMillis();
for (int i = 0; i < NUM_THREADS; i++) {
QueryThread qt = new QueryThread();
queryThreads.add(qt);
Thread t = new Thread(qt);
threads.add(t);
}
System.out.println("Threads created, starting them for " + NUM_ITERS + " iterations...");
for (int i = 0; i < NUM_THREADS; i++) {
threads.get(i).start();
}
// Now wait for everyone to be done
System.out.println("Threads are running, waiting for them to end...");
deathWatch:
while (true) {
for (int i = 0; i < NUM_THREADS; i++) {
QueryThread qt = queryThreads.get(i);
if ( ! qt.done) {
Thread.sleep(100);
continue deathWatch;
}
}
break deathWatch;
}
System.out.println("All done, time elapsed: " + (System.currentTimeMillis() - startTime) + "ms.");
}
}
Typical output:
Creating 100 threads...
Threads created, starting them for 20 iterations...
Threads are running, waiting for them to end...
Incorrect number of rows, expected 3105, but got 3099, thread 49, iteration 4
Incorrect number of rows, expected 3105, but got 3086, thread 17, iteration 6
Incorrect number of rows, expected 3105, but got 3101, thread 64, iteration 8
Incorrect number of rows, expected 3105, but got 2853, thread 57, iteration 12
etc...
Related
My requirement is to encrypt personal identification columns of table. For which I have written a small code which selects the data in batches and insert into new table with few extra columns.
Problem is when i run my code , it works good in begining but stops inserting record in db. It does not print any exception as well.
This is my connection pool config.
private BlockingQueue<EntityManager> getConnectionPool(int poolSize) throws InterruptedException {
// List<EntityManager> list = new ArrayList<>();
BlockingQueue<EntityManager> queue = new ArrayBlockingQueue<>(poolSize);
int i = poolSize;
do
{
EntityManager entityManager = connectionService.getEm();
queue.put(entityManager);
//list.add(entityManager);
i--;
}
while (i != 0);
return queue;
}
This is the class from where everything starts. It calculates the total number of batches and calls one method for executor service.
public void insertData() throws InterruptedException {
key = hash(key);
EntityManager entityManager = connectionService.getEm();
EntityTransaction entityTransaction = entityManager.getTransaction();
BlockingQueue<EntityManager> queue = getConnectionPool( 200);
try {
int batchSize= 1000;
BigInteger totalResults = partnerRepository.getCountCustomerLedger(entityManager);
double totalPages = Math.ceil(totalResults.longValue() / batchSize);
int maxResult = batchSize;
CountDownLatch latch = new CountDownLatch(((Double)totalPages).intValue());
for(int i =1 ; i <= totalPages; i++) {
int firstResult = (i - 1) * batchSize;
if (i == totalPages)
{
batchSize = totalResults.intValue() - firstResult;
}
exectueTask(queue, firstResult, batchSize, latch, i);
}
System.out.println("waiting for latch to finish");
latch.await();
System.out.println("latch exited");
}catch (Exception e) {
e.printStackTrace();
if (entityTransaction.isActive()) {
entityTransaction.rollback();
}
entityManager.close();
}
finally {
int i = poolSize;
do
{
queue.take().close();
i--;
}
while (i != 0);
}
entityManager.close();
}
This calls the executor method
private void exectueTask(BlockingQueue<EntityManager> queue, int firstResult, int batchSize, CountDownLatch latch, int batchNumber) {
taskExecutor.execute(() -> {
try {
try {
run(queue, firstResult, batchSize, latch, batchNumber);
} catch (IOException e) {
e.printStackTrace();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
});
}
Here I executing queries in batches and inserting data into db
private void run(BlockingQueue<EntityManager> queue, int firstResult, int batchSize, CountDownLatch latch, int batchNumber) throws InterruptedException, IOException {
logger.info("batchNumber " + batchNumber + " batchNumber called " + " at " + new Date());
EntityManager entityManager = queue.take();
logger.info("batchNumber " + batchNumber + " batchNumber took " + " following time to get entitymanager " + new Date());
EntityTransaction entityTransaction = entityManager.getTransaction();
List<CustomerLedger> customerLedgerList = partnerRepository.getAllCustomerLedger(entityManager,firstResult, batchSize);
//List<Object[]> customerLedgerList = partnerRepository.getAllCustomerLedgerNative(entityManager,firstResult, batchSize);
entityTransaction.begin();
for (CustomerLedger old :customerLedgerList) {
CustomerLedgerNew ledgerNew = new CustomerLedgerNew();
String customerLedgerJson = objectMapper.writeValueAsString(old);
ledgerNew = customerLedgerToCustomerLedgerNew(customerLedgerJson);
ledgerNew.setFirstName(convertToDatabaseColumn(old.getFirstName(),key));
ledgerNew.setMiddleName(convertToDatabaseColumn(old.getMiddleName(),key));
ledgerNew.setLastName(convertToDatabaseColumn(old.getLastName(),key));
ledgerNew.setAddressLine1(convertToDatabaseColumn(old.getAddressLine1(),key));
ledgerNew.setAddressLine2(convertToDatabaseColumn(old.getAddressLine2(),key));
ledgerNew.setAddressLine3(convertToDatabaseColumn(old.getAddressLine3(),key));
ledgerNew.setAddressLine4(convertToDatabaseColumn(old.getAddressLine4(),key));
ledgerNew.setHomePhone(convertToDatabaseColumn(old.getHomePhone(),key));
ledgerNew.setWorkPhone(convertToDatabaseColumn(old.getWorkPhone(),key));
ledgerNew.setEmail1(convertToDatabaseColumn(old.getEmail1(),key));
ledgerNew.setMobile(convertToDatabaseColumn(old.getMobile(),key));
ledgerNew.setMobileSha(sha256Hash(old.getMobile()));
ledgerNew.setMobileChecksum(getMD5Hash(old.getMobile()));
ledgerNew.setEmailSha(sha256Hash(old.getEmail1()));
ledgerNew.setEmailChecksum(getMD5Hash(old.getEmail1()));
//ledgerNew.setChannel(old.getChannel());
//ledgerNew.setUniqueCustomerId(old.getUniqueCustomerId());
//ledgerNew.setLastModifiedDate(old.getLastModifiedDate());
entityManager.persist(ledgerNew);
}
//System.out.println("commited");
logger.info("batchNumber " + batchNumber + " batchNumber started commiting data at " + new Date());
entityTransaction.commit();
logger.info("batchNumber " + batchNumber + " batchNumber finished commiting data at " + new Date());
queue.put(entityManager);
latch.countDown();
logger.info("batchNumber " + batchNumber + " latch count " + latch.getCount());
}
what I noticed from the logs , at one point It only print the logs
** batchNumber 615 batchNumber started commiting data at Wed Dec 11 17:22:54 IST 201** but does not print the next line logs of commiting data. I am really unable to get this reason.
Thread pool config class
#Configuration
public class ThreadPoolConfiguration {
private final static org.slf4j.Logger LOGGER = org.slf4j.LoggerFactory.getLogger(ThreadPoolConfiguration.class);
private final int defaultCorePoolSize = 200;
private final int defaultMaxPoolSize = 300;
private final int defaultQueueCapacity = 20000;
private final int defaultKeepAlive = 10;
#Bean
#Qualifier("TaskExecutor")
public TaskExecutor taskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(defaultCorePoolSize);
executor.setMaxPoolSize(defaultMaxPoolSize);
executor.setQueueCapacity(defaultQueueCapacity);
executor.setKeepAliveSeconds(defaultKeepAlive);
executor.setAllowCoreThreadTimeOut(true);
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.setThreadNamePrefix("encryption-DEFAULT-");
executor.initialize();
return executor;
}
}
Please forgive me If I am unable to frame it properly
There are multiple possible sources for your problem:
* MySQL table locking deadlock
* MySQL running out of connections (check MySQL logs)
* Out-of-memory situations due to overfull EntityManager buffers
* Deadlock in EntityManager
Since your problem arises when you call entityTransaction.commit(), I'd presume you have a problem with table locking deadlocks. Take a look at this article to analyse posible MySQL deadlock problems.
Your approach looks to me as if you've been working for quite some time on the performance this batch update. Working with multiple threas will give the database a hard time doing table/record locking while not gaining much performance.
I'd recommend doing big batches of work not with an entity manager, but JDBC.
If you have to use JPA, better optimize on the batch size. Take a look at The best way to do batch processing with JPA and Hibernate to get more inspirations.
I've got problems at trying to write data into my oracle table.
So, what I'm trying to do is to receive data from TMDB (tmdb.org) and write those into an Oracle table.
Here's my class where all the magic is happening:
public class Movies3 {
public void execute_to_db() throws SQLException, ClassNotFoundException, RuntimeException {
final String url = "jdbc:oracle:thin:#192.168.XXX.XXX:XXX:XXX";
final String user = "TEST2";
final String password = "XXX";
final String table = "TMDB_TEST";
DatabaseConnect db = new DatabaseConnect();
QueryCreateTable_Movies createtable = new QueryCreateTable_Movies();
try {
db.connect(user, password, url);
ResultSet tablelike = db.processQuery(
"SELECT COUNT(table_name) " + "FROM all_tables " + "WHERE table_name = '" + table + "' ");
PreparedStatement insert_ps = db.prepareStatement("INSERT INTO " + table + " "
+ "(TMDB_ID, IMDB_ID, ORIGINAL_TITLE, TITLE_DE, BUDGET, REVENUE, RELEASE_DATE) "
+ "VALUES (?,?,?,?,?,?,?)");
int tablelike_int = 0;
while (tablelike.next())
tablelike_int = tablelike.getInt(1);
if (tablelike_int == 0)
db.processInsert(createtable.create);
else {
TmdbMovies movies = new TmdbApi("XXX").getMovies();
MovieDb latest_movie = movies.getLatestMovie();
int tmdb_max_id = latest_movie.getId();
try {
int id_exist = 0;
for (int i = 1; i < tmdb_max_id; i++) {
ResultSet id_existq = db
.processQuery("SELECT (tmdb_id) FROM " + table + " WHERE tmdb_id = " + i);
while (id_existq.next())
id_exist = id_existq.getInt(1);
if (id_exist == 0) {
try {
MovieDb movie_name_en = movies.getMovie(i, "en");
MovieDb movie_name_de = movies.getMovie(i, "de");
String original_title = movie_name_en.getOriginalTitle();
String title_de = movie_name_de.getTitle();
String imdb_id = movie_name_en.getImdbID();
int budget_en = (int) movie_name_en.getBudget();
int revenue_en = (int) movie_name_en.getRevenue();
String release_date_en = movie_name_en.getReleaseDate();
insert_ps.setInt(1, i);
insert_ps.setString(2, imdb_id);
insert_ps.setString(3, original_title);
insert_ps.setString(4, title_de);
insert_ps.setInt(5, budget_en);
insert_ps.setInt(6, revenue_en);
insert_ps.setString(7, release_date_en);
insert_ps.executeUpdate();
/** Start Output **/
double percent = (i * 100) / tmdb_max_id;
StringBuilder string = new StringBuilder(140);
int percent_int = (int) percent;
long total = (long) tmdb_max_id;
long current = (long) i;
string.append('\r').append(String.join("",
Collections.nCopies(percent_int == 0 ? 2 : 2 - (int) (Math.log10(percent_int)),
" ")))
.append(String.format(" %d%% [", percent_int))
.append(String.join("", Collections.nCopies((percent_int / 2), "=")))
.append('>')
.append(String.join("",
Collections.nCopies((100 / 2) - (percent_int / 2), " ")))
.append(']')
.append(String.join("",
Collections.nCopies(
(int) (Math.log10(total)) - (int) (Math.log10(current)), " ")))
.append(String.format(" %d/%d | TMDB_ID: %d | Movie: %s", current, total, i,
original_title));
System.out.flush();
System.out.print(string);
/** End Output **/
i++;
tmdb_max_id = latest_movie.getId();
} catch (RuntimeException e) {
continue;
} catch (SQLException sqle) {
System.err.println(sqle + " SQL ERROR at movie with ID" + i);
throw sqle;
} finally {
id_existq.close();
insert_ps.close();
tablelike.close();
}
} else
i++;
}
} catch (SQLException sqle2) {
throw sqle2;
} catch (RuntimeException e2) {
throw e2;
} finally {
insert_ps.close();
tablelike.close();
}
}
db.disconnect();
} catch (SQLException sqle_con) {
throw sqle_con;
}
catch (ClassNotFoundException clnf) {
throw clnf;
} finally {
}
}
}
When executing, I receive ORA-00604 and ORA-01000.
Exception in thread "main" java.sql.SQLException: ORA-00604: Fehler auf rekursiver SQL-Ebene 1
ORA-01000: Maximale Anzahl offener Cursor überschritten
ORA-00604: Fehler auf rekursiver SQL-Ebene 1
ORA-01000: Maximale Anzahl offener Cursor überschritten
ORA-01000: Maximale Anzahl offener Cursor überschritten
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)
at oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
at database.DatabaseConnect.processQuery(DatabaseConnect.java:31)
at tmdb_api_to_db.Movies3.execute_to_db(Movies3.java:75)
at tmdb_api_to_db.Main.main(Main.java:22)
I'm pretty sure, that the problem occurs because I got a fallacy with those try-catch-finally constructions - especially when closing my statements - but can't find my mistake and I simply feeling like a dog chasing its tail...
I'm using Eclipse Neon2 on Java 8 and Oracle 11g.
If there is further information needed, I'd be happy to provide.
Please consider that I'm not very experienced and thus be forgiving, if my question hurts any feelings... :)
The error message tranlated into English says
Maximum number of open cursors exceeded
The problem is that you are leaking cursors. One place where this could happen is in the inner for loop.
When the body of
if (id_exist == 0) {
is not executed, the try / finally where you should be closing the ResultSet is never executed. That will leak a cursor. Eventually, Oracle won't let you open any more ...
I'm going to recommend that you read up on the "try with resources" constructed that has been supported by Java since Java 7.
It allows you to write resource cleanup code that is easier to read AND less error prone.
Also, if you find that you have a method where the majority of the code is indented 9 levels deep. That should be a sign to you that you need to refactor it. Seriously, you don't need to do all of that in a single monolithic method. For a start, refactoring your code will make it easier for you (and everyone else) to understand.
package Simple;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Time;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Date;
import java.util.List;
import java.text.*;
public class CurrentProg {
//connecting to the database
private static final String DB_DRIVER = "com.mysql.jdbc.Driver";
private static final String DB_CONNECTION ="jdbc:mysql://localhost:3306/db?autoReconnect=true";
private static final String DB_USER = "root";
private static final String DB_PASSWORD = "root";
static Connection dbConnection = null;
static Statement statement = null;
static int total=1;
//Searching between startdate and enddate
public static java.util.LinkedList searchBetweenDates(java.util.Date startDate, java.util.Date endDate) {
java.util.Date begin = new Date(startDate.getTime());
java.util.LinkedList list = new java.util.LinkedList();
list.add(new Date(begin.getTime()));
java.util.Date end = new Date(endDate.getTime());
endDate.setTime(endDate.getTime() + 24*3600*1000);
Calendar cal = Calendar.getInstance();
cal.setTime(begin);
dbConnection = getDBConnection();
while(begin.compareTo(endDate)<0){
begin = new Date(begin.getTime() + 86400000);
list.add(new Date(begin.getTime()));
Timestamp timestamp = new Timestamp(new Date().getTime());
//For a single day calculation: 24hours*60mins=1440 /2 (2 mins time difference as per the requirement) = 720
for (int j = 0; j < 720; j++) {
cal.add(Calendar.MINUTE, 2);
timestamp = new Timestamp(cal.getTime().getTime());
String S = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(timestamp);
String[] parts = S.split(" ");
String date=parts[0];
String time=parts[1];
cal.getTime().toString();
// To create data loop into a List
List<String> records = new ArrayList<String>();
StringBuffer record = new StringBuffer();
for (int i = 1; i <= total; i++) {
records = new ArrayList<String>(total);
int a2 = 220 + j % 31; // 230 - 244 by 1
String wString = Integer.toString(a2);
String a = String.valueOf(a2);
double b2 = 0.00 + j % 3.7 ; // 1.3 - 3.9 by 0.1
String aString = Double.toString(b2);
String b = String.valueOf(b2);
b = b.substring(0, Math.min(b.length(), 5));
record.delete(0, record.length());
record.append(a + "," + b + ",'"+ date + "', '"+ time + "'");
record.append("\t\t");
record.append("\n");
records.add(record.toString());
//Insert Query
String insertTableSQL = "INSERT INTO cmd1"
+ "(a, b, date, time) " + "VALUES"
+ "("+record.toString()+")";
System.out.println("insertTableSQL - " + insertTableSQL); // Statement.executeUpdate(insertTableSQL);
try {
statement = dbConnection.createStatement();
statement.executeUpdate(insertTableSQL);
System.out.println("Record is inserted into Db table!");
} catch (SQLException e) {
System.out.println(e.getMessage());
}
try {
// dbConnection = getDBConnection();
statement = dbConnection.createStatement();
statement.executeUpdate(insertTableSQL);
System.out.println("Record is inserted into Db table!");
} catch (SQLException e) {
System.out.println(e.getMessage());
}
finally {
// httpPost.releaseConnection()
try{
if(statement!=null)
statement.close();
}
finally{
}
try{
if(dbConnection!=null)
dbConnection.close();
}
finally{
}
}
}
}
}
return list;
}
#SuppressWarnings("unused")
public static void main(String[] args) throws Exception {
//To enter startDate and enddate
// EntityManagerFactory.getCache().evictAll;
SimpleDateFormat startDate=new java.text.SimpleDateFormat("yyyy-MM-dd");
SimpleDateFormat endDate=new java.text.SimpleDateFormat("yyyy-MM-dd");
java.util.LinkedList hitList = searchBetweenDates(
startDate.parse("2016-01-01"),
endDate.parse("2016-03-01"));
String[] combo = new String[hitList.size()];
for(int i=0; i<hitList.size(); i++)
combo[i] = new java.text.SimpleDateFormat("yyyy-MM-dd").format(((java.util.Date)hitList.get(i)));
}
private static void insertRecordIntodb() {
//
}
private static Connection getDBConnection() {
Connection dbConnection = null;
try {
Class.forName(DB_DRIVER);
} catch (ClassNotFoundException e) {
System.out.println(e.getMessage());
}
try {
dbConnection = DriverManager.getConnection(DB_CONNECTION, DB_USER, DB_PASSWORD);
return dbConnection;
} catch (SQLException e) {
System.out.println(e.getMessage());
}
return dbConnection;
}
}
Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: java.net.SocketException: No buffer space available (maximum connections reached?): connect'.
** BEGIN NESTED EXCEPTION **
java.net.SocketException
MESSAGE: java.net.SocketException: No buffer space available (maximum connections reached?): connect
STACKTRACE:
java.net.SocketException: java.net.SocketException: No buffer space available (maximum connections reached?): connect
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2717)
at com.mysql.jdbc.Connection.<init>(Connection.java:1509)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at Simple.CurrentProg.getDBConnection(CurrentProg.java:186)
at Simple.CurrentProg.searchBetweenDates(CurrentProg.java:126)
at Simple.CurrentProg.main(CurrentProg.java:164)
** END NESTED EXCEPTION **
Attempted reconnect 3 times. Giving up.
Exception in thread "main" java.lang.NullPointerException
at Simple.CurrentProg.searchBetweenDates(CurrentProg.java:137)
at Simple.CurrentProg.main(CurrentProg.java:164)
Here I am trying to connect java program with database but when i am trying to insert large data say for 1 month so its only fetching 16000 records not more than that i want the data should be inserted as per the given date range what should i do to get that . In stacktrace its showing an exception as no buffer space available maximum connection reached. Thanks In advance
Here is the modified code. I removed a lot of code which is not necessary:
public static LinkedList<Date> searchBetweenDates(Date startDate, Date endDate) throws SQLException {
Date begin = new Date(startDate.getTime());
LinkedList<Date> list = new LinkedList<Date>();
list.add(new Date(begin.getTime()));
endDate.setTime(endDate.getTime() + 24 * 3600 * 1000);
Calendar cal = Calendar.getInstance();
cal.setTime(begin);
dbConnection = getDBConnection();
PreparedStatement ps = dbConnection.prepareStatement("INSERT INTO cmd1(aaaa, bbbb, datee, timee) VALUES(?, ?, ?, ?)");
while (begin.compareTo(endDate) < 0) {
begin = new Date(begin.getTime() + 86400000);
list.add(new Date(begin.getTime()));
Timestamp timestamp = new Timestamp(new Date().getTime());
// For a single day calculation: 24hours*60mins=1440 /2 (2 mins time
// difference as per the requirement) = 720
for (int j = 0; j < 720; j++) {
cal.add(Calendar.MINUTE, 2);
timestamp = new Timestamp(cal.getTime().getTime());
String S = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(timestamp);
String[] parts = S.split(" ");
String date = parts[0];
String time = parts[1];
cal.getTime().toString();
// To create data loop into a List
for (int i = 1; i <= total; i++) {
int a2 = 220 + j % 31; // 230 - 244 by 1
String a = String.valueOf(a2);
double b2 = 0.00 + j % 3.7; // 1.3 - 3.9 by 0.1
String b = String.valueOf(b2);
b = b.substring(0, Math.min(b.length(), 5));
ps.setString(1, a);
ps.setString(2, b);
ps.setString(3, date);
ps.setString(4, time);
ps.execute();
}
}
}
if (ps != null)
ps.close();
if (dbConnection != null)
dbConnection.close();
return list;
}
What I changed:
I removed your try/catch's because I wanted to make the code short so that I can edit it here easily. But you should handle exception correctly. And don't swallow exceptions ever. I mean the following is no go:
} catch (SQLException e) {
System.out.println(e.getMessage());
}
It is very bad Idea because you'll not see the real cause of the problem in this case; at least do e.printStackTrace() eventhough it is not recommended in a real project.
I exchanged the Statement object for PreparedStatement because it is much more efficient.
Removed package names and put import statements instead because it is not necessary to do so unless you have different classes from different packages with the same name.
I changed column names because my DB does not want to accept them. Column names like a are very bad. Choose instead descriptive names so that you might understand what it is for a couple of months later. Don't use column names like date because they are reserved words for some databases systems I know.
Don't create database resources like Connection in a loop unless it is absolutely needed! Otherwise, your program will go out of resources because these resources are very expensive! That is exactly what you are experiencing right now.
Hope it helps, otherwise, drop me a comment.
I'm need to update a table with data from a CSV. All data is validated before the update takes place: a validation method (witch is not presented bellow) checks if some assumptions are true and "flags" the object as valid or invalid. I've already test it a lot and it's working exactly as I want.
Even so, I would like to guarantee that all Statements will be executed even if there's a fail on a batch, something that I was not able to think about. If this happens, I want the batch in witch this fail statement is to be skipped and that the next one is executed.
public void updateTable(List<PersonBean> personList) {
Connection connection = null;
PreparedStatement ps = null;
String updateDBPersonSQL = "UPDATE Person set merge_parent_id = ? WHERE id = ?";
try {
logger.info("DATA UPDATING STARTED");
input = new FileInputStream("resources/propertiesFiles/applications.properties");
properties.load(input);
final int batchSize = Integer.parseInt(properties.getProperty("batchSize"));
connection = DBConnection.getConnection();
connection.setAutoCommit(false);
int validObj = 0;
ps = connection.prepareStatement(updateDBPersonSQL);
for (int i = 0; i < personList.size(); i++) {
PersonBean person = personList.get(i);
if (person.getValidationStatus().equals("valid")) {
ps.setInt(1, person.getMerge_parent_id());
ps.setInt(2, person.getId());
ps.addBatch();
validObj++;
if (validObj % batchSize == 0 && validObj != 0) {
ps.executeBatch();
connection.commit();
logger.info((batchSize) + " rows updated");
}
}
}
int [] batchCount = ps.executeBatch();
connection.commit();
logger.info(batchCount.length + " rows updated");
writeValidationStatusToCSV(personList);
} catch (BatchUpdateException e) {
int [] updateCount = e.getUpdateCounts();
for (int i = 0; i < updateCount.length; i++) {
if (updateCount[i] >= 0) {
logger.info(updateCount.length + " objects updated.");
} else if (updateCount[i] == Statement.EXECUTE_FAILED) {
?????
}
}
logger.error(updateCount.length);
logger.error("BatchUpdateException: " + e);
logger.error("getNextException: " + e.getNextException());
try {
connection.rollback();
} catch (SQLException e1) {
logger.error("Rollback error: " + e1, e1);
}
} finally {
if (ps!= null) {
try {
ps.close();
} catch (SQLException e) {
logger.info(e);
}
}
}
logger.info("DATA UPDATING FINISHED");
}
I saw a lot of material about how to handle the exception, but none explained or pointed me to the direction of how to retry the next Statements, it means, how to execute the next batch.
How do I manage to do this?
EDIT: I'm using Postgresql
I manage to retry the next batches by surrounding the batch execution with try and catch statements. This way I'm able to catch the BatchUpdateException and call a continue statement.
try {
ps.executeBatch();
connection.commit();
/*Some more code*/
} catch (BatchUpdateException e) {
connection.rollback();
/*Some more code*/
continue;
}
I also used some control logic to "flag" the statements and batches that were already executed and logged them, making it easier to troubleshoot if some statement fails.
Here is the full code:
public void updateTable(List<PersonBean> personList) throws Exception {
logger.info("TABLE UPDATE STARTED");
List <PersonBean> personListValidated = createValidStmtList(personList);
Connection connection = null;
PreparedStatement ps = null;
String updatePersonSQL = "UPDATE Person SET merge_parent_id = ? WHERE id = ?";
input = new FileInputStream("resources/propertiesFiles/applications.properties");
properties.load(input);
final int batchSize = Integer.parseInt(properties.getProperty("batchSize"));
/*A list was used to "flag" the batches that were already executed. BatchStatus objs have only two parameters, number (incremented as the batches are being executed) and status (success or fail).*/
List <BatchStatus> batchStatusList = new ArrayList<BatchStatus>();
/*This variables will be used to help flag the batches and statements that were already executed.*/
int batchCount = 0;
int stmtAddedToBatchCount = 0;
try {
connection = DBConnection.getConnection();
connection.setAutoCommit(false);
ps = connection.prepareStatement(updatePersonSQL);
/*personListValidated contains the objects that will be updated in the table. Instead of doing the validation on the update method, I decomposed
* this part in other 2 methods, making it easier to control of the statements added to the batch.
*/
for (int i = 0; i < personListValidated.size(); i++) {
PersonBean personValid = personListValidated.get(i);
ps.setInt(1, personValid.getMerge_parent_id());
ps.setInt(2, personValid.getId());
ps.addBatch();
personValid.setToBatch("true");
stmtAddedToBatchCount++;
logger.info("Row added to batch (count: " + stmtAddedToBatchCount + ")");
if (stmtAddedToBatchCount % batchSize == 0) {
batchCount++;
try {
ps.executeBatch();
connection.commit();
for (int j = stmtAddedToBatchCount - batchSize; j < stmtAddedToBatchCount; j++){
personValid = personListValidated.get(j);
personValid.setValidationStatus("success");
}
BatchStatus batchStatusObj = new BatchStatus(batchCount, "sucess");
batchStatusList.add(batchStatusObj);
logger.info(batchStatusList.get(batchCount - 1));
} catch (BatchUpdateException e) {
connection.rollback();
for (int j = stmtAddedToBatchCount - batchSize; j < stmtAddedToBatchCount; j++){
personValid = personListValidated.get(j);
personValid.setValidationStatus("fail");
}
BatchStatus batchStatusObj = new BatchStatus(batchCount, "fail");
batchStatusList.add(batchStatusObj);
logger.info(batchStatusList.get(batchCount - 1));
logger.error("Bacth execution fail: " + e, e);
continue;
}
}
}
} catch (SQLException e) {
logger.error(e, e);
}
int[] lastBatchCount = null;
/*Try and catch to handle the statements executed on the last batch*/
try {
lastBatchCount = ps.executeBatch();
connection.commit();
for (int j = batchStatusList.size() * batchSize; j < stmtAddedToBatchCount; j++){
PersonBean personValid = personListValidated.get(j);
personValid.setValidationStatus("success");
}
logger.info(lastBatchCount.length + " rows inserted on the last batch");
logger.info("Last batch excuted");
} catch (BatchUpdateException e) {
connection.rollback();
for (int j = batchStatusList.size() * batchSize; j < stmtAddedToBatchCount; j++){
PersonBean personValid = personListValidated.get(j);
personValid.setValidationStatus("fail");
}
logger.error("Last batch fail to execute: " + e, e);
}
writeValidationStatusToCSV(personList);
logger.info("TABLE UPDATE FINISHED");
}
I am making a practice program using java and an access database.
the program is an ultimate tictactoe board and the databse is meant for keeping track of the names of the players and their scores.
the trouble i am having is that i keep getting these errors.
Exception in thread "main" java.lang.NullPointerException
at AccessDatabaseConnection.getName(AccessDatabaseConnection.java:39)
at ultimate.<init>(ultimate.java:39)
at ultimate.main(ultimate.java:82)
with further research i also found this:
[Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified
here is my code. the math is a little unfinished in the sql statements but im not really worried about that yet. i need to get this connection between the program and the database.
here is the area of code in my constructor for the program that connects to the accessdatabaseconnections class:
AccessDatabaseConnection DB = new AccessDatabaseConnection();
Font f = new Font("Dialog", Font.BOLD, 80);
public ultimate() {
super("Testing Buttons");
String dbname = DB.getName();
String wins = DB.getWins();
String losses = DB.getLosses();
Container container = getContentPane();
container.setLayout(null);
ButtonHandler handler = new ButtonHandler();
for (int j = 0; j < 9; j++) {// set the rows
x = 10;
for (int i = 0; i < 9; i++) {// set the columns
button[j][i] = new JButton();
container.add(button[j][i]);
button[j][i].setName(Integer.toString(j) + "_"
+ Integer.toString(i));
button[j][i].addActionListener(handler);
button[j][i].setSize(100, 100);
button[j][i].setVisible(true);
button[j][i].setFont(f);
button[j][i].setText(null);
if ((i > 2 && j < 3 && i < 6) || (j > 2 && j < 6 && i < 3)
|| (j > 2 && j < 6 && i < 9 && i > 5)
|| (j > 5 && j < 9 && i < 6 && i > 2)) {
button[j][i].setBackground(Color.LIGHT_GRAY);
} else {
button[j][i].setBackground(Color.WHITE);
}
button[j][i].setLocation(x, y);
x = x + 110;
}
y = y + 110;
}
setSize(1024, 1050);
setVisible(true);
container.setBackground(Color.BLACK);
}
public static void main(String args[]) {
ultimate application = new ultimate();
application.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
PlayerOne = JOptionPane.showInputDialog("Player 1: Enter Your Name");
PlayerTwo = JOptionPane.showInputDialog("Player 2: Enter Your Name");
while(PlayerOne == PlayerTwo){
PlayerTwo = JOptionPane.showInputDialog("Player 2: Re-Enter Your Name (Cannot be the same!)");
}
}
and here is the code for accessing the database:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.logging.Level;
import java.util.logging.Logger;
public class AccessDatabaseConnection {
public static Connection connect() {
Connection con;
try {
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
String database ="jdbc:odbc:Driver{Microsoft Access Driver (*.accdb)};DBQ=C:\\Users\\McKenzieC\\Documents\\tictactoeRecords.accdb;";
con = DriverManager.getConnection(database, "", "");
} catch (Exception ex) {
return null;
}
return con;
}
public void addData(String nameOne, int win, String nameTwo,int loss){
try {
Statement stmt = connect().createStatement();
stmt.executeQuery("INSERT INTO t_Records (Name, Wins) " +
"VALUES (" + nameOne + ", " + Integer.toString(win));
/*stmt.executeQuery("INSERT INTO t_Records (Name, Wins) " +
"VALUES (" + nameTwo + ", " + Integer.toString(loss));
+ ", " + Integer.toString(loss)*/
}
catch (SQLException ex) {
}
}
public String getName() {
try {
Statement stmt = connect().createStatement();
ResultSet rset = stmt.executeQuery("SELECT * FROM t_Records");
if (rset.next()) {
String name = rset.getString("Name");
return name;
}
} catch (SQLException ex) {
}
return null;
}
public String getWins() {
try {
Statement stmt = connect().createStatement();
ResultSet rset = stmt.executeQuery("SELECT * FROM t_Records");
if (rset.next()) {
String wins = rset.getString("Wins");
return wins;
}
} catch (SQLException ex) {
}
return null;
}
public String getLosses() {
try {
Statement stmt = connect().createStatement();
ResultSet rset = stmt.executeQuery("SELECT * FROM t_Records");
if (rset.next()) {
String losses = rset.getString("Losses");
return losses;
}
} catch (SQLException ex) {
}
return null;
}
public static void main(String[] args) {
}
}
I assume that you can't see the real error because you're hiding the real error:
Never do this:
catch (Exception ex) {
return null;
}
You can change for this at least (again not recommended but better than the above code):
catch (Exception ex) {
ex.printStackTrace();
return null;
}
//After this change the program will fail again but you will got a better error message
But you always must manage the Exception:
Print a error message
Put a log message (java logging, log4j and so on)
Deal with the error
Re-throw the exception
And son on
The statement...
String database ="jdbc:odbc:Driver{Microsoft Access Driver (*.accdb)};DBQ=C:\\Users\\McKenzieC\\Documents\\tictactoeRecords.accdb;";
...has two problems:
You are missing the equal sign (=) after the Driver keyword.
There is no ODBC driver named Microsoft Access Driver (*.accdb).
Try this instead:
String database ="jdbc:odbc:Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\\Users\\McKenzieC\\Documents\\tictactoeRecords.accdb;";
Is Statement stmt equal null after connect().createStatement();? If yes, then stmt.executeQuery will cause null ptr exception.