I'm developing tool to continuously export changes from MongoDb to Oracle database.
I have problem with execution batch operation(Oracle).
static void save(List result) {
withBatchConnection { Statement stm ->
result.each { String line ->
stm.addBatch(line)
}
}
}
static withConnection(Closure closure) {
def conn = null
boolean success = false
while (!success) {
try {
conn = getConnection()
closure.call(conn)
success = true
} catch (e) {
log.error('Connection problem', e)
log.error(e, e)
log.info('Retrying for 30 sec')
sleep(30000)
} finally {
conn?.close()
}
}
}
static withTransactionConnection(Closure closure) {
withConnection { Sql sql ->
OracleConnection conn = sql.getConnection() as OracleConnection
conn.setAutoCommit(false)
closure.call(conn)
conn.commit()
}
}
static withBatchConnection(Closure closure) {
withTransactionConnection { Connection conn ->
def statement = conn.createStatement()
closure.call(statement)
statement.executeBatch()
statement.close()
}
}
Problem is i cant use prepared statement because order of operations is very important.
When I'm saving to MySql with Rewrite Batched Statements its like 10k operations per second. For Oracle is 400 operations/s
Is any chance to make it faster?
I'm using OJDBC 7 and groovy 2.4.7
Please set the array size from client side to maximum and try
Related
I have a use case where I would like to mix a jdbc transaction with jooq context.
The JDBC code looks like that:
public void inTransaction(InTransaction lambda) {
DataSource ds = dataSource.get();
try (Connection connection = ds.getConnection()) {
try {
logger.info("set autocommit to false");
connection.setAutoCommit(false);
try (Statement statement = connection.createStatement()) {
lambda.execute(statement);
logger.info("commiting transaction");
connection.commit();
}
} catch (RuntimeException e) {
logger.info("rolling back transaction");
connection.rollback();
throw e;
} finally {
logger.info("set autocommit to true");
connection.setAutoCommit(true);
}
} catch (SQLException e) {
throw new TilerException(e);
}
}
#FunctionalInterface
public interface InTransaction {
void execute(Statement statement) throws SQLException;
}
And I would like the lambda parameter to be able to work with both jdbc and jooq.
For jdbc using a statement is pretty straight-forward. For example something like this tutorail:
inTransaction(stmt -> {
String SQL = "INSERT INTO Employees " +
"VALUES (106, 20, 'Rita', 'Tez')";
stmt.executeUpdate(SQL);
String SQL = "INSERTED IN Employees " +
"VALUES (107, 22, 'Sita', 'Singh')";
stmt.executeUpdate(SQL);
});
In order to execute jooq queries on the same transaction I have to obtain a context. I found an api to get a DSLContext from datasource/connection.
What is not clear to me is if/how to create a jooq DSLContext from a statement?
A solution to the problem you described
You can do all of this with jOOQ's transaction API:
// Create this ad-hoc, or inject it, or whatever
DSLContext ctx = DSL.using(dataSource, dialect);
And then:
public void inJDBCTransaction(InJDBCTransaction lambda) {
ctx.transaction(config -> {
config.dsl().connection(connection -> {
try (Statement statement = connection.createStatement()) {
lambda.execute(statement);
}
});
});
}
public void inJOOQTransaction(InJOOQTransaction lambda) {
ctx.transaction(config -> lambda.execute(config.dsl()));
}
#FunctionalInterface
public interface InJDBCTransaction {
void execute(Statement statement) throws SQLException;
}
#FunctionalInterface
public interface InJOOQTransaction {
void execute(DSLContext ctx);
}
Your final code:
inJDBCTransaction(stmt -> {
String SQL = "INSERT INTO Employees " +
"VALUES (106, 20, 'Rita', 'Tez')";
stmt.executeUpdate(SQL);
String SQL = "INSERTED IN Employees " +
"VALUES (107, 22, 'Sita', 'Singh')";
stmt.executeUpdate(SQL);
});
inJOOQTransaction(ctx -> {
ctx.insertInto(EMPLOYEES).values(106, 20, "Rita", "Tez").execute();
ctx.insertInto(EMPLOYEES).values(107, 22, "Sita", "Singh").execute();
});
I'm not too convinced about the need for this abstraction over jOOQ and JDBC. jOOQ never hides JDBC from you. You can always access the JDBC API as shown above when using the DSLContext.connection() method. So, as shown above:
The jOOQ transaction API does exactly what you're planning to do. Wrap a lambda in a transactional context, commit if it succeeds, rollback if it fails (your version's rollback doesn't work because it catches the wrong exception).
If the "JDBC escape hatch" is needed, jOOQ can offer that
Side note
In many RDBMS, you don't want to run queries on a static JDBC Statement. You'll want to use PreparedStatement instead because:
You'll profit from execution plan caching (and less contention on the cache)
You'll avoid syntax errors (in case your real query is dynamic)
You'll avoid SQL injection trouble
If you want to get the query string from jOOQ you can call
String sqlString = query.getSQL()
and then use this string in your statement:
stmt.executeUpdate(sqlString);
I'm currently working on a college project, and I'm creating a very simple e-commerce style website.
I'm using JDBC driver manager and connection pool for the connection to the db, while using Tomcat 9.0 as the container.
The problem is: when I modify some product through the website (let's say the amount available for example), the website doesn't always reflect the changes, while I can always see the data correctly in MySql Workbench.
It actually works one time out of two on the same query:
I run the query for the first time after the changes -> it shows the old value
I run the query for the second time after the changes -> it shows the new value
I run the query for the third time after the changes -> it shows the old value
And so on.
I've already tried to set caching off (from the query, using the SQL_NO_CACHE), but it didn't seem to solve the problem, I've tried to use Datasource instead, but it causes other problems that most likely I won't have the time to solve.
This is the connection pool file, which I think might be problem, I'm not that sure tho:
public class DriverManagerConnectionPool {
private static List<Connection> freeDbConnections;
static {
freeDbConnections = new LinkedList<Connection>();
try {
Class.forName("com.mysql.cj.jdbc.Driver");
} catch (ClassNotFoundException e) {
System.out.println("DB driver not found:"+ e.getMessage());
}
}
private static synchronized Connection createDBConnection() throws SQLException {
Connection newConnection = null;
String ip = "localhost";
String port = "3306";
String db = "storage";
String username = "root";
String password = "1234";
newConnection = DriverManager.getConnection("jdbc:mysql://"+ ip+":"+ port+"/"+db+"?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC", username, password);
newConnection.setAutoCommit(false);
return newConnection;
}
public static synchronized Connection getConnection() throws SQLException {
Connection connection;
if (!freeDbConnections.isEmpty()) {
connection = (Connection) freeDbConnections.get(0);
freeDbConnections.remove(0);
try {
if (connection.isClosed())
connection = getConnection();
} catch (SQLException e) {
connection.close();
connection = getConnection();
}
} else {
connection = createDBConnection();
}
return connection;
}
public static synchronized void releaseConnection(Connection connection) throws SQLException {
if(connection != null) freeDbConnections.add(connection);
}
}
I really hope you can help me, I haven't found any solution online!
I guess it is because of auto-commit is disabled. Please try using #Transactional or set auto-commit to true. You can also try to use db.commit after each statement.
As per your connection pool implementation, all connection in your pool seems to be auto committed false.
Please check you have properly committed the connection after executing the query or not.
So it might be the case that, when executing the query after changes with same connection it reflects those changes, done earlier and on other connections, old values are might get returned.
I need to make a special treatment when a connection problem to the database is occurring like database server down and not an sql problem.
In the source code we can get various exceptions but which ones are belonging to the connection ones ?
We would like if this kind of problem occurs to make less logs.
EDITED
I have many methods that perform connection to the database but all get the session from the same method (initSession):
Here an example:
private Session initSession(HibernateUtil hibernateUtil) {
Session oSession = null;
try {
oSession = hibernateUtil.getSession();
} catch (Exception e) {
log.error("unable to log, Please check the details of your database");
}
return oSession;
}
public List findAlerts(int pFirstLine, int pNbElement) throws AnalyzerException {
List oAlerts = new ArrayList();
Session oSession = initSession(lHibernateUtil);
try {
oAlerts = AlertFinders.instance().findAlertByStatus(oSession, false, pFirstLine, pNbElement);
Iterator iterAlerts = oAlerts.iterator();
while (iterAlerts.hasNext()) {
...
}
} catch (UnableToLocateObjectException eU) {
throw new AnalyzerException(eU.getMessageSource(), eU.getClassNameSource(), eU.getMethodSource(), eU);
} finally {
oSession.close();
}
return oAlerts;
}
Multiple possible ways.
Use Java Connection isValid method.
Use connection pool - All major connection
pools support this functionality (including c3p0 and dbcp).They can
throw SQLException has getErrorCode() and getSQLState() methods
Write Java code & poll frequently - sample code below
Run arguments sample: jdbc:oracle:thin:#localhost:1521:XE system mypassword123 oracle.jdbc.driver.OracleDriver
public class DbConnCheck {
public static void main(String[] args) throws Exception {
String url = args[0];
String username = args[1];
String password = args[2];
String driver = args[3];
Class.forName(driver);
Connection conn = DriverManager.getConnection(url, username, password);
try {
Statement statement = conn.createStatement();
ResultSet rs = statement.executeQuery("SELECT SYSDATE FROM DUAL");
while(rs.next()) {
System.out.println(rs.getObject(1));
}
} finally {
conn.close();
}
}
}
Edit : Adding details on hibernate part
Not done in Hibernate but to be precise you can check in connection pool configuration.
If using c3p0 then check how you can best use setting like idle_test_period, preferredTestQuery and testConnectionOnCheckout;
If using dbcp then validationQuery can do the job.
If you want to use c3p0 with Hibernate and Spring check this link
I have a method in build.gradle that basically creates an oracle database connection and returns it. But while running the script, it shows build failed as seen in the screenshot below.
def ext.getOracleConnection = { ->
java.sql.Connection conn = null;
try{
configurations.jdbcdriver.files.each {
groovy.sql.Sql.classLoader.addURL(it.toURI().toURL())
}
configurations.xdb.files.each {
groovy.sql.Sql.classLoader.addURL(it.toURI().toURL())
}
//load the jdbc driver and create the connection.
java.sql.DriverManager.registerDriver(groovy.sql.Sql.classLoader.loadClass("oracle.jdbc.OracleDriver").newInstance())
readProperites();
conn = java.sql.DriverManager.getConnection(db_url, dbUser, dbPassword);
}
catch(Exception e){
e.printStackTrace();
}
return conn;
}
I tried changing the return type to java.sql.Connection but it doesn't work. Can anyone suggest a solution?.
This is a compilation error as shown in the failure message.
You need to remove def from the declaration.
This is a question relating more to coding standards than anything else.
The problem that I am having is that I am struggling to use my prepared statements as a class/constructor (I am from an informix background btw java is still new to me).
Usually when I code I like to keep scripting out of the main block as much as possible and then call in functions as I need them like in the example I will show. I am also exaggerating the structure with lots of forward slashes.
public class Script {
///////////////////////////////////////////////////////////////// start main
public static void main(String[] args) {
System.out.println("Script Is Starting"); // basic message
classCONN conn = new classCONN(); // connect class
Connection cnct = null; // connect variable
//
try { // try connect
conn.func_driverCheck(); //
cnct = conn.func_dbConnect(); //
} catch(SQLException log) { //
System.out.println(log); //
} //
*i would like to call the prepare*
*statements function once for the*
*rest of the script*
classSQL sql = new classSQL(); // prepare statements
sql.func_prep(cnct); //
users_sel.setString(1, "zoh"); // insert with prepared
users_sel.setString(2, "my"); // statements
users_sel.setString(3, "goodness"); //
row = users_sel.executeQuery(); //
}
///////////////////////////////////////////////////////////////// end main
///////////////////////////////////////////////////////////////// start classes
class classCONN {
public void func_driverCheck() {*code to check driver*}
public Connection func_dbConnect() {*code to connect to db*}
}
class classSQL {
*I would like to prepare my statements here*
public void f_prep(Connection cnct) {
lv_sql = "INSERT INTO users " +
"VALUES(?, ?, ?)";
PreparedStatement users_ins = cnct.prepareStatement(lv_sql);
}
}
///////////////////////////////////////////////////////////////// end classes
}
so my question being is there a way to get code like this to work so that the statements are prepared and then I can executeUpdate them from inside different classes or in the main or anything like that without actually preparing the statements completely in the main block
Here you have greate Example on how to use PreparedStatement.
JDBC PreparedStatement Example – Select List Of The Records
Ok I have come up with an answer to my own question and I would just like to post it here in case anyone can benefit from it or has anything to add to it because that's what this community is for.
public class Prog {
// static variables ---------------------------------------------------
static Connection conn;
static ResultSet row;
// main ---------------------------------------------------------------
public static void main(String[] args) {
// connect to db ---------------------------------------------------
try {
DBConnect cl_conn = new DBConnect();
conn = cl_conn.f_connect();
} catch(Exception log) {
System.out.println("FAIL")
}
// prepare statements ----------------------------------------------
SQLPrep prep = new SQLPrep(conn);
// execute statement 01 --------------------------------------------
try {
prep.users_sel.setInt(1, 2); // pass values to stmnt
row = prep.users_sel.executeQuery(); // execute stmnt
} catch(SQLException log) {
System.out.println("FAIL");
}
}
}
class DBConnect {
***code to connect to db***
}
// all prepared stmnts in one place ----------------------------------------
class SQLPrep {
static PreparedStatement users_sel = null; // select from users
static PreparedStatement access_sel = null; // select from access
try {
sp_sql = "SELECT * FROM USERS WHERE u_id = ?";
users_sel = conn.prepareStatement(sp_sql);
sp_sql = "SELECT * FROM ACCESS WHERE a_id = ?";
access_sel = conn.prepareStatement(sp_sql);
} catch(SQLException log) {
System.out.println("FAIL");
}
}
This may look strange to some people but I find this to be a very clean and tidy way of structuring code (keeping as much code out of the main as possible). Even the block where the statement is executed can be moved into a separate function and would only need to be passed 'prep.users_prep' to work.