I'm running some load tests on our system and I notice a massive number of "SET autocommit=0" and "SET autocommit=1" queries being executed. Something around 25,000 within 1 minute. I'm trying to figure out what is causing this and how to get rid of it.
We use the following technologies:
MySQL
Hibernate
Hikari
Spring
Tomcat
I have tried the following but it did not seem to help:
"SET autocommit = 0" in MySQL
Added the elideSetAutoCommits property in the db connection URL. "jdbc:mysql://localhost/db_name?useUniCode=true&characterEncoding=UTF-8&pinGlobalTxToPhysicalConnection=true&elideSetAutoCommits=true"
Could someone point me towards what might be causing these queries?
Could someone point me towards what might be causing these queries?
Your queries are the consequence of Connection#setAutoCommit(boolean) which is used to switch from the default mode that is auto-commit mode to transactional mode in order to insert/update/delete/read data within a transaction.
The common code is :
// Switch to transactional mode which actually triggers a SET autocommit = 0
con.setAutoCommit(false);
try {
// Some operations on the db
con.commit();
} finally {
// Switch back to auto-commit mode which actually triggers a SET autocommit = 1
con.setAutoCommit(true);
}
Here is a good link that explains how transactions work in JDBC.
If you know that your pool of connections will always be used to get connections in transactional mode, you can set the default mode in the configuration of Hikari thanks to the parameter autoCommit to set to false this way the connections will be already in transactional mode such that it won't be needed anymore to modify the mode.
This property controls the default auto-commit behavior of connections
returned from the pool. It is a boolean value. Default: true
More details about the configuration of Hikari here.
Related
How can I check that a connection to db is active or lost using spring data jpa?
Only the way is to make a query "SELECT 1"?
Nothing. Just execute your query. If the connection has died, either your JDBC driver will reconnect (if it supports it, and you enabled it in your connection string--most don't support it) or else you'll get an exception.
If you check the connection is up, it might fall over before you actually execute your query, so you gain absolutely nothing by checking.
That said, a lot of connection pools validate a connection by doing something like SELECT 1 before handing connections out. But this is nothing more than just executing a query, so you might just as well execute your business query.
our best chance is to just perform a simple query against one table, e.g.:
select 1 from SOME_TABLE;
https://docs.oracle.com/javase/6/docs/api/java/sql/Connection.html#isValid%28int%29
if you can use Spring Boot. Spring Boot Actuator is useful for you.
actuator will configure automatically and after it has activated ,
you can get know database status to request "health"
http://[CONTEXT_ROOT]/health
and it will return , database status like below
{"status":"UP","db":{"status":"UP","database":"PostgreSQL","hello":1}}
My hibernate config:
Properties properties = new Properties();
properties.put("hibernate.dialect", "org.hibernate.dialect.MySQLDialect");
properties.put("hibernate.hbm2ddl.auto", "validate");
properties.put("hibernate.show_sql", "true");
properties.put("hibernate.id.new_generator_mappings", "false");
properties.put("hibernate.connection.autocommit", "true");
properties.put("hibernate.connection.driver_class", "com.mysql.jdbc.Driver");
properties.put("hibernate.connection.url", DBConnection.url);
properties.put("hibernate.connection.username", DBConnection.username);
properties.put("hibernate.connection.password", DBConnection.password);
Code example:
// pattern 1
Session s = sessionFactory.openSession();
ObjectA A = s.load(ObjectA.class, pk);
A.setAttr("abc");
s.update(A);
s.close();
// pattern 2
Session s = sessionFactory.openSession();
s.beginTransaction();
ObjectA A = s.load(ObjectA.class, pk);
A.setAttr("abc");
s.update(A);
s.close();
// pattern 3
Session s = sessionFactory.openSession();
Transaction tx = s.beginTransaction();
ObjectA A = s.load(ObjectA.class, pk);
A.setAttr("abc");
s.update(A);
tx.commit();
s.close();
Please ignore my compilation error. I am using hibernate in web application (without spring), and without using transaction, because I am using MySql database and MySql autocommit is true, so in turn, in hibernate, I make it as autocommit true as well. (I am using mysql-connector-java-5.1.23-bin.jar too).
Three of the pattern, I am only able to get pattern 3 works. I am totally confused now. I have few questions below:
1) I can't understand why pattern 1 is not working, all my select (via hibernate CriteriaBuilder or load) and insert (via hibernate session.save) works but only update doesn't work.
2) OK then I try using transaction like pattern 2, my hibernate auto-commit is true, so I assume when I close the session, the transaction should auto-commit but it doesn't work. Why?
3) Pattern 3 works, why I need transaction manager here? I want the jdbc to execute each single query in each transaction (one sql in one transaction), I don't worry the performance, but I have to include transaction here, why?
For pattern 1 and 2, I found that the update script is not even generated (based on hibernate log), the problem is not because script is generated but commit failed. Don't understand why? Please help...
PS:
Just wrap up some points for future reference after some trial and error:
1) Hibernate will only generate sql script upon the session.flush() is called but not tx.commit(), and session.flush() have to be called in Transaction block. without Transaction, it leads to exception. Explicit flush is not needed if the flush mode is auto, commit() will trigger flush.
2) Hibernate Transaction is not equivalent to database transaction, after some tries, I found that, if hibernate autocommit is false, yes, they are functionally equivalent and corresponding begin transaction script is generated via JDBC and send over to database (my guess only). If hibernate autocommit is true, no begin transaction is started although we declare it in hibernate Transaction tx = s.beginTransaction(), all the query will be autocommit and rollback will not work.
3) The reason of my case, session.save() (and also select) work without Transaction, it is a bit special because save have to be triggered in order to get the table identifier(primary key) and so sql script generated even without flush.
4) For pattern 2, I miss-understood, autocommit doesn't mean autocommit upon session closed, its true meaning should be autocommit upon each sql reach database. so pattern 2 will not work because there is no tx.commit, meaning there is no flush, so no sql script is generated. (whether tx.commit will be called automatically upon session.close, it depend on vendor implementation, some will be rollback.)
Conclusion, Transaction block is needed in Hibernate not matter what.
I think you have a bit of confusion. The transaction (org.hibernate.transaction) is not exactly a DB transaction.
Such Object are used by hibernate when you flush the Session (Session.flush) to bound the instruction in a single db transaction. In other word do not confuse Hibernate Session with DB session, nevertheless do not confue hibernate Sessio with db connection.
Most important is that by specificatio hibernate generate sql code only for what is included between a hibernate transaction. That's why pattern A and B doesn't work and doesn't generate sql code. More specifically the auto-commit in pattern B has no influence since the sql cod is never generated. Moreover, according with hibernate best pracitces, you have to remember to open and close a transaction even for simple select instruction. By the way a select should work even without transaction, but you may have some trouble.
To better understand the concept we can resume the architecture:
hibernate session: is a container, wich hold your hibernate object and your db operations as java objects, and many other things.
the hibernate transaction: is a transaction object referred to an hibernate session.
db connection: is your connection to DB
conenction pool: is a set ofdb connection.
What appen when a session is flushed can be resumed with the followoing step:
a connection is get from the connection pool
for each committed
transaction in your session a db connection is get from the pool,
the sql commands are generated and sent to DB
the db connection is put back on the pool
it is just a small recap, but hope this help
r.
I am using hive jdbc 1.0 in my java application to create connection with hive server and execute query. I want to set the idle hive connection timeout from java code. Like say, user first creates the hive connection, and if the hive connection remains idle for next 10 minutes, then this connection object should get expired. If user uses this same connection object after 10 minutes for executing query, then hive jdbc should throw error. Can you please tell me the way to achieve this through java code.
I know there is a property hive.server2.idle.session.timeout in hive, but I don't know whether this is the right property required to be set from java code or there is some other property. I tried setting this property in jdbc connection string but it did not worked.
try {
Class.forName("org.apache.hive.jdbc.HiveDriver");
} catch (ClassNotFoundException e) {
LOG.error(ExceptionUtils.getStackTrace(e));
}
String jdbcurl = "jdbc:hive2://localhost:10000/idw?hive.server2.idle.session.timeout=1000ms";
Connection con;
con = DriverManager.getConnection(jdbcurl,"root","");
Thread.sleep(3000);
Now below I am using connection object, hive jdbc should throw error here as I used connection object after 3000 ms but I had set the idle timeout as 1000ms but hive jdbc had not thrown error
ResultSet rs = con.createStatement().executeQuery("select * from idw.emp");
Need help on this.
The hive.server2.idle.session.timeout causes a session to be terminated when it is not accessed for the specified duration. However, hive.server2.idle.session.timeout needs to be specified with hive.server2.session.check.interval set to
a positive value. basically, we need to be specify a number of session checks within the timeout interval to cause the session to be closed.
More details can be checked here
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.session.check.interval
You are looking at the problem the wrong way. These properties are not set in the URL, as described in the Hive AdminManual...
In the Hadoop ecosystem, server defaults are set in XML config files -- in that case /etc/hive/conf/hive-site.xml and/or hiveserver2-site.xml
Once your session is open, you can set custom values with the set <prop>=<value> statement (somewhat similar to the Oracle ALTER SESSION).
Caveat: some properties are defined as "final" in the config files and cannot be overriden. Check with your Hadoop Admin if in doubt.
We are having two instances of oracle (two physical machines) and one schema for our application.
we are using weblogic application server. application uses datasource for which XA transaction is enabled.
I am having one bean managed EJB, where i do -
update some data in table and then commit
submit oracle job
again update some data in table and then commit
Here i am getting error - java.sql.SQLException: could not use local transaction commit in a global transaction.
strangely this error is not commig every execution, it is comming 1 in 7-8 executions.
Now my questions are
What is significance of bean managed transaction if i use XA enabled transaction ?
why it is not encountering in every execution ?
Thanks.
below is code -
DataObject.updateDataAndReturnCount(" UPDATE EOD_Trn_BatchProcess SET iJobNo = ?, szParameters = ? WHERE iProcessSeqNo = ? ", conn, new String[]{null, strParameters, (String)mapParameters.get("__PROCESS_SEQ_NO")});
conn.commit();
String strStatement = "{? = call submitProcAsJob(?, ?)}";
//String strStatement = "begin ? := submitProcAsJob(?, ?); end;";
CallableStatement pStmt = conn.prepareCall(strStatement);
pStmt.registerOutParameter(1, OracleTypes.NUMBER);
pStmt.setObject(2, strJobName);
pStmt.setObject(3, strInstanceNo);
pStmt.execute();
vString strJobNo = pStmt.getString(1);
vpStmt.close();
DataObject.updateData(" UPDATE EOD_Trn_BatchProcess SET iJobNo = ?, szParameters = ? WHERE iProcessSeqNo = ? ", conn, new String[]{strJobNo, strParameters, (String)mapParameters.get("__PROCESS_SEQ_NO")});
conn.commit();
here first commit is required because i want to save parameters used during call, even if job submission fails.(or any thing ahead.)
The reason for the exception is that you can not mannaully call commit()/rollback under a global transaction,you can only marked it for rollback.You have three options:
Throw an exception,which depends on the ejb-jar.xml/weblogic-ejb-jar.xml,the default is for any RuntimeException the transaction is marked for rollback;
call the EJBContext.setRollbackOnly() method in case of CheckedException or whenever you need;
If none of above happend for all the resouces under the same transaction,it will be commited sooner or later by the transaction manager.
The transaction manager is responsible for commit()/rollback() the transaction for you, so that the it has a chance to co-operate with
different resources(two oralce db for example).You can check the detail by gooble the key word "two phased transaction" or "global transaction",here is what I found:
Global Transaction
As for your question
What is significance of bean managed transaction if i use XA enabled transaction ?
The bean-managed transaction is a "Global transaction" if the transaction-attribute in in ejb-jar.xml enable the transaction propagation. A global transaction need the datasource to be XA enabled,that is the jdbc driver itself is XA kind driver such as oracle.jdbc.xa.client.OracleXADataSource ,or the thin driver oracle.jdbc.OracleDriver with XA enabled (a simulation of two phased transaction,but not real one )
why it is not encountering in every execution ?
I am not sure why,I guess the driver have some mechanism to check whether the ruled is breaked.Or the transaction-attribute is configured to Supports ,so if caller has a transaction context,then your ejb is under global transaction,otherwise not.
I wish my answer to be helpful,good luck!
I've encountered the same problem,and when I set local transaction to auto commit false solved the problem.
Connection.setAutoCommit(false)
I guess you are using a 2-phrase transaction,What if you do not commit in you 1st step?
I have recently been playing with the sqlite4java library.
I think I have it pretty much figured out. The only think that bothers me is that I do not know how to switch off the auto-commit using this library. Can anyone please help?
A code example would be much appreciated.
Thanks in advance,
Boro
Jefromi and king_nak are correct - you just need to issue SQL statements that begin and end a transaction.
SQLiteConnection con = new SQLiteConnection();
con.exec("BEGIN");
// do transaction work - auto-commit is disabled
con.exec("COMMIT");
// auto-commit is enabled again
Edit: I confused sqlite4java with the sqliteJDBC package. So the below code will not help. I'm keeping it for reference nevertheless.
After you obtained the connection, just call setAutoCommit(false)
java.sql.Connection con = DriverManager.getConnection("jdbc:sqlite:/your/db/file", "user", "password");
con.setAutoCommit(false);
Referring to SQLite's C interface description:
The sqlite3_get_autocommit() interface returns non-zero or zero if the given database connection is or is not in autocommit mode, respectively. Autocommit mode is on by default. Autocommit mode is disabled by a BEGIN statement. Autocommit mode is re-enabled by a COMMIT or ROLLBACK.
So you disable auto-commit by a BEGIN TRANSACTION statement. No separate API function is present for this