Develop Reference

SQL Server JDB connection issue with Java 8 and Tomcat 7.0.53 in Azure Web App

I am calling a web service and after frequently I am getting the below exception, how do I resolve it
com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
peer: socket write error\r\n\tat
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1745)\r\n\tat
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1732)\r\n\tat
com.microsoft.sqlserver.jdbc.TDSChannel.write(IOBuffer.java:1842)\r\n\tat
com.microsoft.sqlserver.jdbc.TDSWriter.flush(IOBuffer.java:4161)\r\n\tat
com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(IOBuffer.java:4062)\r\n\tat
com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(IOBuffer.java:3107)\r\n\tat
com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:6700)\r\n\tat
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:424)\r\n\tat
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:372)\r\n\tat
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:6276)\r\n\tat
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1793)\r\n\tat
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:184)\r\n\tat
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:159)\r\n\tat
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:315)\r\n\tat
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)\r\n\tat
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate
I am using Apache Commons BasicDataSource for connection pooling, and below is the code for the same
ds = new BasicDataSource();
ds.setDriverClassName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
ds.setUsername(username);
ds.setPassword(password);
ds.setUrl(database_url);
// the settings below are optional -- dbcp can work with defaults
ds.setMinIdle(10);
ds.setMaxIdle(50);
ds.setMaxActive(100);
ds.setInitialSize(10);

HikariCp pool getting Unknown system variable 'tx_isolation' exception

I have an application where user can choose any type and version of database(relational) provide URL, username, password and create datasource, connect to database, then run SELECT queries. To connect database I am using HikariCp for my connection pool(before I used DBCP2 as connection timeout didn't work I changed to HikariCp). Here is my configuration
HikariConfig config = new HikariConfig();
config.setDriverClassName(StringUtils.trimToEmpty(driver));
config.setJdbcUrl(StringUtils.trimToEmpty(url));
config.setUsername(username);
config.setPassword(password);
config.setMaximumPoolSize(POOL_MAX_ACTIVE);
config.setMinimumIdle(POOL_INITIAL_SIZE);
config.setConnectionTestQuery(validationQuery);
config.setConnectionTimeout(TimeUnit.SECONDS.toMillis(jdbcTimeout));
HikariDataSource dataSource = new HikariDataSource(config);
Connection connection = dataSource.getConnection();
When I am trying to connect MYsql database I am getting the following error:
HikariPool-8 - Starting...
2019-04-10 18:45:49 WARN com.zaxxer.hikari.pool.PoolBase:472 - ReqId:60a79ef60413424e96a4fa73010fc861 - HikariPool-8 - Default transaction isolation level detection failed ((conn=14) Unknown system variable 'tx_isolation').
2019-04-10 18:45:49 ERROR com.zaxxer.hikari.pool.HikariPool:574 - ReqId:60a79ef60413424e96a4fa73010fc861 - HikariPool-8 - Exception during pool initialization.
java.sql.SQLException: (conn=14) Unknown system variable 'tx_isolation'
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:198)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)
What is wrong in the configuration? with DBCP2 it works fine
I am using mariadb-java-client, version: '2.2.1' driver to connect to Mysql database.

CannotCreateTransactionException in Spring Data JPA batch process

I'm trying to process about 10 million records of MySQL DB1, then save them to MySQL DB2.
I use about 50 threads to achieve this. The producer fetch 500 records from
MySQL DB1 one time, then put them into a queue. About 50 consumer will process them and then insert into MySQL DB2.
In most cases, it works fine. But for about 50 thousands records, it fails.
After I analysing the log, I found CannotCreateTransactionException were thrown:
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.TransactionException: JDBC begin transaction failed:
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:450) ~[spring-orm-5.0.9.RELEASE.jar!/:5.0.9.RELEASE]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378) ~[spring-tx-5.0.9.RELEASE.jar!/:5.0.9.RELEASE]
There are some similar questions on this site and I have tried some of them:
retry for 5 time after MyRepository.saveAll() throws exception
use BasicDataSource with a larger maxActive and setValidationQuery, setTestOnBorrow:
BasicDataSource dataSource = new BasicDataSource();
dataSource.setMaxActive(100);
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("***");
dataSource.setUsername("***");
dataSource.setPassword("***");
dataSource.setTestOnBorrow(true);
dataSource.setValidationQuery("SELECT 1");
dataSource.setMaxActive(30);
It still does not work. How can I fix this?

I searched a lot and tried different solutions.
Finally, after changing my config to this, the exception does not show anymore:
#Bean
public DataSource poiDataSource() {
BasicDataSource dataSource = new BasicDataSource();
dataSource.setMaxActive(100);
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("**");
dataSource.setUsername("**");
dataSource.setPassword("**");
dataSource.setTestOnBorrow(true);
dataSource.setValidationQuery("SELECT 1");
dataSource.setMaxActive(30);
dataSource.setRemoveAbandoned(true);
dataSource.setMaxWait(60000);
dataSource.setTimeBetweenEvictionRunsMillis(300000);
dataSource.setMinEvictableIdleTimeMillis(300000);
dataSource.setTestWhileIdle(true);
dataSource.setLogAbandoned(true);
return dataSource;
}

Redshift server closes connection after 10 minutes

I have a statement that takes about 20 minutes to run, which is of the form:
create table new_table diststyle key distkey(column1) sortkey(column2)
as (select ....);
When I run it using an SQL IDE or with the psql command line client, the statement executes successfully but when I run it from my Java program, the server closes the connection after 10 minutes with the following exception:
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [create table new_table diststyle key distkey(column1) sortkey(column2) as (select ....);];
SQL state [HY000]; error code [600001]; [Amazon](600001) The server closed the connection.;
nested exception is java.sql.SQLException: [Amazon](600001) The server closed the connection.
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:538) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at com.abc.mypackage.MyClass.myMethod(Myclass.java:123) [classes/:?]
Caused by: java.sql.SQLException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
Caused by: com.amazon.support.exceptions.GeneralException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
I'm using org.apache.commons.dbcp2.BasicDataSource to create connections. I've tried extending the timeout via defaultQueryTimeout, maxConnLifetimeMillis and socketTimeout but to no avail. The server keeps closing the connection after the same 10 minutes.
dataSource = new BasicDataSource();
dataSource.setUsername(dbUser);
dataSource.setPassword(dbPassword);
dataSource.setUrl(dbUrl);
dataSource.setDefaultAutoCommit(true);
dataSource.setTestOnBorrow(true);
dataSource.setTestOnReturn(true);
dataSource.setDriverClassName("com.amazon.redshift.jdbc41.Driver");
dataSource.setDefaultQueryTimeout(7200);
dataSource.setMaxConnLifetimeMillis(7200000);
dataSource.addConnectionProperty("socketTimeout", "7200");
How do I keep the connection alive for longer?
P.S. I do not have any problems establishing connections and running queries that take less than 10 minutes to finish.

You might want to extend your socket timeout.
Current it is 7200ms only:
dataSource.addConnectionProperty("socketTimeout", "7200");

check if the redshift server have a workload management policy that is timing out queries after 10 minutes.
your java code might be setting this policy

You need to set the tcpKeepAlive time to 1 min or less while getting the connection to redshift cluster.
Properties props = new Properties();
props.setProperty("user", user);
props.setProperty("password", password);
props.setProperty("tcpKeepAlive", "true");
props.setProperty("TCPKeepAliveMinutes", "1");
DriverManager.getConnection("jdbc:redshift://"+endpoint+":"
+port+"/"+database, props);

OP here- I was able to make it work by writing wrappers over BasicDataSource and Connection to poll active connection with isValid(int) every few minutes (any frequency more than once-per-10-minutes works). In hindsight, it seems that most timeout-related properties on BasicDataSource apply to connections which are in the pool but are not being used. setDefaultQueryTimeout and tcpKeepAlive + TCPKeepAliveMinutes did not work.
P.S. It has been a while since I resolved this problem and I do not have the code for the wrappers now. Here's a brief description of the wrappers.
WrappedConnection class takes a Connection object (conn) and a TimerTask object (timerTask) in its constructor and implements the Connection interface by simply calling the methods from conn. timerTask calls this.isValid(100) every few minutes as long as the connection is active. WrappedConnection.close stops timerTask and then calls conn.close.
WrappedBasicDataSource implements the DataSource interface, redirecting methods to a BasicDataSource object. BasicDataSourceWrapper.getConnection gets a connection from the aforementioned BasicDataSource and generates a WrappedConnection using the connection and a new TimerTask object.
I might have missed explaining some details but this is the gist of it.

Connecting to as400 JDBC on Play Framework

I'm having issues making a connection to an AS400 database inside of Play!.
My application.conf looks like:
db.default.driver="com.ibm.as400.access.AS400JDBCDriver"
db.default.url="jdbc:as400://SERVER;libraries=A,B,C;toolbox trace=all;trace=true"
db.default.username="user"
db.default.password="password"
I've set up jt400 in the classpath, and I can see under "external libraries" that it shows up and is available. But essentially I get an error message about failing to connect (on user/password I know works) and failure to execute isValid(), which is a function that can not be found inside of AS400JDBCConnection class.
[error] c.z.h.p.PoolBase - HikariPool-1 - Failed to execute isValid() for connection, configure connection test query. (com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z)
[error] application -
! #72265nf0a - Internal server error, for (GET) [/] ->
play.api.Configuration$$anon$1: Configuration error[Cannot connect to database [default]]
at play.api.Configuration$.configError(Configuration.scala:154)
at play.api.Configuration.reportError(Configuration.scala:806)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:48)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
at scala.collection.immutable.List.foreach(List.scala:381)
at play.api.db.DefaultDBApi.connect(DefaultDBApi.scala:42)
at play.api.db.DBApiProvider.get$lzycompute(DBModule.scala:72)
at play.api.db.DBApiProvider.get(DBModule.scala:62)
at play.api.db.DBApiProvider.get(DBModule.scala:58)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
Caused by: play.api.Configuration$$anon$1: Configuration error[Failed to initialize pool: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z]
at play.api.Configuration$.configError(Configuration.scala:154)
at play.api.PlayConfig.reportError(Configuration.scala:996)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:70)
at play.api.db.PooledDatabase.createDataSource(Databases.scala:199)
at play.api.db.DefaultDatabase.dataSource$lzycompute(Databases.scala:123)
at play.api.db.DefaultDatabase.dataSource(Databases.scala:121)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:138)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:44)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:512)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:58)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:54)
at play.api.db.PooledDatabase.createDataSource(Databases.scala:199)
at play.api.db.DefaultDatabase.dataSource$lzycompute(Databases.scala:123)
at play.api.db.DefaultDatabase.dataSource(Databases.scala:121)
Caused by: java.lang.AbstractMethodError: com.ibm.as400.access.AS400JDBCConnection.isValid(I)Z
at com.zaxxer.hikari.pool.PoolBase.checkDriverSupport(PoolBase.java:400)
at com.zaxxer.hikari.pool.PoolBase.setupConnection(PoolBase.java:375)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:346)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:506)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:58)
at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:54)
I'm able to connect in other java-based projects using something like:
try {
Class.forName("com.ibm.as400.access.AS400JDBCDriver");
Connection con = DriverManager.getConnection("jdbc:as400://" +
ApplicationAuthentication.server + "/" +
ApplicationAuthentication.library,
ApplicationAuthentication.user,
ApplicationAuthentication.password
);
} catch (Exception e) {
System.err.println(e);
throw new WebApplicationException(genericError, Response.Status.UNAUTHORIZED);
}

Guessing from the stacktrace, it appears that the connection returned from your driver is not playing well with the connection Hikari Connection Pool. Hikari is default connection pool in playframework.
Specifically, your exception trace shows that the Hikari CP is attempting to call isValid method on the connection object returned by your JDBC driver and then failing with java.lang.AbstractMethodError.
You can try switching to BoneCP connection pool and see if it helps. You can also check comments on this issue on hikari github issue list

Try adding the following to application.config
db.default.hikaricp.connectionTestQuery="SELECT 1"
Not tested in Play Framework but I had similar issue on spring framework and solved in that way.

Use liquibase datasource like below with connection-test-query
#Bean
#LiquibaseDataSource
public DataSource liquibaseDataSource() {
HikariDataSource dataSource = (HikariDataSource) DataSourceBuilder.create().url("url")
.username("username")
.password("password")
.type(HikariDataSource.class).build();
dataSource.setConnectionTestQuery("select 1 from sysibm.sysdummy1");
return dataSource;
}