Develop Reference

SSLSocket duplex close failed java.net.SocketException: Socket is closed while trying to connect to DB2 over TLS

I'm trying to understand why this error occurs:
javax.net.ssl|WARNING|01|main|2021-07-04 12:08:30.668 CEST|SSLSocketImpl.java:497|SSLSocket duplex close failed (
"throwable" : {
java.net.SocketException: Socket is closed
at java.base/java.net.Socket.shutdownInput(Socket.java:1538)
at java.base/sun.security.ssl.BaseSSLSocketImpl.shutdownInput(BaseSSLSocketImpl.java:216)
at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:751)
at java.base/sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:701)
at java.base/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:562)
at java.base/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:486)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.close(SSLSocketImpl.java:1034)
at com.ibm.db2.jcc.t4.a0.j(a0.java:343)
at com.ibm.db2.jcc.t4.b.freeTransport_(b.java:5523)
at com.ibm.db2.jcc.t4.a.close_(a.java:455)
at com.ibm.db2.jcc.am.Agent.close(Agent.java:345)
at com.ibm.db2.jcc.t4.b.b(b.java:965)
at com.ibm.db2.jcc.t4.b.a(b.java:804)
at com.ibm.db2.jcc.t4.b.a(b.java:441)
at com.ibm.db2.jcc.t4.b.a(b.java:414)
at com.ibm.db2.jcc.t4.b.<init>(b.java:352)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:233)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:200)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:182)
at com.example.MainApplication.main(MainApplication.java:36)}
)
javax.net.ssl|ALL|01|main|2021-07-04 12:08:30.668 CEST|SSLSocketImpl.java:1217|Closing output stream
Exception in sql: com.ibm.db2.jcc.am.SqlNonTransientConnectionException
DB2 SQL Error: SQLCODE=-20157, SQLSTATE=08004, SQLERRMC=WEBADMIN;QUIESCE DATABASE;;, DRIVER=4.25.13
com.ibm.db2.jcc.am.SqlNonTransientConnectionException: DB2 SQL Error: SQLCODE=-20157, SQLSTATE=08004
The exception is thrown when I'm invoking getConnection() method of the DriverManager:
connection = DriverManager.getConnection(DB_URL, properties);
I'm using Java 11.0.11 from Oracle (non-OpenJDK).

Talk with your DBA team or whoever manages the database - you get this exception because someone (or some job) has put the database into a specific state that is used for maintenance activity.
Normally this is a temporary situation, and the database (or Db2-instance) needs to be brought back to normal mode by an unquiesce action, when the maintenance activity is completed. After the unquiesce action, your connection should complete as normal.
The SQLCODE (-20157) and SQLERRMC ( SQLERRMC=WEBADMIN;QUIESCE DATABASE;) that are in the message tell you the cause of this exception.
Lookup SQL20157N in the docs to get the detailed explanation.

java.net.SocketException: Socket closed: handle connection interruption

I have a weak connection with database server and time consuming query which is sometimes fails with exception:
Caused by: java.sql.SQLException: I/O Error: Socket closed
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2481)
at net.sourceforge.jtds.jdbc.TdsCore.getNextRow(TdsCore.java:805)
at net.sourceforge.jtds.jdbc.JtdsResultSet.next(JtdsResultSet.java:611)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:885)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:731)
at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:477)
at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:146)
at net.sourceforge.jtds.jdbc.TdsData.readData(TdsData.java:901)
at net.sourceforge.jtds.jdbc.TdsCore.tdsRowToken(TdsCore.java:3175)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2433)
How can connection interruption be handled? currently I have to manually re-run the operation until it executes successfully, maybe it could be done at jdbc driver level
ps socketTimeout property doesn't seem to affect this

Are you sure this is because of query/procedure taking long time ? As Socket closed error normally means something went wrong with the network connection itself, basically something that your driver could not control...
In general, I would suggest to move to a Pool based mechanism, which allows greater control on your persistence layer interaction.

Redshift server closes connection after 10 minutes

I have a statement that takes about 20 minutes to run, which is of the form:
create table new_table diststyle key distkey(column1) sortkey(column2)
as (select ....);
When I run it using an SQL IDE or with the psql command line client, the statement executes successfully but when I run it from my Java program, the server closes the connection after 10 minutes with the following exception:
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [create table new_table diststyle key distkey(column1) sortkey(column2) as (select ....);];
SQL state [HY000]; error code [600001]; [Amazon](600001) The server closed the connection.;
nested exception is java.sql.SQLException: [Amazon](600001) The server closed the connection.
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:538) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at com.abc.mypackage.MyClass.myMethod(Myclass.java:123) [classes/:?]
Caused by: java.sql.SQLException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
Caused by: com.amazon.support.exceptions.GeneralException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
I'm using org.apache.commons.dbcp2.BasicDataSource to create connections. I've tried extending the timeout via defaultQueryTimeout, maxConnLifetimeMillis and socketTimeout but to no avail. The server keeps closing the connection after the same 10 minutes.
dataSource = new BasicDataSource();
dataSource.setUsername(dbUser);
dataSource.setPassword(dbPassword);
dataSource.setUrl(dbUrl);
dataSource.setDefaultAutoCommit(true);
dataSource.setTestOnBorrow(true);
dataSource.setTestOnReturn(true);
dataSource.setDriverClassName("com.amazon.redshift.jdbc41.Driver");
dataSource.setDefaultQueryTimeout(7200);
dataSource.setMaxConnLifetimeMillis(7200000);
dataSource.addConnectionProperty("socketTimeout", "7200");
How do I keep the connection alive for longer?
P.S. I do not have any problems establishing connections and running queries that take less than 10 minutes to finish.

You might want to extend your socket timeout.
Current it is 7200ms only:
dataSource.addConnectionProperty("socketTimeout", "7200");

check if the redshift server have a workload management policy that is timing out queries after 10 minutes.
your java code might be setting this policy

You need to set the tcpKeepAlive time to 1 min or less while getting the connection to redshift cluster.
Properties props = new Properties();
props.setProperty("user", user);
props.setProperty("password", password);
props.setProperty("tcpKeepAlive", "true");
props.setProperty("TCPKeepAliveMinutes", "1");
DriverManager.getConnection("jdbc:redshift://"+endpoint+":"
+port+"/"+database, props);

OP here- I was able to make it work by writing wrappers over BasicDataSource and Connection to poll active connection with isValid(int) every few minutes (any frequency more than once-per-10-minutes works). In hindsight, it seems that most timeout-related properties on BasicDataSource apply to connections which are in the pool but are not being used. setDefaultQueryTimeout and tcpKeepAlive + TCPKeepAliveMinutes did not work.
P.S. It has been a while since I resolved this problem and I do not have the code for the wrappers now. Here's a brief description of the wrappers.
WrappedConnection class takes a Connection object (conn) and a TimerTask object (timerTask) in its constructor and implements the Connection interface by simply calling the methods from conn. timerTask calls this.isValid(100) every few minutes as long as the connection is active. WrappedConnection.close stops timerTask and then calls conn.close.
WrappedBasicDataSource implements the DataSource interface, redirecting methods to a BasicDataSource object. BasicDataSourceWrapper.getConnection gets a connection from the aforementioned BasicDataSource and generates a WrappedConnection using the connection and a new TimerTask object.
I might have missed explaining some details but this is the gist of it.

Start Java/Play2.5 REST API without DB

I would like to start my Java/Play! Service even though the Database is not yet reachable. So far, if the database is not available the service stops. Is it possible in Play to try and reconnect every second or so without completely stopping the service and reconnecting to db as soon as connection is available.
Our application.conf currently looks like this:
play.evolutions.enabled=false
//play.db.pool=bonecp
db {
default.driver=org.postgresql.Driver
default.url="jdbc:postgresql://postgresdb.postgres:5432/postcode"
default.username="<username>"
default.password="<strong-password>"
}
play.db {
config = "db"
prototype.hikaricp = {
connectionTimeout = 250 milliseconds
initializationFailFast = false
readOnly = true
}
}
ebean.default = ["models.*"]
Currently I get the following logs:
2017-05-15 13:26:54.525 [error] postcode-play: Connection error:
org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
Ort: Datei: postinit.c, Routine: InitPostgres, Zeile: 779.
Server SQLState: 53300
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2538)
at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:122)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:227)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:431)
at org.postgresql.Driver.connect(Driver.java:247)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
2017-05-15 13:26:54.626 [error] postcode-play: Connection error:
org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
Ort: Datei: postinit.c, Routine: InitPostgres, Zeile: 779.
Server SQLState: 53300
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2538)
at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:122)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:227)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:431)
at org.postgresql.Driver.connect(Driver.java:247)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
.
.
.
Indicating that the server does not stop altogether but tries to connect to db multiple times which is as desired. But if the database becomes available the following logs are written:
2017-05-15 14:08:43.748 [error] postcode-play:
! #7430ed7gf - Internal server error, for (GET) [/?postcode=88682&country=DE] ->
play.api.Configuration$$anon$1: Configuration error[Cannot connect to database [default]]
at play.api.Configuration$.configError(Configuration.scala:155)
at play.api.Configuration.reportError(Configuration.scala:808)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:48)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
at scala.collection.immutable.List.foreach(List.scala:392)
at play.api.db.DefaultDBApi.connect(DefaultDBApi.scala:42)
at play.api.db.DBApiProvider.get$lzycompute(DBModule.scala:72)
at play.api.db.DBApiProvider.get(DBModule.scala:62)
at play.api.db.DBApiProvider.get(DBModule.scala:58)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-4 - Connection is not available, request timed out after 5001ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:138)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:44)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
at scala.collection.immutable.List.foreach(List.scala:392)
at play.api.db.DefaultDBApi.connect(DefaultDBApi.scala:42)
Caused by: org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:265)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:431)
at org.postgresql.Driver.connect(Driver.java:247)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:193)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:430)
Caused by: java.net.ConnectException: Verbindungsaufbau abgelehnt (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:62)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194
)
The server cannot reach out to the database even though it is available now. If I restart the server the database will be found and can be worked with, but the database lways has to be there on startup. How can I fix that, so that the app can be started independently of the database?
If the database becomes unavailable after the application has started it is no problem after the db becomes available again. it is automatically reconnected.
Any help is greatly apreciated!

It seems like you are leaving some connections open to Postgres. These connections must be closed or else Postgres will hit its max_connections limit.
You could increase max_connections but it's not recommended as performance of Postgres will degrade if it's set too high. Also, as you are using HikariCP it wouldn't be a bad idea to take a look at the HikariCP docs, in particular the idleTimeout and the minimumIdle config variables should help you.
If the above doesn't work then you must confirm exactly how many connections are being made to Postgres. Maybe even check the number of sessions you personally are making to Postgres via SSH or through other apps, not just the number of Play Framework connections.

DBCP Connection properties

We are having hard time figuring out the properties defined, minIdle, maxIdle etc.
we are seeing the following error with the following setting
InitialSize=5
maxActive=50
maxIdle=40
maxWait=2000
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1001)
at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1064)
... 36 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:148)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 39 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1134)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

This is resolved. The issue was connection leak from the bad application code. The application code was written to access the underlying native connection to use Oracle Array Types and not managed properly. Changed the code to using SimpleNativeJdbcExtractor
http://static.springsource.org/spring/docs/2.0.x/api/org/springframework/jdbc/support/nativejdbc/SimpleNativeJdbcExtractor.html