Context
Spring Batch Application & Postgres are hosted on the same server.
Spring Batch Step is running in a multi-threaded manner executing SELECT queries.
The Spring Batch Step executes correctly for awhile, then randomly hits the below error while executing a query against Postgres.
Error
2021-04-29 20:57:31.974 [SimpleAsyncTaskExecutor-1] ERROR org.springframework.batch.core.step.AbstractStep - Encountered an error executing step ECIP_TN in job ECIP_tom_JOB
com.lit.tom.core.exception.ApplicationException: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection to 127.0.0.1:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at com.lit.tom.core.driver.retriever.sql.SqlRetriever.getData(SqlRetriever.java:98) ~[tom-engine-1.2.11-SNAPSHOT.jar!/:1.2.11-SNAPSHOT]
at com.lit.tom.core.driver.retriever.LegacyDataRetriever.retrieveLegacyRecords(LegacyDataRetriever.java:48) ~[tom-engine-1.2.11-SNAPSHOT.jar!/:1.2.11-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection to 127.0.0.1:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:371) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:452) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:462) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:490) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at com.lit.tom.core.driver.retriever.sql.SqlRetriever.getData(SqlRetriever.java:59) ~[tom-engine-1.2.11-SNAPSHOT.jar!/:1.2.11-SNAPSHOT]
... 78 more
Caused by: org.postgresql.util.PSQLException: Connection to 127.0.0.1:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:303) ~[postgresql-42.2.19.jar!/:42.2.19]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.2.19.jar!/:42.2.19]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223) ~[postgresql-42.2.19.jar!/:42.2.19]
at org.postgresql.Driver.makeConnection(Driver.java:465) ~[postgresql-42.2.19.jar!/:42.2.19]
at org.postgresql.Driver.connect(Driver.java:264) ~[postgresql-42.2.19.jar!/:42.2.19]
... 78 more
Caused by: java.net.ConnectException: Cannot assign requested address (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_292]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_292]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_292]
Investigative Work
Postgres
Postgres is running, and accepting data on that port.
#Relevant postgres configuration
#listen_addresses = 'localhost'
max_connections = 100
port = 5432
Java
The error is reached faster with more threads.
8, 6, 4, and 2 threads have failed.
Single threaded runs fine.
I'm lost on what to look at, any help would be appreciated.
I discovered the problem, and it was in line with the comments on this post.
The max number of connections was being reached.
The error was coming from a child project, and they had changed the DataSource from HikariCP to the DriverManagerDataSource.
Relevant documentation snippet:
NOTE: This class is not an actual connection pool; it does not actually pool Connections. It just serves as simple replacement for a full-blown connection pool, implementing the same standard interface, but creating new Connections on every call.
The solution was to change back to HikariCP.
Related
I wish to use Hikari Connection Pool in my Spring application to connect to the Oracle Database
Below is the connection string
JDBC URL:
jdbc:oracle:thin:#ldaps://abc-def-dmz.demo.com:876/SOM_APP,cn=oraclecontext,dc=demo,dc=com
Getting the following error
Error Message: Failed to initialize pool: IO Error: JNDI Package
failure javax.naming.CommunicationException: localhost:876[Root
exception is java.net.ConnectException: Connection refused: connect]
How do I fix this?
java.net.ConnectException: Connection refused: connect
The error implies that your connection is being blocked, most likely by a firewall. Check with your DBA and system or network administrators to make sure the network path to the server is open, and that you are using the right port numbers.
Can't connect to postgres from scala slick app, jdbc connection on VPS.
But I can connect
from local computer to VPS using dbeaver,
and I can connect using psql localhost from inside VPS,
and I can connect to VPS db launching app on my home computer.
So looks like everything working except app on VPS. Port is open, configs are same, Java version 1.8 on home computer and vps, connection string not empty and looks like
jdbc:postgresql://localhost:{custom port}/{db_name}
Error log:
DEBUG c.zaxxer.hikari.pool.HikariPool - database.whale - Cannot acquire connection from data source
org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:211)
at org.postgresql.Driver.makeConnection(Driver.java:458)
at org.postgresql.Driver.connect(Driver.java:260)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:136)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:706)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:692)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.postgresql.core.PGStream.<init>(PGStream.java:75)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
Actually restart the VPS make all working.
I tried to restart firewalls(ufw, iptables etc), postgres, apps, another components, nothing happend. But restart full VPS save the day.
I think it was like a bug in OS, but don't know what kind of...
I wrote a multi-server chat system based on Java under Windows. At the security part, I created one keystore to create the SSLSocket. When I launch 3 servers, it works on Windows(Win10 14393.321) but fails on OS X(Version 10.12 (16A323)) and Linux(Ubuntu 14.04.4 LTS). It really confused me. Here is the keystore part:
System.setProperty("javax.net.ssl.keyStore",keyFilepath);
System.setProperty("javax.net.ssl.trustStore",keyFilepath);
System.setProperty("javax.net.ssl.keyStorePassword","password");
System.setProperty("javax.net.ssl.trustStorePassword", "password");
And when I run the third server on OS X or Linux, it shows:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:589) at
sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at
sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:427) at
sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
at server.AuthorizeServer.MessageReceive(AuthorizeServer.java:99) at
server.AuthorizeServer.main(AuthorizeServer.java:64) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
This is my first time asking on StackOverflow and I really looking forward to your kind help.
Thanks!
java.net.ConnectException: Connection refused
Connection refused is an error message from the TCP stack and means that it could not connect with TCP to the other side. Since SSL/TLS is a layer on top of TCP and is only started once the TCP connect succeeded it means that the problem is not caused by different behavior at the SSL/TLS layer.
That this is not cause by the SSL layer but the TCP layer can also be seen by the stacktrace: Connection refused at java.net.PlainSocketImpl.socketConnect
More likely is that there is something blocking the TCP connection (firewall) or that you've tried to listen/connect to the wrong IP address (e.g. trying to reach a server listening on 127.0.0.1 on Windows from the Linux system). But is impossible to say from the currently provided information what exactly is the case.
I have a java servlet application running on Tomcat8 over Windows Server 2012 r2.
Oracle 11g r2 is serving on different machine with same subnet, so no firewall exist between java application and oracle.
The java application is multi-threaded and has a haavy load on peak times.
my question is; when everything is alright and working fine below exception occurs.
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor189.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.mulberry.request.MulberryRequestProcessor.run(MulberryRequestProcessor.java:168)
at com.mulberry.request.MulberryRequestProcessor.lambda$run$0(MulberryRequestProcessor.java:78)
at com.mulberry.request.MulberryRequestProcessor$$Lambda$32/461612128.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:743)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:277)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:181)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:699)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:633)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:186)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127)
at com.mulberry.data.connection.ConnectionManager.getConnection(ConnectionManager.java:86)
at com.mulberry.data.MulberryOrm.getConnection(MulberryOrm.java:103)
at com.mulberry.data.MulberryOrm.insertMulti(MulberryOrm.java:201)
at com.mulberry.data.MulberryOrm.upsertMulti(MulberryOrm.java:192)
at com.aril.profile.service.impl.LprBulkdataService.saveLoadProfileData(LprBulkdataService.java:86)
... 11 more
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:506)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:595)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:230)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1452)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:496)
... 25 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:161)
at oracle.net.nt.ConnOption.connect(ConnOption.java:159)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:428)
... 30 more
When this exception occured on oracle site;
dedicated user count : ~5200
(Also more than 1 tomcat instace is running with same purpose)
oracle parameter sessions is setted 12000
oracle running machine is also windows server 2012 with 64 cpu and 96GB ram
I am using tomcat jdbc connection pool and applications getting connections under tomcat pool.
Connection string is
jdbc:oracle:thin:#[IPADDRESS]:1521/[SERVICENAME]
Ethernet between servers are 10gbit so dont think about network delay.
Can anyone help me to detect my problem ?
you can set timeout ;
Properties props = new Properties();
props.setProperty("user", "dbuser");
props.setProperty("password", "dbpassword");
props.setProperty(OracleConnection.CONNECTION_PROPERTY_THIN_NET_CONNECT_TIMEOUT, "2000");
Connection con = DriverManager.getConnection("<JDBC connection string>", props);
When I try to connect to cassandra within my java application I receive the exception at the end of this message. I am using pelops library to access database. Database is working on linux and my development environment is on windos 7. Intresting line is
WARNING: 10.0.0.7 NodeContext killing all pooled connections for session 44
10.0.0.7 is the IP of the cassandra node. And I can connect this node with cassandra-cli. After a few hours of googling I decided to ask, what i am doing wrong?
Complete stack trace is below.
SEVERE: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.wyki.cassandra.pelops.ThriftPool$Connection.open(ThriftPool.java:329)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext.createConnection(ThriftPool.java:438)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext.access$5(ThriftPool.java:429)
at org.wyki.cassandra.pelops.ThriftPool$NodeContext$1.run(ThriftPool.java:494)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 7 more
WARNING: 10.0.0.7 NodeContext killing all pooled connections for session 44
"Connection refused" means that Cassandra isn't listening on the interface you're connecting to. By default Cassandra listens on port 9160 on localhost. You will need to change that if you want to connect externally; read the comments for ThriftAddress in the configuration file.
Pelops isn't doing anything remotely tricky when it comes to making connections to Cassandra so it seems very unlikely it's Pelops specific.
I've just had a quick look at the Hector connection code and it looks pretty much the same:
https://github.com/rantav/hector/blob/master/core/src/main/java/me/prettyprint/cassandra/connection/HThriftClient.java
The cassandra-cli code looks pretty much the same as well:
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cli/CliMain.java#L57
p.s. Cassandra should not be configured to listen on 0.0.0.0. See the "Why can't I make Cassandra listen on 0.0.0.0 (all my addresses)?" on the Cassandra FAQ.
It seems that this is a problem with pelops, which is a library on top of cassandra's thrift client. It has some problems for connection to Cassandra working on linux and client on a windows machine.
I've got the same exception connecting with "any" client to cassandra remotely:
ConnectException: Connection refused
Looking through SO I couldn't find a quick solution for that, just reading carefully through cassandra docs and comments within cassandra.yaml - they are very handy!
So for me worked following:
change rpc_address value to the hostname or ipaddress you are using to connect to the cassandra instance with the client