We are facing one strange issue with sql server 2008r2 and hibernate. Frequently getting read timeout error from the database, here is the error details:
10:02:11,368 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-/55.2.0.180:443-36) SQL Error: 0, SQLState: 08S01
10:02:11,368 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-/55.2.0.180:443-36) Read timed out
10:02:11,368 ERROR [stderr] (http-/55.2.0.180:443-36) org.hibernate.exception.JDBCConnectionException: Read timed out10:02:11,368 ERROR [stderr] (http-/55.2.0.180:443-36) at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:131)10:02:11,368 ERROR [stderr] (http-/55.2.0.180:443-36)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)10:02:11,368 ERROR [stderr] (http-/55.2.0.180:443-36)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)10:02:11,368 ERROR [stderr] (http-/55.2.0.180:443-36)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)10:02:11,384 ERROR [stderr] (http-/55.2.0.180:443-36)
at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:129)10:02:11,384 ERROR [stderr] (http-/55.2.0.180:443-36)
... 131 more10:02:11,727 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-/55.2.0.180:443-36) SQL Error: 0, SQLState: null
10:02:11,727 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-/55.2.0.180:443-36) The connection is closed.
10:02:11,727 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/CastrolCubedUnit].[spring]] (http-/55.2.0.180:443-36) JBWEB000236: Servlet.service() for servlet spring threw exception: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
Few More information if its related:
We are using C3P0 connection pooling with hibernate. And configured
connection testing as well. We have checked the DB with(in the time
of exception occurring) exec sp_lock command DB is only holding S or
IS lock
We are not getting this in local environment in local its running
fine
Application is hosted in Amazon aws with two application server and
one load balancer where the issue persist.
We are stuck into this help is much appreciated.
MS SQl server keeps statistical information about all queries. table sys.dm_exec_query_stats having the statistics about each query but before that I suggest you to run DBCC command which restarts logging of the query running time and some cache clear work is done.
DNDC Command DBCC FREEPROCCACHE
And following is the query to find long query
SELECT DISTINCT TOP 10
t.TEXT QueryName,
s.execution_count AS ExecutionCount,
s.max_elapsed_time AS MaxElapsedTime,
ISNULL(s.total_elapsed_time / s.execution_count, 0) AS AvgElapsedTime,
s.creation_time AS LogCreatedOn,
ISNULL(s.execution_count / DATEDIFF(s, s.creation_time, GETDATE()), 0) AS FrequencyPerSec
FROM sys.dm_exec_query_stats s
CROSS APPLY sys.dm_exec_sql_text( s.sql_handle ) t
ORDER BY
s.max_elapsed_time DESC
GO
You can go through This Link for optimizing query.
Related
I have a JDBC connection, using java 1.8. My database is PostgreSQL 9.4 and my application server is jboss-eap 7.0.
Although everything seems to work fine, im getting a recursive warning:
WARN org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory - IJ030027: Destroying connection that is not valid, due to the following exception: org.postgresql.jdbc4.Jdbc4Connection#77eec781
org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
17 Mar 2019 12:56:34,828 - WARN org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory - IJ030027: Destroying connection that is not valid, due to the following exception: org.postgresql.jdbc4.Jdbc4Connection#3ac6ea3d
org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:281)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker.isValidConnection(PostgreSQLValidConnectionChecker.java:61)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:1272)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getInvalidConnections(BaseWrapperManagedConnectionFactory.java:1085)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.validateConnections(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1463)
at org.jboss.jca.core.connectionmanager.pool.validator.ConnectionValidator$ConnectionValidatorRunner.run(ConnectionValidator.java:277)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1700)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
... 11 more
The JDBC driver that im using for postgre is
postgresql-8.3-603.jdbc3.jar
And I have the following reference on my standalone.xml:
<driver name="postgresql-8.3-603.jdbc3.jar" module="com.postgre">
<xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
</driver>
Does anyone know what is happening? Why I get this recursive warning on my server?
Thanks in advance! :)
The backend runs in a docker container. It runs with spring boot & security and works fine. The frontend runs in another container as well. All works fine until out of nowhere this exception is thrown. I only noticed this exception after I idled a few minutes (normaly between 5 and 10 min).
WARN 1 --- [nio-8080-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08S01
ERROR 1 --- [nio-8080-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : I/O Error: Connection reset
ERROR 1 --- [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: could not extract ResultSet; nested exception is org.hibernate.exception.JDBCConnectionException: could not extract ResultSet] with root cause
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209) ~[na:1.8.0_111-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_111-internal]
at java.io.DataInputStream.readFully(DataInputStream.java:195) ~[na:1.8.0_111-internal]
.....
The connection to the Database is available all the time. The weird thing is, that this error is only thrown when I run my project with docker. If I run it with eclipse & node.js the error don't occurs. The only thing I noticed was, that a few minutes after I loged in and idled, I got automaticly loged out when I tried to do something. I don't know if this has any connection with the orgiginal error.
application.properties
spring.datasource.url=databaseURL
spring.datasource.username=SQLusername
spring.datasource.password=Password
spring.datasource.driver-class-name=net.sourceforge.jtds.jdbc.Driver
spring.datasource.initialSize=100
spring.datasource.minIdle=10
spring.jpa.hibernate.naming_strategy=org.hibernate.cfg.EJB3NamingStrategy
spring.jpa.hibernate.naming.implicit-strategy=org.hibernate.boot.model.naming.ImplicitNamingStrategyLegacyJpaImpl
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
spring.datasource.tomcat.max-active=250
spring.datasource.tomcat.max-wait=30000
spring.datasource.test-on-borrow=true
spring.datasource.test-while-idle=true
spring.datasource.minEvictableIdleTimeMillis=21600000
spring.datasource.timeBetweenEvictionRunsMillis=43200000
spring.datasource.connection-test-query=SELECT 1
spring.datasource.validationQuery=SELECT 1
I'm glad for every suggestion that could help to fix this proplem.
I had the same problem months ago!
This happens because the jvm doesn’t have memory enough to accept this request!
This error is intermittent and is very difficult to find your root cause!
Try this:
Put in your Dockerfile some flags to solve it
ENTRYPOINT ["java","-XX:+TieredCompilation","-XX:TieredStopAtLevel=1", "-jar","yourJar.jar"]
More details here: https://jpbempel.github.io/2020/05/22/startup-containers-tieredcompilation.html
Sometimes the problem can be memory. When you start your container with java, the jvm default heap memory is 1/4 of total memory of container.
Then you can use the flag -XX:MaxRAMFraction where your value has a range 1 to 4.
More details here: https://merikan.com/2019/04/jvm-in-a-container/#java-8u131-and-java-9
Good luck!
During my development, server(jboss4.0) was restarted because of some changes in java source and server failed to restart with the below stacktrace.
[JBossManagedConnectionPool] Throwable while attempting to get a new connection: null
org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Io exception: Connection refused(DESCRIPTION=(TMP=)(VSNNUM=186647552)(ERR=12514)(ERROR_STACK=(ERROR=(CODE=12514)(EMFI=4)))))
at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:179)
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:565)
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:250)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:491)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:341)
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:301)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:396)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88)
at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:69)
at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:72)
at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:1881)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1174)
Caused by: java.sql.SQLException: Io exception: Connection refused(DESCRIPTION=(TMP=)(VSNNUM=186647552)(ERR=12514)(ERROR_STACK=(ERROR=(CODE=12514)(EMFI=4))))
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:179)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:333)
at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:404)
at oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:468)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:314)
at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:171)
... 150 more
17:51:19,271 WARN [SettingsFactory] Could not obtain connection metadata
org.jboss.util.NestedSQLException: Could not create connection; - nested throwable: (java.sql.SQLException: Io exception: Connection refused(DESCRIPTION=(TMP=)(VSNNUM=
I have even successfully tested connection with a standalone java program and with ping. Also it is connecting from my oracle sql developer. I'm using oracle 9i.
What could be the reason as its suddenly getting into this error?
We have a stand-alone single user desktop application that has a Swing front-end and uses an embedded Derby database (10.8.2.2 1181258). Spring (3.0.4) is used as the go between the application and the database. A customer (in Hebrew/Israel locale iw_IL) running the application using JRE 1.7.0_21 ran into an issue. The following stack trace was logged:
ERROR [index-stat-thread] stderr - Exception in thread "index-stat-thread"
ERROR [index-stat-thread] stderr - java.lang.ClassCastException: org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeScan.positionAtNextPage(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeForwardScan.fetchRows(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeScan.fetchNextGroup(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl$KeyComparator.fetchRows(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.updateIndexStatsMinion(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.generateStatistics(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.processingLoop(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.run(Unknown Source)
ERROR [index-stat-thread] stderr - at java.lang.Thread.run(Thread.java:722)
Any idea what might cause such an error?
Subsequently, Spring logged the following:
org.springframework.jdbc.support.SQLErrorCodesFactory - Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: No current connection.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:293)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:320)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:212)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:141)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:104)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:636)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:665)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:673)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:796)
at org.springframework.jdbc.core.simple.SimpleJdbcTemplate.queryForList(SimpleJdbcTemplate.java:235)
Given that the application uses an embedded Derby database on the same machine and file system as the application, the JDBC connection error is baffling. Could the first Derby error be the reason for this?
Following this is another error from Spring while loading entries for one table, which essentially seems to be the same as the first error from Derby:
ERROR [main] org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [select * from XYZ where scresult_id=?]; SQL state [XJ001]; error code [0]; Java exception:
'org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow: java.lang.ClassCastException'.; nested exception is java.sql.SQLException: Java exception:
'org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow: java.lang.ClassCastException'.
The same application has been used successfully by other customers for several months. This is the first time a customer has run into these problems.
It looks like your database has become corrupt. Those are internal messages that you should never see.
Here is information about how to use your backups and logs to recover your database: http://db.apache.org/derby/docs/10.10/adminguide/cadminhubbkup98797.html
Before you restore the database, you should retain a copy of the damaged database, the derby.log file, and any other information about the circumstances under which the database became corrupt, as they may give you important clues about how to prevent this problem in the future.
In general, it should not be possible to corrupt a Derby database like this; hence arriving at such a situation might indicate a bug in Derby itself. But there could be other possible explanations, such as damaged or misconfigured hardware.
I assume from your initial question that you do not know how to reproduce this problem? In that case, preserving as much information as you can about the incident, and about what was occurring at the moment that the corruption occurred (did the power fail? Was the machine hard-rebooted? did a disk become full? was a virus-checker running on the machine? There are a million possible questions like this to ask...) may give you clues about how to reproduce the problem yourself.
we are getting error when restarting the app server.
generally when we restart the app server it will create connections itself.
but it was throwing below error.
2013-11-05 04:06:00,029 WARN [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (pool-11-thread-1) Throwable while attempting to get a new connection: null
org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: null, message from server: "Host 'jade2.ftdi.com' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'")
Observed below log also:
2013-11-05 04:07:18,074 INFO [STDOUT] (pool-11-thread-1) DEBUG 2013-11-05 04:07:18,074 [pool-11-thread-1] com.ftd.core.datasource.model.DataSourceHealthMonitor: doDBServerMaintenance path to db server maintenance files jboss.server.data.dir
2013-11-05 04:07:18,074 ERROR [STDERR] (Thread-12) java.lang.OutOfMemoryError: Java heap space
2013-11-05 04:07:20,844 INFO [STDOUT] (http-0.0.0.0-8713-2) ERROR 2013-11-05 04:07:16,673 [http-0.0.0.0-8713-2] com.ftd.core.datasource.exceptionhandlers.NormalExecutionHandler: Caught TransactionException.org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.ftd.core.datasource.exceptions.ConnectionException: nested exception is com.ftd.core.datasource.exceptions.DataSourceException: All Databases are down!!!
After this error Our DBA'S issued flush hosts; in DB side but no luck still we are getting same error.how fix it
java.lang.OutOfMemoryError: Java heap space
Maybe you should try to increase the heap size of VM