I am using hibernate 3.6.7 to do mapping. I set the connection part as static.
I need the service program runs forever, and another service will call some methods of this service to query database.
When I leave the service running, the first day it works well but when I call it the next day, it gives:
org.hibernate.exception.GenericJDBCException: could not execute query
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2545)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:459)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:365)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
at com.myCompany.Query.myMethod(Query.java:99)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.ebayopensource.ginger.server.internal.container.MessageProcessor.processRequest(MessageProcessor.java:93)
at org.ebayopensource.ginger.server.container.GingerServletShell.service(GingerServletShell.java:80)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.SQLException: Closed Connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:208)
at oracle.jdbc.driver.PhysicalConnection.prepareStatement(PhysicalConnection.java:864)
at oracle.jdbc.driver.PhysicalConnection.prepareStatement(PhysicalConnection.java:783)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
at org.hibernate.loader.Loader.doQuery(Loader.java:801)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2542)
It seems the connection gets closed.
Could anyone please give me some suggestion?
Thank you very much:)
Sounds like either the database has closed the connection or some network device has terminated the socket. There are many ways you can work around this problem:
You can issue some sort of "keep alive" type of query (ex: SELECT 1) on the connection every so often to keep it alive. This assumes that it got closed because it was idle.
You can re-open the connection every so often.
If you get a connect closed exception then you can just reopen the connection. Duh.
You can use a connection pool which can do the keep-alive and the reconnection for you. Apache's DBCP is a favorite of many.
I recommend the last one. You would use DBCP something like:
BasicDataSource ds = new BasicDataSource();
ds.setDriverClassName("oracle.jdbc.driver.OracleDriver");
ds.setUsername("scott");
ds.setPassword("tiger");
ds.setValidationQuery("SELECT 1"); // this is database specific
ds.setTestWhileIdle(true); // test the connections every so often
ds.setUrl(connectURI);
...
while (!shutdown) {
Connection conn = dataSource.getConnection();
Statement stmt = conn.createStatement();
...
stmt.close();
// this returns the connection back to the pool instead of really closing
// the connection
conn.close();
}
So instead of passing around a Connection you pass around the BasicDataSource and call getConnection() when you need the connection and conn.close() when you are done with it. The pool does all of the validation work, etc. It also will allow multiple threads to use the database.
Hope this helps.
Edit: You also cannot hold a hibernate session open for a long time. To show you how short lived the session is supposed to be, I'll quote from the docs:
The lifecycle of a Session is bounded by the beginning and end of a logical transaction. (Long transactions might span several database transactions.)
A session holds a database connection so you never want to keep it around for any length of time. You should do a couple of database requests (like a single web request with queries and updates) and then close the session so the underlying database connection can be returned to the pool.
That maybe the structure of the JAVA Model is changed,but the DB table update incorrect! what make the Model and the DB table are not correspondence completely.
You can do like this:
drop the old table
restart the project to rebuild the table
Recently I encountered this error in SpringBoot project.
Reason: Entity class and their respective data in Table had mismatched types. In my case one of the fields which was LocalDate type had value "0000-00-00".
Solution:
In case you are building a new project, drop the old table and restart the project to rebuild the table.
In case your project is old and you do not want to let go of your data, inspect the database table and fix the corresponding value manually.
In my case, I was getting exactly same error when using native Update query.
I resolved this error by using #Modifying (pkg: org.springframework.data.jpa.repository.Modifying) as well as #Transactional annotation (pkg: javax.transaction.Transactional) in repository method.
If we are using Insert, Update, Delete query then #Modifying is required in addition to #Transactional.
#Modifying
#Transactional
#Query(value="Update TBL_EMPLOYEE SET STATUS = :status where ID = :id", nativeQuery = true)
void updateEmployee(#Param("status") String status, #Param("id") String id)
Related
My issue is I am handle ehcache by my self not using hibernate.
eg. Whenever I required any Object First I get from cache if not found then get from db and put into cache.so, next time It is available in cache.
issue : when two thread try to get that object at that time first of all both have not found object in cache. first both get from db and then both try to put object in cache.At that time second thread gave error of No vote.
can you please suggest any best way to manage ehcache manually. If you not getting my question feel free to contact.
Error
org.springframework.transaction.UnexpectedRollbackException: JTA
transaction unexpectedly rolled back (maybe due to a timeout); nested
exception is javax.transaction.RollbackException: Prepare: NO vote
at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1026)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:518)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:292)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
This exception means that when a transaction is requested to commit but has already been rolled back due to a timeout.You can increase the transaction timeout to resolve this.
I have a statement that takes about 20 minutes to run, which is of the form:
create table new_table diststyle key distkey(column1) sortkey(column2)
as (select ....);
When I run it using an SQL IDE or with the psql command line client, the statement executes successfully but when I run it from my Java program, the server closes the connection after 10 minutes with the following exception:
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [create table new_table diststyle key distkey(column1) sortkey(column2) as (select ....);];
SQL state [HY000]; error code [600001]; [Amazon](600001) The server closed the connection.;
nested exception is java.sql.SQLException: [Amazon](600001) The server closed the connection.
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:538) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at com.abc.mypackage.MyClass.myMethod(Myclass.java:123) [classes/:?]
Caused by: java.sql.SQLException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
Caused by: com.amazon.support.exceptions.GeneralException: [Amazon](600001) The server closed the connection.
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source) ~[?:?]
I'm using org.apache.commons.dbcp2.BasicDataSource to create connections. I've tried extending the timeout via defaultQueryTimeout, maxConnLifetimeMillis and socketTimeout but to no avail. The server keeps closing the connection after the same 10 minutes.
dataSource = new BasicDataSource();
dataSource.setUsername(dbUser);
dataSource.setPassword(dbPassword);
dataSource.setUrl(dbUrl);
dataSource.setDefaultAutoCommit(true);
dataSource.setTestOnBorrow(true);
dataSource.setTestOnReturn(true);
dataSource.setDriverClassName("com.amazon.redshift.jdbc41.Driver");
dataSource.setDefaultQueryTimeout(7200);
dataSource.setMaxConnLifetimeMillis(7200000);
dataSource.addConnectionProperty("socketTimeout", "7200");
How do I keep the connection alive for longer?
P.S. I do not have any problems establishing connections and running queries that take less than 10 minutes to finish.
You might want to extend your socket timeout.
Current it is 7200ms only:
dataSource.addConnectionProperty("socketTimeout", "7200");
check if the redshift server have a workload management policy that is timing out queries after 10 minutes.
your java code might be setting this policy
You need to set the tcpKeepAlive time to 1 min or less while getting the connection to redshift cluster.
Properties props = new Properties();
props.setProperty("user", user);
props.setProperty("password", password);
props.setProperty("tcpKeepAlive", "true");
props.setProperty("TCPKeepAliveMinutes", "1");
DriverManager.getConnection("jdbc:redshift://"+endpoint+":"
+port+"/"+database, props);
OP here- I was able to make it work by writing wrappers over BasicDataSource and Connection to poll active connection with isValid(int) every few minutes (any frequency more than once-per-10-minutes works). In hindsight, it seems that most timeout-related properties on BasicDataSource apply to connections which are in the pool but are not being used. setDefaultQueryTimeout and tcpKeepAlive + TCPKeepAliveMinutes did not work.
P.S. It has been a while since I resolved this problem and I do not have the code for the wrappers now. Here's a brief description of the wrappers.
WrappedConnection class takes a Connection object (conn) and a TimerTask object (timerTask) in its constructor and implements the Connection interface by simply calling the methods from conn. timerTask calls this.isValid(100) every few minutes as long as the connection is active. WrappedConnection.close stops timerTask and then calls conn.close.
WrappedBasicDataSource implements the DataSource interface, redirecting methods to a BasicDataSource object. BasicDataSourceWrapper.getConnection gets a connection from the aforementioned BasicDataSource and generates a WrappedConnection using the connection and a new TimerTask object.
I might have missed explaining some details but this is the gist of it.
here an error which drives me crazy, which seems to have tons of related entries but non solution worked so far. Maybe there is something special about my configuration I haven't found so far...
Im running a Java application, using Hibernation (5.0.3.Final) connected an Azure SQL DB (v12) via JNDI ressources (tomcat 8.0.28).
I am receiving, as it looks randomly, Connection reset by peer: socket write error's followed by The connection is closed errors. The app does not recover until restart...
This happens despite the fact that i have configured a validationQuery and also set testOnCreate, testOnBorrow and testWhileIdle to true.
I'll give you the whole configuration below. This really drives me crazy as this seems to be a bug which is well known but not solved since years?!
Here's the configuration:
server.xml:
<Resource name="jdbc/booking"
global="jdbc/booking"
auth="Container"
username="XXX"
password="YYY"
type="javax.sql.XADataSource"
driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"
url="jdbc:sqlserver://mydatabase.database.windows.net:1433;database=booking;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
maxWait="10000"
validationQuery="SELECT 1"
testOnCreate="true"
testOnBorrow="true"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="60000"
validationInterval="1000"
removeAbandonedOnMaintenance="true"
fastFailValidation="true"/>
context.xml (in web/META-INF/ of the war)
<ResourceLink name="jdbc/booking" global="jdbc/booking"/>
hibernate.cfg
<property name="connection.driver_class">com.microsoft.sqlserver.jdbc.SQLServerDriver</property>
<property name="connection.datasource">java:comp/env/jdbc/booking</property>
I'm setting up the Hibernate session factory like this:
private void setUp() {
final StandardServiceRegistry registry = new StandardServiceRegistryBuilder().configure( AzureDB.configFile ).build();
try {
this.sessionFactory = new MetadataSources( registry ).buildMetadata().buildSessionFactory();
} catch ( final Exception e ) {
System.err.println( e.getMessage() );
StandardServiceRegistryBuilder.destroy( registry );
System.exit( 666 );
}
}
and use it via injection in my services like this:
#Inject private AzureDB bookingDB;
...
this.bookingDB.getCurrentSession().beginTransaction();
final Hotel hotel = this.bookingDB.getCurrentSession().get( Hotel.class, hotelId );
One request will finish successfully, the next will run into the Connection rest by peer error. A full stracktrace will look like this:
16-Nov-2015 09:26:59.028 WARN [http-apr-8080-exec-13] org.hibernate.engine.jdbc.spi.SqlExceptionHelper.logExceptions SQL Error: 0, SQLState: 08S01
16-Nov-2015 09:26:59.028 ERROR [http-apr-8080-exec-13] org.hibernate.engine.jdbc.spi.SqlExceptionHelper.logExceptions Connection reset by peer: socket write error
16-Nov-2015 09:26:59.044 INFO [http-apr-8080-exec-13] org.hibernate.event.internal.DefaultLoadEventListener.onLoad HHH000327: Error performing load command : org.hibernate.exception.JDBCConnectionException: could not extract ResultSet
16-Nov-2015 09:26:59.060 SEVERE [http-apr-8080-exec-13] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [com.nekst.service.nekstAPI] in context with path [] threw exception [org.hibernate.exception.JDBCConnectionException: could not extract ResultSet] with root cause
com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by peer: socket write error
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1748)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1732)
at com.microsoft.sqlserver.jdbc.TDSChannel.write(IOBuffer.java:1842)
at com.microsoft.sqlserver.jdbc.TDSWriter.flush(IOBuffer.java:4161)
at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(IOBuffer.java:4062)
at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(IOBuffer.java:3107)
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:6700)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:424)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:372)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:6276)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1793)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:184)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:159)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:284)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.getResultSet(AbstractLoadPlanBasedLoader.java:433)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeQueryStatement(AbstractLoadPlanBasedLoader.java:185)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:120)
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:85)
at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:167)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3954)
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:488)
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:453)
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:196)
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:258)
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:134)
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1071)
at org.hibernate.internal.SessionImpl.access$2600(SessionImpl.java:164)
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2638)
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:955)
at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:338)
at com.sun.proxy.$Proxy138.get(Unknown Source)
at com.nekst.service.rest.Kauai.getLocation(Kauai.java:227)
at com.nekst.service.rest.Kauai.getRecommendation(Kauai.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:217)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at com.googlecode.psiprobe.Tomcat80AgentValve.invoke(Tomcat80AgentValve.java:45)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:673)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2503)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2492)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
followed by com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed. on any succeeding request.
There can be weeks of undisturbed service and then (like currently) I see this errors continuously... Has anybody experience in this concrete setting or any other remarks to my situation? Help is appreciated a lot...
UPDATE: I also tried adding custom registry entries as suggested here: https://msdn.microsoft.com/en-us/library/hh290696(v=SQL.110).aspx, no luck, I can see the keepalives in wireshark, but still getting those errors
Per my experience, the issue was generally caused by using the unsuitable configuration for the JDBC connection pool in JNDI.
There is a offical doc for the Azure SQL Database resource limits https://azure.microsoft.com/en-us/documentation/articles/sql-database-resource-limits/.
I checked your JNDI configuration. So I suggest you can refer to your Azure service tier's limits and the Tomcat doc for JDBC data source settings in JNDI https://tomcat.apache.org/tomcat-7.0-doc/jndi-resources-howto.html#JDBC_Data_Sources to set the configuration properties initialSize, maxActive,minIdle and maxIdle to suitable values like maxIdle=100.
Update:
The initialSize & minIdle default value is 0. The minIdle property control the pool connection minimal count when the pool release the connection with low concurrent access. If the pool connection released till 0, the new access will cause the issue connection reset.
Updated at 2015.11.19:
There is an excerpt explained the issue Connection reset by peer.
[10054] Connection reset by peer
Connection reset by peer is a tough one because it can be caused by so many things. In all cases, the server determines that the socket is no longer good and closes it from its side.
Write Error
Scenario: Mary was trying to talk to Joe but didn't think she was getting through, so she hung rather than lose his messages (data).
A write error occurs when a server cannot successfully write to a user's client. When the server receives information, it usually responds with information of its own. When the server receives an error when writing to a client, it then disconnects the user, resulting in a write error quit message similar to the read error format.
Try to set the Idle timeout value to some large value. Do you observe any pattern as to when this issue occurs? Is there a idle state after which you see this problem? Check - http://www.coderanch.com/t/633566/JDBC/databases/microsoft-sqlserver-jdbc-SQLServerException-Connection
This also explains the same behavior due to timeout - DBCP returns closed connections
Try to turn on the logging for the driver by following the link - https://msdn.microsoft.com/en-us/library/ms378517(v=SQL.110).aspx
Use the finest level logging and check if you can get the details of the problem.
My Application(java spring-core) has several threads running concurrently and accessing db, I am getting exception in some peaktime
07:43:33,400 WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 1213, SQLState: 40001
07:43:33,808 ERROR [org.hibernate.util.JDBCExceptionReporter] Deadlock found when trying to get lock; try restarting transaction
07:43:33,808 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
org.hibernate.exception.LockAcquisitionException: could not insert: [com.xminds.bestfriend.frontend.model.Question]
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:107)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2436)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2856)
at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:79)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:184)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:147)
at com.xminds.bestfriend.consumers.Base.onMessage(Base.java:96)
at org.springframework.jms.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:339)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:535)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2450)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2371)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2355)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2416)
... 25 more
My code looks
try
{
this.consumerTransactionTemplate.execute(new TransactionCallbackWithoutResult(){
#Override
protected void doInTransactionWithoutResult(
TransactionStatus status)
{
process();
}
});
}
catch(Exception e){
logger.error("Exception occured " , e);
//TODO: Exception handling
}
MySQL's InnoDB engine sports row-level locking, which can lead to deadlocks even when your code is inserting or updating a single row (specially if there are several indexes on the table being updated). Your best bet is to design the code around this in order to retry a transaction if it fails due to a deadlock. Some useful info about MySQL deadlock diagnose and possible workarounds is available here.
An interesting implementation of deadlock retry via AOP in Spring is available here. This way you just need to add the annotation to the method you want to retry in case of deadlock.
Emir's answer is great and it describes the problem that you are getting. However I suggest you to try spring-retry.
It's a brilliant framework that implements the retry pattern via annotation(s).
Example:
#Retryable(maxAttempts = 4, backoff = #Backoff(delay = 500))
public void doSomethingWithMysql() {
consumerTransactionTemplate.execute(
new TransactionCallbackWithoutResult(){
#Override
protected void doInTransactionWithoutResult(
TransactionStatus status)
{
process();
}
});
}
In case of exception, it will retry (call) up to 4 times the method doSomethingWithMysql() with a backoff policy of 500ms
If you are using JPA/Hibernate then simple just follow below steps to avoid dead lock. Once you have acquired the lock, don't made any call on db with same id anywhere in the transaction (I mean to say you should not get entity again on sameid), on locking object you modify and save no issues.
service level:-
employee=empDao.getForUpdate(id);
Dao level:-
public employee getForUpdate(String id)
return mySqlRepository.getForUpdate(id)
Repository(interface):-
#Lock(LockModeType.PESSIMITSIC_WRITE)
#Query("select e from employee e where id=?1")
public employee getForUpdate(String id)
Here is an example with plain Spring and no extra frameworks.
TransactionTemplate transactionTemplate = new TransactionTemplate(transactionManager); // autowired
transactionTemplate.setIsolationLevel(TransactionDefinition.ISOLATION_SERIALIZABLE); // this increases deadlocks, but prevents multiple simultaneous similar requests from inserting multiple rows
Object o = transactionTemplate.execute(txStatus -> {
for (int i=0; i<3; i++) {
try {
return findExistingOrCreate(...);
} catch (DeadlockLoserDataAccessException e) {
Logger.info(TAG, "create()", "Deadlock exception when trying to find or create. Retrying "+(2-i)+" more times...");
try { Thread.sleep(2^i*1000); } catch (InterruptedException e2) {}
}
}
return null;
});
if (o == null) throw new ApiException(HttpStatus.SERVICE_UNAVAILABLE, "Possible deadlock or busy database, please try again later.");
Using serializable transaction isolation level is specific to my situation because it converts SELECT to SELECT ... IN SHARE MODE / SELECT ... FOR UPDATE and locks those rows. The findExistingOrCreate() is doing a lot of complicated searching for existing rows and auto-generating names and checking for bad words, etc. When many of the same request came in at the same time, it would create multiple rows. With the serializable transaction isolation level, it is now idempotent; it now locks the rows, creates a single row, and all subsequent requests return the new existing row.
When you face this kind of error "deadlock detected". You should inspect your queries execution and verify if two or more concurrent transactions can cause a deadlock.
These transactions should acquire database locks in the same order in order to avoid deadlock.
This can happen on none-concurrent applications with one thread inserting records consecutively, too. In case a table has a unique constraint MySQL "builds" that constraint after the commit. That locks the table and might disturb the next insert leading to the above mentioned deadlock. Although I only noticed that error on Windows.
Like on all other answers, repeating the insert solved the issue.
With other databases - PostgreSQL, Oracle or H2 - it works without this workaround.
We are using Spring 2.5.6 and the Hibernate that comes with it.
As part of our architecture, we have a service and a manager. The service calls the manager to provide a unit of work. We place an AOP interceptor around the manager to manage the transaction.
In our case, the manager is creating an image and a note that links that image to a person, then returning back to the service. These are brand new functions.
When I run this I get the following stack trace:
2013-07-11 08:06:52,969 3773870 ERROR org.hibernate.event.def.AbstractFlushingEventListener | Could not synchronize database state with session
org.hibernate.exception.SQLGrammarException: Could not execute JDBC batch update
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:90)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:266)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:167)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:365)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:655)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy50.attachImage(Unknown Source)
at com.acs.gs.juror.service.person.impl.PersonServiceImpl$1.retry(PersonServiceImpl.java:557)
at com.acs.gs.juror.service.Service.withRetry(Service.java:362)
at com.acs.gs.juror.service.person.impl.PersonServiceImpl.attachImage(PersonServiceImpl.java:562)
at com.xerox.tclg.juror.servlet.StoreImageServlet.handleRequestInternal(StoreImageServlet.java:73)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:879)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:600)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1703)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected NUMBER got BINARY
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10345)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:230)
at org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
... 41 more
When I searched on the error, I found a lot entries about queries with null values. The thing is that I've already done the inserts and that worked fine. It's throwing this exception during the commit.
Both image and note are existing fields that I've used in the past. I have recently put a link on note to image so that a note may reference an image.
Just because of the error message, I'm suspicious of the image. Here is the definition of the image data field on the image:
#Column( name = "IMAGE_DATA" )
#Lob()
private byte[ ] imageData;
I'd appreciate any advice as to how to proceed.
The key to the answer is in the lower part of the stacktrace:
ORA-00932: inconsistent datatypes: expected NUMBER got BINARY
Hard to say which field this is about, but since you're working with images there may be something wrong in how you configured persistence for that class / field.
Maybe you should hint Hibernate how to store the field by annotating it with
#Lob(type = LobType.BLOB)
The problem ended up involving a missing annotation.
I had just added an image to note entry and set it up to point to the UUID of the image, but left off #OneToOne. As such, it appeared to be trying to store the image data in a field for UUID.
I Added the #OneToOne and the problem went away. Nathan was also correct in saying that it wasn't actually getting inserted until the commit. To find the issue, I moved these inserts outside of the transaction so that they got inserted immediately. This helped me track down the issue.