The backend runs in a docker container. It runs with spring boot & security and works fine. The frontend runs in another container as well. All works fine until out of nowhere this exception is thrown. I only noticed this exception after I idled a few minutes (normaly between 5 and 10 min).
WARN 1 --- [nio-8080-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08S01
ERROR 1 --- [nio-8080-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : I/O Error: Connection reset
ERROR 1 --- [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: could not extract ResultSet; nested exception is org.hibernate.exception.JDBCConnectionException: could not extract ResultSet] with root cause
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209) ~[na:1.8.0_111-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_111-internal]
at java.io.DataInputStream.readFully(DataInputStream.java:195) ~[na:1.8.0_111-internal]
.....
The connection to the Database is available all the time. The weird thing is, that this error is only thrown when I run my project with docker. If I run it with eclipse & node.js the error don't occurs. The only thing I noticed was, that a few minutes after I loged in and idled, I got automaticly loged out when I tried to do something. I don't know if this has any connection with the orgiginal error.
application.properties
spring.datasource.url=databaseURL
spring.datasource.username=SQLusername
spring.datasource.password=Password
spring.datasource.driver-class-name=net.sourceforge.jtds.jdbc.Driver
spring.datasource.initialSize=100
spring.datasource.minIdle=10
spring.jpa.hibernate.naming_strategy=org.hibernate.cfg.EJB3NamingStrategy
spring.jpa.hibernate.naming.implicit-strategy=org.hibernate.boot.model.naming.ImplicitNamingStrategyLegacyJpaImpl
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
spring.datasource.tomcat.max-active=250
spring.datasource.tomcat.max-wait=30000
spring.datasource.test-on-borrow=true
spring.datasource.test-while-idle=true
spring.datasource.minEvictableIdleTimeMillis=21600000
spring.datasource.timeBetweenEvictionRunsMillis=43200000
spring.datasource.connection-test-query=SELECT 1
spring.datasource.validationQuery=SELECT 1
I'm glad for every suggestion that could help to fix this proplem.
I had the same problem months ago!
This happens because the jvm doesn’t have memory enough to accept this request!
This error is intermittent and is very difficult to find your root cause!
Try this:
Put in your Dockerfile some flags to solve it
ENTRYPOINT ["java","-XX:+TieredCompilation","-XX:TieredStopAtLevel=1", "-jar","yourJar.jar"]
More details here: https://jpbempel.github.io/2020/05/22/startup-containers-tieredcompilation.html
Sometimes the problem can be memory. When you start your container with java, the jvm default heap memory is 1/4 of total memory of container.
Then you can use the flag -XX:MaxRAMFraction where your value has a range 1 to 4.
More details here: https://merikan.com/2019/04/jvm-in-a-container/#java-8u131-and-java-9
Good luck!
Related
I am currently being denied access with the following error:
INSERT command denied to user 'root'#'172.25.0.3' for table 'users'
ERROR 40 --- Servlet.service() for servlet [dispatcherServlet] in context with
path [] threw exception [Request processing failed;
nested exception is
org.springframework.dao.InvalidDataAccessResourceUsageException:
could not execute statement; SQL [n/a];
nested exception is org.hibernate.exception.SQLGrammarException:
could not execute statement] with root cause
java.sql.SQLSyntaxErrorException: INSERT command denied to user
'root'#'172.25.0.3' for table 'users'
This was all working fine with my last change being adding a column into the table. This worked fine for a while but then broke, since it was the last change I've reverted it but I still have an issue yet it all works as expected on local host (even with the new changes).
I would appreciate any help on this. I am sure the syntax is correct and so would love any suggestions of potential avenues to explore.
NB - My browser just shows a 500 error.
*EDIT - I am able to reproduce the error whenever I add columns to my table (adding or deleting rows appears to be fine). The issue will work on the immediate posting and then shortly after it will start causing the issue. My guess is that the generated statements used by java spring don't get updated whilst the old volume is still there, if that is true then my new question is how I would get the correct queries firing without wiping my volume. Still don't have a solution other then wiping the volume - I will be launching to production shortly so this won't be a viable solution once this happens.
The issue was solved by running docker compose -v - explanation can be seen here - MYSQL_ROOT_PASSWORD is set but getting "Access denied for user 'root'#'localhost' (using password: YES)" in docker container - It unfortunately meant losing my data - this was fine for my usecase but backup your volume before hand.
I am trying to setup mysql to my keycloak instance and couldn't get that working due to the following exception:
Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
("core-service" => "management"),
("management-interface" => "http-interface")
]'
14:34:42,546 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler#65d34517 for operation add-deployer-chains at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:522)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1518)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1472)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1445)
at org.jboss.as.controller.AbstractOperationContext$Step.access$400(AbstractOperationContext.java:1319)
at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:876)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:726)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:521)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:472)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:434)
at org.jboss.as.server.ServerService.boot(ServerService.java:435)
at org.jboss.as.server.ServerService.boot(ServerService.java:394)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:374)
at java.lang.Thread.run(Thread.java:748)
14:34:42,549 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler#65d34517 for operation add-deployer-chains at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException.
https://developer.jboss.org/thread/272010 in this question they also had a similar exception and got it solved after changing timeout from 300 to 600.
but where can i change this setting.
BTW i changed the transaction timeout as below but still facing the issue.
<coordinator-environment default-timeout="7200" />
As most of the people suggested, adding this parameter should do the fix for you.
-Djboss.as.management.blocking.timeout=3600
I use ubuntu 20.04 LTS on both my desktop and laptop. I noticed that mysql running on the docker container on the laptop while everything was fine on the desktop is running extremely slow, see MySQL bug#46959. If you're having a problem like mine, this solution might work.
I deployed this Spring application]1 on a kubernetes cluster with kompose up. Once it was deployed, the pods for a number of the services would enter a CrashLoopBackoff state. The services I've had problems with are the account service, monitoring service, notification service, and statistics service. It appears that when the services start up, they run into the following error:
2017-11-15 22:10:52.857 INFO 1 --- [ main] c.p.monitoring.MonitoringApplication : Started MonitoringApplication in 0.511 seconds (JVM running for 8.579)
2017-11-15 22:10:52.882 INFO 1 --- [ main] o.s.c.s.b.r.RabbitMessageChannelBinder : declaring queue for inbound: springCloudHystrixStream.anonymous.oCdNGHmBRPO_Z3LMusKS2Q, bound to: springCloudHystrixStream
2017-11-15 22:11:52.953 WARN 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'outputBindingLifecycle'; nested exception is org.springframework.context.ApplicationContextException: Failed to start bean 'inputBindingLifecycle'; nested exception is org.springframework.amqp.AmqpIOException: java.net.SocketTimeoutException: connect timed out
The full log can be found here. Any help is greatly appreciated. I assume this is a Spring/Java related issue but it may be a Kubernetes related issue as well given that the application was designed to run on docker swarm.
We have a stand-alone single user desktop application that has a Swing front-end and uses an embedded Derby database (10.8.2.2 1181258). Spring (3.0.4) is used as the go between the application and the database. A customer (in Hebrew/Israel locale iw_IL) running the application using JRE 1.7.0_21 ran into an issue. The following stack trace was logged:
ERROR [index-stat-thread] stderr - Exception in thread "index-stat-thread"
ERROR [index-stat-thread] stderr - java.lang.ClassCastException: org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeScan.positionAtNextPage(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeForwardScan.fetchRows(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.store.access.btree.BTreeScan.fetchNextGroup(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl$KeyComparator.fetchRows(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.updateIndexStatsMinion(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.generateStatistics(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.processingLoop(Unknown Source)
ERROR [index-stat-thread] stderr - at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.run(Unknown Source)
ERROR [index-stat-thread] stderr - at java.lang.Thread.run(Thread.java:722)
Any idea what might cause such an error?
Subsequently, Spring logged the following:
org.springframework.jdbc.support.SQLErrorCodesFactory - Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: No current connection.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:293)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:320)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:212)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:141)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:104)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:636)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:665)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:673)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:796)
at org.springframework.jdbc.core.simple.SimpleJdbcTemplate.queryForList(SimpleJdbcTemplate.java:235)
Given that the application uses an embedded Derby database on the same machine and file system as the application, the JDBC connection error is baffling. Could the first Derby error be the reason for this?
Following this is another error from Spring while loading entries for one table, which essentially seems to be the same as the first error from Derby:
ERROR [main] org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [select * from XYZ where scresult_id=?]; SQL state [XJ001]; error code [0]; Java exception:
'org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow: java.lang.ClassCastException'.; nested exception is java.sql.SQLException: Java exception:
'org.apache.derby.impl.store.access.btree.BranchControlRow cannot be cast to org.apache.derby.impl.store.access.btree.LeafControlRow: java.lang.ClassCastException'.
The same application has been used successfully by other customers for several months. This is the first time a customer has run into these problems.
It looks like your database has become corrupt. Those are internal messages that you should never see.
Here is information about how to use your backups and logs to recover your database: http://db.apache.org/derby/docs/10.10/adminguide/cadminhubbkup98797.html
Before you restore the database, you should retain a copy of the damaged database, the derby.log file, and any other information about the circumstances under which the database became corrupt, as they may give you important clues about how to prevent this problem in the future.
In general, it should not be possible to corrupt a Derby database like this; hence arriving at such a situation might indicate a bug in Derby itself. But there could be other possible explanations, such as damaged or misconfigured hardware.
I assume from your initial question that you do not know how to reproduce this problem? In that case, preserving as much information as you can about the incident, and about what was occurring at the moment that the corruption occurred (did the power fail? Was the machine hard-rebooted? did a disk become full? was a virus-checker running on the machine? There are a million possible questions like this to ask...) may give you clues about how to reproduce the problem yourself.
When I deploy an application.war that contains a servlet to my weblogic server it runs fine. The servlet uses an instance of UniversalConnectionPoolManagerMBean and starts up without any problems.
In the administration console I use the "stop when work completes" command and in my destroy() method it calls:
UniversalConnectionPoolManagerImpl.getUniversalConnectionPoolManager()
.startConnectionPool(mConnectionPoolName);
When I then click on "start servicing requests" command in the administration console I get this exception:
java.sql.SQLException: Unable to start the Universal Connection Pool: java.sql.SQLException: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: MBean exception occurred while registering or unregistering the MBean
at com.exzac.dal.jdbc.Database.getConnection(Database.java:134)
at com.exzac.dal.jdbc.Database.executeQuery(Database.java:161)
at com.exzac.profilesystem.ProfileSystemConfiguration.retrieveConfigurationTable(ProfileSystemConfiguration.java:214)
... 49 more
Caused by: java.sql.SQLException: Unable to start the Universal Connection Pool: java.sql.SQLException: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: MBean exception occurred while registering or unregistering the MBean
at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:541)
at oracle.ucp.jdbc.PoolDataSourceImpl.throwSQLException(PoolDataSourceImpl.java:587)
at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:276)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:646)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:613)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:607)
at com.exzac.dal.jdbc.Database.getConnection(Database.java:132)
... 51 more
Caused by: java.sql.SQLException: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: MBean exception occurred while registering or unregistering the MBean
at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:541)
at oracle.ucp.jdbc.PoolDataSourceImpl.throwSQLException(PoolDataSourceImpl.java:587)
at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:247)
... 55 more
Caused by: oracle.ucp.UniversalConnectionPoolException: MBean exception occurred while registering or unregistering the MBean
at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:421)
at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:389)
at oracle.ucp.admin.UniversalConnectionPoolManagerMBeanImpl.getUniversalConnectionPoolManagerMBean(UniversalConnectionPoolManagerMBeanImpl.java:149)
at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:242)
... 55 more
Caused by: java.security.PrivilegedActionException: javax.management.InstanceAlreadyExistsException: oracle.ucp.admin:name=UniversalConnectionPoolManagerMBean
at java.security.AccessController.doPrivileged(Native Method)
at oracle.ucp.admin.UniversalConnectionPoolManagerMBeanImpl.getUniversalConnectionPoolManagerMBean(UniversalConnectionPoolManagerMBeanImpl.java:136)
... 56 more
Caused by: javax.management.InstanceAlreadyExistsException: oracle.ucp.admin:name=UniversalConnectionPoolManagerMBean
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$27.run(WLSMBeanServerInterceptorBase.java:714)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.registerMBean(WLSMBeanServerInterceptorBase.java:709)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.registerMBean(JMXContextInterceptor.java:448)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$27.run(WLSMBeanServerInterceptorBase.java:712)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.registerMBean(WLSMBeanServerInterceptorBase.java:709)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.registerMBean(WLSMBeanServer.java:462)
at oracle.ucp.admin.UniversalConnectionPoolManagerMBeanImpl$2.run(UniversalConnectionPoolManagerMBeanImpl.java:142)
... 58 more
From then on I hit this exception, even when I stop the server and restart and try to deploy. Only cleaning all published resources seems to clear the exception. I understand that a published resource was not properly cleaned in the destroy method, and I am thinking that the resource is the UniversalConnectionPoolManagerMBean instance, but I thought it would be clean up by the destroyConnectionPool method:
http://docs.oracle.com/cd/B28359_01/java.111/e11990/oracle/ucp/admin/UniversalConnectionPoolManagerMBean.html#destroyConnectionPool_java_lang_String_
How do I avoid hitting this exception?
Just as an interesting (yet unsafe) workaround: For some reason when I insert this into the initialization:
try {
UniversalConnectionPoolManagerImpl.getUniversalConnectionPoolManager().destroyConnectionPool(mConnectionPoolName);
} catch (final Exception e1) {
LOGGER.error(e1);
}
the exception is thrown and caught saying that it could not find a connectionPool with this name to destroy, but the original problem is gone. javax.management.InstanceAlreadyExistsException: oracle.ucp.admin:name=UniversalConnectionPoolManagerMBean is not thrown
This bug seems to have been fixed in versions 11.2.0.4 and 12.1.0.2. I couldn't find any documentation about it, but for me the error disappeared with the above mentioned versions.
JDBC/UCP Download Page:
http://www.oracle.com/technetwork/database/features/jdbc/index-091264.html
This is from Oracle's 11.2.0.4 UCP.jar README.txt.
Fixes in 11.2.0.4
16805157 The name of the UniversalConnectionPoolManagerMBean is made
unique per classloader and hence unique to every UCP instance
even within the same JVM
This fixes the javax.management.InstanceAlreadyExistsException described in the original question. Every connection pool created with have a unique name and will not result in InstanceAlreadyExistsException. However, since you are restarting, you should look into figuring out how to close the connection pool MBean cleanly so you won't run into this problem.