I am running a batch java application. The application runs every 10/20 minutes in my Production and UAT environment and I get database alerts like this:
Thu Feb 06 15:15:08 2014
opiodr aborting process unknown ospid (28246400) as a result of ORA-609
After researching a bit on the internet the suggested fix for these alerts is to change INBOUND_CONNECT_TIMEOUT as:
Sqlnet.ora: SQLNET.INBOUND_CONNECT_TIMEOUT=180
Listener.ora: INBOUND_CONNECT_TIMEOUT_listener_name=120
We have changed the setting on the database server side but don't know where to change in the client application. We are using c3p0 to create a connection pool and we are setting only these parameters:
dataSource.setAcquireRetryDelay(30000);
dataSource.setMaxPoolSize(50);
dataSource.setMinPoolSize(20);
dataSource.setInitialPoolSize(10);
We have other web services running on the same server as the batch application and they use Tomcat's DBCP pool and they don't seem to create any alerts. Also strangely enough, our batch application doesn't generate the alerts in lower test environments. They happen once in a while but the UAT and PROD environments get these alerts very frequently based on the schedule. Any suggestions what configurations to set in the c3p0 pool or should I try changing to another pool API like DBCP?
Update: I have added a few more parameters in the datasource and the frequency of alerts has reduced. I added the following and the number of alerts have gone down from 15 an hour to 4 an hour.
dataSource.setLoginTimeout(120);
dataSource.setAcquireRetryAttempts(10);
dataSource.setCheckoutTimeout(60000);
I moved to DBCP connection pooling and it seems to have fixed the issue. I tried changing a few more c3p0 settings mentioned above but nothing changed. The alerts were reduced but didn't go completely. So we decided to try DBCP. I am using all default values in DBCP except for the pool size. I'm using the tomcat version of DBCP available in tomcat's lib folder (tomcat-dbcp.jar).
Related
I working on upgrade the oracle jdbc driver from 11g (ojdbc6.jar) to 19c (ojdbc8.jar) in my java application, driver used is Instant Client (instantclient-basiclite-nt-19.11) with JRE1.8.0_271. After change to 19c, my application keep hit "ORA-02396: exceeded maximum idle time, please connect again" or "ORA-03113: end-of-file" error.
In oracle database properties, there are some limitation set, Idle-Time = 2 minutes and Connection-Time = 10 minutes. But I will not do any change on database because this may cause high CPU if many users are using the application at the same time.
In Java application, connection is stored in connection pool, I put the logging and can see that connection is closed and return to connection pool after finish execute. But when I run the application again after 2 minutes, the oracle error raised.
If I switch back to 11g, I don't get such error and application is working fine after 2 minutes. No change in code.
Is this BUG in latest oracle driver? I saw there is UCP.jar (Universal Connection Pool) package available in 19c but not in 11g, is it I have implement this? and how?
Your problem has nothing to do with the driver but with a database profile setting that limits the maximum allowed idle_time. Normally this is done to get rid of forgotten sessions.
You can check this using
select a.username,b.profile,b.RESOURCE_NAME,b.LIMIT from dba_users a, dba_profiles b where b.resource_name='IDLE_TIME' and a.profile=b.profile;
find which profile is used for your user and see if your dba is willing to change the limit.
if this happens to be the default profile it could be changed to unlimited by
ALTER PROFILE DEFAULT LIMIT IDLE_TIME UNLIMITED;
but it might be better to create a custom profile for your user.
If the IDLE_TIME limit can not be changed, run a query every once and a while, like select 'keep me alive from dual;
This also prevents closure by firewalls.
So, I have had a working Tomcat JDBC session storage solution for some time. I deployed what I consider to be an unrelated change (and I rolled back to previous war and the new issue still exists).
The problem
I can reproduce the problem like this
systemctl restart tomcat
wait for wars to load, manager to respond etc
access web app page that fetches for session, syserr:
java.lang.NullPointerException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.catalina.session.JDBCStore.open(JDBCStore.java:955)
(this part of apache's JDBCStore imple tries to load db driver using class name)
I keep accessing the web page -> same error to syserr
I keep doing this for a minute or so
Error suddenly no longer appears and session storage works as it should (persists new sessions all the way to db correctly, fetches sessions since prev restart correctly)
What I have tried
I have checked context.xml (that has the JDBCStore configuration) has not been changed by updates (and surely it is ok since the app does work eventually)
I have double checked jdbc driver is in tomcat/lib folder, has permissions etc (and surely it is since the app does work eventually)
I have tried to think of any change on the server that might cause this but I'm drawing blank
Killing Tomcat with kill -9 <pid> instead of systemctl restart does not make a difference
Stopping Tomcat, waiting for ~3 minutes and starting up: no difference, still have to wait ~1 minute before session storage works
What I suspect it that somehow JDBCStore (which is "internal Tomcat stuff", not part of my app) can not load the driver with Class.forName, but some other app/connection poll that I have running on the same Tomcat succeeds, and after that JDBCStore also works. What I don't know is why this has come up now but not lets say a month ago. In any case, any hints would be nice.
Tomcat8
openjdk version "1.8.0_265"
RHEL 7.9
If tomcat could not open a connection to the database it will throw NPE, check this commit that fixes it. Update your tomcat that will be released which includes this latest patch and you won't get the NPE.
But check why you are not able to connect to the database.
I'm using Wildfly 8.2 and fire a series of DB requests when a certain web page is opened. All queries are invoked thru JPA Criteria API, return results as expected - and - none of them delivers a warning, error or exception. It all runs in Parallel Plesk.
Now, I noticed that within 2 to 3 days the following error appears and the site becomes unresponsive. I restart and I wait approx another 3 days till it happens again (depending on the number of requests I have).
I checked the tcpsndbuf on my linux server and I noticed it is constantly at max. Unless I restart Wildfly. Apparently it fails to release the connections.
The connections are managed by JPA/Hibernate and the Wildfly container. I don't do any special or custom transaction handling e.g. open, close. etc. I leave it all to Wildfly.
The MySQL Driver I'm using is 5.1.21 (mysql-connector-java-5.1.21-bin.jar)
In the standalone.xml I have defined the following datasource datasource values (among others):
<transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
<pool>
<min-pool-size>3</min-pool-size>
<max-pool-size>10</max-pool-size>
</pool>
<statement>
<prepared-statement-cache-size>32</prepared-statement-cache-size>
<shared-prepared-statements>true</shared-prepared-statements>
</statement
Has anyone experience the same rise of tcpsndbuf values (or this error)? In case you require more config or log files, let me know. Thanks!
UPDATE
Despite the following additional timeout settings, it still runs into the hanger. And thus, it will then use 100% CPU time, whenever the max tcpsndbuf is reached.,
Try adding this Hibernate property:
<property name="hibernate.connection.release_mode">after_transaction</property>
By default, JTA mandates that connection should be released after each statement, which is undesirable for most use cases. Most Drivers don't allow multiplexing a connection over multiple XA transactions anyway.
Do you use openvz? I think this question should be asked on serverfault. It is related to linux configuration. You can read: tcpsndbuf. You should count opened sockets and check condition:
I am trying, for the first time, to run a java EE 7 web app without an IDE and struggling through the learning curve. To keep things simple I have started with Glassfish 4.1 and Derby 10.11.1.2 which is what I used for development.
My current problem is an error configuring the connection pool on the Glassfish web interface. The error is 'Connection authentication failure occurred. Reason: Userid or password invalid'. However I have ij runing in a dos prompt and the Glashfish web console in Chrome.
from ij I can type 'connect 'jdbc:derby:localhost:1527/databasename;user=username;password=password'; and I connect fine and can look at tables, etc. In the Glassfish web console I have name/value pairs with the same 'user' and 'password' values and it fails (exact same letters & case). I've tried changing the 'databaseName' property to make sure the error wasn't misleading and confirmed it has found the database correctly.
At face value, the error message seems wrong as I have proven from the DOS window. I am sure I am doing something wrong but am lost at how to diagnose it. Any suggestions?
More details of the steps taken: (Although I am sure some of these steps superceed others my lack of success has me in the mindset that redundancy is better than omission - all-in-all it seems much more complicated than it needs to be for a basic setup). My target is Wildfly but Glassfish documentation is better so I figured I would start there.
relevant Windows environment variables:
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_40
DERBY_HOME=%JAVA_HOME%\db
DERBY_INSTALL=%DERBY_HOME%
JAVADB_HOME=%DERBY_HOME%
GLASSFISH=C:\Program Files\Java\GlassFish-4.1
Path=%JAVA_HOME%\bin;%DERBY_HOME%\bin;%GLASSFISH%\bin;%GLASSFISH%\glassfish
CLASSPATH=%DERBY_HOME%\lib\derbynet.jar;%DERBY_HOME%\lib\derbytools.jar;%DERBY_HOME%\derbyclient.jar;%DERBY_HOME%\lib\derbyrun.jar
WILDFLY_HOME=C:\program files\java\wildfly\8.2.0.Final
sysinfo shows all the right paths and versions
Copied many of the derby jars from %derby_home%\lib to %glassfish%\glassfish\domains\domain1\lib\ext per a post by BalusC. I think classpath duplicates this it but can't hurt
Started Domain 'asadmin start-domain' defaulting to domain 1
Started DB by changing directories and using java. Wanted to use 'asadmin start-database --dbhome DB path' but this always started the wrong version. Glassfish and JDK both package derby/javadb and I couldn't figure out how to upgrade Glassfish or start the correct version any other way
cd \users\john\.netbeans-derby (root of the DB created by Netbeans)
Java –jar “%derby_home%\lib\derbyrun.jar" server start
connect browser to 'localhost:4848' to create JDBC connection pool and resource
connection pool: pool name=connectionPool, resource type=javax.sql.DataSource, DB driver Vendor=Derby, introspect was not enabled (found very little documentation to explain what it is) - step 2 was left as defaults
editing connectionPool - enabled Ping on General tab to identify errors, Additional Properties tab: changed User to DB username, left other attributes as default, added... password=db password, databaseName=ClubScoring
errors were presented along the way which prompted setting each attribute. No error was presented for lack of URL but tested steps below with and without URL=jdbc:derby://localhost:1527/ClubScoring
jdbc resources. Added jdbc/ClubScoring, pool=connectionPool => selected and enabled
starting db from correct directory using 'asadmin start-database' does let the application find the correct DB but it crashes since the DB was created with version 10.11.1.2 and this command starts the Glassfish version of 10.10.2.0
Sorry for making this so long but I haven't found anywhere that lays out all of the steps in a simple fashion and figured the details would save time in the long run.
I was dealing with this issue today and found a solution:
https://db.apache.org/derby/docs/10.14/ref/rrefattribsecmech.html
Basically the only security options allowable (as far as 10.14 is concerned) is 3, 4 and 9.
4 is the default and means "Just UserID". You would think this means that as long as a UserID is given, you could connect. But having both set appears to be an issue.
In order to use a (clear text) password and userID, you have to specify 3. It works also if the attribute is deleted or 0 as well but I don't think you get any security that way.
I have a long-running method which executes a large number of native SQL queries through the EntityManager (TopLink Essentials). Each query takes only milliseconds to run, but there are many thousands of them. This happens within a single EJB transaction. After 15 minutes, the database closes the connection which results in following error:
Exception [TOPLINK-4002] (Oracle TopLink Essentials - 2.1 (Build b02-p04 (04/12/2010))): oracle.toplink.essentials.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Closed Connection
Error Code: 17008
Call: select ...
Query: DataReadQuery()
at oracle.toplink.essentials.exceptions.DatabaseException.sqlException(DatabaseException.java:319)
.
.
.
RAR5031:System Exception.
javax.resource.ResourceException: This Managed Connection is not valid as the phyiscal connection is not usable
at com.sun.gjc.spi.ManagedConnection.checkIfValid(ManagedConnection.java:612)
In the JDBC connection pool I set is-connection-validation-required="true" and connection-validation-method="table" but this did not help .
I assumed that JDBC connection validation is there to deal with precisely this kind of errors. I also looked at TopLink extensions (http://www.oracle.com/technetwork/middleware/ias/toplink-jpa-extensions-094393.html) for some kind of timeout settings but found nothing. There is also the TopLink session configuration file (http://download.oracle.com/docs/cd/B14099_19/web.1012/b15901/sessions003.htm) but I don't think there is anything useful there either.
I don't have access to the Oracle DBA tables, but I think that Oracle closes connections after 15 minutes according to the setting in CONNECT_TIME profile variable.
Is there any other way to make TopLink or the JDBC pool to reestablish a closed connection?
The database is Oracle 10g, application server is Sun Glassfish 2.1.1.
All JPA implementations (running on a Java EE container) use a datasource with an associated connection pool to manage connectivity with the database.
The persistence context itself is associated with the datasource via an appropriate entry in persistence.xml. If you wish to change the connection timeout settings on the client-side, then the associated connection pool must be re-configured.
In Glassfish, the timeout settings associated with the connection pool can be reconfigured by editing the pool settings, as listed in the following links:
Changing timeout settings in GlassFish 3.1
Changing timeout settings in GlassFish 2.1
On the server-side (whose settings if lower than the client settings, would be more important), the Oracle database can be configured to have database profiles associated with user accounts. The session idle_time and connect_time parameters of a profile would constitute the timeout settings of importance in this aspect of the client-server interaction. If no profile has been set, then by default, the timeout is unlimited.
Unless you've got some sort of RAC failover, when the connection is terminated, it will end the session and transaction.
The admins may have set into some limits to prevent runaway transactions or a single job 'hogging' a connection in a pool. You generally don't want to lock a connection in a pool for an extended period.
If these queries aren't necessarily part of the same transaction, then you could try terminating and restarting a new connection.
Are you able to restructure your code so that it completes in under 15 minutes. A stored procedure in the background may be able to do the job a lot quicker than dragging the results of thousands of operations over the network.
I see you set your connection-validation-method="table" and is-connection-validation-required="true", but you do not mention that you specified the table you were validating on; did you set validation-table-name="any_table_you_know_exists" and provide any existing table-name? validation-table-name="existing_table_name" is required.
See this article for more details on connection validation.
Related StackOverflow article with similar problem - he wants to flush the entire invalid connection pool.