My team has to make some changes and renew an old web application. This application has one main thread and 5 to 15 daemon threads used as workers to retrieve and insert data in a DB.
All those threads have this design (here simplified for convenience):
public MyDaemon implements Runnable {
// initialization and some other stuffs
public void run() {
...
while(isEnabled) {
Engine.doTask1();
Engine.doTask2();
...
Thread.sleep(someTime);
}
}
}
The Engine class provides a series of static methods used to maipulate other methods of DataAccessor classes, some of those methods been static:
public Engine {
public static doTask1() {
ThisDataAccessor.retrieve(DataType data);
// some complicated operations
ThisDataAccessor.insertOrUpdate(DataType data);
}
public static doTask2() {
ThatDataAccessor da = new ThatDataAccessor();
da.retrieve(DataType data);
// etc.
}
...
}
DataAccessor classes usually interact with DB using simple JDBC statements enclosed in synchronized methods (static for some classes). DataSource is configured in the server.
public ThatDataAccessor {
public synchronized void retrieve(DataType data) {
Connection conn = DataSource.getConnection();
// JDBC stuff
conn.close();
}
...
}
The problem is that the main thread needs to connect to DB and when these daemon threads are working we run easily out of available connections from the pool, getting "waiting for connection timeout" exceptions. In addition, sometimes even those daemon threads get the same exception.
We have to get rid of this problem.
We have a connection pool configured with 20 connections, and no more can be added since that "20" is our production environment standard. Some blocks of code need to be synchronized, even if we plan to move the "synchronized" keyword only where really needed. But I don't think that it would make really the difference.
We are not experienced in multithreading programming and we've never faced this connection pooling problem before, that's why I'm asking: is the problem due to the design of those threads? Is there any flaw we haven't noticed?
I have profiled thread classes one by one and as long as they are not running in parallel it seems that there's no bottleneck to justify those "waiting for connection timeout".
The app is running on WebSphere 7, using Oracle 11g.
You are likely missing a finally block somewhere to return the connections back to the pool. With hibernate, I think this is probably done when you call close() or possibly for transactions, when you call rollback(). But I would call close anyway.
For example, I wrote a quick and dirty pool myself to extend an old app to make it multithreaded, and here is some of the handling code (which should be meaningless to you except the finnally block):
try {
connection = pool.getInstance();
connection.beginTransaction();
processFile(connection, ...);
connection.endTransaction();
logger_multiThreaded.info("Done processing file: " + ... );
} catch (IOException e) {
logger_multiThreaded.severe("Failed to process file: " + ... );
e.printStackTrace();
} finally {
if (connection != null) {
pool.releaseInstance(connection);
}
}
It is fairly common for people to fail to use finally blocks properly... For example, look at this hibernate tutorial, and skip to the very bottom example. You will see that in the try{} he uses tx.commit() and in the catch{} he uses tx.rollback(), but he has no session.close(), and no finally. So even if he added a "session.close()" in try and in catch, if his try block threw something other than a RuntimeException, or his catch caused an additional Exception before the try or a non-HibernateException before the rollback(), his connection would not be closed. And without session.close(), I don't think that is actually very good code. But even if the code is seemingly working, a finally gives you assurance that you are protected from this type of problem.
So I would rewrite his methods that use Session to match the idiom shown on this hibernate documentation page. (and also I don't recommend his throwing a RuntimeException, but that is a different topic).
So if you are using Hibernate, I think the above is good enough. But otherwise, you'll need to be more specific if you want specific code help, but otherwise the simple idea that you should use a finally to ensure the connection is closed is enough.
Related
Is there a way to close java.net.http.HttpClient to instantly release resources held by it?
Internally it holds a selector, a connection pool and an Executor (when default one is used). However it does not implement Closeable/AutoCloseable.
I had similar problem when I was redeploying a war file into Tomcat. War application had an HttpClient, that was running scheduled jobs issuing http requests and processing results.
I was seeing warnings from Tomcat when redeploying war file quite often on dev enrionment about hanging threads that may cause memory leaks. The stack trace was pointing to HttpClient threads. After several attempts I solved this problem in this way:
HttpClient is created only when necessary to execute the job. It is not created as a field of a class or serivec, only as a local variable inside the scheduled method.
HttpClient is created using builder and populated with a ThreadPool Executor, thus I keep the link to Executor and have control on it.
ExecutorService executor = Executors.newSingleThreadExecutor();
HttpClient client = HttpClient.newBuilder().followRedirects(Redirect.ALWAYS).connectTimeout(Duration.ofSeconds(5)).executor(executor).build();
When job is done in try-catch block, finally section has these two lines that: close the thread pool explicitly and set null to the httpClient local variable:
executor.shutdownNow();
client = null;
System.gc();
Notes, have short connection timeout to limit the time of execution. keep number of threads small. I use threadPool of 1 thread.
After all these changes warnings about memory leaks disappeared from Tomcat logs.
As you've noticed, java.net.http.HttpClient does not implement Closeable or AutoCloseable. So I can think of only 2 options, but neither of them are really bulletproof or even good:
You could eliminate every strong reference to the HttpClient that your program is holding and request a garbage collection. However there's a real risk that something beyond your direct control is holding onto it or one of its components. Any remaining strong references would prevent the referenced object, and any objects it holds a strong reference to, from being garbage collected. Nevertheless, this is arguably the more idiomatic option than the alternative.
I also found another option.
final class HttpClientImpl extends HttpClient implements Trackable {
...
// Called from the SelectorManager thread, just before exiting.
// Clears the HTTP/1.1 and HTTP/2 cache, ensuring that the connections
// that may be still lingering there are properly closed (and their
// possibly still opened SocketChannel released).
private void stop() {
// Clears HTTP/1.1 cache and close its connections
connections.stop();
// Clears HTTP/2 cache and close its connections.
client2.stop();
}
...
}
I wouldn't feel comfortable using this unless I had no other choice. Your reference is probably of type HttpClient, so you'd need to cast it to HttpClientImpl. It's bad to rely on the concrete implementation, which could change in future releases, rather than the HttpClient interface. The method is also private. There are ways around this but it's messy.
In Java 11, each HttpClient spawn a daemon thread called selmgr which supposed to take care of in fly requests. This thread will be closed when there is no reference to the HttpClient in the code. However, in my experience, it is not reliable. Especially, when you use asynchronous methods with future timeouts.
Here is a piece of code I wrote using reflection to reliably close the HttpClient
static void shutDownHttpClient(HttpClient httpClient)
{
ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) httpClient.executor().get();
threadPoolExecutor.shutdown();
try {
Field implField = httpClient.getClass().getDeclaredField("impl");
implField.setAccessible(true);
Object implObj = implField.get(httpClient);
Field selmgrField = implObj.getClass().getDeclaredField("selmgr");
selmgrField.setAccessible(true);
Object selmgrObj = selmgrField.get(implObj);
Method shutDownMethod = selmgrObj.getClass().getDeclaredMethod("shutdown");
shutDownMethod.setAccessible(true);
shutDownMethod.invoke(selmgrObj);
}
catch (Exception e) {
System.out.println("exception " + e.getMessage());
e.printStackTrace();
}
}
As you can see, this is implementation-dependent and may not works with future Java versions. It is tested with Java 11 and Java 12.
Also, you need to add --add-opens java.net.http/jdk.internal.net.http=ALL-UNNAMED to your java command.
Obviously HttpClient is designed to be self managed. So it responsible to maintain connection pool, cache ttl by itself.
In HttpClientCode We could find the following code:
if (!owner.isReferenced()) {
Log.logTrace("{0}: {1}",
getName(),
"HttpClient no longer referenced. Exiting...");
return;
}
this is a graceful way to exit from SelectorManager loop and clean all resources.
#Override
public void run() {
...
try {
while (!Thread.currentThread().isInterrupted()) {
...
if (!owner.isReferenced()) {
Log.logTrace("{0}: {1}",
getName(),
"HttpClient no longer referenced. Exiting...");
return;
}
...
}
} catch (Throwable e) {
...
} finally {
...
shutdown();
}
}
final boolean isReferenced() {
HttpClient facade = facade();
return facade != null || referenceCount() > 0;
}
So when your HttpClient object will not be referenced, then it will clean all resources.
UPD: also you should tune your requests by passing timeouts
If it is just to gracefully close HttpClient at the end of the application lifecycle, System.exit(0) shall just work.
public static void main(String[] args) {
...
System.exit(0);
}
I think it sends a interrupt signal to all threads in the JVM and HttpClient selmgr daemon does pick this up and shutdown itself.
final class HttpClientImpl extends HttpClient implements Trackable {
...
// Main loop for this client's selector
private final static class SelectorManager extends Thread {
...
#Override
public void run() {
...
try {
...
while (!Thread.currentThread().isInterrupted()) {...}
} catch (Throwable e) {...}
finally {
...
shutdown();
}
This is a bit late, but I just want to highlight that the comment from Jacob G. (Dec 25, 2018) contained a solution that worked for me:
Creating the httpClient:
myExecutorService = Executors.newCachedThreadPool();
HttpClient myHttpClient = HttpClient.newBuilder()
.executor(executor)
....
.build();
Shutting down:
myExecutorService.shutDown();
Instead of having to wait for 90 seconds for the client to drop its connection, it happened "instantly".
I am getting below error in my PROD environment. Some DB call is working fine but some call is throwing exception due to package is discarded/state is invalidate and many reason. But I don't have control to re-compile package. i.e i have only read-only access. I am using Java8, JDBI and Oracle database with connection pooling.
ORA-04068: existing state of packages has been discarded
ORA-04061: existing state of package body "USER.PKG_MY_PACKAGE" has been invalidated
ORA-04065: not executed, altered or dropped package body "USER.PKG_MY_PACKAGE"
ORA-06508: PL/SQL: could not find program being called: "USER.PKG_MY_PACKAGE"
ORA-06512: at line 34
I would like do to re-try options(2 times) by handle this exception in catch block and re-try from catch block. I referred many forums, connection has old compiled stored procedure or state of object as dirty. So, If I re-try with fresh connection for 3 times i may not get this error. Please suggest me fresh connection will work or fresh session will work. I am going to try this below
public void callDB(int userName, int retryCount) throws Exception {
try(Handle handle = dbInstance.open()) {
OutputParameters parameters = handle.createCall(MY_STORED_PROC).bind(0,userName).bind(1,Oracle.CURSOR);
Employee employee=parameters.getObject(2);
} catch(Exception e) {
logger.error(e);
if(retryCount!=2 && e.getMessage().contains("ORA-04068")) {
retryCount=retryCount+1;
callDB(userName, retryCount);
} else {
throw e;
}
}
}
I am getting from connection pooling via JDBI API. Please let me know this above code will work or not. Or Shall I need to re-try different way.
Also, suggest me when i re-try do i need create a new session or connection from connection pool is enough.
I'm still an undergrad just working part time and so I'm always trying to be aware of better ways to do things. Recently I had to write a program for work where the main thread of the program would spawn "task" threads (for each db "task" record) which would perform some operations and then update the record to say that it has finished. Therefore I needed a database connection object and PreparedStatement objects in or available to the ThreadedTask objects.
This is roughly what I ended up writing, is creating a PreparedStatement object per thread a waste? I thought static PreparedStatments could create race conditions...
Thread A stmt.setInt();
Thread B stmt.setInt();
Thread A stmt.execute();
Thread B stmt.execute();
A´s version never gets execed..
Is this thread safe? Is creating and destroying PreparedStatement objects that are always the same not a huge waste?
public class ThreadedTask implements runnable {
private final PreparedStatement taskCompleteStmt;
public ThreadedTask() {
//...
taskCompleteStmt = Main.db.prepareStatement(...);
}
public run() {
//...
taskCompleteStmt.executeUpdate();
}
}
public class Main {
public static final db = DriverManager.getConnection(...);
}
I believe it is not a good idea to share database connections (and prepared statements) between threads. JDBC does not require connections to be thread-safe, and I would expect most drivers to not be.
Give every thread its own connection (or synchronize on the connection for every query, but that probably defeats the purpose of having multiple threads).
Is creating and destroying PreparedStatement objects that are always the same not a huge waste?
Not really. Most of the work happens on the server, and will be cached and re-used there if you use the same SQL statement. Some JDBC drivers also support statement caching, so that even the client-side statement handle can be re-used.
You could see substantial improvement by using batched queries instead of (or in addition to) multiple threads, though. Prepare the query once, and run it for a lot of data in a single big batch.
The threadsafety is not the issue here. All looks syntactically and functionally fine and it should work for about half a hour. Leaking of resources is however the real issue here. The application will crash after about half a hour because you never close them after use. The database will in turn sooner or later close the connection itself so that it can claim it back.
That said, you don't need to worry about caching of preparedstatements. The JDBC driver and the DB will take care about this task. Rather worry about resource leaking and make your JDBC code as solid as possible.
public class ThreadedTask implements runnable {
public run() {
Connection connection = null;
Statement statement = null;
try {
connection = DriverManager.getConnection(url);
statement = connection.prepareStatement(sql);
// ...
} catch (SQLException e) {
// Handle?
} finally {
if (statement != null) try { statement.close(); } catch (SQLException logOrIgnore) {}
if (connection != null) try { connection.close(); } catch (SQLException logOrIgnore) {}
}
}
}
To improve connecting performance, make use of a connection pool like c3p0 (this by the way does not mean that you can change the way how you write the JDBC code; always acquire and close the resources in the shortest possible scope in a try-finally block).
You're best to use a connection pool and get each thread to request a connection from the pool. Create your statements on the connection you're handed, remembering to close it and so release it back to the pool when you're done. The benefit of using the pool is that you can easily increase the number of available connections should you find that thread concurrency is becoming an issue.
I have a Java app that opens a connection to a database at the beginning, and closes it at the end. However, the program doesn't always finish, because an exception is thrown or I am debugging it and stop it halfway through.
Will this cause open connections to pile up and slow the database, or will it be cleaned up automatically?
A database Connection is owned and managed by the database, the class just gives you access to that database resource. If you don't close the connection then the Java class may be garbage collected, but the database may not be able to tell that the connection is no longer in use which may result in database resources being wasted (until a timeout on the database side) or even leak.
So, when you're done with using your Connection, you should be sure to explicitly close it by calling its close() method. This will allow the garbage collector to recollect memory as early as possible and, more important, it releases any other database resources (cursors, handles, etc) the connection may be holding on to.
The traditional way to do this in Java is to close your ResultSet, Statement, and Connection (in that order) in a finally block when you are done with them and the safe pattern looks like that:
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
try {
// Do stuff
...
} catch (SQLException ex) {
// Exception handling stuff
...
} finally {
if (rs != null) {
try {
rs.close();
} catch (SQLException e) { /* ignored */}
}
if (ps != null) {
try {
ps.close();
} catch (SQLException e) { /* ignored */}
}
if (conn != null) {
try {
conn.close();
} catch (SQLException e) { /* ignored */}
}
}
The finally block can be slightly improved into (to avoid the null check):
} finally {
try { rs.close(); } catch (Exception e) { /* ignored */ }
try { ps.close(); } catch (Exception e) { /* ignored */ }
try { conn.close(); } catch (Exception e) { /* ignored */ }
}
But, still, this is extremely verbose so you generally end up using an helper class to close the objects in null-safe helper methods and the finally block becomes something like that:
} finally {
DbUtil.closeQuietly(rs);
DbUtil.closeQuietly(ps);
DbUtil.closeQuietly(conn);
}
And, actually, the Apache Commons DbUtils has a DbUtils class which is precisely doing that so there is no need to write your own.
In your case, this will solve the problem of the exception, but not the debugging one (and you will waste database resources until the timeout occur on the database side). So 1. don't debug your code using a production database 2. try to execute your debug session until the end.
Here's what Sun (err...Oracle?) says:
It is recommended that programmers explicitly close connections and statements they have created when they are no longer needed.
A programmer writing code in the Java programming language and not using any outside resources does not need to worry about memory management. The garbage collector automatically removes objects when they are no longer being used and frees the memory they were using. When memory is running low, it will recycle discarded objects, making the memory they currently occupy available for quick reuse.
However, if an application uses external resources, as it does when it accesses a DBMS with the JDBC API, the garbage collector has no way of knowing the status of those resources. It will still recycle discarded objects, but if there is lots of free memory in the Java heap, it may garbage collect infrequently, even though the (small) amount of Java garbage is holding open large amounts of expensive database resources. Therefore, it is recommended that programmers explicitly close all connections (with the method Connection.close) and statements (with the method Statement.close) as soon as they are no longer needed, thereby freeing DBMS resources as early as possible. This applies especially to applications that are intended to work with different DBMSs because of variations from one DBMS to another.
I would put the database access in a try block and make sure to close all statements and connections in a finally block.
Your db server will have a timeout setting. It will close the connection and roll back any uncommitted transactions. This has been happening for decades on any production capable db product.
If you want to do it properly use a try { ..your code..} finally { ..close connections..}
Nope.
If your program continues and your connections is alive, then the BD simply rejected your sentence.
If something happened with your connection (by example, a timeout) then the BD was who closed that connection and it's not consuming resources.
If you released your connection, and the garbage collector was called (it can be a while) the connection will close itself before get freed.
If your program terminated without closing your connection, then all the (operating system) process will release its native resources, and between them, the native resource that connected to the BD (probable a network socket). The BD will then receive the connection aborted/closed and release your connection.
The only thing that could happen is that one only execution would connect many times to the BD and do things very bad to keep them open, ocuppying all the connections available. But it's not your case I think.
Edit: in general BD's are made bad-client-behavior-proof
Sometimes when I call connect() on a third-party proprietary JDBC driver, it never returns and a stack trace shows that it is stuck waiting for a socket read (usually). Is there a generic way to forcibly cancel this operation from another thread? It's a blocking I/O call so Thread.interrupt() won't work, and I can't directly close the socket because I don't have access to it since it's created inside the proprietary code.
I'm looking for generic solutions because I have a heterogeneous DB environment (Oracle, MySQL, Sybase etc). But driver-specific suggestions are also welcome. Thanks,
There is no standard JDBC interface to set connection or read timeouts, so you are bound to use proprietary extensions, if the JDBC driver supports timeouts at all. For the Oracle JDBC thin driver, you can e.g. set the system properties "oracle.net.CONNECT_TIMEOUT" and or "oracle.jdbc.ReadTimeout" or pass a Properties instance to DriverManager.getConnection with these properties set. Although not particulary well documented, the Oracle specific properties are listed in the API documentation.
For other JDBC drivers, the documentation should contain the relevant references.
Ah ... the joys of using closed-source libraries ...
If interrupt() doesn't work, and you cannot set some kind of timeout, then I think there is no safe way to do it. Calling Thread.kill() might do the job, but the method is deprecated because it is horribly unsafe. And this is the kind of scenario where the unsafe-ness of Thread.kill() could come back and bite you.
I suggest that you simply code your application to abandon the stuck thread. Assuming that your application doesn't repeatedly try to connect to the DB, a stuck thread isn't a huge overhead.
Alternatively use a better JDBC driver. (And on your way out of the door, complain to the supplier about their driver being too inflexible. There is a slight chance that someone might listen to you ...)
At least one JDBC driver (not one of those you listed, though) will cleanly close the connection if the thread this connection attempt is running on is interrupted. I don't know if this will work for all drivers though.
This is a problem with Java, not the JDBC driver. In certain circumstances, the socket connect call ignores the timeout parameters and can take minutes to return. This happens to us when firewall blocks the port. It happens to all TCP connections (HTTP, RMI).
The only solution I find is to open connection in a different thread like this,
private static final ExecutorService THREADPOOL
= Executors.newCachedThreadPool();
private static <T> T call(Callable<T> c, long timeout, TimeUnit timeUnit)
throws InterruptedException, ExecutionException, TimeoutException
{
FutureTask<T> t = new FutureTask<T>(c);
THREADPOOL.execute(t);
return t.get(timeout, timeUnit);
}
try {
Data data = call(new Callable<Data>() {
public Data call() throws Exception
{
// Open connection, get data here
return data;
}, 2, TimeUnit.SECONDS);
} catch (TimeoutException e) {
System.err.println("Data call timed-out");
}