I am writing a webservice that allows users to post files and then retrieve them at a URL (basically think of it as the RESTful Amazon S3). The issue I came across was rather then return a byte[] from my Oracle query (Spring JDBC) I am returning an InputStream and then streaming the data back to the client in chunks. This (IMO) is a much better idea since I put no size restriction on the file and I don't want 2GB byte arrays in memory.
At first it seemed to work fine, but I ran into a case during heavy load that sometimes a Connection would get reused before the previous servlet could send the file. It seems after the JDBC call that returned the InputStream, the Connection would be returned to the pool (Spring would call conn.close(), but not clear the associated ResultSet). So if no other request was given that Connection then the InputStream would still be valid and could be read from, but if the Connection was given to a new request then the InputStream would be null and the previous request would fail.
My solution was to create a subclass of InputStream that also takes a Connection as a constructor arg, and in the overridden public close() method also close the Connection. I had to ditch the Spring JDBC and just make a normal PreparedStatement call, otherwise Spring would always return the connection to the pool.
public class ConnectionInputStream extends InputStream {
private Connection conn;
private InputStream stream;
public ConnectionInputStream(InputStream s, Connection c) {
conn = c;
stream = s;
}
// all InputStream methods call the same method on the variable stream
#Override
public void close() throws IOException {
try {
stream.close();
} catch (IOException ioex) {
//do something
} finally {
try {
conn.close();
} catch (SQLException sqlex) {
//ignore
}
}
}
}
Does anyone have a more elegant solution, or see any glaring problems with my solution? Also this code wasn't cut/paste from my actual code so if there is a typo just ignore it.
Unfortunately, my imagination went wild when you asked this question. I don't know if this solution is considered more elegant. However, these classes are simple and easily re-usable so you may find a use for them if they are not satisfactory. You will see everything coming together at the end...
public class BinaryCloseable implements Closeable {
private Closeable first;
private Closeable last;
public BinaryCloseable(Closeable first, Closeable last) {
this.first = first;
this.last = last;
}
#Override
public void close() throws IOException {
try {
first.close();
} finally {
last.close();
}
}
}
BinaryCloseable is used by CompositeCloseable:
public class CompositeCloseable implements Closeable {
private Closeable target;
public CompositeCloseable(Closeable... closeables) {
target = new Closeable() { public void close(){} };
for (Closeable closeable : closeables) {
target = new BinaryCloseable(target, closeable);
}
}
#Override
public void close() throws IOException {
target.close();
}
}
The ResultSetCloser closes ResultSet objects:
public class ResultSetCloser implements Closeable {
private ResultSet resultSet;
public ResultSetCloser(ResultSet resultSet) {
this.resultSet = resultSet;
}
#Override
public void close() throws IOException {
try {
resultSet.close();
} catch (SQLException e) {
throw new IOException("Exception encountered while closing result set", e);
}
}
}
The PreparedStatementCloser closes PreparedStatement objects:
public class PreparedStatementCloser implements Closeable {
private PreparedStatement preparedStatement;
public PreparedStatementCloser(PreparedStatement preparedStatement) {
this.preparedStatement = preparedStatement;
}
#Override
public void close() throws IOException {
try {
preparedStatement.close();
} catch (SQLException e) {
throw new IOException("Exception encountered while closing prepared statement", e);
}
}
}
The ConnectionCloser closes Connection objects:
public class ConnectionCloser implements Closeable {
private Connection connection;
public ConnectionCloser(Connection connection) {
this.connection = connection;
}
#Override
public void close() throws IOException {
try {
connection.close();
} catch (SQLException e) {
throw new IOException("Exception encountered while closing connection", e);
}
}
}
We now refactor your original InputStream idea into:
public class ClosingInputStream extends InputStream {
private InputStream stream;
private Closeable closer;
public ClosingInputStream(InputStream stream, Closeable closer) {
this.stream = stream;
this.closer = closer;
}
// The other InputStream methods...
#Override
public void close() throws IOException {
closer.close();
}
}
Finally, it all comes together as:
new ClosingInputStream(
stream,
new CompositeCloseable(
stream,
new ResultSetCloser(resultSet),
new PreparedStatementCloser(statement),
new ConnectionCloser(connection)
)
);
When this ClosingInputStream's close() method is called, this is effectively what happens (with exception handling omitted for clarity's sake):
public void close() {
try {
try {
try {
try {
// This is empty due to the first line in `CompositeCloseable`'s constructor
} finally {
stream.close();
}
} finally {
resultSet.close();
}
} finally {
preparedStatement.close();
}
} finally {
connection.close();
}
}
You're now free to close as many Closeable objects as you like.
Why not read the entire InputStream/byte[]/whatever from the query before releasing the query yourself? It sounds like you are trying to return data from the query after your code has told Spring / the pool that you are done with the connection.
An alternative approach is to use a callback. Below is kind of the idea.
class MyDao
{
public boolean getData(Function<InputStream, Boolean> processData) {
// Do your SQL stuff to get a ResultSet
InputStream input = resultSet.getBinaryStream(0);
processData.apply(input);
// Do your cleanup if any
}
}
Related
How could I wrote a ThrowingSupplier with an unchecked method that could replace this part of code? I have really no idea how to start with it should it be an interface or rather a class.
try {
// get connection with the database
connection = dataSource.getConnection();
} catch (Exception e) {
throw new UndeclaredThrowableException(e);
}
What I would like to get is something like
Connection connection = ThrowingSupplier.unchecked(dataSource::getConnection).get();
Any ideas how should it looks like? I am not sure if it should be an interface or a class I tried to wrote that, but then I could not create a static method unchecked and I would not to create new instance of that.
If I understand correctly, this is what you want:
public class ThrowingSupplier {
public static <T> Supplier<T> unchecked(Callable<T> callable) {
return () -> {
try {
return callable.call();
}
catch (Exception e) {
throw new UndeclaredThrowableException(e);
}
};
}
// example usage:
public static void main(String[] args) {
DataSource dataSource = null;
Connection connection = ThrowingSupplier.unchecked(dataSource::getConnection).get();
}
}
Auto closing HttpURLConnection (same as DB Connection using try-with-resourse)
here i am looking for closing HttpURLConnection,without closing manually ex:urlConnection.disconnect(); in finally block
It's not exactly the same, but you can write a wrapper class for Autocloseable that kind of does it for you.
class AutocloseWrapper<T> implements Autocloseable {
T wrapped;
Consumer<T> closeMethod;
public AutocloseWrapper(T wrapped, Consumer<T> closeMethod) {
this.wrapped = wrapped; this.closeMethod = closeMethod;
}
public void close() {
closeMethod.accept(wrapped);
}
}
And you'd call that with
private void yourMethod() {
HttpUrlConnection connection = createConnection();
try (AutocloseWrapper wrapper = new AutocloseWrapper(connection, HttpUrlConnection::disconnect)) {
// do your stuff with the connection
}
// connection.disconnect() will have been called here
}
A coworker just unsettled me concerning finally blocks. He claimed that if multiple resources are closed inside a finally block, I do not have to worry about exception handling.
So if I close my resources like this
try {
// do stuff
} catch(Exception e) {
// handle stuff
} finally {
resource1.close();
resource2.close();
}
and an exception occurs at resource1.close(), will the close() method of resource2 get called?
A simple check would confirm:
class MyResource implements AutoCloseable {
private final String name;
MyResource(String name) { this.name = name; }
#Override public void close() throws IOException {
System.out.println("Closing " + name);
throw new IOException();
}
}
public static void main(String[] args) throws IOException {
MyResource a = new MyResource("a");
MyResource b = new MyResource("b");
try {
} finally {
a.close();
b.close();
}
}
This would print "Closing a" and then print a stack trace; "Closing b" would not be printed. In contrast:
try (MyResource a = new MyResource("a");
MyResource b = new MyResource("b")) {
}
would print both.
That depends. If the only exception throwing things (explicitly or potentially) you have inside your try-catch block are close operations, you wouldn't need exception handling. However, most of the times, the close operations are themselves declared as throwing exceptions, thus, you'd need to put them inside a try-catch block anyway.
I have a class which looks like that:
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
public class ConnectionPool {
private HikariDataSource hds;
private final String propertyFileName;
public ConnectionPool(String propertyFileName) {
if (propertyFileName == null) {
throw new IllegalArgumentException("propertyFileName can't be null");
}
this.propertyFileName = propertyFileName;
reloadFile();
}
public void reloadFile() {
if (hds != null) {
hds.close();
}
hds = new HikariDataSource(new HikariConfig(propertyFileName));
}
public HikariDataSource getHikariDataSource() {
return hds;
}
public String getPropertyFileName() {
return propertyFileName;
}
public void executeQuery(final String sql, final CallBack<ResultSet, SQLException> callBack) {
new Thread(new Runnable() {
#Override
public void run() {
Connection connection = null;
PreparedStatement preparedStatement = null;
ResultSet resultSet = null;
try {
connection = hds.getConnection();
preparedStatement = connection.prepareStatement(sql);
resultSet = preparedStatement.executeQuery();
callBack.call(resultSet, null);
} catch (SQLException e) {
callBack.call(null, e);
} finally {
if (resultSet != null) {
try {
resultSet.close();
} catch (SQLException ignored) {}
}
if (preparedStatement != null) {
try {
preparedStatement.close();
} catch (SQLException ignored) {}
}
if (connection != null) {
try {
connection.close();
} catch (SQLException ignored) {}
}
}
}
}).start();
}
public void executeUpdate(final String sql, final CallBack<Integer, SQLException> callBack) {
//TODO
}
public void execute(final String sql, final CallBack<Boolean, SQLException> callBack) {
//TODO
}
public void connection(final String sql, final CallBack<Connection, SQLException> callBack) {
//TODO
}
}
The problem is that the reloadFile() method can be called from a different thread as hds is used. So it's possible that hds is closed while I use a connection object of it in another thread. What's the best way to solve this problem? Should I wait a few seconds after creating the new HikariDataSource object befor closing the old one (until the queries are finished)?
Edit: Another question: Should hds be volatile, so that the changes of hds are visible for all threads?
Have had a very very quick and brief look in the source code in HikariDataSource. In its close(), it is calling its internal HikariPool's shutdown() method, for which it will try to properly close the pooled connections.
If you want to even avoid any chance that in-progress connection from force closing, one way is to make use of a ReadWriteLock:
public class ConnectionPool {
private HikariDataSource hds;
private ReentrantReadWriteLock dsLock = ....;
//....
public void reloadFile() {
dsLock.writeLock().lock();
try {
if (hds != null) {
hds.close();
}
hds = new HikariDataSource(new HikariConfig(propertyFileName));
} finally {
dsLock.writeLock().unlock();
}
}
public void executeQuery(final String sql, final CallBack<ResultSet, SQLException> callBack) {
new Thread(new Runnable() {
#Override
public void run() {
Connection connection = null;
PreparedStatement preparedStatement = null;
ResultSet resultSet = null;
dsLock.readLock().lock();
try {
connection = hds.getConnection();
// ....
} catch (SQLException e) {
callBack.call(null, e);
} finally {
// your other cleanups
dsLock.readLock().unlock();
}
}
}).start();
}
//....
}
This will make sure that
multiple thread can access your datasource (to get connection etc)
Reload of datasource needs to wait until thread using the datasource to complete
No thread is able to use the datasource to get connection when it is reloading.
Why exactly are you trying to cause HikariCP to reload? Many of the important pool parameters (minimumIdle,maximumPoolSize,connectionTimeout,etc.) are controllable at runtime through the JMX bean without restarting the pool.
Restarting the pool is a good way to "hang" your application for several seconds while connections are closed and rebuilt. If you can't do what you need through the JMX interface, Adrian's suggestion seems like quite a reasonable solution.
Other solutions are possible, but have more complexity.
EDIT: Just for my own entertainment, here is the more complex solution...
public class ConnectionPool {
private AtomicReference<HikariDataSource> hds;
public ConnectionPool(String propertyFileName) {
hds = new AtomicReference<>();
...
}
public void reloadFile() {
final HikariDataSource ds = hds.getAndSet(new HikariDataSource(new HikariConfig(propertyFileName)));
if (ds != null) {
new Thread(new Runnable() {
public void run() {
ObjectName poolName = new ObjectName("com.zaxxer.hikari:type=Pool (" + ds.getPoolName() + ")");
MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
HikariPoolMXBean poolProxy = JMX.newMXBeanProxy(mBeanServer, poolName, HikariPoolMXBean.class);
poolProxy.softEvictConnections();
do {
Thread.sleep(500);
} while (poolProxy.getActiveConnections() > 0);
ds.close();
}
}).start();
}
}
public HikariDataSource getHikariDataSource() {
return hds.get();
}
public void executeQuery(final String sql, final CallBack<ResultSet, SQLException> callBack) {
new Thread(new Runnable() {
#Override
public void run() {
...
try {
connection = getHikariDataSource().getConnection();
...
}
}
}).start();
}
}
This will swap out the pool (atomically) and will start a thread that waits until all active connections have returned before shutting down the orphaned pool instance.
This assumes that you let HikariCP generate unique pool names, i.e. do not set poolName in your properties, and that registerMbeans=true.
A few options:
Synchronize all access to the data source so that only one thread can ever be messing with it. Not scaleable, but workable.
Roll your own connection pooling, such as Apache Commons Pooling so that each access, regardless of thread, requests a data source and pooling creates one as necessary. Can mess with data ACID, just depends on whether dirty data is needed, when data is flushed, transactionality, etc.
Each thread could also have its own data source using ThreadLocal so that each thread is totally independent of each other. Again, quality of data might be an issue, resources might be an issue if you've got "lots" of threads (depends on your definition) and too many open connections cause resource issues on either the client or server.
I have these 4 methods which each close a different connection type, meaning each of those has a different input object. Each method calls the close() method on the provided input object. Is there a way to combine those methods into one, taking in a generic object? I have no way to implement an interface on the provided objects or extend them.
import java.sql.Statement;
import javax.jms.Connection;
import javax.mail.Transport;
private void close(Statement stm) {
if(stm == null) {
return;
}
try {
stm.close();
} catch (SQLException ex) {
logger.error("Error while closing statement", ex);
}
}
private void close(java.sql.Connection con) {
if(con == null) {
return;
}
try {
con.close();
} catch (SQLException ex) {
logger.error("Error while closing connection", ex);
}
}
private void close(javax.jms.Connection con) {
if(con == null) {
return;
}
try {
con.close();
} catch(JMSException ex) {
logger.error("Error while closing JMS connection", ex);
}
}
private void close(Transport transport) {
if(transport == null) {
return;
}
try {
transport.close();
} catch (MessagingException ex) {
logger.error("Error while closing mail transport", ex);
}
}
EDIT:
Thank you for your answers regarding Java 1.7. Unfortunately our servers are running Java 1.6, so is there any solution for that?
You can use reflection.
First sample give you support for try() with resources:
#Test
public void testRes() throws Exception {
try(ClosableWrapper<StringWriter> rs = new ClosableWrapper<>(new StringWriter())){
Writer wr = rs.getResource();
}
}
static class ClosableWrapper<T> implements AutoCloseable{
private T resource;
private String closeMethod = "close";
public ClosableWrapper(T resource) {
this.resource = resource;
}
public ClosableWrapper(T resource, String closeMethod) {
this.resource = resource;
this.closeMethod = closeMethod;
}
public T getResource() {
return resource;
}
#Override
public void close() throws Exception {
if(resource!=null){
Method m = resource.getClass().getMethod(closeMethod);
m.invoke(resource);
}
}
}
or just one method:
public void close(Object resource) throws Exception {
if(resource!=null){
Method m = resource.getClass().getMethod("close");
m.invoke(resource);
}
}
Assuming these are your classes, use the AutoCloseable interface and put them in a try-with-resource.