We are running an AWS RDS Aurora/MySQL database in a cluster with a writer and a reader instance where the writer is replicated to the reader.
The application accessing the database is a standard java application using a HikariCP Connection Pool. The pool is configured to use a "SELECT 1" test query on checkout.
What we noticed is that once in a while RDS fails over the writer to the reader. The failover can also be replicated manually by clicking "Instance Actions/Failover" in the AWS console.
The connection pool is not able to detect the failover and the fact that it is now connected to a reader database, as the "SELECT 1" test queries still succeed. However any subsequent database updates fail with "java.sql.SQLException: The MySQL server is running with the --read-only option so it cannot execute this statement" errors.
It appears that instead of a "SELECT 1" test query, the Connection Pool can detect that it is now connected to the reader by using a "SELECT count(1) FROM test_table WHERE 1 = 2 FOR UPDATE" test query.
Has anybody experienced the same issue?
Are there any downsides on using "FOR UPDATE" in the test query?
Are there any alternate or better approaches of handling an AWS RDS cluster writer/reader failover?
Your help is much appreciated
Bernie
I've been giving this a lot of thought in the two months since my original reply...
How Aurora endpoints work
When you start up an Aurora cluster you get multiple hostnames to access the cluster. For the purposes of this answer, the only two that we care about are the "cluster endpoint," which is read-write, and the "read-only endpoint," which is (you guessed it) read-only. You also have an endpoint for each node within the cluster, but accessing nodes directly defeats the purpose of using Aurora, so I won't mention them again.
For example, if I create a cluster named "example", I'll get the following endpoints:
Cluster endpoint: example.cluster-x91qlr44xxxz.us-east-1.rds.amazonaws.com
Read-only endpoint: example.cluster-ro-x91qlr44xxxz.us-east-1.rds.amazonaws.com
You might think that these endpoints would refer to something like an Elastic Load Balancer, which would be smart enough to redirect traffic on failover, but you'd be wrong. In fact, they're simply DNS CNAME entries with a really short time-to-live:
dig example.cluster-x91qlr44xxxz.us-east-1.rds.amazonaws.com
; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> example.cluster-x91qlr44xxxz.us-east-1.rds.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40120
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;example.cluster-x91qlr44xxxz.us-east-1.rds.amazonaws.com. IN A
;; ANSWER SECTION:
example.cluster-x91qlr44xxxz.us-east-1.rds.amazonaws.com. 5 IN CNAME example.x91qlr44xxxz.us-east-1.rds.amazonaws.com.
example.x91qlr44xxxz.us-east-1.rds.amazonaws.com. 4 IN CNAME ec2-18-209-198-76.compute-1.amazonaws.com.
ec2-18-209-198-76.compute-1.amazonaws.com. 7199 IN A 18.209.198.76
;; Query time: 54 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Fri Dec 14 18:12:08 EST 2018
;; MSG SIZE rcvd: 178
When a failover happens, the CNAMEs are updated (from example to example-us-east-1a):
; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> example.cluster-x91qlr44xxxz.us-east-1.rds.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27191
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;example.cluster-x91qlr44xxxz.us-east-1.rds.amazonaws.com. IN A
;; ANSWER SECTION:
example.cluster-x91qlr44xxxz.us-east-1.rds.amazonaws.com. 5 IN CNAME example-us-east-1a.x91qlr44xxxz.us-east-1.rds.amazonaws.com.
example-us-east-1a.x91qlr44xxxz.us-east-1.rds.amazonaws.com. 4 IN CNAME ec2-3-81-195-23.compute-1.amazonaws.com.
ec2-3-81-195-23.compute-1.amazonaws.com. 7199 IN A 3.81.195.23
;; Query time: 158 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Fri Dec 14 18:15:33 EST 2018
;; MSG SIZE rcvd: 187
The other thing that happens during a failover is that all of the connections to the "cluster" endpoint get closed, which will fail any in-process transactions (assuming that you've set reasonable query timeouts).
The connections to the "read-only" endpoint don't get closed, which means that whatever node gets promoted will get read-write traffic in addition to read-only traffic (assuming, of course, that your application doesn't just send all requests to the cluster endpoint). Since read-only connections are typically used for relatively expensive queries (eg, reporting), this may cause performance problems for your read-write operations.
The Problem: DNS Caching
When failover happens, all in-process transactions will fail (again, assuming that you've set query timeouts). There will be a short amount of time that any new connections will also fail, as the connection pool attempts to connect to the same host before it's done with recovery. In my experience, failover takes around 15 seconds, during which time your application shouldn't expect to get a connection.
After that 15 seconds (or so), everything should return to normal: your connection pool attempts to connect to the cluster endpoint, it resolves to the IP address of the new read-write node, and all is well. But if anything prevents resolving that chain of CNAMEs, you may find that your connection pool makes connections to a read-only endpoint, which will fail as soon as you try an update operation.
In the case of the OP, he had his own CNAME with a longer timeout. So rather than connect to the cluster endpoint directly, he would connect to something like database.example.com. This is a useful technique in a world where you would manually fail-over to a replica database; I suspect it's less useful with Aurora. Regardless, if you use your own CNAMEs to refer to database endpoints, you need them to have short time-to-live values (certainly no more than 5 seconds).
In my original answer, I also pointed out that Java caches DNS lookups, in some cases forever. The behavior of this cache depends on (I believe) the version of Java, and also whether you're running with a security manager installed. With OpenJDK 8 running as an application, it appears that the JVM will delegate all naming lookups and not cache anything itself. However, you should be familiar with the networkaddress.cache.ttl system property, as described in this Oracle doc and this SO question.
However, even after you've eliminated any unexpected caches, there may still be times where the cluster endpoint is resolved to a read-only node. That leaves the question of how you handle this situation.
Not-so-good solution: use a read-only test on checkout
The OP was hoping to use a database connection test to verify that his application was running on a read-only node. This is surprisingly hard to do: most connection pools (including HikariCP, which is what the OP is using) simply verify that the test query executes successfully; there's no ability to look at what it returns. This means that any test query has to throw an exception to fail.
I haven't been able to come up with a way to make MySQL throw an exception with just a stand-alone query. The best I've come up with is to create a function:
DELIMITER EOF
CREATE FUNCTION throwIfReadOnly() RETURNS INTEGER
BEGIN
IF ##innodb_read_only THEN
SIGNAL SQLSTATE 'ERR0R' SET MESSAGE_TEXT = 'database is read_only';
END IF;
RETURN 0;
END;
EOF
DELIMITER ;
Then you call that function in your test query:
select throwIfReadOnly()
This works, mostly. When running my test program I could see a series of "failed to validate connection" messages, but then, inexplicably, the update query would run with a read-only connection. Hikari doesn't have a debug message to indicate which connection it hands out, so I couldn't identify whether it had allegedly passed validation.
But aside from that possible problem, there's a deeper issue with this implementation: it hides the fact that there's a problem. A user makes a request, and maybe waits for 30 seconds to get a response. There's nothing in the log (unless you enable Hikari's debug logging) to give a reason for this delay.
Moreover, while the database is inaccessible Hikari is furiously trying to make connections: in my single-threaded test, it would attempt a new connection every 100 milliseconds. And these are real connections, they simply go to the wrong host. Throw in an app-server with a few dozen or hundred threads, and that could cause a significant ripple effect on the database.
Better solution: use a read-only test on checkout, via a wrapper Datasource
Rather than let Hikari silently retry connections, you could wrap the HikariDataSource in your own DataSource implementation and test/retry yourself. This has the benefit that you can actually look at the results of the test query, which means that you can use a self-contained query rather than calling a separately-installed function. It also lets you log the problem using your preferred log levels, lets you pause between attempts, and gives you a chance to change pool configuration.
private static class WrappedDataSource
implements DataSource
{
private HikariDataSource delegate;
public WrappedDataSource(HikariDataSource delegate) {
this.delegate = delegate;
}
#Override
public Connection getConnection() throws SQLException {
while (true) {
Connection cxt = delegate.getConnection();
try (Statement stmt = cxt.createStatement()) {
try (ResultSet rslt = stmt.executeQuery("select ##innodb_read_only")) {
if (rslt.next() && ! rslt.getBoolean(1)) {
return cxt;
}
}
}
// evict connection so that we won't get it again
// should also log here
delegate.evictConnection(cxt);
try {
Thread.sleep(1000);
}
catch (InterruptedException ignored) {
// if we're interrupted we just retry
}
}
}
// all other methods can just delegate to HikariDataSource
This solution still suffers from the problem that it introduces a delay into user requests. True, you know that it's happening (which you didn't with the on-checkout test), and you could introduce a timeout (limit the number of times through the loop). But it still represents a bad user experience.
The best (imo) solution: switch into "maintenance mode"
Users are incredibly impatient: if it takes more than a few seconds to get a response back, they'll probably try to reload the page, or submit the form again, or do something that doesn't help and may hurt.
So I think the best solution is to fail quickly and let them know that somethng's wrong. Somewhere near the top of the call stack you should already have some code that responds to exceptions. Maybe you just return a generic 500 page now, but you can do a little better: look at the exception, and return a "sorry, temporarily unavailable, try again in a few minutes" page if it's a read-only database exception.
At the same time, you should send a notification to you ops staff: this may be a normal maintance window failover, or it may be something more serious (but don't wake them up unless you have some way of knowing that it's more serious).
set connection pool idle connection timeout in your java code datasource. set around 1000ms
Aurora failover
As Sayantan Mandal hints in his comments. When using Aurora just use the MariaDb driver it has support for failover.
It is documented here:
https://aws.amazon.com/blogs/database/using-the-mariadb-jdbc-driver-with-amazon-aurora-with-mysql-compatibility/
And here:
https://mariadb.com/kb/en/failover-and-high-availability-with-mariadb-connector-j/#aurora-endpoints-and-discovery
Your connection string will start with jdbc:mariadb:aurora// or jdbc:mysql:aurora//.
The connection pool normally calls JDBC4Connection#isValid which should correctly return false with this driver when on a read only replica.
No custom code required.
DNS Caching
As for DNS caching (networkaddress.cache.ttl) depending on your JVM the default is 30 or 60s depening of whether a security manager is present.
You can retrieve the value at runtime with this snippet if unsure:
Class.forName("sun.net.InetAddressCachePolicy").getMethod("get").invoke(null)
With 30s DNS cache, your connection will start to arrive at the read-write replica at most 30s after the failover happens.
I am working with a Spring LDAP application for which I did none of the setup of the LDAP workings, but now I need to add a failover feature.
We supply our ContextSource with two space-seperated URLs:
String theseUrls = primaryLdapUrl + " " + secondaryLdapUrl;
environment.put("java.naming.provider.url", theseUrls);
ilc = new InitialLdapContext(environment, null);
If the primary URL is functional, then it connects to that. If not, it connects to the secondary just fine. The connections are then pooled, however I am having trouble figuring out the exact mechanics. But, as it is, due to the pooling if the established connection goes down, the whole application shits the bed.
Is there a way to disable pooling, or create a short timeout for it? I have done some research but can't find exact mechanics that have worked for me (including trying to call setPooled(false)). Ideally, the secondary server is only queried if the first is down. When the first is restored, then it will go back to that.
NOTE: This URL (http://forum.spring.io/forum/spring-projects/data/ldap/34643-switching-ldap-contexts-for-failover) has given me a lot of ideas, but I can't get anything to work.
I am attempting to upgrade our code base from Hibernate 3.6 to 4.0 (just as a first step into getting us more up to date). I am hitting an issue where a COMMIT is not being issued even though we are calling the commit() after making sure the transaction is isActive().
When running some queries successfully against the Postgres database, I see this in the Postgres logs:
2014-12-30 20:09:39 CST LOG: execute <unnamed>: SET extra_float_digits=3
2014-12-30 20:09:39 CST LOG: execute S_1: BEGIN
2014-12-30 20:09:39 CST LOG: execute <unnamed>: -- This script........
Notice the BEGIN there, and then here's an example of the simple commit call:
if (sf.getCurrentSession().getTransaction().isActive()) {
sf.getCurrentSession().getTransaction().commit();
}
I have debugged down into AbstractTransactionImpl and am looking at a Jdbc4Connection commit() method, and see that the actual COMMIT call is being skipped.....but I don't know why. Here's the if statement that this is failing on (it is in AbstractJdbc2Connection).
if (protoConnection.getTransactionState() != ProtocolConnection.TRANSACTION_IDLE)
executeTransactionCommand(commitQuery);
So, apparently our transactionstate is == ProtocolConnection.TRANSACTION_IDLE. However, I'm not entirely sure what that entails, and why are we getting this issue when the transaction says it isActive()?
NOTE: this same exact code worked for us on Hibernate 3.6.
Thanks.
UPDATE: I debugged further, and it looks like there are many ProtocoalConnectionImpl objects being constructed, does that indicate a problem on our software side that is doing something it shouldn't? Like does that indicate that we're opening connections that are just hanging around? Thanks for any more insight.
So it turns out I was doing something incorrect (surprise). There's a good reason that the TransactionState was showing as TRANSACTION_IDLE and not issuing a commit.
I had seen in multiple places on the internetz where people were getting access to a JDBC connection in the new Hibernate 4.x land by getting access to the underlying ConnectionProvider (sort of like this: https://stackoverflow.com/a/21271019/115694).
However, the problem with doing that is that you are getting a BRAND NEW connection....instead of the connection associated with the current session. This is a pretty big no-no, or at least for us it was.
You do have to in fact do what the person says in this answer:
https://stackoverflow.com/a/3526626/115694
Basically, you MUST use the Work API.
I am currently working on a java project which implements web-scraping and I am facing a weird issue so far.
Here is what I do :
Get an URL Connection with a page of a website
Parse the HTML code to get some content (OpenData)
Add the content in my database
Move onto the next page and go back to Step 1
This is actually very long and it can last for days so I need to let the script running. The problem is that sometimes, it stops for no reason (no errors, no messages, no window close ; It just litterally stops and I need to press one of my button to restart it). I have implemented a short code which restarts the application from where it stopped. I believe it's a connection problem to the database so I would like to know how could I fix it.
I use a static class which creates an instance of this class at the beginning of the application and then I use static methods from this class to run my queries like this for example :
ConnexionBDD.con.prepareStatement(query);
public static Connection loadDriver() {
try {
Class.forName(Driver);
con = DriverManager.getConnection(ConnectionString, user, pwd);
} catch (ClassNotFoundException e) {
System.err.println("Classe not found : Class.forName(...)");
} catch (SQLException e) {
System.err.println(e.getMessage());
}
return con;
}
I am not sure I am doing the right thing to make my connection lasts forever (in theory) and eventually close it when It has finished to iterate over my links.
You're jumping the gun a bit here. There's no evidence that the database connection is actually the problem. Usually if you were having DB connection issues you'd be getting an exception from the connection when you try to perform operations on it, a timeout, etc.
You need to:
Add detailed logging to your application, so you can see what it's doing as it progresses, and what it's trying to do when it stops; and
Run it with -Xdebug and other suitable options for remote debugging, so you can attach a debugger to it when it stops and examine its state to see what it is doing at the time. Use the debugger user interface from NetBeans, Eclipse, or whatever you prefer to attach to the program when the logging indicates that it's stopped progressing.
For logging, you can use java.util.logging. See the javadoc and the logging overview docs.
Here's an example of how to do remote debugging with Eclipse. You'll be able to find similar guides for your chosen IDE. Java also has a command line debugger, but it's pretty painful.
You also need to check to see whether the program might be crashing or exiting, rather than just stopping working. You should capture any standard error output from the program and check the program's error return code from the shell. Also look for hs_error files in the directory the program runs in, in case there's a JVM crash, though that should generate output on stderr as well.
You should also:
Set an application_name when you establish a connection to PostgreSQL, so you can easily see what your client is doing with the database. You can specify application_name as a JDBC connection parameter, or run a SET application_name = 'blah' statement after connecting.
When logging (or however you currently tell that your program is no longer progressing) indicates that the program has stopped working, examine pg_stat_activity in the server, looking at the entry/entries for your application. See if the connection is idle, idle in transaction, or running a statement, and what that statement is. If it's running a statement, query against pg_locks to see if it's blocked on an ungranted lock.
I presume i have a close to default HicariConfiguration with MaximumPoolSize(5).
The problem i faced with is there're a lot of attempts to connect to database even the first one failed. I mean, for instance, the password i'm going to use to connect to Oracle is wrong and connection fails, but then we have one more attempts to connect to database which lock the account as a result.
Question: What HicariCP setting is supposed to be used to limit up to 1 number of attempt to connect?
Thanks for any information!
### UPDATE
env.conf:
jdbc {
test1 {
datasourceClassName="oracle.jdbc.pool.OracleDataSource"
dataSourceUrl=.....jdbc url
dataSourceUser=USER
dataSourcePassword=password
setMaximumPoolSize = 5
setJdbc4ConnectionTest = true
}
}
Conf file is read by means of ConfigFactory, and create HicariConfig based on conf file (setDriverClassName etc).
Output of HikariConfig:
autoCommit.....................true
connectionTimeOut..............30000
idleTimeOut....................600000
initializationFailFast.........false
isolateInternalQueries.........false
jdbc4ConnectionTest............test
maxLifetime....................1800000
minimumIdle....................5
https://github.com/brettwooldridge/HikariCP/issues/312, As explained at the end of this issue, HikariCP will keep trying to acquire a connection. It removed the acquireRetries parameters deliberately. so the way is to configure the right username/password, since DB only lock after authenticaions failures.
Here's extracted from the issue. HikariCP intends to retry forever.
Back to acquireRetries... Without a concept of acquireRetries, how
long does the dedicated thread continue to try to create a new
connection? Forever. The background creation thread will continue to
try to add a connection to the pool forever, or until one of three
conditions is met: