I have created a Storm topology which connects to Redis Cluster using Jedis library. Storm component always expects that Redis is up and running and only then it connects to Redis and subscribes the events.Currently we use pub-sub strategy of Redis.
Below is the code sample that explains my Jedis Connectivity inside Storm to for Redis.
try {
jedis.psubscribe(listener, pattern);
} catch(Exception ex) {
//catch statement here.
} finally {
pool.returnResource(jedis);
}
....
pool = new JedisPool(new JedisPoolConfig(), host, port); //redis host port
ListenerThread listener = new ListenerThread(queue, pool, pattern);
listener.start();
EXPECTED BEHAVIOUR
Once Redis dies and comes back online, Storm is expected to identify the status of Redis. It must not need a restart in case when Redis die and come online.
ACTUAL BEHAVIOUR
Once Redis restarts due to any reason, I always have to restart the Storm topology as well and only then it starts listening back to Redis.
QUESTION
How can I make Storm listen and reconnect to Redis again after Redis is restarted? any guidance would be appreciated, viz. docs, forum answer.
Catch the exception for the connection lost error and set the pool to null
(Assume that you doing this in Spout) Use an if-else statement to check if pool is null then create a new instance of JedisPool() assigning to the pool like in your code:
pool = new JedisPool(new JedisPoolConfig(), host, port); //redis host port
If pool not null (means connected) then continue your work
This is a common issue with apache-storm where connection thread is alivein stale condition, although the source from where you are consuming is down/restarted. Ideally it should retry to create new connection thread instead reusing the existing one. Hence the Idea is to have it automated it by by detecting the Exception (e.g. JMSConnectionError in case of JMS).
refer this Failover Consumer Example which will give you brief idea what to do in such cases.(P.S this is JMS which would be JMS in redis your case.)
The Steps would be something like this.
Catch Exception in case of ERROR or connection lost.
Init connection (if not voluntarily closed by program) from catch.
If Exception got to step 1.
Related
I am trying to insert data into a table in SQL Server hosted on AWS RDS.
It was working fine and suddenly I started getting an issue. It seems like an intermittent issue but I am unable to see why it is this happening
Fail to read any response from the server, the underlying connection might get lost unexpectedly.
This is how I am creating the database connection:
public static MSSQLPool createMssqlDbPool(Vertx vertx, ConfigModel configModel) {
MSSQLConnectOptions connectOptions = new MSSQLConnectOptions()
.setHost(System.getenv().getOrDefault("DB_HOST", configModel.getDbConfig().getHost()))
.setPort(Integer.parseInt(System.getenv().getOrDefault("DB_PORT", configModel.getDbConfig().getPort())))
.setDatabase(System.getenv().getOrDefault("DB_NAME", configModel.getDbConfig().getDatabase()))
.setUser(System.getenv().getOrDefault("DB_USER", configModel.getDbConfig().getUser()))
.setPassword(System.getenv().getOrDefault("DB_PASSWORD", configModel.getDbConfig().getPassword()));
// Pool options
PoolOptions poolOptions = new PoolOptions()
.setMaxSize(4);
LOG.info("DB connection : {}", connectOptions.toJson());
return MSSQLPool.pool(vertx, connectOptions, poolOptions);
}
I have read threads on GitHub about adding timeout but they are not definitive.
If you see this error it is likely a pooled connection has been idle for too long and closed by some intermediate proxy.
Change your pool options to close idle connections eagerly:
// Pool options
PoolOptions poolOptions = new PoolOptions()
.setMaxSize(4)
.setIdleTimeout(5)
.setIdleTimeoutUnit(TimeUnit.MINUTES);
5 minutes is just an example. The longer a connection lives, the better.
I am trying to get Lettuce to connect to the newly promoted master (former slave) after the old one failed. But all writes stop. The writes continue after the failed host reconnects, now as a slave. And it continues to write to the new master (former slave).
I tried setting periodic topology refreshes, as well as adaptive ones on all events but it didn't help. Is there another setting I have to use?
This is how I configured the client:
final List<RedisURI> redisURIs = buildRedisURIs(redisServerSettings.getNodes());
final RedisClusterClient client = RedisClusterClient.create(clientResources, redisURIs);
final ClusterTopologyRefreshOptions refreshOptions =
ClusterTopologyRefreshOptions.builder()
.enableAllAdaptiveRefreshTriggers()
.adaptiveRefreshTriggersTimeout(Duration.ofMinutes(2))
.refreshTriggersReconnectAttempts(2)
.enablePeriodicRefresh(Duration.ofMinutes(10))
.build();
client.setOptions(ClusterClientOptions.builder().topologyRefreshOptions(refreshOptions).build());
I solved the problem.
Because lettuce doesn't have a timeout normally, it waited forever for the response from the server. Setting the timeout caused some transactions to fail but after the failed transactions, the reads and writes continued.
I want to know the number of queues on my ActiveMQ and I have this Code, which works with my local installation of ActiveMQ:
ActiveMQSslConnectionFactory connectionFactory = new ActiveMQSslConnectionFactory(url);
//connectionFactory.setTrustStore("truststore.ts");
//connectionFactory.setTrustStorePassword("password");
ActiveMQConnection connection = (ActiveMQConnection) connectionFactory.createConnection();
connection.start();
DestinationSource destinationQueues = connection.getDestinationSource();
Set<ActiveMQQueue> queues = destinationQueues.getQueues();
System.out.println(queues.size());
But when I'm using it on our ActiveMQ on the server (with the truststore) it will return zero queues. I know that the connection works because I can consume messages from one exact queue. I already tried tips from other threads like sleep but it still won't work.
Is it maybe because of the SSL Connection, do I have to change something in this case? Thank you in advance.
Found the problem. It's a setting of the ActiveMQ which is found in the activemq.xml. The part advisorySupport="false" caused the problem. After commenting this setting out, the query returns the true amount of queues.
The Destination Source bits make no guarantee of timely return of the full set of Queues etc. It could be you have configured the SSL connector on the broker differently or it could just be that due to the much slower throughput of SSL on JDK implementations the data hasn't arrived.
Currently i'm working on a standalone Java apps that connects to a Websphere MQ to send and receive messages.
The flow is in asynchronous mode, which we implemented using MessageListener class to retrieve the messages from the queue when they are ready. The code to initialize the consumer with the listener is as follow:
if(connection == null)
connection = getJmsConnection();
try {
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
if (isTopic) {
destination = session.createTopic(destinationName);
} else {
destination = session.createQueue(destinationName);
}
consumer = session.createConsumer(destination);
consumer.setMessageListener(listener);
} catch (JMSException e) {
e.printStackTrace();
}
The getJmsConnection() method will return a connection from a pool, implemented using Apache Commons Pool library.
My question is, will the connection assign to the listener from the pool be active and tied to that listener as long as the program is running? Or the connection is used intermittently and can be reuse by other processes? Our idea is to have the sending and receiving process to reuse the connection from the pool, but i'm not sure how the MessageListener deal with the connection they are assigned with.
Thank you.
The key object here is the session rather than the connection; the session is on the one that will be doing the primary work here with the message consumption (async or otherwise).
It is advisable to try and share out the connection as widely as possible. Temporary destinations are scoped on the connection level. So the use of pooling is a good idea; it will be perfectly possible to share that connection around.
However I would also say that it might be worth considering pooling the sessions. With the code here a new session will be created, each time through that code, that will mean a new connection to the WebSphere MQ queue manager will be created. It's not clear what the scope of that will be, but if that is closed quickly it could become a bottleneck.
I presume i have a close to default HicariConfiguration with MaximumPoolSize(5).
The problem i faced with is there're a lot of attempts to connect to database even the first one failed. I mean, for instance, the password i'm going to use to connect to Oracle is wrong and connection fails, but then we have one more attempts to connect to database which lock the account as a result.
Question: What HicariCP setting is supposed to be used to limit up to 1 number of attempt to connect?
Thanks for any information!
### UPDATE
env.conf:
jdbc {
test1 {
datasourceClassName="oracle.jdbc.pool.OracleDataSource"
dataSourceUrl=.....jdbc url
dataSourceUser=USER
dataSourcePassword=password
setMaximumPoolSize = 5
setJdbc4ConnectionTest = true
}
}
Conf file is read by means of ConfigFactory, and create HicariConfig based on conf file (setDriverClassName etc).
Output of HikariConfig:
autoCommit.....................true
connectionTimeOut..............30000
idleTimeOut....................600000
initializationFailFast.........false
isolateInternalQueries.........false
jdbc4ConnectionTest............test
maxLifetime....................1800000
minimumIdle....................5
https://github.com/brettwooldridge/HikariCP/issues/312, As explained at the end of this issue, HikariCP will keep trying to acquire a connection. It removed the acquireRetries parameters deliberately. so the way is to configure the right username/password, since DB only lock after authenticaions failures.
Here's extracted from the issue. HikariCP intends to retry forever.
Back to acquireRetries... Without a concept of acquireRetries, how
long does the dedicated thread continue to try to create a new
connection? Forever. The background creation thread will continue to
try to add a connection to the pool forever, or until one of three
conditions is met: