Develop Reference

Ignite Server Node with CacheJdbcPojoStore / Delay Client Node

I'm using Spring and Ignite Spring to run a pretty simple cluster consisting of one server node and multiple client nodes with varying domain logic.
Ignite configuration:
All nodes connect via TcpCommuncationSpi, TcpDiscoverySpi and TcpDiscoveryVmIpFinder. TcpDiscoveryVmIpFinder.setAddresses only contain the server node.
The server node has serveral CacheJdbcPojoStore configured, the database data is loaded after calling Ignition.start(cfg) using ignite.cache("mycachename").loadCache(null).
After a client node is connected to the server node, it does some domain specific checks to verify data integrity. This works very well if I first start the server node, wait for all data to be loaded and then start the client nodes.
My problem: if I first start the client nodes, they can't connect to the server node as it is not yet started. They patiently wait for the server node to come up. If I now start the server node, the client nodes connect directly after Ignition.start(cfg) is done on the server node, but BEFORE the CacheJdbcPojoStores are done loading their data. Thus the domain specific integrity checks do fail as there is no data present in the caches yet.
My goal: I need a way to ensure the client nodes are only able to connect to the server node AFTER all data is loaded in the server node. This would simply the deployment process as well as local development a lot, as there would be no strict ordering of starting the nodes.
What I tried so far: Fiddling around with ignite.cluster().state(ClusterState.INACTIVE) as well as setting up a manually controlled BaselineTopology. Sadly, both methods simple declare the cluster as not being ready yet and thus I can't even create the caches, let alone load data.
My question: Is there any way to achieve either:
hook up into the the startup process of the server node so I can load the data in BEFORE it joins the cluster
Declare the server node "not yet ready" until the data is loaded.

Use one of the available data structures like AtomicLong or Semaphore to simulate readiness state. There is no need for internal hooks.
Added an example below:
Client:
while (true) {
// try get "myAtomic" and check its value
IgniteAtomicLong atomicLong = ignite.atomicLong("myAtomic", 0, false);
if (atomicLong != null && atomicLong.get() == 1) {
// initialization is complete
break;
}
// not ready
Thread.sleep(1000);
}
Server:
...
//loading is complete, create atomic sequence and set its value to 1
ignite.atomicLong("myAtomic", 1, true);
...
UPDATE#2
instead of having a self-written-loop including Thread.sleep, latch.await() can be called. See: countdownlatch

Hazelcast in SpringBoot Admin to run 2 instances via Docker Swarm

I'm very new to SpringBoot Admin, HazelCast and Docker Swarm...
What I'm trying to do is to run 2 instances of SpringBoot Admin Server, in Docker Swarm.
It works fine with one instance. I have every feature of SBA working well.
If I set the number of replicas to "2", with the following in swarm, then the logging in page doesn't work (it shows up but I can't log in, with no error in the console):
mode: replicated
replicas: 2
update_config:
parallelism: 1
delay: 60s
failure_action: rollback
order: start-first
monitor: 60s
rollback_config:
parallelism: 1
delay: 60s
failure_action: pause
order: start-first
monitor: 60s
restart_policy:
condition: any
delay: 60s
max_attempts: 3
window: 3600s
My current HazelCast config is the following (as specified in SpringBoot Admin doc):
#Bean
public Config hazelcast() {
// This map is used to store the events.
// It should be configured to reliably hold all the data,
// Spring Boot Admin will compact the events, if there are too many
MapConfig eventStoreMap = new MapConfig(DEFAULT_NAME_EVENT_STORE_MAP).setInMemoryFormat(InMemoryFormat.OBJECT)
.setBackupCount(1).setEvictionPolicy(EvictionPolicy.NONE)
.setMergePolicyConfig(new MergePolicyConfig(PutIfAbsentMapMergePolicy.class.getName(), 100));
// This map is used to deduplicate the notifications.
// If data in this map gets lost it should not be a big issue as it will atmost
// lead to
// the same notification to be sent by multiple instances
MapConfig sentNotificationsMap = new MapConfig(DEFAULT_NAME_SENT_NOTIFICATIONS_MAP)
.setInMemoryFormat(InMemoryFormat.OBJECT).setBackupCount(1).setEvictionPolicy(EvictionPolicy.LRU)
.setMergePolicyConfig(new MergePolicyConfig(PutIfAbsentMapMergePolicy.class.getName(), 100));
Config config = new Config();
config.addMapConfig(eventStoreMap);
config.addMapConfig(sentNotificationsMap);
config.setProperty("hazelcast.jmx", "true");
// WARNING: This setups a local cluster, you change it to fit your needs.
config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(true);
TcpIpConfig tcpIpConfig = config.getNetworkConfig().getJoin().getTcpIpConfig();
tcpIpConfig.setEnabled(true);
// NetworkConfig network = config.getNetworkConfig();
// InterfacesConfig interfaceConfig = network.getInterfaces();
// interfaceConfig.setEnabled( true )
// .addInterface( "192.168.1.3" );
// tcpIpConfig.setMembers(singletonList("127.0.0.1"));
return config;
}
```
I guess these inputs are not enough for you to properly help, but since I don't really weel understand the way HazelCast is workging, I don't really know what is useful or not. So please don't hesitate to ask me for what is needed to help! :)
Do you guys have any idea of what I'm doing wrong?
Many thanks!

Multicast is not working in Docker Swarm in default overlay driver (at least it stated here).
I have tried to make it run with weave network plugin but without luck.
In my case, it was enough to switch Hazelcast to the TCP mode and provide the network I like to search for the other replicas.
Something like that:
-Dhz.discovery.method=TCP
-Dhz.network.interfaces=10.0.251.*
-Dhz.discovery.tcp.members=10.0.251.*

Cassandra behavior on contact point based on data center

Cassandra setup in 3 data-center (dc1, dc2 & dc3) forming a cluster
Running a Java Application on dc1.
dc1 application has Cassandra connectors pointed to dc1 (ips of cassandra in dc1 alone given to the application)
turning off the dc1 cassandra nodes application throws exception in application like
All host(s) tried for query failed (no host was tried)
More Info:
cassandra-driver-core-3.0.8.jar
netty-3.10.5.Final.jar
netty-buffer-4.0.37.Final.jar
netty-codec-4.0.37.Final.jar
netty-common-4.0.37.Final.jar
netty-handler-4.0.37.Final.jar
netty-transport-4.0.37.Final.jar
Keyspace : Network topology
Replication : dc1:2, dc2:2, dc3:2
Cassandra Version : 3.11.4

Here are some things I have found out with connections and Cassandra (and BTW, I believe Cassandra has one of the best HA configurations of any database I've worked with over the past 25 years).
1) Ensure you have all of the components specified in your connection connection. Here is an example of some of the connection components, but there are others as well (maybe you've already done this):
cluster = Cluster.builder()
.addContactPoints(nodes.split(","))
.withCredentials(username, password)
.withPoolingOptions(poolingOptions)
.withLoadBalancingPolicy(
new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder()
.withLocalDc("MYLOCALDC")
.withUsedHostsPerRemoteDc(1)
.allowRemoteDCsForLocalConsistencyLevel()
.build()
)
).build();
2) Unless the entire DC you're "working in" is down, you could receive errors. Cassandra doesn't fail over to alternate DCs unless every node is down in the DC. If less than all nodes are down and your client can't satisfy the client CL settings, you will receive errors. I was actually hoping, when I did testing a while back, that if you couldn't achieve client CL in the LOCAL DC (even if some nodes in the current DC were up) and alternate DCs could, that it would automatically fail over, but this is not the case (since I last tested).
Maybe that helps?
-Jim

How to detect when Apache Zookeeper sessions are lost or timed out?

Assume we have an Apache Zookeeper quorum up and running and n client nodes connected (using Apache Curator). Is it possible to receive notifications on one of the nodes (the one we are observing) from zookeeper when any of the other nodes sessions are terminated or a timeout is reached? If so, how is this accomplished?

The answer is fairly simple and can be accomplished using Ephemeral nodes and PathChildrenCache. Zookeeper will detect when a node times out (in this example we set the timeout to 10s) and the associated ephemeral node will disappear from the tree. This will fire off an event which we can listen for.
First establish a connection with the curator client and start it up on all nodes
CuratorFramework curator =
CuratorFrameworkFactory
.newClient(zkConnectionString, 10000, 10000, retryPolicy);
curator.start();
curator.getZookeeperClient().blockUntilConnectedOrTimedOut();
Next use PathChildrenCache to assign listeners for zookeeper events. Event types include CHILD_ADDED, CHILD_UPDATED, and CHILD_REMOVED. The event object in the callback will contain the relevant information (and possible associated payload) of the node which went down.
PathChildrenCache pathCache = new PathChildrenCache(curator, "/nodes", true);
pathCache
.getListenable()
.addListener((curator, event) -> {
if (event.getType() == Type.CHILD_REMOVED) {
System.out.println("Child has been removed");
}
});
pathCache.start();
Now on a remote node, add the ephemeral node (here we give it an ID of 33 without a payload)
curator
.create()
.creatingParentsIfNeeded()
.withMode(CreateMode.EPHEMERAL)
.forPath("/nodes/33");
Now pull the plug on the remote node and the event should be detected where the listeners have been assigned.

Cassandra Hector Load balancing

I have setup a Cassandra cluster with two nodes recently. The replication factor is set to 2 and they both seem to be working well if both the nodes are turned on.
Now how can I use hector in such a way so that it keeps working as far as atleast one node is up? As of now I have something like following.
CassandraHostConfigurator cassandraHostConfigurator = new CassandraHostConfigurator(
"localhost:9160,xx.xx.13.22:9160");
cassandraHostConfigurator.setMaxActive(20);
cassandraHostConfigurator.setMaxIdle(5);
cassandraHostConfigurator.setCassandraThriftSocketTimeout(3000);
cassandraHostConfigurator.setMaxWaitTimeWhenExhausted(4000);
Cluster cluster = HFactory.getOrCreateCluster("structspeech",
cassandraHostConfigurator);
Keyspace keyspace = HFactory.createKeyspace("structspeech", cluster);
....
Let's say if host xx.xx.13.22 goes down then I am getting the following message in my console and all my inserts are failing untill that node comes up.
Downed xx.xx.13.22(xx.xx.13.22):9160 host still appears to be down: Unable to open transport to xx.xx.13.22(xx.xx.13.22):9160 , java.net.ConnectException: Connection refused: connect
This is how my keyspace is defined
update keyspace structspeech with placement_strategy =
'org.apache.cassandra.locator.SimpleStrategy'
and strategy_options =[{replication_factor:2}];
I am sure I am missing something very trivial, any help will be greatly appreciated.
Thanks

By default Hector uses a consistency level of Quorum so if one of your nodes is down this level cannot be satisfied.
When RF = 2 quorum means you need to read and write to both nodes, so if one of them is down you can't execute.
Here's a nice online tool that demonstrates NRW (N = replication factor, R = read consistency and W = write consistency) http://www.ecyrd.com/cassandracalculator/
To change the consistency level while writing/reading use, for example AllOneConsistencyLevelPolicy HFactory.createKeyspace(String, Cluster, ConsistencyLevelPolicy)

What consistency level are you using when you insert? If you are writing at QUORUM or ALL, you need both nodes to be up to write with a replication factor of 2 (a quorum for 2 nodes is 2, that's why typical cassandra clusters use an odd number for replication factor)