I have a running Ignite cluster and I use AWS S3 for node discovery:
TcpDiscoveryS3IpFinder ipFinder = new TcpDiscoveryS3IpFinder();
BasicAWSCredentials awsCredentials =
new BasicAWSCredentials(igniteAccessKey, igniteSecretAccessKey);
ipFinder.setAwsCredentials(awsCredentials);
ipFinder.setBucketName(igniteBucketName);
ipFinder.setBucketEndpoint("s3.eu-central-1.amazonaws.com");
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setIpFinder(ipFinder);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setClientMode(true);
cfg.setDiscoverySpi(spi);
Ignition.start(cfg);
It works very well and I can connect to this cluster using Apache Ignite Client nodes.
But what about Apache Ignite Thin Client? Thin client uses ClientConfiguration class (instead of IgniteConfiguration) that requires a list of IP addresses of cluster nodes. AFAIK one can only hardcode that list of IP addresses. From official documentation:
ClientConfiguration cfg = new ClientConfiguration().setAddresses("127.0.0.1:10800");
try (IgniteClient client = Ignition.startClient(cfg)) {
ClientCache<Integer, String> cache = client.cache("myCache");
// Get data from the cache
}
So I have questions:
How should I handle situations when a list of IP addresses change?
Is there any way to use node discovery for Thin clients?
You can use domain names as well as IP addresses. But you can't use node discovery with thin clients.
Related
server.propereties setup:
listeners=PLAINTEXT://:29092, SSL://:29093
SSL related set too done.
so that we can connect 29092 for plaintext and 29093 along with SSL setup.
Here am trying to produce data into port 29093 as below
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, System.getProperty("kafkaPort", "localhost:29093"));
//SSL related setup too done in props
Producer<Long, String> producer = new KafkaProducer<>(props, new LongSerializer(), new KafkaSerializer());
final ProducerRecord<Long, String> record = new ProducerRecord<Long, String>(System.getProperty("kafkaTopic", "dqerror"),
content);
RecordMetadata metadata = producer.send(record).get();
After publishing dqerror topic created in both also data get published in both
Data is published into two topic.
Actually, am trying to find is any possible to restrict to drop data into a specific port ?
Data is not published in "both" ports. There is only one Kafka cluster that is listening on two ports. There is one set of disks that the data is written into on your one broker.
Also, from what I can tell, there is only one topic used in your code.
If you want to restrict TCP traffic on any port, that would be a firewall rule from the OS, rather than any Kafka settings or Java code.
To configure a kafkaSpout, it Takes BrokerHosts which in turn take one zookeeper host.
BrokerHosts host = new ZkHosts("server-1:2181");
SpoutConfig spoutConfig = new SpoutConfig(host, TopologyConstants.KAFKA_QUEUE.SOURCE,
"/" + TopologyConstants.KAFKA_QUEUE.SOURCE, ID);
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
Problem is, If i have a zookeeper quorum(cluster of 3 zk servers), how do i configure KafkaSpout to take all the members of quorum instead of only 1.
As one zookeeper server may get down and the whole topology will be unavailable.
Found answer to my question:
connection string format is "host1:port1,host2:port2,host3:port3..." (storm-kafka uses Curator under the hood), so just supply multiple Zookeeper server urls to the ZkHosts constructor
I was trying to connect to JedisCluster (ElastiCache Redis) from java. But I was getting JedisConnectionException with No reachable node in the cluster.
Here was my code to connect to JedisCluster
public static void main(String[] args) throws IOException{
final GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
poolConfig.setMaxWaitMillis(2000);
poolConfig.setMaxTotal(300);
Set<HostAndPort> jedisClusterNode = new HashSet<HostAndPort>();
jedisClusterNode.add(new HostAndPort("mycachecluster.eaogs8.0001.usw2.cache.amazonaws.com",6379));
jedisClusterNode.add(new HostAndPort("mycachecluster.eaogs8.0002.usw2.cache.amazonaws.com",6379));
JedisCluster jedisCluster = new JedisCluster(jedisClusterNode,poolConfig);
System.out.println("Cluster Size...." + jedisCluster.getClusterNodes().size());
try{
jedisCluster.set("foo", "bar");
jedisCluster.get("foo");
}
catch(Exception e){
e.printStackTrace();
}
finally{
jedisCluster.close();
}
}
The exception I got after running this
redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:57)
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:74)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:116)
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31)
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103)
I have checked
telnet mycachecluster.eaogs8.0001.usw2.cache.amazonaws.com 6379
as mentioned in AWS Doc I got the reply as Connected.
What is the issue here and why I am not able to connect to the JedisCluster using java?
Note :
I am using jedis version 2.9.0
Update:
In AWS Encryption in-transit and Encryption at-rest are activated
So
Jedis jedis = null;
try{
jedis = new Jedis(URI.create("rediss://mycachecluster.eaogs8.0001.usw2.cache.amazonaws.com:6379"));
System.out.println(jedis.ping());
System.out.println("XXXXX: "+jedis.get("c"));
}
catch(Exception exception){
exception.printStackTrace();
}
finally{
jedis.close();
}
works fine. But not the jedis cluster.
From URI.create("rediss://..."), it's evident that you are using Redis SSL Scheme to create a successful connection by Jedis. But JedisCluster doesn't have SSL support yet.
There is a pending feature request regarding this.
The JedisCluster had some problems while connecting the redis cluster server with SSL enabled. Even with the latest revision(as of July 2020) we were getting the exception JedisNoReachableClusterNodeException. There are very less articles on the configurations required for various server requirements.
We needed the library in 2 languages, one in Java and the other in Python. For python I used the python redis-py-cluster. While for Java initially we tried with Jedis and then Jedis Cluster but both were not helpful.
So another library I found is
Lettuce
For a redis cluster server with SSL support the configuration is pretty straight forward and supports a builder pattern to construct the connection object with optional parameters. Here is the sample to create and connect to redis-cluster server
RedisURI redisURI = RedisURI.Builder.redis("<<Redis Server primary endpoint>>", 6379).withSsl(true).withVerifyPeer(false).build();
RedisClusterClient redisClient = RedisClusterClient.create(redisURI);
StatefulRedisClusterConnection<String, String> conn = redisClient.connect();
List<KeyValue<String, String>> res_1= conn.sync().mget(keys...)_
conn.close();
But note that if the redis server is a single node instance then even Jedis library is also good to use.
i am using ElasticSearch node client to perform queries but i want to connect to different IP address then localhost, I want to achieve this
Client client = TransportClient.builder().build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("host1"), 9300))
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("host2"), 9300));
thing in node client
Node node = nodeBuilder().clusterName("yourcluster").client(true).node();
Client client = node.client();
please help me, I want to use Node Client with other hosts not TransportClient
Node client automatically discovers all the nodes of the cluster because it's actually a part of the cluster itself, you don't need to list the hosts manually. But if your cluster resides in another network which can't be discovered automatically you may also specify at least one cluster host with settings.
Settings.Builder settings = Settings.builder()
.put("path.home", ".")
.put("discovery.zen.ping.unicast.hosts","192.168.1.1");
Node node = NodeBuilder.nodeBuilder().settings(settings)
.clusterName("elasticsearch")
.client(true)
.node();
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html
I have implemented elasticsearch with native client. This is my implementation:
Settings settings = Settings.settingsBuilder().put("cluster.name", "elasticsearch").build();
TransportClient client = TransportClient.builder().settings(settings).build();
Now I want to implement the same with Jest client. I have created the client, but I am missing cluster name:
JestClientFactory factory = new JestClientFactory();
factory.setHttpClientConfig(
new HttpClientConfig
.Builder("http://127.0.0.1:9301")
.multiThreaded(true)
.build()
);
JestClient client = factory.getObject();
Is there any way to implement with cluster name?
Jest uses the HTTP protocol since it's hitting the REST API, hence you don't need to specify the cluster name like you do with the native TransportClient does.
Also make sure to use the port 9201 and not 9301