Develop Reference

apache pulsar Direct approach with Apache spark

There are 2 approaches of integrating apache kafka with apache spark as mentioned here with pros and cons.
From what i have read there are 2 spark connector for apache pulsar
org.apache.pulsar:pulsar-spark
io.streamnative.connectors:pulsar-spark-connector_2.12
Can someone please help me understand if
they are following receiver based approach or direct approach?
What is the core difference between both the connectors?
Is there a possibility of data loss if my job fails while it is processing some batch.

Migrate Elasticsearch 2.4 to higher version along with Spring Framework 4.2.5 to Springboot 2.x. What will be best approach?

I want to upgrade my elasticsearch version & spring framework to Springboot 2.x version. Which elasticsearch & sprinbboot version should be used considering the spring-data-elasticsearch. I'm new to migrating application, would like to know the things that needs to taken into consideration.

I haven't done any Elasticsearch migration before. But 2.4 is the very old version, I guess you need to rewrite the ingestion logic in the migration.
For your reference, we are using the following tech stack in the project:
Spring boot: 2.3.x
AWS Elasticsearch Service: version 7.1
spring-data-elasticsearch 3.2.10

Spring data Jest version and Amazon ES versions, correlation between them

I'm trying to use Spring data Jest in order to interact with an Amazon ES cluster. Right now, 18/04/2018, Amazon documentation says that the available versions of ES are "6.2, 6.0, 5.5, 5.3, 5.1, 2.3, and 1.5".
According to Spring Data Jest documentation, the ES supported versions are "1.5.2, 2.2.0, 2.4.3, 2.4.4, and 5.5.0".
So this means that I'm only capable of using Spring Data Jest library on ES 1.5 and 5.5.0? This ones are the only match that I see but perhaps I'm missing something.
I really want to use the latest version available but without using Spring Boot 2.X...

You can use the ElasticSearch HighLevel RestClient API instead of JEST. The latest version of the API is 6.2
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html
HighLevelRest Client is a straight forward API and easy to use.

Lagom Cassandra Driver

According to some research about Lagom and Cassandra, I found that:
Lagom uses DataStax Java Driver for Cassandra, and
DataStax Java Driver only supports Cassandra 3.0.x (link)
So, if I want to use Cassandra 3.11 and Lagom, what should I do:
Should I configure Lagom with another Cassandra driver like Achilles, PlayORM, ... (link). Is that possible?
is DataStax support Cassandra 3.11 in the enterprise edition?
Any help, please?

DataStax Java Driver 3.2.0 that is used by Lagom should work with Cassandra 3.11 out of the box (just checked it myself using simple queries).
Even if it won't work out of the box, you can explicitly override driver version in Maven or other build system.

How to Use DCAwareRoundRobinPolicy in java using Java Driver (2.0.2,3.1) and cassandra 3.7

I have configured cassandra cluster with 2 datacenters, and 3 nodes each. i wanted to use DCAwareRoundRobinPolicy to specify the local datacenter. i tried using both Datastax java driver 2.0.2 and 3.1. but with 2.0.2 there no compile time error rather in run-time i am getting NoSuchMethodError and 3.1 giving me the DCAwareRoundRobinPolicy() constructor is not visible.
Can anyone please let me know how to fix this issue?
Thanks in advance.

For the Java Cassandra Driver 3.1 you now use a builder to create your DCAwareRoundRobinPolicy
DCAwareRoundRobinPolicy dcAwareRoundRobinPolicy = DCAwareRoundRobinPolicy.builder()
.withLocalDc("my-dc")
.withUsedHostsPerRemoteDc(1)
.build();
I would also suggest using TokenAware as well so when you are building your cluster add
Cluster.Builder()
.withLoadBalancingPolicy(new TokenAwarePolicy(dcAwareRoundRobinPolicy))