With the deprecation of SearchTye.SCAN and the newly Reindex API we want to migrate our elasticsearch cluster and clients from 2.1.1 to 2.3.3.
We use java and the appropiate libraries to access elasticsearch. To access the cluster we use the TransportClient, for embedded Unittests we use the NodeClient.
Unfortunatly the Reindex API is provided as plugin, which the NodeClient seems to be unable to deal with.
So the question is how to use the NodeClient with the Reindex-Plugin?
I already tried exposing the protected NodeClient constructor to pass the ReindexPlugin class as argument without success.
Using the NodeClient to start an embedded ElasticSearch and using the TransportClient with the added ReindexPlugin didn't worked either. All I get here is an Exception: ActionNotFoundTransportException[No handler for action [indices:data/write/reindex]]
Dependencies of interest:
org.elasticsearch:elasticsearch:2.3.3
org.elasticsearch.module:reindex:2.3.3
org.apache.lucene:lucene-expressions:5.5.1
org.codehaus.groovy:groovy:2.4.6
Starting the NodeClient:
Settings.Builder settings = Settings.settingsBuilder();
settings.put("path.data", "/some/path/data");
settings.put("path.home", "/some/path/home");
//settings.put("plugin.types", ReindexPlugin.class.getName()); > No effect
settings.put("http.port", 9299);
settings.put("transport.tcp.port", 9399);
node = NodeBuilder.nodeBuilder()
.clusterName("testcluster")
.settings(settings)
.local(true)
.node();
// also tested with local(false), then no transport port is available, resulting in NoNodeAvailableException
Using TransportClient to access the Node:
Settings settings = Settings.settingsBuilder()
.put("cluster.name", "testcluster")
.put("discovery.zen.ping.multicast.enabled", false)
.build();
InetSocketTransportAddress[] addresses = new InetSocketTransportAddress[]
{new InetSocketTransportAddress(new InetSocketAddress("localhost", 9399))};
client = TransportClient.builder()
.settings(settings)
.addPlugin(ReindexPlugin.class)
.build()
.addTransportAddresses(addresses);
Main part of triggering reindex:
ReindexRequestBuilder builder = ReindexAction.INSTANCE.newRequestBuilder(getClient())
.source(indexFrom)
.destination(indexTo)
.refresh(true);
I was able to solve this, by combining both approaches described above.
So creating the NodeClient involves overriding the Node:
class ExposedNode extends Node {
public ExposedNode(Environment tmpEnv, Version version, Collection<Class<? extends Plugin>> classpathPlugins) {
super(tmpEnv, version, classpathPlugins);
}
}
And using it when starting the NodeClient:
Settings.Builder settings = Settings.settingsBuilder();
settings.put("path.data", "/some/path/data");
settings.put("path.home", "/some/path/home");
settings.put("http.port", 9299);
settings.put("transport.tcp.port", 9399);
// Construct Node without NodeBuilder
List<Class<? extends Plugin>> classpathPlugins = ImmutableList.of(ReindexPlugin.class);
settings.put("node.local", false);
settings.put("cluster.name", "testcluster");
Settings preparedSettings = settings.build();
node = new ExposedNode(InternalSettingsPreparer.prepareEnvironment(preparedSettings, null), Version.CURRENT, classpathPlugins);
node.start();
After that you can use the TransportClient which adds the ReindexPlugin, as described in the question.
Nevertheless this is a dirty hack, which may break in a future release, and shows how poorly Elasticsearch supports plugin development imo.
Related
We have a service that receive the queue name and if the queue exist push data on the queue.
I'm looking to find a way to check if the queue exist on azure in java.
So far, using IQueueClient I can onl know if the queue exist when sending data via sendAsync with return a exception.
I want to found out when the client is created or even before.
The best answer I've found so far is using the NamespaceManager.
in this fashion:
Uri uri = ServiceManagementHelper.GetServiceUri();
TokenProvider tokenProvider = ServiceManagementHelper.GetTokenProvider(uri);
NamespaceManager nm = new NamespaceManager(uri, tokenProvider);
if (!nm.QueueExists("TestQueue")) return;
source:
http://www.cloudcasts.net/devguide/Default.aspx?id=12018
https://www.pmichaels.net/2017/06/11/reading-message-azure-service-bus-queue/
However this class seem not to be part of azure-core or azure-servicebus librairy and I can seem to find which lib to add to the project to have access to those class.
However this class seem not to be part of azure-core or azure-servicebus librairy and I can seem to find which lib to add to the project to have access to those class.
You mentioned code is related to .net SDK. For Java sdk we could use the following code check if the queue exist. ManagementClient seems only available in the 2.0.0-PREVIEW version. We also could get the source code from GitHub.
String connectionString = "Endpoint=sb://xxxx.servicebus.windows.net/;SharedAccessKeyName=SharedAccessKeyName;SharedAccessKey=xxxxxxxx=";
ConnectionStringBuilder connectionStringBuilder = new ConnectionStringBuilder(connectionString);
ManagementClient client = new ManagementClient(connectionStringBuilder);
boolean isExist = client.queueExists("queueName");
System.out.println(isExist);
Test Result:
I am trying to implement ATPlus scan consistency within my couchbase java application. I have updated my queries to include the consistentWith(mutationState):
RawJsonDocument courseJsonDocument = toRawJsonDocument(course, true);
RawJsonDocument insertedJsonDocument = bucket.insert(courseJsonDocument);
MutationState insertMutationState = MutationState.from(insertedJsonDocument);
.....
N1qlQuery.simple(GET_COURSE_BY_ID_QUERY, N1qlParams.build().consistentWith(mutationState));
I'm trying to achieve read-your-own-write, but when I run the query immediately after inserting the document, nothing is found, so I must be doing something wrong. I think what I am missing is actually enabling enhanced durability on the client configuration.
I see examples of how to do it in .NET, but I can't figure out how to 'enable enhanced durability' in JAVA. Here is my cluster configuration:
Cluster cluster = CouchbaseCluster.create(DefaultCouchbaseEnvironment.builder()
.queryServiceConfig(QueryServiceConfig.create(1, 100))
.mutationTokensEnabled(true)
.observeIntervalDelay(Delay.fixed(100, TimeUnit.MICROSECONDS))
.connectTimeout(timeout)
.build(),
clusterHost);
I have problem with vertx HttpClient.
Here's code which shows that tests GET using vertx and plain java.
Vertx vertx = Vertx.vertx();
HttpClientOptions options = new HttpClientOptions()
.setTrustAll(true)
.setSsl(false)
.setDefaultPort(80)
.setProtocolVersion(HttpVersion.HTTP_1_1)
.setLogActivity(true);
HttpClient client = vertx.createHttpClient(options);
client.getNow("google.com", "/", response -> {
System.out.println("Received response with status code " + response.statusCode());
});
System.out.println(getHTML("http://google.com"));
Where getHTML() is from here: How do I do a HTTP GET in Java?
This is my output:
<!doctype html><html... etc <- correct output from plain java
Feb 08, 2017 11:31:21 AM io.vertx.core.http.impl.HttpClientRequestImpl
SEVERE: java.net.UnknownHostException: failed to resolve 'google.com'. Exceeded max queries per resolve 3
But vertx can't connect. What's wrong here? I'm not using any proxy.
For reference: a solution, as described in this question and in tsegismont's comment here, is to set the flag vertx.disableDnsResolver to true:
-Dvertx.disableDnsResolver=true
in order to fall back to the JVM DNS resolver as explained here:
sometimes it can be desirable to use the JVM built-in resolver, the JVM system property -Dvertx.disableDnsResolver=true activates this behavior
I observed this DNS resolution issue with a redis client in a kubernetes environment.
I had this issue, what caused it for me was stale DNS servers being picked up by the Java runtime, i.e. servers registered for a network the machine was no longer connected to. The issue is first in the Sun JNDI implementation, it also exists in Netty which uses JNDI to bootstrap its list of name servers on most platforms, then finally shows up in VertX.
I think a good place to fix this would be in the Netty layer where the set of default DNS servers is bootstrapped. I have raised a ticket with the Netty project so we'll see if they agree with me! Here is the Netty ticket
In the mean time a fairly basic workaround is to filter the default DNS servers detected by Netty, based on whether they are reachable or not. Here is a code Sample in Kotlin to apply before constructing the main VertX instance.
// The default set of name servers provided by JNDI can contain stale entries
// This default set is picked up by Netty and in turn by VertX
// To work around this, we filter for only reachable name servers on startup
val nameServers = DefaultDnsServerAddressStreamProvider.defaultAddressList()
val reachableNameServers = nameServers.stream()
.filter {ns -> ns.address.isReachable(NS_REACHABLE_TIMEOUT)}
.map {ns -> ns.address.hostAddress}
.collect(Collectors.toList())
if (reachableNameServers.size == 0)
throw StartupException("There are no reachable name servers available")
val opts = VertxOptions()
opts.addressResolverOptions.servers = reachableNameServers
// The primary Vertx instance
val vertx = Vertx.vertx(opts)
A little more detail in case it is helpful. I have a company machine, which at some point was connected to the company network by a physical cable. Details of the company's internal name servers were set up by DHCP on the physical interface. Using the wireless interface at home, DNS for the wireless interface gets set to my home DNS while the config for the physical interface is not updated. This is fine since that device is not active, ipconfig /all does not show the internal company DNS servers. However, looking in the registry they are still there:
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces
They get picked up by the JNDI mechanism, which feeds Netty and in turn VertX. Since they are not reachable from my home location, DNS resolution fails. I can imagine this home/office situation is not unique to me! I don't know whether something similar could occur with multiple virtual interfaces on containers or VMs, it could be worth looking at if you are having problems.
Here is the sample code which works for me.
public class TemplVerticle extends HttpVerticle {
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
// Create the web client and enable SSL/TLS with a trust store
WebClient client = WebClient.create(vertx,
new WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
.setDefaultHost("www.w3schools.com")
);
client.get("www.w3schools.com")
.as(BodyCodec.string())
.send(ar -> {
if (ar.succeeded()) {
HttpResponse<String> response = ar.result();
System.out.println("Got HTTP response body");
System.out.println(response.body().toString());
} else {
ar.cause().printStackTrace();
}
});
}
}
Try using web client instead of httpclient, here you have an example (with rx):
private val client: WebClient = WebClient.create(vertx, WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
)
open fun <T> get(uri: String, marshaller: Class<T>): Single<T> {
return client.getAbs(host + uri).rxSend()
.map { extractJson(it, uri, marshaller) }
}
Another option is to use getAbs.
while loading Map from external data source using MapLoader Hazelcast cluster(multicast discovery) gives error as
WARNING: [<IP>]:5702 [<cluster_name>] [3.8-EA] Received data format is invalid. (An old version of Hazelcast may be running here.)
com.hazelcast.nio.serialization.HazelcastSerializationException: Problem while reading DataSerializable, namespace: 0, id: 0, class: 'com.hazelcast.cluster.impl.JoinRequest', exception: com.hazelcast.cluster.impl.JoinRequest
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.rethrowReadException(DataSerializableSerializer.java:178)
...
Caused by: java.lang.ClassNotFoundException: com.hazelcast.cluster.impl.JoinRequest
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
I have tested this on hazelast 3.5.4 version .. It is working fine.
We can ignore this warning but not sure what is the impact of it. Also it floods the log.
The old and new versions of Hazelcast are not compatible in terms of multicast discovery since the internal protocol changed. That said, the new Hazelcast version cannot identify the old versions discovery packet.
Please change the multicast group according to the documentation found under: http://docs.hazelcast.org/docs/3.8-EA/manual/html-single/index.html#multicast-element
For those that may be running into this problem in an OSGi environment, you may be getting bit by a nuance of the com.hazelcast.util.ServiceLoader.findHighestReachableClassLoader() method sometimes picking the wrong class loader during Hazelcast initialization (as it won't always pick the class loader you set on the config). The following shows a way to work around that problem by taking advantage of Java's context class loader:
private HazelcastInstance createHazelcastInstance() {
// Use the following if you're only using the Hazelcast data serializers
final ClassLoader classLoader = Hazelcast.class.getClassLoader();
// Use the following if you have custom data serializers that you need
// final ClassLoader classLoader = this.getClass().getClassLoader();
final com.hazelcast.config.Config config = new com.hazelcast.config.Config();
config.setClassLoader(classLoader);
final ClassLoader previousContextClassLoader = Thread.currentThread().getContextClassLoader();
try {
Thread.currentThread().setContextClassLoader(classLoader);
return Hazelcast.newHazelcastInstance(config);
} finally {
if(previousContextClassLoader != null) {
Thread.currentThread().setContextClassLoader(previousContextClassLoader);
}
}
}
Using JClouds, up to version 1.6.x it was possible to access to the native EC2 provider API by using the following idiom:
AWSEC2Client ec2Client = AWSEC2Client.class.cast(context.getProviderSpecificContext().getApi());
Actually, I copied from the documentation page: http://jclouds.apache.org/guides/aws/
It turns out that in the latest release this method has been removed. Is there an alternative method/way to access to the provider specific features (security groups, key-pairs, etc)?
Unwrapping the API from the ComputeServiceContext
ComputeServiceContext context = ContextBuilder.newBuilder("aws-ec2")
.credentials("accessKey", "secretAccessKey")
.buildView(ComputeServiceContext.class);
ComputeService computeService = context.getComputeService();
AWSEC2Api ec2Api = context.unwrapApi(AWSEC2Api.class);
Building the API directly
AWSEC2Api ec2Api = ContextBuilder.newBuilder("aws-ec2")
.credentials("accessKey", "secretAccessKey")
.buildApi(AWSEC2Api.class);