Develop Reference

What should it look like when one of the host failed for zookeeper when building zk connections in java

I have the code like the following:
ZooKeeperInstance inst = new ZooKeeperInstance(this.instance, this.zkServers);
Connector this.conn = inst.getConnector(this.user, new PasswordToken(this.password));
(this.zkServers is a list of zk host separated by comma;
ZookeeperInstance is from the package of "org.apache.accumulo.core.client")
When one of the host from this.zkServers is died, but the other two host still work, these two lines just failed to connect. Does anyone know how to explain this? Thanks.
Does that mean that accumulo connection needs all the hosts in zookeeper to keep alive?

I have also noticed this behavior. You should be able to connect if you remove the dead server from your zkServers connection string.
It is unfortunate that Accumulo does not fail over to the next server, perhaps someone should file a bug report.

Get results from JMeter remote test in Java

I´m trying to run JMeter tests directly from my Java code. The load should be produced via a JMeter server. In the code, I use a ClientJMeterEngine instance that connects to the JMeter daemon running on the server. This works so far, I can start and run my test. What I currently don´t understand is how to get the results from the test run after it has completed. My current code looks like this:
// create ClientJMeterEngine instance
ClientJMeterEngine jmeter = new ClientJMeterEngine("myHost:4000");
// load test
File file = new File("C:/myTest.jmx");
// configure test
HashTree testPlanTree = SaveService.loadTree(file);
jmeter.configure(testPlanTree);
// run test
jmeter.runTest();
If I run the test using the StandardJMeterEngine object, I use a ResultCollector to get the results from my test runs. This works as expected. When I try to run the tests using the described server-based approach, I get the following exception on the machine where the server is running:
2016/03/15 09:24:35 ERROR - jmeter.samplers.BatchSampleSender: testEnded(host) java.rmi.ConnectException: Connection refused to host: myHost; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:631)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:228)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:214)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:238)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:190)
at com.sun.proxy.$Proxy1.processBatch(Unknown Source)
The server thus wants to call back to the client from which I start the test run. It fails, because I don´t have anything waiting for the remote call. I´ve worked my way through the JMeter source, but haven´t found an answer, what I need to do on my Java side to make this work.
EDIT
In the meantime I´ve made some progress. What I´ve found is the following:
The client thread must be kept in state running, so the ResultCollector on the client can receive the results from the server. I now do this by "busy waiting" in my client Java code.
The issue now is to know when the test execution on the server is finished, so I can also finish the client thread. The ResultCollector does have the capabilities to recognize the test is finished, but no public methods for this.
The solution would be to extend the ResultCollector and override the testEnded method: This method sets a public boolean variable "testFinished" to true.
Though this feels a little bit "hacky". Does anybody have a better solution here?

I had the same problem and tried your solution with overriding testEnded() method of ResultCollector class. My test was endlessly waiting for response from remote JMeter slave server.
The actual problem is that remote server - Slave - succeed in replying of requests from Master at early stage. However, the connection dropped at some point and the test finished without receiving any report from a Slave. Slave is unable to start a new TCP connection, and can't send results when it is ready with his testing report as its hostname (IP) hasn't been set/sent to Master correctly.
Therefore, the solution is to start a Slave with the correct parameter in command line or in the jmeter.properties.
I started my slave like this:
c:\JMeter_new\apache-jmeter-3.1\bin>jmeter-server -Djava.rmi.server.hostname=IP
where IP is of the Slave
PS If you run tests in GUI, run Master with the parameter as well:
D:\JMeter_new\apache-jmeter-3.1\bin>jmeterw -Djava.rmi.server.hostname=IP_of_master

Kafka Connect implementation errors

I was running through the tutorial here: http://kafka.apache.org/documentation.html#introduction
When I get to "Step 7: Use Kafka Connect to import/export data" and attempt to start two connectors I am getting the following errors:
ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding messages, 1 left
ERROR Failed to commit offsets for WorkerSourceTask
Here is the portion of the tutorial:
Next, we'll start two connectors running in standalone mode, which means they run in a single, local, dedicated process. We provide three configuration files as parameters. The first is always the configuration for the Kafka Connect process, containing common configuration such as the Kafka brokers to connect to and the serialization format for data. The remaining configuration files each specify a connector to create. These files include a unique connector name, the connector class to instantiate, and any other configuration required by the connector.
bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties
I have spent some time looking for a solution, but was unable to find anything useful. Any help is appreciated.
Thanks!

The reason I was getting this error was because the first server I created using the config/server.properties was not running. I am assuming that because it is the lead of the topic, the messages could not be flushed and the offsets could not be committed.
Once I started the kafka server using the server propertes (config/server.properties) This issue was resolved.

You need to start Kafka server and Zookeeper before running Kafka Connect.
You need to exec the cmds in "Step 2: Start the server" below first:
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
from here:https://mail-archives.apache.org/mod_mbox/kafka-users/201601.mbox/%3CCAK0BMEpgWmL93wgm2jVCKbUT5rAZiawzOroTFc_A6Q=GaXQgfQ#mail.gmail.com%3E

You need to start zookeeper and kafka server first before running that line.
start zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
start multiple kafka servers
bin/kafka-server-start.sh config/server.properties
bin/kafka-server-start.sh config/server-1.properties
bin/kafka-server-start.sh config/server-2.properties
start connectors
bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties
Then you will see some lines are written into test.sink.txt:
foo
bar
And you can start the consumer to check it:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}

If you configure your Kafka Broker with a hostname such as my.sandbox.com make sure that you modify the config/connect-standalone.properties accordingly:
bootstrap.servers=my.sandbox.com:9092
On Hortonworks HDP the default port is 6667, hence the setting is
bootstrap.servers=my.sandbox.com:6667
If Kerberos is enabled you will need the following settings as well (without SSL):
security.protocol=PLAINTEXTSASL
producer.security.protocol=PLAINTEXTSASL
producer.sasl.kerberos.service.name=kafka
consumer.security.protocol=PLAINTEXTSASL
consumer.sasl.kerberos.service.name=kafka

Get Accumulo instance name

I want to use GeoMesa (GIS extension of Accumulo) and virtualized it using Docker just like this repo. Now I want to connect to the Accumulo instance using Java using:
Instance i = new ZooKeeperInstance("docker_instance",zkIP:port);
Connector conn = i.getConnector(user, new PasswordToken(password));
The connetion does not get established and hangs (just like in this question). I can connect to the ZooKeeper instance using using
./zkCli.sh -server ip:port
So i guess the instance_name is wrong. I used the one noted in the repo linked first. However I don't know where how to check the instance_name needed.
To make my problem reproducable I did setup a digital ocean server with all necessary dependencies and accumulo. I tested that the connection to zookeeper is possible using zkCli and checked the credentials using accumulo shell on the server.
Instance i = new ZooKeeperInstance("DIGITAL_OCEAN","46.101.199.216:2181");
// WARN org.apache.accumulo.core.client.ClientConfiguration - Found no client.conf in default paths. Using default client configuration values.
System.out.println("This is reached");
Connector conn = i.getConnector("root", new PasswordToken("mypassw"));
System.out.println("This is not reached");

As a troubleshooting step, you may be able to extract the instance name by using HdfsZooInstance.getInstance().getInstanceName() or by connecting directly to ZooKeeper and listing the instance names with ls /accumulo/instances/

There are multiple easy ways to get the instance_name: Ether just look to the top of the accumulo status page as elserj noted in the comments or use zkCli to connect to Zookeeper and use ls /accumulo/instances / as Christopher answered.
However I could not manage to connect to accumulo using the ordinary Java Connector. Nevertheless I managed to connect to Accumulo using the Proxy-Settings which is a valid solution for me, even that I still would have liked to find the problem.

kafka + zookeeper remote = error

I am trying to install a kafka & zookeeper instance on a remote server. I only need 1 node of each actually because i only want to provide remote kafka for test purposes.
Kafka and Zookeeper are running from the Apache Kafka tarball you can find there (v0.0.9), inside a Docker image.
Trying to consume / produce using the provided scripts. And trying to produce using own java application. Everythinf is working fine if Kafka & ZK are installed on the local server.
Here is the error I get while trying to produce :
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
No partition metadata for topic RSS due to kafka.common.LeaderNotAvailableException}] for topic [RSS]: class kafka.common.LeaderNotAvailableException
Kafka properties tested
First :
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=localhost:<PORT>
Second:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
Third:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Last:
borker.id=0
port=9092
host.name=</etc/host name>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Here is my "/etc/hosts"
127.0.0.1 kafka kafka
127.0.0.1 localhost
I followed the Getting Started, which if I understood is a localhost / signle server configurations. I cannot understand what I have to do to get this work with remote calls...
Thanks for your help !
EDIT 1
host.name=localhost
advertised.host.name=politik.cm-cloud.fr
Seems to allow a local consumer (on the server) and producer. But if we want to do the same from a remote server we get
[2015-12-09 12:44:10,826] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host

The error does not look like connectivity problem with Zookeeper / Kafka.
Just follow the instruction in "quickstart" from http://kafka.apache.org/
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
Additionally the error indicates there is no partition info i.e topic not yet created . Try creating topics first and then try to produce/consume because when producing to a non existent topic kafka will create the topic based on auto.create.topics.enable in server.properties but remotely it is better to create topics rathen than relying on auto create