Develop Reference

How to solve java.net.UnknownHostException: mosquitto in docker container?

I am running a Java application which successfully connects to a MQTT broker locally like this:
ws://localhost:9001 and then has to do the same in a docker container for ws://mosquitto:9001. This used to work fine but I found now that I get this issue:
Failed to connect to the MQTT broker with address ws://mosquitto:9001 MqttException (0) - java.net.UnknownHostException: mosquitto
I receive no other error so I wonder what could be the cause of this since it used to work fine before.

you may go to https://github.com/eclipse/mosquitto/issues/2074
where scott has mentioned
This broke for me when Watchtower upgraded this to 2.0.7 also. Adding the following to the config file and a restart fixed things for me:
listener 1883

Drill not finding drillbit from zookeeper?

I am trying out the drill sample in my project using this example.
https://github.com/vicenteg/DrillJDBCExample/blob/master/src/main/java/com/mapr/drill/DrillJDBCExample.java
I have started drillbits on all my datanodes with the same "cluster-id" and I specify "zk.connect" to point to "zookeeper1,zookeeper2,zookeeper3" in my drill-override.conf (picked up by default i believe).
I am getting the following error:
java.lang.IllegalStateException: No DrillbitEndpoint can be found
Am I supposed to start drillbits on my zookeeper nodes too in addition to my datanodes? Or what is wrong?
My drill-override is as follows:
drill.exec: {
cluster-id: "testcluster",
zk.connect: "zookeeper1:2181,zookeeper2:2181,zookeeper3:2181"
}

if you are trying to connect to a different machine where drill is installed. Then while connecting from windows, give ip address of the machine where drill is running as .
Important
If in drill-override.conf(Linux machine where Drill is running) you have written "zookeeper1" as node name, then you should modify your "c:\Windows\System32\Drivers\etc\hosts" file in your client machine, and gave a DNS name of that ip.
Example :
192.168.32.84 zookeeper1 .

kafka + zookeeper remote = error

I am trying to install a kafka & zookeeper instance on a remote server. I only need 1 node of each actually because i only want to provide remote kafka for test purposes.
Kafka and Zookeeper are running from the Apache Kafka tarball you can find there (v0.0.9), inside a Docker image.
Trying to consume / produce using the provided scripts. And trying to produce using own java application. Everythinf is working fine if Kafka & ZK are installed on the local server.
Here is the error I get while trying to produce :
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
No partition metadata for topic RSS due to kafka.common.LeaderNotAvailableException}] for topic [RSS]: class kafka.common.LeaderNotAvailableException
Kafka properties tested
First :
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=localhost:<PORT>
Second:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
Third:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Last:
borker.id=0
port=9092
host.name=</etc/host name>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Here is my "/etc/hosts"
127.0.0.1 kafka kafka
127.0.0.1 localhost
I followed the Getting Started, which if I understood is a localhost / signle server configurations. I cannot understand what I have to do to get this work with remote calls...
Thanks for your help !
EDIT 1
host.name=localhost
advertised.host.name=politik.cm-cloud.fr
Seems to allow a local consumer (on the server) and producer. But if we want to do the same from a remote server we get
[2015-12-09 12:44:10,826] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host

The error does not look like connectivity problem with Zookeeper / Kafka.
Just follow the instruction in "quickstart" from http://kafka.apache.org/
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
Additionally the error indicates there is no partition info i.e topic not yet created . Try creating topics first and then try to produce/consume because when producing to a non existent topic kafka will create the topic based on auto.create.topics.enable in server.properties but remotely it is better to create topics rathen than relying on auto create

Getting Permission denied (publickey). when starting hadoop cluster on AWS

I am getting "Permission denied (publickey)" error when starting hadoop multi node cluster in AWS. But when i do ssh to each individual slave node without starting the cluster then i am able to access them. I did all the settings correct and checked twice.Any help on what may be wrong?

The problem was i created a new user i.e. hduser and then configured hadoop in it.
I did all the setup(hadoop configuration) in Ubuntu user(default for ec2 Ubuntu instances) and it worked. I think its better to use default users's in AWS instances then creating any new one and then struggling to get permissions and other errors.

Protobuf RPC not available on Hadoop 2.2.0 single node server?

I am trying to run a hadoop 2.2.0 mapreduce job on my local single node cluster installed by following this tutorial:
http://codesfusion.blogspot.co.at/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1
Though on the server side the following exception is thrown:
org.apache.hadoop.ipc.RpcNoSuchProtocolException: Unknown protocol: org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:527)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:566)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
Is there a way for me to configure Protobuf RPC to be available on server side? Do I need the hadoop native libraries for this? Or can I switch somehow on the client side to Writables/Avro RPC?

Ok, found the reason, I connected to the wrong port for the yarn resourcemanager. The correct configuration is:
yarn.resourcemanager.address=localhost:8032

In my case, I was getting the same error in the logs when there was not enough memory between the Application Master and the YARN containers. Reduced the yarn.app.mapreduce.am.resource.mb property and it worked on my single node installation.