Develop Reference

Starting my Topology with Storm 2.2.0 : Could not find leader nimbus from seed hosts [localhost]

I'm working with storm 2.2.0 and my configuration file is:
#storm.yaml
storm.local.hostname: "10.100.7.109"
storm.zookeeper.servers:
- "10.100.242.107"
nimbus.seeds: ["localhost"]
supervisor.slots.ports:
- 6701
- 6702
- 6703
- 6704
- 6705
- 6706
- 6707
- 6708
- 6709
- 6710
- 6711
- 6712
- 6713
- 6714
- 6715
- 6716
- 6717
The storm nimbus work correctly but when starting one topology i have an error related with nimbus.seeds configuration:
20:58:05.488 [main] WARN o.a.s.v.ConfigValidation - task.heartbeat.frequency.secs is a deprecated config please see class org.apache.storm.Config.TASK_HEARTBEAT_FREQUENCY_SECS for more information.
20:58:05.563 [main] WARN o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from localhost. will retry with a different seed host.
org.apache.storm.thrift.TApplicationException: Internal error processing getLeader
...
Exception in thread "main" org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
When running the command storm list, I receive the same error:
root#storm-nimbus-60:/usr/share/storm-2.2.0/conf# storm list
Running: /usr/lib/jvm/java-8-oracle/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/share/storm-2.2.0 -Dstorm.log.dir=/usr/share/storm-2.2.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 -Dstorm.conf.file= -cp /usr/share/storm-2.2.0/*:/usr/share/storm-2.2.0/lib/*:/usr/share/storm-2.2.0/extlib/*:/usr/share/storm-2.2.0/extlib-daemon/*:/usr/share/storm-2.2.0/conf:/usr/share/storm-2.2.0/bin org.apache.storm.command.ListTopologies
21:09:39.153 [main] INFO o.a.s.v.ConfigValidation - Will use [class org.apache.storm.DaemonConfig, class org.apache.storm.Config] for validation
21:09:39.438 [main] WARN o.a.s.v.ConfigValidation - task.heartbeat.frequency.secs is a deprecated config please see class org.apache.storm.Config.TASK_HEARTBEAT_FREQUENCY_SECS for more information.
21:09:39.669 [main] WARN o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from localhost. will retry with a different seed host.
org.apache.storm.thrift.TApplicationException: Internal error processing getLeader
at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.generated.Nimbus$Client.recv_getLeader(Nimbus.java:1183) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.generated.Nimbus$Client.getLeader(Nimbus.java:1171) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:222) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:179) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.NimbusClient.withConfiguredClient(NimbusClient.java:127) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.NimbusClient.withConfiguredClient(NimbusClient.java:117) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.command.ListTopologies.main(ListTopologies.java:27) [storm-core-2.2.0.jar:2.2.0]
Exception in thread "main" org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:250)
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:179)
at org.apache.storm.utils.NimbusClient.withConfiguredClient(NimbusClient.java:127)
at org.apache.storm.utils.NimbusClient.withConfiguredClient(NimbusClient.java:117)
at org.apache.storm.command.ListTopologies.main(ListTopologies.java:27)
Any solutions for my errors? Thanks all

Your error looks like you haven't any nimbus service running.
It happens also if the nimbus can't connect to the zookeeper server.
Apache Storm 2.2.0 required a zookeeper service, even your run it locally.
I recommend you to run these services locally for easy development and debug:
zookeeper
storm supervisor
storm nimbus
storm UI
My storm.yaml is:
storm.zookeeper.servers:
- "127.0.0.1"
storm.local.dir: "/opt/apache-storm-1.2.2/data/"
nimbus.host: "127.0.0.1" # for nimbus
nimbus.seeds: ["localhost"] # for ui and supervisor
supervisor.slots.ports:
- 6700 # no need to more than one topology running
supervisor.memory.capacity.mb: 8192.0 #i have 16G ram, half for storm
supervisor.cpu.capacity: 600.0 # and 6 core cpu
# your storm dashboard will be at http://localhost:8081/
ui.port: 8081
ui.host: "0.0.0.0"

DataNode cannot connect with Name Node - "org.apache.hadoop.ipc.Client: Retrying connect to server"

I've deployed a Hadoop 3.1.2 cluster with 1 Namenode and 2 Datanodes. NameNode is UP, secondaryNameNode and ResourceManager also up for Master Node, however DataNode cannot connect with the NameNode, thus no capacity is shown.
I've been trying to find out what the error might be, but haven't succeed so far.
Removed domain resolutions as I was getting odd errors:
WARNING: Attempting to start all Apache Hadoop daemons as hadoop in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [server]
lim_sbo_bigdata_master: ERROR: Cannot set priority of namenode process 11606
Starting datanodes
Starting secondary namenodes [server]
lim_sbo_bigdata_master: ERROR: Cannot set priority of secondarynamenode process 11825
Starting resourcemanager
Starting nodemanagers
* SELinux is disabled
* IPtables is OPEN for all traffic:
hadoop#lim_server]$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Servers belong to the same network.
NameNode:
[hadoop#server ~]$ hadoop version
Hadoop 3.1.2
Source code repository https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a
Compiled by sunilg on 2019-01-29T01:39Z
Compiled with protoc 2.5.0
From source with checksum 64b8bdd4ca6e77cce75a93eb09ab2a9
This command was run using /home/hadoop/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar
[hadoop#server ~]$ jps
27089 Jps
26760 ResourceManager
26491 SecondaryNameNode
26239 NameNode
[hadoop#server ~]$ hdfs dfsadmin -report
Configured Capacity: 0 (0 B)
Present Capacity: 0 (0 B)
DFS Remaining: 0 (0 B)
DFS Used: 0 (0 B)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
DataNode Error
[hadoop#server_2]$ jps
17052 DataNode
17166 NodeManager
17406 Jps
2019-08-27 05:46:09,086 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9867
2019-08-27 05:46:09,229 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:9867
2019-08-27 05:46:09,243 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2019-08-27 05:46:09,251 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2019-08-27 05:46:09,260 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to /10.30.17.228:9000 starting to offer serv
ice
2019-08-27 05:46:09,265 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-08-27 05:46:09,265 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9867: starting
2019-08-27 05:46:10,330 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 10.30.17.228/10.30.17.228:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountW
ithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-08-27 05:46:11,331 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 10.30.17.228/10.30.17.228:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountW
ithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Try changing "localhost" to the actual hostname or IP of the namenodes.

Unable to set up hadoop on my local machine

I am trying to install Hadoop as a pseudo stand alone application on my macbook, and I have been seeing errors.
When I try to execute sbin/start-dfs.sh, I get the following error.
$ sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: Connection closed by ::1 port 22
Starting datanodes
localhost: Connection closed by ::1 port 22
Starting secondary namenodes [Kartiks-MacBook-Pro.local]
Kartiks-MacBook-Pro.local: Connection closed by 100.110.189.236 port 22
2018-01-22 00:20:19,441 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I used the following pages as my reference
1) http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
2) https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation
3) http://zhongyaonan.com/hadoop-tutorial/setting-up-hadoop-2-6-on-mac-osx-yosemite.html
None of the URLs to view either the namenode load for me
1) http://localhost:9870/ (from the Hadoop's website)
2) http://localhost:50070/ (from http://zhongyaonan.com/hadoop-tutorial/setting-up-hadoop-2-6-on-mac-osx-yosemite.html)
I am using my personal login id from my Macbook.

Hadoop Docker Setup - WordCount Tutorial

I was following the tutorial to run WordCount.java mentioned in here and when I run the following line in the tutorial
hadoop jar wordcount.jar org.myorg.WordCount /user/cloudera/wordcount/input /user/cloudera/wordcount/output
I get the following error -
17/09/04 01:57:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/09/04 01:57:30 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
The docker image that I used was docker pull cloudera/quickstart
There were no setup tutorials for Hadoop with Docker so it would be helpful if you could tell me the configurations that are to be made to overcome these issues.

That tutorial assumes you are in the cluster with the Hadoop client command available, the Hadoop Services are started, and properly configured.
0.0.0.0:8032 is the default YARN resource manager, so you need to configure your HADOOP_CONF_DIR XML files (specifically yarn-site for this error) to point at the Docker container for the correct addresses of YARN. core and hdfs-site will need configured to point at HDFS as well.

NoRouteToHostException while hadoop fs -copyFromLocal

I installed hadoop 2.5.1 on CentOS7.0
and I'm using 3 computers with below hosts file, the same as all 3 computers
I'm not using DNS.
XXX.XXX.XXX.65 mccb-com65 #server
XXX.XXX.XXX.66 mccb-com66 #client01
XXX.XXX.XXX.67 mccb-com67 #client02
127.0.0.1 localhost
127.0.1.1 mccb-com65
I execute the command
$hadoop fs -copyFromLocal /home/hadoop/hdfs/hdfs/s_corpus.txt hdfs://XXX.XXX.XXX.65:9000/tmp/
I met below error message
INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at
org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1526)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1328)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)
15/02/27 16:57:40 INFO hdfs.DFSClient: Abandoning
BP-1257634566-XXX.XXX.XXX.65-1425014347197:blk_1073741837_1013
15/02/27 16:57:40 INFO hdfs.DFSClient: Excluding datanode
XXX.XXX.XXX.67:50010 <-- the same as another salve node XXX.XXX.XXX.66
I turn off all firewall of both computers mccb-com66 and mccb-com67 as below state shows.
$systemctl status iptables
iptables.service - IPv4 firewall with iptables Loaded: loaded
(/usr/lib/systemd/system/iptables.service; disabled)
Active:
inactive (dead)
and Additionally I also turn off selinux.
datanode and nodemanager are alive in both machines
I can check the state
jps and
http://mccb-com65:50070 and
http://mccb-com65:8088
What I'm missing?
Could you anybody help me???

Even though I turn off the iptables, it's not valid solution.
After I open port one by one with firewall-cmd, it works..
for all slaves (66 and 67)
$firewall-cmd --zone=public --add-port=8042/tcp
$firewall-cmd --zone=public --add-port=50010/tcp
$firewall-cmd --zone=public --add-port=50020/tcp
$firewall-cmd --zone=public --add-port=50075/tcp
$firewall-cmd --reload
and then it works.
However, since I cannot open all ports which need to run Hadoop App,
turn off firewalld is reasonable such as
$systemctl stop firewalld
$systemctl disable firewalld
and check the status
$Systemctl status firewalld

your /etc/hosts should contain:
XXX.XXX.XXX.65 mccb-com65 #server
XXX.XXX.XXX.66 mccb-com66 #client01
XXX.XXX.XXX.67 mccb-com67 #client02
Remove
127.0.0.1 localhost
127.0.1.1 mccb-com65