Here is what I have done in a nutshell:
STEP1: I have successfully configured hadoop 2.6 on my laptop (single node) and ran a sample mapreduce job.
STEP2: I cloned tez repository and successfully built the 0.8.0 version and copied the jarfiles into HDFS and exports the required variables. I also changed the value of variable mapreduce.framework.name to yarn-tez in the mapred-site.xml.
But when I want to run a tez orderedwordcount job, I got this error:
15/07/04 18:45:03 INFO ipc.Client: Retrying connect to server: hostname/hostIP:57339.
Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/07/04 18:45:12 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
I have checked resource manager and it is listening on port 8030.
But it seems the client tries to connect to a random port. is it correct?
What can I do to get it work correctly?
It seems that it was the problem of this version (0.8.0) connecting to the resource manager. I compiled and integrated the previous stable release (0.7.0) and everything is good to go now. I hope that they will figure the problem out.
From your logs it seems a Firewall issue rather than issue with Tez version. And it is irrespective of Tez, even if you run Hadoop only you can face this.
Hadoop uses multiple ports for communication with clients and between service components. To enable Hadoop communication, open the specific ports that Hadoop uses.
To open specific ports, you can set the access rules in Windows. For example, the following command will open up port 80 in the active Windows Firewall:
netsh advfirewall firewall add rule name=AllowRPCCommunication dir=in action=allow protocol=TCP localport=80
For more see here http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0-Win/bk_HDP_Install_Win/content/ref-79239257-778e-42a9-9059-d982d0c08885.1.html
Related
I've some problems during configuration of remote Tomcat Server in Intellij
I've got a java web app that need to run in a cluster configuration, so I've got 3 Linux VMS in which I've installed tomcat server.
The problem is that I need to use this 3 tomcat servers together in debug mode in order to test and check issues.
I've tried some things but I was not able to get a solution.
OPTION A: configuring Remote JVM Debug:
I've tried to configure a Remote JVM Debug on my intellij configuring Catalina to listen to another port (2001) and launch it with the command: sh catalina.sh jpda start -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:2001 but it stucks on this message "Connected to the target VM, address: 'IP:2001', transport: 'socket'"
OPTION B: remote Tomcat server:
I've tried configuring a remote Tomcat server (that is the thing I prefer) configuring remote host with SFTP, but in this case Tomcat service start, it works for some minutes and after that it goes in TimeoutException
[](https://i.stack.imgur.com/XmDKx.png)
I hope anyone can help me, I read a lot of thing on the internet but no-one works.
When using apache SSHD scp client to copy files from local to remote, I get the following error:
flush(ChannelOutputStream[ChannelExec[id=0, recipient=0]-ClientSessionImpl[uxxxxxx#Hostname.domain.com/192.163.23.68:45018]] SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of len=24576: waitForCondition(Window[client/remote](ChannelExec[id=0, recipient=0]-ClientSessionImpl[uxxxxxx#Hostname.domain.com/192.163.23.68:45018])) timeout exceeded: 30000
Here is how I have set up the SSHServer and the ScpClient:
How to upload/download files using apache SSHD ScpClient
This SCPClient is running in a linux host and there are multiple SSHServers that are running amongs linux and windows hosts.
I use this SCPClient to copy files to both linux and win SSHServers. What I am doing is I create some 20 odd akka actors that take care of copying to the respective remote hosts which are a combination of win and linux. So this does put some strain on the localhost when copying.
However I get this error only when copying to some WIN servers in which the SSHServer is running.
I did notice the copying is very slow but I am not sure what exactly is the issue and how I can fix it?
I have a vague idea that it has to do something with this param:
https://github.com/apache/mina-sshd/blob/sshd-2.5.0/sshd-core/src/main/java/org/apache/sshd/common/channel/ChannelOutputStream.java#L43
But I am not exactly sure where I can configure this when creating the client?
any pointers would be helpful.
We can use the PropertyResolverUtils to update properties of any config
PropertyResolverUtils.updateProperty(sshClient, ChannelOutputStream.WAIT_FOR_SPACE_TIMEOUT, 120000);
I just brought up CDH 5.4 and installed zookeeper. I used zkCli successfully many times before. This time the command line launch stops before getting to the prompt with
Welcome to ZooKeeper!
JLine support is disabled
2015-05-04 18:18:33,936 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread#975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2015-05-04 18:18:33,952 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread#852] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2015-05-04 18:18:33,985 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread#1235] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x34d12349d0a15cf, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
I know the usual print out is
JLine support is enabled
Is that what's getting it stuck? I don't see any way to change this one the Cloudera manager config page.
Disclaimer: please note answer is from 2015. It is probably not relevant anymore but may serve historical purpose and have some light on similar 'modern' cases.
SHORT
Cloudera has broken zookeeper 3.4.5-cdh5.4.0 in several places. Service is working but CLI is dead. No workaround other than roll back.
LONG
Assign a bounty on this ;-). I have stepped on this mine too and was angry enough to find the reason:
Zookeeper checks JLine during ZooKeeperMain.run(). There is try-catch block which loads number of classes. Any exception during class loading fails whole block and JLine support is reported to be disabled.
But here is why this happens with CDH 5.4.0:
Current opensource Zookeeper-3.4.6 works against jline-0.9.94. Has no such issue.
In CDH 5.4 Cloudera has applied the following patch:
roman#node4:$ diff zookeeper-3.4.5-cdh5.3.3/src/java/main/org/apache/zookeeper/ZooKeeperMain.java zookeeper-3.4.5-cdh5.4.0/src/java/main/org/apache/zookeeper/ZooKeeperMain.java
305,306c305,306
< Class consoleC = Class.forName("jline.ConsoleReader");
< Class completorC =
---
> Class consoleC = Class.forName("jline.ConsoleReader");
> Class completorC =
316,317c316,317
< Method addCompletor = consoleC.getMethod("addCompletor",
< Class.forName("jline.Completor"));
---
> Method addCompletor = consoleC.getMethod("addCompleter",
> Class.forName("jline.console.completer.Completer"));
CDH 5.4 uses jline-2.11.jar for ZooKeeper and it has no jline.ConsoleReader class (from 2.11 it is jline.console.ConsoleReader).
Jline 0.9.94 in turn has no jline.console.completer.Completer.
So there is incompatibility with any existing JLine. Any Cloudera CDH 5.4 user can run zookeeper-client on his/her cluster and find it does not work.
Open source zookeeper-3.4.6 depends on jline-0.9.94 which has no such patches. Don't know why Cloudera engineers have done such a mine.
I see no clean way to fix it with 3.4.5-cdh5.4.0. I stayed with 3.4.5-cdh5.3.3 dependency where I need CLI and have production clusters.
It seemed to me both jline-0.9.94.jar and jline.2.11.jar in classpath for zookeeper will fix the problem. But just have found Cloudera made another 'fix' in ZK for CDH 5.4.0, they have renamed org.apache.zookeeper.JLineZNodeCompletor class to org.apache.zookeeper.JLineZNodeCompleter.
But here is the code from ZooKeeperMain.java
Class<?> completorC = Class.forName("org.apache.zookeeper.JLineZNodeCompletor");
And of course it meaan practically it is not possible to start ZK CLI in CDH 5.4.0 proper way. Awful work. :-(
you can go to usr/lib/hbase/bin/hbase and passe this code :
# Add libs to CLASSPATH
for f in $HBASE_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
if [ "$COMMAND" = "zkcli" ] ; then
CLASSPATH=`echo $CLASSPATH | sed 's/jruby-cloudera-1\.0\.0\.jar//g'`
CLASS="org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServer" ;
Fi
done
it works for me :)
When I am using WAS Liberty profile V8.5 beta, I can not find anywhere to change default service port 9080 in RAD, I tried to add httpendpoint section in server.xml, liberty server reports configuration update successful, but got failure when running web application.
Anybody knows how to solve this? Thanks!
There was a bug in the beta which prevented some port changes from being preserved when the server was restarted. There is a forum thread discussing the issue with the cached port value, and the answer is to start the server with the --clean option (from the command line), there is an equivalent checkbox available when you start the server via the tools.
[1] https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14770100
I would start there.
I'am installing ArcGisServer for the Java plataform on Centos 5.5 x86_64, this is not a supported platform but I have overcome almost every problem preventing the success of the installation. It uses exhaustively Sun Directory Server. The last error i receive was:
ldap_simple_bind: Can't connect to the ldap server - No route to host
It happens in other applications which makes uses of it, so it seems to be an specific problem of Sun Directory Server on linux and solaris. There is no reported solution. Usually I search the problem as much as I can but this time I have reached my patience and I need it working as soon as posible. I recognize this as an excellent forum because of it's community and quality of answers, ¿can anybody help me with this?
The "No route to host" error suggests that the issue is one of network connectivity between your ArcGIS server (the Sun Directory Server component, as you mention) and the LDAP server. So, a few things to examine, in order:
Do you have an LDAP server set up and running?
Is your LDAP server reachable from your Centos machine outside of the ArcGis server?
Is your ArcGis configured with the correct address to the LDAP server - should be in the web.config file? Example below:
<connectionStrings>
<add name="ADConnectionString"
connectionString="LDAP://SERVER_LDAP:389/ou=Sigestredi,o=Sicondef,dc=aplicaciones,o=mdef,c= es" />
</connectionStrings>
Disclaimer: I don't know anything about the ArcGIS server per se - I'm just diagnosing the "no route to host error" with a few snippets I picked up from some quick searches of the ArcGis forums.
ArcGIS includes a Sun Directory Server on it, so arcgis server and ldap are on the same machine. The port is set to 62000. When I run the diagnostic tool the DG028 fails:
DG028 - check LDAP server: is listening
I made a prove using nmap as:
nmap localhost -p62000
And it says it is opened. I don't know how to verify if LDAP is up and running, the startup log doesn't show anything wrong. I have found a config file named ldap.conf:
url ldap://name.subdomain.domain:62000/dc=name,dc=arcgis
admnm agsadmin
And my /etc/hosts is:
127.0.0.1 localhost localhost
ip_direction name.subdomain.domain name.subdomain.domain
I don't have an alias for "name", so:
ping name.subdomain.domain
Works
But:
ping name
Doesn't work
I have never used ldap so I don't know what should be on "dc". Could my hosts file be malformed or is my ldap.conf?
Another information is that the computer I am using is part of a domain. When I installed Centos, /etc/hosts file had an alias for the loopback interface as localhost.localdomain but i removed it.
I would appreciate any help.
I have solved my problem, the problem was on my /etc/hosts file. I added an alias for my ip direction:
127.0.0.1 localhost localhost
ip_direction name.subdomain.domain name
Then I run the ServerConfig script. This is a successful installation of ArcGIS Server for the Java platform on Linux. Thanks Greg for your guide.