I have installed Hadoop and Hbase on a VirtualBox VM running Ubuntu; both Hadoop and Hbase are running successfully in pseudo-distributed mode. I have disabled IPv6 on Ubuntu and changed the localhost to 127.0.0.1 in the hosts file on the VM.
I am trying to write some basic Java code on a Windows machine in Eclipse to connect to the Hbase instance, create a table, insert and retrieve data, etc. The code fails with an error that it cannot connect to the master. However, it makes the Zookeeper connection to the VM just fine.
On the Windows machine, I am able to connect to the Hbase instance info via the web browser via the same IP address and port that I specify in the Java code.
I have searched everywhere and tried everything that I could find, but it is still failing to connect to the master after it makes the zookeeper connection.
I have read that others have had this problem too, but no one has posted a solution.
Please help! Thanks!
The IP and Port used to view information are not the one used to read/write from/into HBase. To do so you need to use either the REST API (included in HBase) or Apache Thrift (2 thrift servers are included in HBase - thrift & thrift2)
I would recommend you to use Apache Thrift (thrift2)
To start REST use :
$HBASE-INSTALL-DIR/bin/hbase-deamon.sh start rest
To start Thrift use :
$HBASE-INSTALL-DIR/bin/hbase-deamon.sh start thrift
To start Thrift (v2) use :
$HBASE-INSTALL-DIR/bin/hbase-deamon.sh start thrift2
To use the Thrift client from Java for example you will need to install thrift on the server and then generate the Java Classes using the hbase thrift file included with HBase.
By default Thrift will be listening on the 9090 port and REST on the 8080
Usefull Links :
HBase Thrift
HBase REST
Ok -- Someone gave me some 1-1 help that fixed the problem and I wanted to pass it along. It turned out to be an IP addressing issue with the VM and with my Windows machine. First, in the etc/hosts file on the VM, I had to take out '127.0.0.1 locahost' and instead insert ' localhost'. Second, on my Windows hosts file, I had to add ' '. Thankfully, that fixed the problem. Please let me know this is unclear since I have seen this problem posted quite a few times without suitable resolution. Also, since I am writing Java code to access the HBase instance in the VM, there was no need to use Thrift or REST -- the Java API was sufficient.
Related
I have to send some applications in python to a Apache Spark cluster. There is given a Clustermanager and some worker nodes with the addresses to send the Application to.
My question is, how to setup and to configure Spark on my local computer to send those requests with the data to be worked out to the cluster?
I am working on Ubuntu 16.xx and already installed java and scala. I have searched the inet but the most find is how to build the cluster or some old advices how to do it, which are out of date.
i assume you remote cluster is running and you are able to submit jobs on it from remote server itself. what you need is ssh tuneling. Keep in mind that it does not work with aws.
ssh -f user#personal-server.com -L 2000:personal-server.com:7077 -N
read more here: http://www.revsys.com/writings/quicktips/ssh-tunnel.html
your question is unclear. If the data are on your local machine, you should first copy your data to the cluster on HDFS filesystem. Spark can works in 3 modes with YARN (are u using YARN or MESOS ?): cluster, client and standalone. What you are looking for is client-mode or cluster mode. But if you want to start the application from your local machine, use client-mode. If you have an SSH access, you are free to use both.
The simplest way is to copy your code directly on the cluster if it is properly configured then start the application with the ./spark-submit script, providing the class to use as an argument. It works with python script and java/scala classes (I only use python so I don't really know)
I would like to connect with an SFTP client to an FTP server using java. I know the two technologies have nothing to do with each-other. What I'm trying to accomplish is to connect to an FTP server via the internet with out using two ports or changing the server configuration.
Is there any SFTP->FTP bridge in java it would be great. If not, how can I accomplish that ?
I would like to incorporate this in an exciting java server so hence, java based solutions are preferred. If there is some standalone software which you can control via code than it should support windows and *nix.
(Since SFTP is just the means here, a similar WebDav solution will also work)
You could go with trial and error. Try this out see if it works.
How to retrieve a file from a server via SFTP?
I am using Zero MQ PUB/SUB model where PUB and SUB are two different applications deployed on the same machine on webpshere 6.1. This model works fine on my local machine but when I deploy it on a remote unix box server it isn't working. My SUB never receives a message from PUB. I tried all the options suggested i could find on the web (localhost, 127.0.0.1) but no luck. Appreciate any help on this. I am using jeroMq 3.2.2.
Thanks
Akash
If you're using multicast, then you need to enable loopback on the socket. As a result of this, the sender app will get the data as well if it's listening for it.
We also faced same issue and it was fixed by using below settings:
Publisher side use * (star) : tcp:// *.port number
Subscriber side use machine name : tcp://machine name of publisher.port number
I'm trying to connect to remote hadoop cluster, which isn't accessible just through HDFS. Right now it is being used in that way: user connects to Jump box through SSH (e.g. ssh user#somejumboxhost.com), then from jump box server we do connect to hadoop also with ssh (e.g. ssh user#realhadoopcluster1.com). What I'm trying to do is to access files from my Scala/Java application using HDFS client. Now I'm feeling like in Matrix - "I must go deeper" and don't know how to reach the server.
May be someone had similar experience? Right now I'm trying to connect to first server with SSH client from my app, but then I don't know how to call the HDFS client.
Any ideas will be appreciated, thanks!
I can think of something like this . There is "ganymed-ssh2" api which helps you to connect to some server using ssh and run unix command from there. Using this you can connect to your jumo box.
And from there you can run command as " ssh user#realhadoopcluster1.com hadoop fs somthing"
As we can run commands with ssh like this.
From your jump box, setup a password less ssh to your hadoopcluster machine. or you can use sshpass with password.
You can visit following link to check how to use this api:
http://souravgulati.webs.com/apps/forums/topics/show/8116298-how-to-execute-unix-command-from-java-
Hadoop is implemented in Java, so you could just run the Hadoop cluster directly from your application. Use Java RMI if it's a remote cluster. This extra pipework you're trying to do makes no sense.
I just want to know how I can start derby in network server mode and still be able to get an embedded connection?
Thank you.
You need to launch Derby in "embedded server mode". If you are already using Derby in embedded mode, this can be enabled by providing the necessary files in your classpath, then specifying a handful of command line arguments when launching the application.
First make sure the following jars are in your application's runtime classpath.
derby.jar derbynet.jar
Then add the following command line options to the Java command used to launch your application. If the class files are missing, these options will have no effect.
-Dderby.drda.startNetworkServer=true
-Dderby.drda.portNumber=8011
I'm running Derby from within a servlet hosted by Tomcat, so I added these options to the catalina.bat file.
Start up your application and check the list of open network sockets.
netstat -an | find "8011"
You should now see Derby listening for connections on 8011. Its now possible to connect to the database using Derby's client driver (derbyclient.jar). The instructions at http://docs.oracle.com/javadb/10.3.3.0/adminguide/radminembeddedserverex.html cover this part pretty well.
It was hinted that running Derby in this mode may be discouraged. I don't believe that to be the case. Your application will continue to access the database using the embedded driver, while other software is now permitted access using the client driver.
The Embedded Server mode sounds like what you are asking for. It allows you to start a network server when you start the embedded database.
It sounds contradictory that you want to start derby in network server mode and get the embedded driver. Even if this might be possible, it is definitely discouraged. You should decide on whether you want to use Apache Derby in the network mode using the DRDA or as an embedded driver and stick to that decision.
Here you'll find a tutorial on how to use the network driver:
http://db.apache.org/derby/papers/DerbyTut/ns_intro.html
Some one correct me if i am wrong, Both will run on separte ports. So you can connect to the required one using the proper connectionName, right?
#pawelocue: Sorry, but this is wrong. Using the embedded server mode is perfectly alright and sometimes very useful. It is definitely not discouraged.