I'm trying to connect to remote hadoop cluster, which isn't accessible just through HDFS. Right now it is being used in that way: user connects to Jump box through SSH (e.g. ssh user#somejumboxhost.com), then from jump box server we do connect to hadoop also with ssh (e.g. ssh user#realhadoopcluster1.com). What I'm trying to do is to access files from my Scala/Java application using HDFS client. Now I'm feeling like in Matrix - "I must go deeper" and don't know how to reach the server.
May be someone had similar experience? Right now I'm trying to connect to first server with SSH client from my app, but then I don't know how to call the HDFS client.
Any ideas will be appreciated, thanks!
I can think of something like this . There is "ganymed-ssh2" api which helps you to connect to some server using ssh and run unix command from there. Using this you can connect to your jumo box.
And from there you can run command as " ssh user#realhadoopcluster1.com hadoop fs somthing"
As we can run commands with ssh like this.
From your jump box, setup a password less ssh to your hadoopcluster machine. or you can use sshpass with password.
You can visit following link to check how to use this api:
http://souravgulati.webs.com/apps/forums/topics/show/8116298-how-to-execute-unix-command-from-java-
Hadoop is implemented in Java, so you could just run the Hadoop cluster directly from your application. Use Java RMI if it's a remote cluster. This extra pipework you're trying to do makes no sense.
Related
We are implementing an university project: a car-pooling service in Java.
We need to solve a problem linked "how to manage a postgres server":
the PostgreSQL Database is configured in a lab server called "golem" (130.136.4.sth) reachable only through terminals in the same subnet (130.136.4.0).
We have four account (ours) through we can establish a ssh connection to an host.
Is it possible to make SQL queries through SSH towards Postgres DB in JAVA?
Thank you :)
Davide
If this is just for development, you can use ssh port forwarding to access the database as if it was installed locally. How port forwarding is enabled depends on the client software you use, openssh for example has a command line switch for it (-L):
ssh user#host -L localport:remotehost:remoteport
This command would make the remoteport on remotehost, though accessible only through host, available on localport on your computer.
Take a look at the other suggested answers as they seem easier to accomplish what you need.
However, if you really need to implement the command submission with Java for your lab assignment, you can take a look at the JSch (Java Secure Channel) library found here: http://www.jcraft.com/jsch/ Examples are here http://www.jcraft.com/jsch/examples/
With it you can submit ssh commands and perform any kind of operation via a Java API
If you run "ssh" followed by any command that command gets executed on the remote host. So you should be able to run pre-baked queries in batch mode via ssh.
Consider doing key-gen and key exchanges to enable passwordless ssh execution.
Example (this just dumps a directory listing to your terminal):
ssh me#mybox ls
I'm currently trying to create an automated test using Java that runs some commands in a remote server the problem is that this remote server has a firewall. Manually I can ssh into the first server (firewall) using putty. Then I enter the details to the Server I wish to execute the commands. I've seen some article with code that mention how to use java code (jsch.jar) that I can ssh into a remote client but I haven't found a good enough explanation when a firewall is present. Can someone give me an explanation of what I should be trying to do and with some code snippet if possible
Not a complete answer, just an idea.
OpenSSH has a feature called ProxyCommand. It allows one to automatically issue a command on a target system, presumably another ssh.
So I have several entries of this sort in my .ssh/config:
Host the.private.host.behind.firewall.net
Hostname 10.0.100.106
User username_on_the_private_host
Compression yes
ProxyCommand ssh the.firewall.net nc -q 1 %h %p
Now I can issue ssh the.private.host.behind.firewall.net and first get to the SSH authentication on the.firewall.net, and then to the second authentication on the target host.
Likely Java implements SSH protocol on its own, but perhaps you could create a construction like this one.
Currently I am using weblogic for application server and it is running in one machine. I have to write the pdf(or any file type) to another Unix server. Can anyone please help me find the solution.
One solution is to mount the remote directory to your local server via smb or NFS (keyword sharing). Then you don't have to put any code for the copy job into your application. Instead the operating system will take care for the operation.
A simple and raw solution would be a SSH connection. There exists several Java libraries for SSH such as http://www.jcraft.com/jsch. Open a connection, write your file, close the connection. Of course you need the corresponding SSH keys and privileges to do this on the remote machine.
My desktop is windows. And I would like to connect a Linux server remotely. The middle target is to using script to ssh(using password, not key pairs) remote linux server from windows and run a command. I have putty installed in my machine. The final target here is that I could use java to collect output from this script(remote connect a server via ssh password and run a command) and then parse the output and extract the data I want and put it in excel sheet. The remote server needs login and password.
Why don't you try JSch?
use jsch of jcraft
here is one example
http://www.jcraft.com/jsch/examples/JTAJSch.java
you can find it on SO as well
Jsch or SSHJ or Ganymed SSH-2?
you could run plink (from PuTTY's developers) for running batches and writing the output to files, then process that with Java.
I'm not sure how simple linking Java to PuTTY would be.
i'm trying to execute a shell script from a j2ee application (made with flash builder 3, spring, apache cxf) et get the result of its execution in my flex interface.
the problem is my application is on a windows 7 station and i don't know how i can execute the script on a distant unix server & get back the result.
i know that ssh apis can help but i've no idea how to get back the result.
any help will be welcome.
thanx
If you have ssh installed on your windows machine, you should be able to execute a command like
ssh user#remote_host ipconfig
This will execute ipconfig on the remote_host as user "user". You will need to do a bit of research into ssh so that you can make it so you can log in without using a password, but google will help with that.
Alternatively you could look in to a java implementation of ssh - jssh for example, although I confess that I have no experience of using that package.
To execute program from windows to unix you really need ssh or telnet.
SSH is more secure. You can do this without running external process. Use one of available pure java SSH libraries (e.g. javassh.org).
See examples. If you use this library your task is trivial. Just call appropriate API.
About the only reasonable and reasonably secure answer I could come up with is to configure ssh on both machines.
*nix boxes usually have ssh server installed by default.
Putty terminal emulation for windows comes with neat ssh client command line utility called plink which can execute shell commands on a remote unix box in a secure manner.