I have written a shell script for automatic
1) start of hadoop services (namenode,datanode,jobtracker,tasktracker,secondary namenode),
2) dropping all tables from hive
3) again importing all tables in hive from SQL SERVER
And I am calling this shel script from java. Below is the code of Shell Script and Java Code
Shell Script:
export HADOOP_HOME=/home/hadoop/hadoop-0.20.2-cdh3u2/
export HIVE_HOME=/home/hadoop/hive-0.7.1/
export SQOOP_HOME=/home/hadoop/sqoop-1.3.0-cdh3u1/
export MSSQL_CONNECTOR_HOME=/home/hadoop/sqoop-sqlserver-1.0
export HBASE_HOME=/home/hadoop/hbase-0.90.1-cdh3u0
export ZOOKEEPER_HOME=/home/hadoop/zookeeper-3.3.1+10
export SQOOP_CONF_DIR=/home/hadoop/sqoop-1.3.0-cdh3u1/conf/
/home/hadoop/hadoop-0.20.2-cdh3u2/bin/hadoop/start-all.sh
/home/hadoop/hadoop-0.20.2-cdh3u2/bin/hadoop -rmr /user/hadoop/*
/home/hadoop/hive-0.7.1/bin/hive -e 'show tables' > TablesToDelete.txt
while read line1
do
echo 'drop table '$line1
/home/hadoop/hive-0.7.1/bin/hive -e 'drop table '$line1
done < TablesToDelete.txt
while read line
do
echo $line" ------------------------------"
/home/hadoop/sqoop-1.3.0-cdh3u1/bin/sqoop-import --connect 'jdbc:sqlserver://192.168.1.1;username=abcd;password=12345;database=HadoopTest' --table line --hive-table $line --create-hive-table --hive-import -m 1 --hive-drop-import-delims --hive-home /home/hadoop/hive-0.7.1 --verbose
done < /home/hadoop/sqoop-1.3.0-cdh3u1/bin/tables.txt
Java Code:
public class ImportTables
{
public static void main(String arsg[])
{
PrintWriter pw=null;
try
{
Formatter formatter = new Formatter();
String LogFile = "Log-"+ formatter.format("%1$tm%1$td-%1$tH%1$tM%1$tS", new Date());
File f=new File("/home/hadoop/"+LogFile);
FileWriter fw1=null;
pw=new PrintWriter(f);
String cmd = "/home/hadoop/sqoop-1.3.0-cdh3u1/bin/TablesToImport.sh"; // this is the command to execute in the Unix shell
// create a process for the shell
ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
pb.redirectErrorStream(true); // use this to capture messages sent to stderr
Process shell = pb.start();
InputStream shellIn = shell.getInputStream(); // this captures the output from the command
int shellExitStatus = shell.waitFor();
// wait for the shell to finish and get the return code
// at this point you can process the output issued by the command
// for instance, this reads the output and writes it to System.out:
int c;
while ((c = shellIn.read()) != -1)
{
System.out.write(c);
}
// close the stream
shellIn.close();
}
catch(Exception e)
{
e.printStackTrace();
e.printStackTrace(pw);
pw.flush();
System.exit(1);
}
}
}
But as I run the program I see nothiing on the console, and program remains in running mode.
And If I put the following code ion shell script:
/home/hadoop/hive-0.7.1/bin/hive -e 'show tables' > TablesToDelete.txt
while read line1
do
echo 'drop table '$line1
/home/hadoop/hive-0.7.1/bin/hive -e 'drop table '$line1
done < TablesToDelete.txt
Then the output come as:
Cannot find hadoop installation: $HADOOP_HOME must be set or hadoop must be in the path
What is the problem in my program/script? Where and How to set HADOOP_HOME and all that path in my script?
The call to waitFor is a blocking call, just as the name implies. It halts further execution until the process is done. But since your code is also the sink for the process's stdout, the whole thing blocks. Just move the waitFor to after you've processed the script's output.
Related
Am trying to get a series of commands on git bash one after the other. I can open the terminal through the code but after that wasn't successful with entering anything. For instance this is the code I tried
String [] args = new String[] {"C:\\Program Files\\Git\\git-bash.exe"};
String something="hi how are you doing";
try {
ProcessBuilder p = new ProcessBuilder();
var proc = p.command(args).start();
var w = new BufferedWriter(new OutputStreamWriter(proc.getOutputStream()));
w.write(something);
} catch (IOException ioException){
System.out.println(ioException);
}
Please let know how to be able to do enter a series of commands into git bash through the code.
The problem is that the command git-bash.exe opens the terminal window but the window's input is still the keyboard, so trying to write to the OutputStream that is returned by method getOutputStream(), in class Process does nothing. Refer to this question.
As an alternative, I suggest using using ProcessBuilder to execute a series of individual git commands. When you do that, your java code gets the command output.
Here is a simple example that displays the git version.
import java.io.IOException;
public class ProcBldT4 {
public static void main(String[] args) {
// C:\Program Files\Git\git-bash.exe
// C:\Program Files\Git\cmd\git.exe
ProcessBuilder pb = new ProcessBuilder("C:\\Program Files\\Git\\cmd\\git.exe", "--version");
pb.inheritIO();
try {
Process proc = pb.start();
int exitStatus = proc.waitFor();
System.out.println(exitStatus);
}
catch (IOException | InterruptedException x) {
x.printStackTrace();
}
}
}
When you run the above code, the git version details will be written to System.out.
Also, if the git command fails, the error details are written to System.err.
You need to repeat the code above for each, individual git command that you need to issue.
Hi I have an elasticsearch index which needs to deleted when a new entry of that type has inserted. I was able create a shellscript file and insert data into elasticsearch with a java appliaction as below,
public static void main(String[] args) {
try {
// Run the process
/*Runtime.getRuntime().exec("cd src/resources");*/
Process p = Runtime.getRuntime().exec("sh src/resources/test.sh");
// Get the input stream
InputStream is = p.getInputStream();
// Read script execution results
int i;
StringBuffer sb = new StringBuffer();
while ((i = is.read()) != -1)
sb.append((char) i);
System.out.println(sb.toString());
} catch (IOException e) {
e.printStackTrace();
}
}
Test.sh
curl -XDELETE http://localhost:9200/pokedex
With my java program it was possible to insert data but I can't delete, I tried to run the above command in command-line and it worked fine. What is happening here? Why is it executing in command-line and not in my program?
Does your Java program know what "sh" is? Have you tried fully qualifying the command:
Process p = Runtime.getRuntime().exec("/bin/sh src/resources/test.sh");
What is the error when it doesn't run? Is it best practice to run a shell script in a Java program? Are there other ways of running commands in Java programs?
java Runtime.exec to run shell script
I need to run the below comand using java but it is running fine in terminal as
svn list http://192.168.0.19/svn/cc/Branch/Jobs/tt/jobs/ --username prasadh --password prasadh2k > output.txt
But when running the same via process builder it is returning empty result.
My code:
ProcessBuilder pb = new ProcessBuilder("cmd", "C:\\Users\\dev112\\output", "svn", "list", "http://192.168.0.19/svn/cadgraf/Branch/Jobs/T0003SATHYABAMAT/Completedjobs", "--username", "prasadh", "--password", "prasadh2k", ">", "output.txt");
pb.redirectErrorStream(true);
try {
Process p = pb.start();
new Thread(new InputConsumerforImageMagick.InputConsumer(p.getInputStream())).start();
try {
System.err.println("Exited with: " + p.getErrorStream());
} catch (Exception ex) {
Logger.getLogger(AddImage.class.getName()).log(Level.SEVERE, null, ex);
}
} catch (IOException ex) {
Logger.getLogger(AddImage.class.getName()).log(Level.SEVERE, null, ex);
}
I/O redirection doesn't work well with ProcessBuilder. You should either call cmd.exe with
new ProcessBuilder("cmd", "/c", "svn ... > output.txt");
(i.e. you have to call cmd with exactly two arguments)
or you must redirect yourself, that is you need to start a background thread which reads stdout from the process and writes it to output.txt. In that case, you should use:
new ProcessBuilder("svn", "list", ...);
The former is brittle when you have spaces in arguments. So I suggest the latter even though the Java code is much more complex.
You should also have a look at Commons Exec which makes it much easier to deal with external processes.
Or with Java 7, you can use pb.redirectOutput();
Don't go through cmd. Just run the command directly:
final Path cwd = Paths.get("c:\\Users\\dev112\\output");
Files.createDirectories(cwd);
final Path outfile = cwd.resolve("output.txt");
final ProcessBuilder pb = new ProcessBuilder("svn", "list",
"http://192.168.0.19/svn/cadgraf/Branch/Jobs/T0003SATHYABAMAT/Completedjobs",
"--username", "prasadh", "--password", "prasadh2k");
pb.directory(cwd.toFile());
pb.redirectOutput(outfile.toFile());
final int retcode = pb.start().waitFor();
What is more, why do you get the process' standard output if you output to a file? Do one or the other, not both. If you output to a file then read the contents of that file after the command is executed.
The sample above outputs to a file; just open a stream to that file afterwards using Files.newInputStream(outfile) (well, that is, if retcode is 0; if it isn't, your command has ended with an error; which also means that you should redirect stderr somewhere, too)
This works for me:
String command = "svn list http://192.168.0.19/svn/cc/Branch/Jobs/tt/jobs/ --username prasadh --password prasadh2k";
ProcessBuilder processBuilder = new ProcessBuilder(command.split());
processBuilder.redirectOutput(new File("C:/Users/dev112/output/", "output.txt"));
processBuilder.start();
I have binary of engine already developed with Hadoop(HDFS,HBASE,MAPREDUCE) and java which is use to generate CSV file. There are some operation performed by this engine like table creation in HBASE and generating the CSV File from this HBASE. But this engine is performed all operation through only command line , as input is given from Linux terminal in form of command. Now my requirement is to connect this Linux terminal Through java program and run the commands But I am not able to successfully run any of the command
There are two option I tried but none of them successfully worked. Please provide any suggestion or solution to solve this problem as I am just beginner of Linux and hadoop to figure out the problem
1st-way
public class EngineTest {
public static void main(String[] args) {
try {
Process process = Runtime
.getRuntime()
.exec("/home/cloudera/PVGproto/Base/ anloss -i ${TOOL_INPUT}/census_10000_col5.csv -d ${TOOL_DEF}/attr_all_def.txt -q k=14,dage=2 -g ${TOOL_RES}/census_100_col8_gen.csv");
process.waitFor();
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(process.getInputStream()));
String line = "";
String output = "";
while ((line = bufferedReader.readLine()) != null) {
output += line + "\n";
}
System.out.println(output);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
2nd-way
I made one ex.sh file, in this file I put the required command for execution and call this . sh file from the same java program but the same thing happened its not running the command through java program
Process process = Runtime.getRuntime().exec("/home/cloudera/PVGproto/Base/ex.sh");
but if I run the same ex.sh from Linux terminal its running all the command successfully.
ex.sh
# !/bin/bash
exec /home/cloudera/PVGproto/Base/ anloss -i ${TOOL_INPUT}/census_10000_col5.csv -d ${TOOL_DEF}/attr_all_def.txt -q k=14,age=2 -g ${TOOL_RES}/census_100_col8_gen.csv
echo command run successfully
what I would like to do is run a batch file multiple times from a java application. Therefore I set up a for-loop that runs this code n times:
for (int i = 0; i < n; i++) {
Runtime.getRuntime().exec("cmd /c start somefile.bat");
}
The problem is that now each time the command is run a new cmd window pops up. However, what I want is just one window that pops up at the beginning and that is used to display all data from the following command calls.
How can I do that?
With && you can execute more than one commands, one after another:
Runtime.getRuntime().exec("cmd /c \"start somefile.bat && start other.bat && cd C:\\test && test.exe\"");
Using multiple commands and conditional processing symbols
You can run multiple commands from a single command line or script using conditional processing symbols. When you run multiple commands with conditional processing symbols, the commands to the right of the conditional processing symbol act based upon the results of the command to the left of the conditional processing symbol.
For example, you might want to run a command only if the previous command fails. Or, you might want to run a command only if the previous command is successful.
You can use the special characters listed in the following table to pass multiple commands.
& [...] command1 & command2
Use to separate multiple commands on one command line. Cmd.exe runs the first command, and then the second command.
&& [...] command1 && command2
Use to run the command following && only if the command preceding the symbol is successful. Cmd.exe runs the first command, and then runs the second command only if the first command completed successfully.
|| [...] command1 || command2
Use to run the command following || only if the command preceding || fails. Cmd.exe runs the first command, and then runs the second command only if the first command did not complete successfully (receives an error code greater than zero).
( ) [...] (command1 & command2)
Use to group or nest multiple commands.
; or , command1 parameter1;parameter2
Use to separate command parameters.
I would use Java's ProcessBuilder or another class which simulates/uses a shell. The following snippet demonstrates the idea (for Linux with bash).
import java.util.Scanner;
import java.io.*;
public class MyExec {
public static void main(String[] args)
{
//init shell
ProcessBuilder builder = new ProcessBuilder( "/bin/bash" );
Process p=null;
try {
p = builder.start();
}
catch (IOException e) {
System.out.println(e);
}
//get stdin of shell
BufferedWriter p_stdin =
new BufferedWriter(new OutputStreamWriter(p.getOutputStream()));
// execute the desired command (here: ls) n times
int n=10;
for (int i=0; i<n; i++) {
try {
//single execution
p_stdin.write("ls");
p_stdin.newLine();
p_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
}
}
// finally close the shell by execution exit command
try {
p_stdin.write("exit");
p_stdin.newLine();
p_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
}
// write stdout of shell (=output of all commands)
Scanner s = new Scanner( p.getInputStream() );
while (s.hasNext())
{
System.out.println( s.next() );
}
s.close();
}
}
Please note that it is only a snippet, which needs to be adapted for Windows, but in general it should work with cmd.exe.
public void TestCommandRun(){
Process process = null;
String[] command_arr = new String[]{"cmd.exe","/K","start"};
ProcessBuilder pBuilder = new ProcessBuilder("C:/Windows/System32/cmd.exe");
try{
process = pBuilder.start();
}
catch(IOException e){
e.printStackTrace();
System.out.println("Process failed");
}
if(null != process){
OutputStream out = process.getOutputStream();
OutputStreamWriter outWriter = new OutputStreamWriter(out);
BufferedWriter bWriter = new BufferedWriter(outWriter);
try{
bWriter.write("dir");
bWriter.newLine();
bWriter.write("ipconfig");
bWriter.flush();
bWriter.close();
}
catch(IOException e){
e.printStackTrace();
System.out.println("bWriter Failed");
}
}
}