I have a Java program where I am triggering a shell scripts.
Java Code Sample is :
ProcessBuilder pb = new ProcessBuilder(cmdList);
p = pb.start();
p.waitFor();
Where cmdList contains all required necessary input argument to execute the shell. This shell script is having a for loop inside and executing some DB scripts in that loop and printing result info & error logs in a file.
Below is sample shell script code :
#!/bin/bash
export PATH=/apps/PostgresPlus/as9.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
set -eE
#################################################### START
TIME_ELAPSED=""
TIME_ELAPSED_IN_HOURS=""
SCRIPT_START_TIME_FORMATTED=date '+%F %T'
SCRIPT_START_TIME_IN_SEC=date +%s
PROCESS_LOG_BASE_PATH="/data/logs/purge_log/"
PROCESS_LOG="$PROCESS_LOG_BASE_PATH/purge.log"
trap 'err=$?; logError 2>&1 "Error occurred during purging. Exiting with status $err at line $LINENO: ${BASH_COMMAND}. Please check logs for more info." >>$PROCESS_LOG' ERR
trap 'logError 2>&1 "Error occurred during purging. Exiting shell script execution as an external interrupt was received. Please check logs for more info." >>$PROCESS_LOG; trap ERR' INT
banner()
{
echo "+------------------------------------------------------------------------------------------------+"
printf "|tput bold[ %-40s tput sgr0|\n" "$1 ] tput setaf 2 $2"
echo "+------------------------------------------------------------------------------------------------+"
}
logError()
{
printf "[ProcessId- $$] [date "+%Y-%m-%d %H:%M:%S"] tput setaf 1 tput bold [ERROR] tput setaf 1 %-40s tput sgr0\n" "$#"
}
logInfo(){
printf "[ProcessId- $$] [date "+%Y-%m-%d %H:%M:%S"] tput setaf 6 bold [INFO] %-40s tput sgr0\n" "$#"
}
logWarn(){
printf "[ProcessId- $$] [date "+%Y-%m-%d %H:%M:%S"] tput setaf 3 tput bold [WARNING] %-40s tput sgr0\n" "$#"
}
logHint(){
printf "[ProcessId- $$] [date "+%Y-%m-%d %H:%M:%S"] tput setaf 5 tput sitm %-40s tput sgr0\n" "$#"
}
main()
{
banner "$SCRIPT_START_TIME_FORMATTED" "Started processing" | tee -a $PROCESS_LOG
logInfo "Started execution at $SCRIPT_START_TIME_FORMATTED" | tee -a $PROCESS_LOG
set PGPASSWORD=$DB_PASSWORD
export PGPASSWORD=$DB_PASSWORD
# Call DB function for audit and category wise data purging, population of schema names
SCHEMA_NAMES_RESULT=$(psql -h $HOST_NAME -d $DB_NAME -U $DB_USER -p $DB_PORT -At -c "SELECT $COMMON_SCHEMA_NAME.purge_audit_and_populate_schema_names('$COMMON_SCHEMA_NAME', $PURGE_DATA_INTERVAL_IN_DAYS,'$SCHEMA_NAMES',$NUM_TOP_CONTRIBUTING_TENANTS)")
SCHEMA_NAMES_RESULT=$(echo "$SCHEMA_NAMES_RESULT" | sed 's/{//g; s/}//g; s/"//g' )
SCHEMA_NAMES=$(echo $SCHEMA_NAMES_RESULT | rev | cut -d"," -f2- | rev)
#Convert comma separated string of tenants to array
SCHEMA_NAMES=($(echo "$SCHEMA_NAMES" | tr ',' '\n'))
# loop for multi schema
for element in "${SCHEMA_NAMES[#]}"
do
logInfo "Effective tenant - $element, Script start time - $SCRIPT_START_TIME_FORMATTED" | tee -a $PROCESS_LOG
# PGSQL call to DB function to execute purging
logInfo "Time elapsed since script execution started - $TIME_ELAPSED" | tee -a $PROCESS_LOG
done
#logInfo "Purge completed!" | tee -a $PROCESS_LOG
logInfo "Purge execution completed successfully at `date '+%F %T'`" | tee -a $PROCESS_LOG
exit 0
}
mkdir -p $PROCESS_LOG_BASE_PATH
main "$#"
#################################################### END
Following is my observation with this program.
When running shell script directly on putty it executes properly without any error.
When triggering shell script through above java program the following behavior I observed.
a. It hangs after a certain iteration in for loop.
b. As I reduce the number of logs entries from shell scripts, iteration (for loop) numbers keeps on increasing.
c. When I removed all info logs and keeps on printing only error log then it completed successfully.
Can someone please help in identifying the reason behind this behavior.
For now, I put check on the number of iteration in for loop but that problem can occur any time when I will start receiving multiple error log.
Regards
Kushagra
You have to consume the process streams or map err and out to file so the native buffers don't fill up. It works better if you create threads to consume each stream. The hacky single thread version is something like this:
ProcessBuilder pb = new ProcessBuilder(cmdList);
p = pb.start();
try (InputStream in = p.getInputStream();
InputStream err = p.getErrorStream();
OutputStream closeOnly = p.getOutputStream()) {
while (p.isAlive()) {
long skipped = 0L;
try {
skipped = in.skip(in.available())
+ err.skip(err.available());
} catch (IOException jdk8155808) {
byte[] b = new byte[2048];
int read = in.read(b, 0, Math.min(b.length, in.available());
if (read > 0) {
skipped += read;
}
read = err.read(b, 0, Math.min(b.length, err.available());
if (read > 0) {
skipped += read;
}
}
if(skipped == 0L) {
p.waitFor(5L, TimeUnit.MILLISECONDS);
}
}
} finally {
p.destroy();
}
The thread way works like this:
public void foo() {
class DevNull implements Runnable {
private final InputStream is;
DevNull(final InputStream is) {
is = Objects.requireNonNull(is);
}
public void run() {
byte[] b = new byte[64];
try {
while (is.read(b) >= 0);
} catch(IOException ignore) {
}
}
}
ExecutorService e = Executors.newCachedThreadPool();
ProcessBuilder pb = new ProcessBuilder(cmdList);
Process p = pb.start();
try (InputStream in = p.getInputStream();
InputStream err = p.getErrorStream();
OutputStream closeOnly = p.getOutputStream()) {
e.execute(new DevNull(in));
e.execute(new DevNull(err));
p.waitFor();
} finally {
p.destroy();
e.shutdown();
}
}
Thanks multi threading one worked for me.
For single thread option it got failed on skip().
Thanks again for helping in resolving the issue.
Related
Environment
Windows 10
Java 1.8
Process
I am running a 7zip's zip task.
The process takes 2 to 3 hours to complete.
Exception
java.lang.IllegalThreadStateException: process has not exited
at java.lang.ProcessImpl.exitValue(ProcessImpl.java:443)
at java.lang.ProcessImpl.waitFor(ProcessImpl.java:452at
My code
int exitValue = -1;
Process start = null;
try
{
ProcessBuilder processBuilder = new ProcessBuilder(commands);
start = processBuilder.start();
try(BufferedReader ipBuf = new BufferedReader(new InputStreamReader(start.getInputStream())))
{
String line = null;
while ((line = ipBuf.readLine()) != null)
{
LOGGER.info(line);
}
}
try(BufferedReader errBuf = new BufferedReader(new InputStreamReader(start.getErrorStream())))
{
String line;
while ((line = errBuf.readLine()) != null)
{
LOGGER.warning(line);
}
}
start.waitFor();
exitValue = start.exitValue();
}
finally
{
if (start != null)
{
start.destroy();
}
}
return exitValue;
I'm unable to find the root cause of this issue.
Note: I've tried this process with a similar demo instance on the same
machine and it works fine.
Please help me resolve this, Thanks.
There are two parts to your problem:
The JDK has a bug which causes an exception to be thrown when a Windows process returns an exit code of 259.
The command that you pass to ProcessBuilder exits with an exit code of 259 when it shouldn't.
Tackling each point in turn:
The bug in the JDK is caused by the following flawed logic in the Windows-specific implementation of Process.waitFor(): First, it waits until the process exits. Then, it calls exitValue() to get the exit value from the process. But unfortunately exitValue() gets the exit value and then checks if it's a special value that indicates the process hasn't exited. Since waitFor() knows that the process has exited, it should get the exit value directly instead of calling this method which does an unwanted check. Hopefully the JDK developers will fix this bug soon.
You should be using the command-line version of 7-zip, 7z.exe which exits with a set of well-defined exit values (so it never returns 259).
If your problem is caused by the Windows ProcessBuilder exit code 259 bug then there are workarounds available: all you need to do to make sure that your sub-process does not exit with status code 259 and Windows JRE won't report java.lang.IllegalThreadStateException.
You can easily reproduce this issue by executing the following command with Runtime.getRuntime().exec(cmd) or ProcessBuilder(cmd):
String[] cmd = {"cmd.exe /c exit /b 259"};
If you have written the code for the sub-process then just edit your code so that exit code is never set to 259.
If you have not written the code for the sub-process then a rather hacky workaround is to wrap your Java sub-process launch with a "CMD.EXE" and mini-script which adapts non-zero sub-process exit back to exit codes 0 or 1:
String[] fixed = new String[] { "cmd.exe", "/c",
"(call "+String.join(" ", cmd)+ ") || (echo !!! DETECTED ERROR!!! && exit 1)" };
Note: I'm no expert on CMD. The above fix definitely won't work for certain commands or punctuation (eg those with quotes / spaces etc), and because it runs under CMD.EXE environment settings the outcome might be different to direct launch from the calling JVM.
Here is an example class you could test with:
/** Examples to test with and without the fix:
java Status259 "cmd.exe /c exit /b 0"
java Status259 "cmd.exe /c exit /b 25"
java Status259 "cmd.exe /c exit /b 259"
java Status259 %JAVA_HOME%\bin\java -cp your.jar Status259$StatusXXX 0
java Status259 %JAVA_HOME%\bin\java -cp your.jar Status259$StatusXXX 33
java Status259 %JAVA_HOME%\bin\java -cp your.jar Status259$StatusXXX 259
*/
public class Status259 {
public static class StatusXXX {
public static void main(String ... args) {
int status = args.length > 0 ? Integer.parseInt(args[0]) : 0;
System.out.println("StatusXXX exit code: "+status);
System.exit(status);
}
}
public static int exec(String[] cmd) throws IOException, InterruptedException {
System.out.println("exec "+Arrays.toString(Objects.requireNonNull(cmd)));
ProcessBuilder pb = new ProcessBuilder(cmd);
// No STDERR => merge to STDOUT - or call redirectError(File)
pb.redirectErrorStream(true);
Process p = pb.start();
// send sub-process STDOUT to the Java stdout stream
try(var stdo = p.getInputStream()) {
stdo.transferTo(System.out);
}
int rc = p.waitFor();
System.out.println("exec() END pid="+p.pid()+" CODE "+rc +' '+(rc == 0 ? "OK":"**** ERROR ****"));
return rc;
}
public static void main(String ... args) throws IOException, InterruptedException {
// COMMENT OUT NEXT LINE TO SEE EFFECT OF DIRECT LAUNCH:
args = fixStatus259(args);
int rc = exec(args);
System.exit(rc);
}
private static String[] fixStatus259(String[] cmd) {
System.out.println("fixStatus259 "+Arrays.toString(cmd));
return new String[] {
"cmd.exe", "/c",
"(call "+String.join(" ", cmd)+ ") || (echo !!! DETECTED ERROR!!! && exit 1)"
};
}
}
I'm trying to do some automation on MacOS with Java.
no problems when running the commands manually from a terminal
i assume it works because of <user.home>/.zprofile
the commands are not found when trying to execute them via ProcessBuilder
How can I execute commands with the same environment as if running a zsh terminal manually?
import java.io.BufferedReader;
import java.io.InputStreamReader;
public class Test {
public static void main(String[] args) throws Exception {
// these commands work
run("/bin/sh", "-c", "echo $PATH");
run("/bin/bash", "-c", "echo $PATH");
run("/bin/zsh", "-c", "echo $PATH");
// these commands all work when I run them manually in a terminal
// but fail here with "zsh:1: command not found: ..."
run("/bin/zsh", "-c", "node -v");
run("/bin/zsh", "-c", "npm -v");
}
private static void run(String... command) throws Exception {
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.redirectErrorStream(true);
processBuilder.command(command);
Process process = processBuilder.start();
try(BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
for(String line = br.readLine(); line != null; line = br.readLine()) {
System.out.println(line);
}
}
System.out.println("return value: " + process.waitFor());
}
}
Output:
/usr/bin:/bin:/usr/sbin:/sbin
return value: 0
/usr/bin:/bin:/usr/sbin:/sbin
return value: 0
/usr/bin:/bin:/usr/sbin:/sbin
return value: 0
zsh:1: command not found: node
return value: 127
zsh:1: command not found: npm
return value: 127
After reading way too many articles about shells, and studying shell init diagrams, I decided to go with Zsh.
The reason was this blog post, which indicated that Zsh seems to have at least one init file that is executed for all possible shell variants (login, non-login, interactive, non-interactive etc).
I moved all my environment setup (PATH and LANG) to /etc/zshenv, deleted /etc/zprofile and all ~/.z* files.
I also changed the shell for both root and my user to Zsh (for the user this can also be done via system preferences):
dscl . -delete /Users/root UserShell && dscl . -create /Users/root UserShell /bin/zsh && dscl . -read /Users/root UserShell
dscl . -delete /Users/reto UserShell && dscl . -create /Users/reto UserShell /bin/zsh && dscl . -read /Users/reto UserShell
Now I get the same environment for:
SSH as root
SSH as user
Terminal.app
Processes started from Java
And pretty much everything else so far
So far so good. Test program output:
/usr/bin:/bin:/usr/sbin:/sbin
return value: 0
/usr/bin:/bin:/usr/sbin:/sbin
return value: 0
/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
return value: 0
v14.17.0
return value: 0
6.14.13
return value: 0
I wrote a java code to execute Vowpal Wabbit in the following way:
System.out.println("Executing command " + command);
final Runtime r = Runtime.getRuntime();
final Process p = r.exec(command);
System.out.println("waiting for the process");
try (final BufferedReader b = new BufferedReader(new InputStreamReader(p.getInputStream()))) {
String line;
while ((line = b.readLine()) != null) {
final T lineResult = textParser.parseLine(line);
parserResultCombiner.addToCombiner(lineResult);
}
}
p.waitFor();
System.out.println("done");
}
where the command is
vw -d input.txt --loss_function=logistic -f model.vw
The disadvantage of this is that it requires writing to disk. After some searching, I learned that vowpal wabbit supports reading data from standard input
example in R
I could not find any example to accomplish this in Java 1.8. Could anyone share one with me?
You need to start vw in daemon mode. This starts a process that listens on the port specified.
$ vw -i model.vw -t --daemon --quiet --port 26542
Once the daemon has started, you can send samples to predict using socket calls
$ echo " abc-example| a b c" | netcat localhost 26542
0.000000 abc-example
$ echo " xyz-example| x y z" | netcat localhost 26542
1.000000 xyz-example
source:
https://github.com/JohnLangford/vowpal_wabbit/wiki/daemon-example
Recently they pushed a java version of the code that interacts with vw using jni
https://github.com/JohnLangford/vowpal_wabbit/tree/master/java
I have written a shell script for automatic
1) start of hadoop services (namenode,datanode,jobtracker,tasktracker,secondary namenode),
2) dropping all tables from hive
3) again importing all tables in hive from SQL SERVER
And I am calling this shel script from java. Below is the code of Shell Script and Java Code
Shell Script:
export HADOOP_HOME=/home/hadoop/hadoop-0.20.2-cdh3u2/
export HIVE_HOME=/home/hadoop/hive-0.7.1/
export SQOOP_HOME=/home/hadoop/sqoop-1.3.0-cdh3u1/
export MSSQL_CONNECTOR_HOME=/home/hadoop/sqoop-sqlserver-1.0
export HBASE_HOME=/home/hadoop/hbase-0.90.1-cdh3u0
export ZOOKEEPER_HOME=/home/hadoop/zookeeper-3.3.1+10
export SQOOP_CONF_DIR=/home/hadoop/sqoop-1.3.0-cdh3u1/conf/
/home/hadoop/hadoop-0.20.2-cdh3u2/bin/hadoop/start-all.sh
/home/hadoop/hadoop-0.20.2-cdh3u2/bin/hadoop -rmr /user/hadoop/*
/home/hadoop/hive-0.7.1/bin/hive -e 'show tables' > TablesToDelete.txt
while read line1
do
echo 'drop table '$line1
/home/hadoop/hive-0.7.1/bin/hive -e 'drop table '$line1
done < TablesToDelete.txt
while read line
do
echo $line" ------------------------------"
/home/hadoop/sqoop-1.3.0-cdh3u1/bin/sqoop-import --connect 'jdbc:sqlserver://192.168.1.1;username=abcd;password=12345;database=HadoopTest' --table line --hive-table $line --create-hive-table --hive-import -m 1 --hive-drop-import-delims --hive-home /home/hadoop/hive-0.7.1 --verbose
done < /home/hadoop/sqoop-1.3.0-cdh3u1/bin/tables.txt
Java Code:
public class ImportTables
{
public static void main(String arsg[])
{
PrintWriter pw=null;
try
{
Formatter formatter = new Formatter();
String LogFile = "Log-"+ formatter.format("%1$tm%1$td-%1$tH%1$tM%1$tS", new Date());
File f=new File("/home/hadoop/"+LogFile);
FileWriter fw1=null;
pw=new PrintWriter(f);
String cmd = "/home/hadoop/sqoop-1.3.0-cdh3u1/bin/TablesToImport.sh"; // this is the command to execute in the Unix shell
// create a process for the shell
ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
pb.redirectErrorStream(true); // use this to capture messages sent to stderr
Process shell = pb.start();
InputStream shellIn = shell.getInputStream(); // this captures the output from the command
int shellExitStatus = shell.waitFor();
// wait for the shell to finish and get the return code
// at this point you can process the output issued by the command
// for instance, this reads the output and writes it to System.out:
int c;
while ((c = shellIn.read()) != -1)
{
System.out.write(c);
}
// close the stream
shellIn.close();
}
catch(Exception e)
{
e.printStackTrace();
e.printStackTrace(pw);
pw.flush();
System.exit(1);
}
}
}
But as I run the program I see nothiing on the console, and program remains in running mode.
And If I put the following code ion shell script:
/home/hadoop/hive-0.7.1/bin/hive -e 'show tables' > TablesToDelete.txt
while read line1
do
echo 'drop table '$line1
/home/hadoop/hive-0.7.1/bin/hive -e 'drop table '$line1
done < TablesToDelete.txt
Then the output come as:
Cannot find hadoop installation: $HADOOP_HOME must be set or hadoop must be in the path
What is the problem in my program/script? Where and How to set HADOOP_HOME and all that path in my script?
The call to waitFor is a blocking call, just as the name implies. It halts further execution until the process is done. But since your code is also the sink for the process's stdout, the whole thing blocks. Just move the waitFor to after you've processed the script's output.
Really simple little function, but does anyone know how to sleep OS X from Java?
Cheers
System.exec("osascript -e 'tell application \"System Events\" to sleep'");
See: Making Mac OS X sleep from the command line
Create a script with the following:
#!/bin/bash
osascript << EOT
tell application "System Events"
sleep
end
EOT
And use system to exec it.
public void gotoSleep(){
try{
logger.finer("Zzz...");
if (preferences.getOS().equals("OSX") == true ){
Process p = Runtime.getRuntime().exec
("/bin/bash");
String command = "osascript -e 'tell application \"System Events\"' "
+ " -e \"sleep\" -e 'end tell'";
OutputStream stdin = p.getOutputStream();
stdin.write( command.getBytes() );
stdin.flush();
stdin.close();
}
}catch( Exception e ) {
logger.warning( e.toString() );
}
}
For Some reason while i was doing it it did not work without executing it through bash.