Develop Reference

How to run a multithreaded python program from command in Java

As I know pretty much nothing about Python, I'm trying to run a python script that I need from Java. I don't need to interface with it's interfaces or anything else, just simply start it with certain arguments.
Following other overflow threads like this one, I am starting the script as follows:
String longitude = "snip"; // Will be a number like +/-xx.xxxxxxx
String latitude = "snip"; // Will be a number like +/-xx.xxxxxxx
String[] command = {"python", "example.py",
"-a", "ptc",
"-u", "snip", // Username
"-p", "snip", // Password
"-l", longitude + " " + latitude,
"-st", "10"
};
ProcessBuilder probuilder = new ProcessBuilder(command);
probuilder.directory(new File("snip")); // Simply a folder where file is
Process p = probuilder.start();
After this I just have a few lines that capture the output of the process and output it to System.out, then run Process.waitFor() so the main thread doesn't return until the python finishes. The issue is, the python script I'm running (found here) uses multiple threads, and I believe it's unable to start all threads. I get output from Flask, but then nothing happens yet Process.waitFor() does not return. Is there a way to correctly run multithreaded python scripts through java?
For reference, here is the output I'm getting, and here is expected output.

How to check if a process is running on Windows?

What I am trying to do it is a program to know if a process it is active on Windows to work with it in Java. I looked on the Internet and found two different solutions:
Option 1
Option 2
But it is not at all what I am looking for.
According to the option 1, I would like a method to make reference to the process (and detect if it is active/running) but without searching in the full tasklist and then searching on it. I am searching if there is a direct method to do that.
I also though to add a filter to the tasklist but could not find any filter to only get the process I am looking for. I saw all the options using tasklist /? on my command line.
Then I searched information about the second option and wmic (that I never heard before) and it seems that wmic allows you to execute tasks on the command line (correct me if I am wrong please).
So, here I have two questions:
Is there a direct method to know if a process is active on Windows with Java? Trying to avoid searching in the full tasklist or using wmic.
If it is impossible, what of the two options that I put above would be better if we talk about efficient programming?
Thanks in advance!

There is no direct way to query general processes as each OS handles them differently.
You kinda stuck with using proxies such as direct OS commands...
You can however, find a specific process using tasklist.exe with /fi parameter.
e.g: tasklist.exe /nh /fi "Imagename eq chrome.exe"
Note the mandatory double quotes.
Syntax & usage are available on MS Technet site.
Same example, filtering for "chrome.exe" in Java:
String findProcess = "chrome.exe";
String filenameFilter = "/nh /fi \"Imagename eq "+findProcess+"\"";
String tasksCmd = System.getenv("windir") +"/system32/tasklist.exe "+filenameFilter;
Process p = Runtime.getRuntime().exec(tasksCmd);
BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
ArrayList<String> procs = new ArrayList<String>();
String line = null;
while ((line = input.readLine()) != null)
procs.add(line);
input.close();
Boolean processFound = procs.stream().filter(row -> row.indexOf(findProcess) > -1).count() > 0;
// Head-up! If no processes were found - we still get:
// "INFO: No tasks are running which match the specified criteria."

With Java 9 you can do it like this:
Optional<ProcessHandle> processHandle = ProcessHandle.of(pid);
boolean isrunning = processHandle.isPresent() && processHandle.get().isAlive();
You are now also able, to get some info about the process directly with java.
Optional<ProcessHandle> processHandle = ProcessHandle.of(pid);
if(processHandle.isPresent()) {
ProcessHandle.Info processInfo = processHandle.get().info();
System.out.println("COMMAND: " + processInfo.command().orElse(""));
System.out.println("CLI: " + processInfo.commandLine().orElse(""));
System.out.println("USER: " + processInfo.user().orElse(""));
System.out.println("START TIME: " + processInfo.startInstant().orElse(null));
System.out.println("TOTAL CPU: " + processInfo.totalCpuDuration().orElse(null));
}

The best way would be launching the process from the java instance. The advantages of this approach are that you can be fully sure that the process that is running, is the process you started. With tasklist, it is possible for normal users to start a process with the name as the target, or reuse the pid if the program closes.
For information how to start a process, see this question.
Process myProcess = Runtime.getRuntime().exec(command);
You can then use the methods of Process to monitor the state of the program, for example myProcess.exitValue() blocks until the program is closed.

Running UNIX Source command in Java

This is my first question on stackoverflow so I'll try to keep it concise and relevant.
I'm currently creating a Java program that is attempting to call an external program located on the system, in order to do this however I am required to call a shell script that sets up links to the relevant libraries to ensure that the system is linked to these before the external program can be executed.
The issue at hand is that I cannot invoke the shell script through Java, I've researched high and low and am aware that of alternative ways such as the use of the ProcessBuilder class. Unfortunately I'm quite new to the world of trying to invoke command line statements through Java and so I'm stuck for answers.
An example of the code I am using can be found below:
private void analyse_JButtonActionPerformed(java.awt.event.ActionEvent evt) {
// Get project path for copying of Fortran program to folder and execution
String projectPath = Newproject_GUI.getProjectPath();
String sourcePath [] = {"/bin/sh ", "-c ","source ~/set_env_WRF_gnu.sh"} ;
Runtime fortranAnalyser = Runtime.getRuntime();
try {
Process p = fortranAnalyser.exec("cp main.exe " + projectPath);
Process k = fortranAnalyser.exec(sourcePath);
BufferedReader is = new BufferedReader(new InputStreamReader(k.getInputStream()));
String line;
while ((line = is.readLine()) != null) {
System.out.println(line);
}
} catch (IOException ex) {
Logger.getLogger(Analyser_GUI.class.getName()).log(Level.SEVERE, null, ex);
}
}
Process p works fine and does indeed copy main.exe to the intended directory when the method is called. Process k however does not and this is where the issue is.
Thanks in advance.

The issue is "source" is internal command of BASH (you are using "sh" but that is just BASH in the simplified mode). So what you do is:
you spawn new process "sh" and source something there (setting some VARIABLES I guess)
the process ends and all VARIABLES are lost
you spawn another process, but VARIABLES are already gone
I am not sure if you use those variables later on, but according to the script name it is probably setting some. Don't do that like this.
By the way if you only want to execute script in bash, you don't need to source it. To get it's side effects, just execute it with:
String sourcePath [] = {"/bin/sh ", "/home/XYZ/set_env_WRF_gnu.sh"} ;
Please note you cannot use ~ in this case, use Java to get your home dir.

(Java) Changed directory (via bash script) not reflected in folder.listfiles

I run a bash script from my Java program which takes a chunk of data, manipulates it, and splits it up.
It's not a question of whether the bash script works -- I can see the split files in the directory.
Say the original file was "bigFile" in data/
Then
try
{
Process proc = Runtime.getRuntime().exec("bash " + SCRIPT_DIR + "/" + SPLIT_SCRIPT_NAME + " " + args[_MESSAGES_PER_UPLOAD_] + " " + args[_MAXIMUM_MESSAGES_PER_FEED_] + " " + (60000*Integer.parseInt(args[_DURATION_BEFORE_EACH_UPLOAD_IN_MINUTES_])/Integer.parseInt(args[_DURATION_OF_EACH_FEED_IN_MILLISECONDS_])));
proc.waitFor();
}
catch(IOException e) { error(e); }
String fileNames;
File folder = new File(DATA_DIR);
File[] filesToUpload = folder.listFiles();
for (int i = 0; i < filesToUpload.length; ++i)
if (filesToUpload[i].isFile())
{
fileNames = filesToUpload[i].getName();
System.out.println(fileNames);
}
Will print bigFile, not...
$ ls data/
dataChunk_00000
dataChunk_00001
dataChunk_00002
dataChunk_00003
dataChunk_00004
dataChunk_00005
dataChunk_00006
dataChunk_00007
dataChunk_00008
dataChunk_00009
dataChunk_00010
dataChunk_00011
dataChunk_00012
dataChunk_00013
dataChunk_00014
dataChunk_00015
dataChunk_00016
dataChunk_00017
dataChunk_00018
dataChunk_00019
dataChunk_00020
dataChunk_00021
dataChunk_00022
dataChunk_00023
dataChunk_00024
dataChunk_00025
dataChunk_00026
dataChunk_00027
as it should. I'm guessing this is a compiler optimization or something.
Edit: If somebody could explain to me why proc.waitFor() isn't working and/or a better way to solve this, I'd much appreciate it.

The problem with this is not compiler optimization or anything like that.
Its because you are invoking your script with a "bash" in front of it . This causes the process to fork -- so your bash command returns successfully immediately , but your script continues to run in the background and terminate.
The proc.waitFor() has nothing to wait for, the rest of the java program executes before your file has been "split".

You cannot change the directory with java.
If you want to "simulate" it, all you need to do is set the property "user.dir".

I am guessing that your bash script is performing actions asynchronously from its own process/thread. This means that the script finishes executing before the work is complete. This would still pass the waitFor() check and continue executing the code.
EDIT:
Kal's answer explains this more clearly, and it was posted first. The problem is the fact that you use the bash command to execute the script.

I suspect your arguments aren't all passed to your script.
Put all your arguments in an ArrayList instance, pass the instance to the ProcessBuilder, then call the start method on the builder instance, which returns the proc on which you call waitFor.
Here's sample Scala code to show what I mean (I can port it to Java if you're really interested ;-):
import java.lang.{ Process => JProcess, ProcessBuilder => JProcessBuilder }
import java.util.{ArrayList => JArrayList, List => JList, Map => JMap}
import java.io.{InputStreamReader, BufferedReader}
def call(args: String*) = {
val command: JList[String] = new JArrayList[String]()
args.foreach {arg =>
command.add(arg)
}
//log.debug("argument list: %s", command.toString)
val builder = new JProcessBuilder(command)
val proc: JProcess = builder.start()
proc.waitFor()
val read = new BufferedReader(new InputStreamReader(proc.getInputStream()));
val sb: StringBuffer = new StringBuffer()
while(read.ready()) {
sb.append(read.readLine)
}
// sb now has the output of the called process...
val exitValue: Int = proc.exitValue
// http://stuffthathappens.com/blog/2007/11/28/crash-boom-too-many-open-files/
read.close
proc.destroy
(exitValue, sb.toString) // return it
}
Example call in REPL:
scala> call("date")
res156: (Int, java.lang.String) = (0,Mon 18 Jul 2011 22:29:58 BST)

There are a number of wrong assumptions with this program:
Every time you do 'exec' you fork a new process, with its own environment, current directory, etc. Any change of the current directory would have been local to that process and will not affect the parent (your Java process). In other words, there is no way to change the current path of an application using a command in a sub-process, there is no Java API for that either - if you really need this, you have to use native call.
The 'cd' command on Unix is a real command, you do not need the shell in order to run it (unlike Windows).
When you fork a process, you need to make sure that you drain the stdout and stderr, or it is going to block when the OS buffer gets full (see next)
Process.waitFor() works. Always.
A better way to approach the problem is to read carefully the File API and as much as possible work with absolute paths. The 'current directory' is something very usefull when you are in shell, but for applications it ends up being more confusing, so the sooner you resolve it to absolute path - the better.

How do I find the process ID (pid) of a process started in java? [duplicate]

This question already has answers here:
How to get PID of process I've just started within java program?
(18 answers)
Closed 9 years ago.
If I get a process object in Java through Runtime.getRuntime().exec(...), or ProcessBuilder.start(), I can wait for it through Process.waitFor(), which is like Thread.join(), or I could kill it with Process.destroy(), which is like the deprecated Thread.stop().
BUT: How do I find the pid of the Process Object? I don't see a method for doing that in The Official Documentation. Can I do this in Java? If so, how?

This guy calls out to bash to get the PID. I'm not sure if there is an java solution to the problem.
/**
* Gets a string representing the pid of this program - Java VM
*/
public static String getPid() throws IOException,InterruptedException {
Vector<String> commands=new Vector<String>();
commands.add("/bin/bash");
commands.add("-c");
commands.add("echo $PPID");
ProcessBuilder pb=new ProcessBuilder(commands);
Process pr=pb.start();
pr.waitFor();
if (pr.exitValue()==0) {
BufferedReader outReader=new BufferedReader(new InputStreamReader(pr.getInputStream()));
return outReader.readLine().trim();
} else {
System.out.println("Error while getting PID");
return "";
}
}
Source:
http://www.coderanch.com/t/109334/Linux-UNIX/UNIX-process-ID-java-program

Similar to the other tools mentioned, there is the jps command line tool that comes with the Java runtime. It spits out the PIDs of all running JVMs. The benefit is the output one needs to parse is confined to only the JVM processes.

Leo, after looking into this issue for about a week myself I think Jhurtado's approach is likely the "best" approach we can manage in Java right now. "best" is in quotes because it has the very nasty side effect of basically being a "guess" at what your child PID is.
If your Java app is spawning native processes quickly in a high-load system, there is NO guarantee that the PID you pickup in your diff calculation is the PID of the Process started by the current Thread or that the PID of the process you pick was even spawned by our app (maybe the host system was already running that process anyway).
That being said, if you are not spawning dozens of processes or the native Process you are spawning is really unique (some custom util you ship with your app) then this approach works fine in which case the PID of the native process you are looking for is the one you want.
On windows you can use 'tasklist' as Jhurtado pointed out to get the full list of PIDs and filter for the one you want (using the /FI filter switch didn't work for me in testing).
On any *nix system you can use "ps ax | grep " where NAME is some process name like 'nginx' or 'httpd' that you want to filter for to get your list.
Additionally, if you need to kill stray processes (for example, on VM exit) on *nix you can of course use "kill -9 " and on Windows, interestingly enough, you can use 'taskkill '.
Hardly optimal unfortunately.

I ran into the same issue as you. I found a pretty decent solution, I recommend a slight sleep before it thought to ensure the process has officially started up.
Process p = Runtime.getRuntime().exec("cmd /c tasklist");
StringWriter writer = new StringWriter();
IOUtils.copy(p.getInputStream(), writer);
String theString = writer.toString();
//System.out.println(theString);
id = findLastString("javaw.exe .{1,} Console 1", theString);
System.out.println(id);
where findLastString is defined as
public static String findLastString(String pattern, String in) {
Pattern p = Pattern.compile(pattern);
Matcher matcher = p.matcher(in);
String it= "";
while(matcher.find()) {
it = matcher.group();
try {
Thread.sleep(10);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
int firstIndex=pattern.indexOf(".{1,}");
int lastIndex=it.substring(firstIndex).indexOf(pattern.substring(pattern.indexOf(".{1,}")+5))+firstIndex;
String dotOn = it.substring(pattern.indexOf(".{1,}"));
it=it.substring(firstIndex, lastIndex);
return it;
}
Basically this will get the list of running processes, and find the most recently ran one with, in this instance, the name javaw.exe (My program was starting a separate java process). You can replace javaw.exe with the name of the process, which you can find by using Task Manager. You will need to get the Apache common IO jar too.

I think in Java your best shot is to get the tasklist before and after spawning your child process. Make a diff and get your PID.
you can get the Tasklist by issuing a Runtime.getRuntime.exec("tasklist");
Notice that tasklist.exe is not included with Windows XP Home edition, but still you can download it.