byte[] bytes = value.getBytes();
Process q = new ProcessBuilder("process","arg1", "arg2").start();
q.getOutputStream().write(bytes);
q.getOutputStream().flush();
System.out.println(q.getInputStream().available());
I'm trying to stream file contents to an executable and capture the output but the output(InputStream) is always empty. I can capture the output if i specify the the file location but not with streamed input.
How might I overcome this?
Try wrapping your streams with BufferedInputStream() and BufferedOutputStream():
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getOutputStream%28%29
Implementation note: It is a good idea for the output stream to be buffered.
Implementation note: It is a good idea for the input stream to be buffered.
Even with buffered streams, it is still possible for the buffer to fill if you're dealing with large amounts of data, you can deal with this by starting a separate thread to read from q.getInputStream(), so you can still be reading from the process while writing to the process.
Perhaps the program you execute only starts its work when it detects the end of its input data. This is normally done by waiting for an EOF (end-of-file) symbol. You can send this by closing the output stream to the process:
q.getOutputStream().write(bytes);
q.getOutputStream().close();
Try this together with waiting for the process.
I dont know if something else may also be wrong here, but the other process ("process") does not even have time to respond, you are not waiting for it (the method available() does not block). To try this out you can first insert a sleep(2000) after the flush(), and if that works you should switch to query'ing q.getInputStream().available() multiple times with short pauses in between.
I think, you have to wait, until the process finished.
I implemented something like this this way:
public class ProcessReader {
private static final int PROCESS_LOOP_SLEEP_MILLIS = 100;
private String result;
public ProcessReader(Process process) {
BufferedReader resultReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder resultOutput = new StringBuilder();
try {
while (!checkProcessTerminated(process, resultReader, resultOutput)) {
}
} catch (Exception ex1) {
throw new RuntimeException(ex1);
}
result = resultOutput.toString();
}
public String getResult(){
return result;
}
private boolean checkProcessTerminated(Process process, BufferedReader resultReader, StringBuilder resultOutput) throws Exception {
try {
int exit = process.exitValue();
return true;
} catch (IllegalThreadStateException ex) {
Thread.sleep(PROCESS_LOOP_SLEEP_MILLIS);
} finally {
while (resultReader.ready()) {
String out = resultReader.readLine();
resultOutput.append(out).append("\n");
}
}
return false;
}
}
I just removed now some specific code, that you dont need, but it should work, try it.
Regards
Related
I'm a small java developer currently working on a discord bot that I made in Java. one of the features of my bot is to simply have a leveling system whenever anyone sends a message (and other conditions but this is irrelevant for the problem I'm encountering).
Whenever someone sends a message an event is fired and a thread is created to compute how much exp the user should gain. and eventually, the function to edit the storage file is called.
which works fine when called sparsely. but if two threads try to write on the file at once, the file usually gets deleted or truncated. either of these two cases being undesired behavior
I then tried to make a queuing system that worked for over 24h but still failed once so it is more stable in a way. I only know the basics of how threads work so I may've skipped over an important thing that causes the problem
the function looks like this
Thread editingThread = null;
public boolean editThreadStarted = false;
HashMap<String, String> queue = new HashMap<>();
public final boolean editParameter(String key, String value) {
queue.put(key, value);
if(!editThreadStarted) {
editingThread = new Thread(new Runnable() {
#Override
public void run() {
while(queue.keySet().size() > 0) {
String key = (String) queue.keySet().toArray()[0];
String value = queue.get(key);
File inputFile = getFile();
File tempFile = new File(getFile().getName() + ".temp");
try {
tempFile.createNewFile();
} catch (IOException e) {
DemiConsole.error("Failed to create temp file");
handleTrace(e);
continue;
}
//System.out.println("tempFile.isFile = " + tempFile.isFile());
try (BufferedReader reader = new BufferedReader(new FileReader(inputFile)); BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile))){
String currentLine;
while((currentLine = reader.readLine()) != null) {
String trimmedLine = currentLine.trim();
if(trimmedLine.startsWith(key)) {
writer.write(key + ":" + value + System.getProperty("line.separator"));
continue;
}
writer.write(currentLine + System.getProperty("line.separator"));
}
writer.close();
reader.close();
inputFile.delete();
tempFile.renameTo(inputFile);
} catch (IOException e) {
DemiConsole.error("Caught an IO exception while attempting to edit parameter ("+key+") in file ("+getFile().getName()+"), returning false");
handleTrace(e);
continue;
}
queue.remove(key);
}
editThreadStarted = false;
}
});
editThreadStarted = true;
editingThread.start();
}
return true;
}
getFile() returns the file the function is meant to write to
the file format is
memberid1:expamount
memberid2:expamount
memberid3:expamount
memberid4:expamount
the way the editing works is by creating a temporary file to which i will write all of the original file's data line by line, checking if the memberid matches with what i want to edit, if it does, then instead of writing the original file's line, i will write the new edited line with the new expamount instead, before continuing on with the rest of the lines. Once that is done, the original file is deleted and the temporary file is renamed to the original file, replacing it.
This function will always be called asynchronously so making the whole thing synchronous is not an option.
Thanks in advance
Edit(1) :
I've been suggested to use semaphores and after digging a little into it (i never heard of semaphores before) it seems to be a really good option and would remove the need for a queue, simply aquire in the beginning and release at the end, nothing more required!
I ended up using semaphores as per user207421's suggestions and it seems to work perfectly
I simply put delays between each line write to artificially make the task longer and make it easier to have multiple threads trying to write at once, and they all wait for their turns!
Thanks
What is the proper way to produce and consume the streams (IO) of external process from Java? As far as I know, java end input streams (process output) should be consumed in threads parallel to producing the process input due the possibly limited buffer size.
But I'm not sure if I eventually need to synchronize with those consumer threads, or is it enough just to wait for process to exit with waitFor method, to be certain that all the process output is actually consumed? I.E is it possible, even if the process exits (closes it's output stream), there is still unread data in the java end of the stream? How does the waitFor actually even know when the process is done? For the process in question, EOF (closing the java end of it's input stream) signals it to exit.
My current solution to handle the streams is following
public class Application {
private static final StringBuffer output = new StringBuffer();
private static final StringBuffer errOutput = new StringBuffer();
private static final CountDownLatch latch = new CountDownLatch(2);
public static void main(String[] args) throws IOException, InterruptedException {
Process exec = Runtime.getRuntime().exec("/bin/cat");
OutputStream procIn = exec.getOutputStream();
InputStream procOut = exec.getInputStream();
InputStream procErrOut = exec.getErrorStream();
new Thread(new StreamConsumer(procOut, output)).start();
new Thread(new StreamConsumer(procErrOut, errOutput)).start();
PrintWriter printWriter = new PrintWriter(procIn);
printWriter.print("hello world");
printWriter.flush();
printWriter.close();
int ret = exec.waitFor();
latch.await();
System.out.println(output.toString());
System.out.println(errOutput.toString());
}
public static class StreamConsumer implements Runnable {
private InputStream input;
private StringBuffer output;
public StreamConsumer(InputStream input, StringBuffer output) {
this.input = input;
this.output = output;
}
#Override
public void run() {
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String line;
try {
while ((line = reader.readLine()) != null) {
output.append(line + System.lineSeparator());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
latch.countDown();
}
}
}
}
}
Is it necessary to use the latch here, or does the waitFor implicate all the output is already consumed? Also, if the output doesn't end/contain new line, will the readLine miss the output, or still read all that is left? Does reading null mean process has closed it's end of the stream - is there any other scenario where null could be read?
What is the correct way to handle streams, could I do something better than in my example?
waitFor signals that the process ended, but you cannot be sure the threads which collect strings from its stdout and stderr finished also, so using a latch is a step in the right direction, but not an optimal one.
Instead of waiting for the latch, you can wait for the threads directly:
Thread stdoutThread = new Thread(new StreamConsumer(procOut, output)).start();
Thread stderrThread = ...
...
int ret = exec.waitFor();
stdoutThread.join();
stderrThread.join();
BTW, storing lines in StringBuffers is useless work. Use ArrayList<String> instead, put lines there without any conversion, and finally retrieve them in a loop.
Your appapproach is right, but is't better to remove CountDownLatch and use ThreadPool, and not create new Thread directly. From ThreadPool you will get two futures, which you can wait after to completion.
But I'm not sure if I eventually need to synchronize with those consumer threads, or is it enough just to wait for process to exit with waitFor method, to be certain that all the process output is actually consumed? I.E is it possible, even if the process exits (closes it's output stream), there is still unread data in the java end of the stream?
Yes, this situation may occurs. Termination and reading IO streams is unrelated processes.
I am trying to receive a telemetry string from arduino board to javafx apllication by jserialcomm.
My arduino output rate is about 100Hz now. in this situation i want to receive data in application about 1Hz. this is what i am doing:
This are only some important parts of code:
Runnable r1 = new Runnable() {
public void run() {
try {
while (true) {
refresher(rx);
Thread.sleep(1000L);
}
} catch (InterruptedException iex) {}
}
};
Thread thr1 = new Thread(r1);
thr1.start();
public void refresher(SerialPort rx){
readRX(rx);
parseString(lastTelemetry);
}
private void readRX(SerialPort rx){
Scanner ss = new Scanner(rx.getInputStream());
while(ss.hasNextLine()){
lastTelemetry = ss.nextLine();
if (lastTelemetry.isEmpty()) continue;
System.out.println(lastTelemetry);
break;
}
}
But the recieved string is not complete. some lines are complete and some or lost . this is what my output look like:
8,0,330,1306.42,86586.00,0,31.36,0,0,0,0,0,0,0,0,62.27,-6.81,4.53,0.00,00
0,0,0,0,0,0,0,0,66.24,-6.81,4.52,-0.30,00
1.36,0,0,0,0,0,0,0,0,70.22,-6.81,4.52,-0.10,00
7098,0,396,1306.33,86587.00,0,31.36,0,0,0,0,0,0,0,0,75.22,-6.81,4.51,-0.10,00
You probably shouldn't be creating a new input stream new Scanner(rx.getInputStream()) every time you read input. It is likely that if data is being buffered it is lost when you make a new stream. Create the input stream once when you open the serial port and pass that as the parameter to the readRX method instead of the SerialPort.
Also, I couldn't find anything in a cursory read of the Javadoc where you specify the buffer size, or what happens if the buffer overflows. That is another factor to consider.
I think there's a problem with the definition of hasNextLine().
See the following example:
byte[] data = "test".getBytes();
ByteArrayInputStream inputStream = new ByteArrayInputStream(data);
Scanner scanner = new Scanner(inputStream);
while (scanner.hasNextLine()) {
String text = scanner.nextLine();
System.out.println(text);
}
There is no \n at the end or something like this. But it still outputs "test". So I expect you assume a different behaviour of the next line part.
Also es mentioned by someone else, cachine might be an additional failure source.
im trying to come up with a design for a wrapper for use when invoking command line utilities in java. the trouble with runtime.exec() is that you need to keep reading from the process' out and err streams or it hangs when it fills its buffers. this has led me to the following design:
public class CommandLineInterface {
private final Thread stdOutThread;
private final Thread stdErrThread;
private final OutputStreamWriter stdin;
private final History history;
public CommandLineInterface(String command) throws IOException {
this.history = new History();
this.history.addEntry(new HistoryEntry(EntryTypeEnum.INPUT, command));
Process process = Runtime.getRuntime().exec(command);
stdin = new OutputStreamWriter(process.getOutputStream());
stdOutThread = new Thread(new Leech(process.getInputStream(), history, EntryTypeEnum.OUTPUT));
stdOutThread.setDaemon(true);
stdOutThread.start();
stdErrThread = new Thread(new Leech(process.getErrorStream(), history, EntryTypeEnum.ERROR));
stdErrThread.setDaemon(true);
stdErrThread.start();
}
public void write(String input) throws IOException {
this.history.addEntry(new HistoryEntry(EntryTypeEnum.INPUT, input));
stdin.write(input);
stdin.write("\n");
stdin.flush();
}
}
And
public class Leech implements Runnable{
private final InputStream stream;
private final History history;
private final EntryTypeEnum type;
private volatile boolean alive = true;
public Leech(InputStream stream, History history, EntryTypeEnum type) {
this.stream = stream;
this.history = history;
this.type = type;
}
public void run() {
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String line;
try {
while(alive) {
line = reader.readLine();
if (line==null) break;
history.addEntry(new HistoryEntry(type, line));
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
my issue is with the Leech class (used to "leech" the process' out and err streams and feed them into history - which acts like a log file) - on the one hand reading whole lines is nice and easy (and what im currently doing), but it means i miss the last line (usually the prompt line). i only see the prompt line when executing the next command (because there's no line break until that point).
on the other hand, if i read characters myself, how can i tell when the process is "done" ? (either complete or waiting for input)
has anyone tried something like waiting 100 millis since the last output from the process and declaring it "done" ?
any better ideas on how i can implement a nice wrapper around things like runtime.exec("cmd.exe") ?
Use PlexusUtils it is used by Apache Maven 2 to execute all external processes.
I was looking for the same thing myself, and I found a Java port of Expect, called ExpectJ. I haven't tried it yet, but it looks promising
I would read the input in with the stream and then write it into a ByteArrayOutputStream. The byte array will continue to grow until there are no longer any available bytes to read. At this point you will flush the data to history by converting the byte array into a String and splitting it on the platform line.separator. You can then iterate over the lines to add history entries. The ByteArrayOutputStream is then reset and the while loop blocks until there is more data or the end of stream is reached (probably because the process is done).
public void run() {
ByteArrayOutputStream out = new ByteArrayOutputStream();
int bite;
try {
while((bite = stream.read()) != -1) {
out.write(bite);
if (stream.available() == 0) {
String string = new String(out.toByteArray());
for (String line : string.split(
System.getProperty("line.separator"))) {
history.addEntry(new HistoryEntry(type, line));
}
out.reset();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
This will make sure you pick up that last line of input and it solves your problem of knowing when the stream is ended.
I have a class that manages the creation of RTF documents and a method in that class that calls the RTF editor with a XML file for display.
All but one user can access this editor without any issues. This one user consistently runs into an issue where their application just hangs. There are no errors in any logs. Normally this kind of problem is easily identified, reproduced and corrected, however, I can't for the life of my reproduce it so my attempts at debugging are failing.
Basically the code is as follows:
int exitVal = CUBSRTFEditor.runRTFEditor("c:\\tmp\\control"+ap_doc_id+".xml", xml,"I:\\AppealsLetters.exe /process \"c:\\tmp\\control"+ap_doc_id+".xml\"");
public static int runRTFEditor(String xmlLocation, String xmlContent, String executePath)
{
int exitVal = 0;
createLocalFile(xmlLocation, xmlContent);
try
{
System.out.println("executePath must = "+executePath);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(executePath);
System.out.println("after executePath runs");
//exhaust that stream before waiting for the process to exit
InputStream inputstream = proc.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
// read the ls output
String line;
while ((line = bufferedreader.readLine())!= null)
{
System.out.println(line);
}
exitVal = proc.waitFor();
}
catch (Throwable t)
{
t.printStackTrace();
}
CUBSRTFEditor.deleteTempFile(xmlLocation);
return exitVal;
}
The last output is the first System.out. When I take the xml file and execute this on any other PC it executes without issue. I see no useful info in proc.getErrorStream() or proc.getOutputStream().
The JDK's Javadoc documentation on this problem (exec hanging):
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
I try exhausting that stream before waiting for the process to exit and that does not seem to help as it never seems to get to that point (the second System.out is not displayed)
Have I implemented this incorrectly? Am I missing something important? Any ideas on how to get more info out of the process would be great.
I am stuck....
Runtime.exec() is a deceptively nasty little spud to work with. I found this article (old, but still relevant) to be quite helpful. You can always skip to Page 4 for some highly gankable sample code. :-)
At a glance, your code needs to handle both proc.getOutputStream() and proc.getErrorStream(), which is a good reason to handle those streams in separate threads.
I wanted to update this because the change went into Production today and worked. Based off of BlairHippo's suggestions I got it to work with an anonymous inner class to create a separate thread to exhaust both the Error and Input streams.
new Thread(new Runnable(){
public void run()
{
try
{
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
String line;
while ((line = br.readLine())!= null)
{
System.out.println(line);
}
}
catch (Throwable t)
{
t.printStackTrace();
}
}
}).start();