I am running into this problem.
my program invokes Runtime.getRuntime().exec(cmd); on a windows platform. I read the error and output stream and do something with this output. This method gets called in a loop after each 4-5 seconds and it goes on till the program is terminated.
Now what happens, each time i read the output, the previous output is appended to the new output and as such with each iteration the result grows bigger and bigger. Is there anyway to stop this thing. The command executed is "tasklist" with some filtering parameters.
I have made a method (which returns String output) for this Runtime.getTuntime().exec(cmd) in which i am also closing the process after execution but when it is called from within the loop, each time previous output is appended to the new one.
Here is the code:
class Track implements Runnable {
static int size = 0;
public void run() {
String cmd1 = "tasklist /fo list /fi \"imagename eq java.exe\"";
String cmd2 = "tasklist /fo list /fi \"imagename eq javaw.exe\"";
String text = "";
int i=1, j=0;
while(size < 100000){
try{
text = fList.pList(cmd2, 1);
if (text.indexOf("javaw.exe")== -1){
text = fList.pList(cmd1, 1);
}
if(j==22) System.out.println(text);
if (text.charAt(0)!= '0') continue;
i = text.lastIndexOf("Mem Usage: ")+14;
text = text.substring(i);
text = text.substring(0,text.lastIndexOf(" K"));
text = text.replaceFirst(",", "");
size = Integer.parseInt(text);
System.out.println(size);
Thread.sleep(3000);
j++;
} catch(Exception e){
System.out.println(e);
}
}
System.out.println("Memory utlization exceeded the permissible limit");
System.out.println("Now terminating the Program\n");
System.exit(1);
}
static void memoryCheck(int size) throws Exception{
(new Thread(new Track())).start();
}
}
in class fList is the method pList:
static String pList(String cmd, int eval) throws Exception{ //can execute external command
String out = "";
int val = 5; // should not be zero, to verify returned code zero for normal exec.
try
{
//String osName = System.getProperty("os.name" );
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd);
// any error message?
eProcList error = new eProcList(proc.getErrorStream());
// any output?
eProcList output = new eProcList(proc.getInputStream());
// kick them off
error.start();
output.start();
// any error???
int exitVal = proc.waitFor();
out = eProcList.procList();
val = exitVal;
proc.destroy();
proc.getInputStream().close();
proc.getErrorStream().close();
} catch (Throwable t)
{
t.printStackTrace();
}
if (eval==1) return val + out;
return out;
}
class eProcList extends Thread
{
InputStream iStream;
static String oPut = "";
eProcList(InputStream iStream)
{
this.iStream = iStream;
}
public void run()
{
try
{
InputStreamReader isr = new InputStreamReader(iStream);
BufferedReader br = new BufferedReader(isr);
String line=null;
while ( (line = br.readLine()) != null)
oPut = oPut + line+"\n";
} catch (IOException e)
{
e.printStackTrace();
}
}
static public String procList(){
return oPut;
}
}
you asked so iv'e copied all here.
You made oPut a static field - its initialized to "" once when the class is loaded and then shared between every new instance of eProcList, i.e. never cleared of the previous run. Either don't make it static (why is it static?) or clear it in the constructor.
Related
I have a large data file say dataset.txt where data is in the format -
1683492079 kyra maharashtra 18/04/2017 10:16:17
1644073389 pam delhi 18/04/2017 10:16:17
.......
The fields are id, name, state, and timestamp.
I have around 50,000 lines of data in the .txt data file.
My requirement is to read the data from this data file in batch size of 10.
So in first batch I need to read from 0 to 9th elements. Next batch from 10th to 19th elements and so on...
Using BufferedReader I have managed to read the whole file:
import java.io.*;
public class ReadDataFile {
public static void main(String args[]) throws IOException {
BufferedReader br = new BufferedReader(new FileReader("dataset.txt"));
String line;
while((line = br.readLine())!= null)
{
System.out.println(line);
}
br.close();
}
}
But my requirement is to read the file in batch size of 10. I am new to Java so would really appreciate if some one can help me in simple terms.
As per #GhostCat answer - this what I have got -
public class ReadDataFile {
public static void main(String args[]) throws IOException {
BufferedReader br = new BufferedReader(new FileReader("dataSetExample.txt"));
readBatch(br,10);
}
public static void readBatch(BufferedReader reader, int batchSize) throws IOException {
List<String> result = new ArrayList<>();
for (int i = 0; i < batchSize; i++) {
String line = reader.readLine();
if (line != null) {
// result.add(line);
System.out.println(line);
}
}
// return result;
return ;
}
}
The file is read in the readBatch method so how do I know in the main method that the end of file is reached to call the next 10 records? Kindly help.
Your requirements aren't really clear; but something simple to get you started:
A) your main method shouldn't do any reading; it just prepare that BufferedReader object
B) you use that reader with a method like:
private static List<String> readBatch(Reader reader, int batchSize) throws IOException {
List<String> result = new ArrayList<>();
for (int i = 0; i < batchSize; i++) {
String line = reader.readLine();
if (line != null) {
result.add(line);
} else {
return result;
}
}
return result;
}
To be used in your main:
BufferedReader reader = ...
int batchSize = 10;
boolean moreLines = true;
while (moreLines) {
List<String> batch = readBatch(reader, batchSize);
... do something with that list
if (batch.size() < batchSize) {
moreLines = false;
}
This is meant as "suggestion" how you could approach this. Things missing from my answer: probably you should use a distinct class, and do parsing right there (and return a List<DataClass> instead of moving around those raw "line strings".
And of course: 50000 lines isn't really much of data. Unless we are talking an embedded device, there is really not much point regarding "batch style".
And finally: the term batch processing has a very distinct meaning; also in Java, and if you intend to go there, see here for further reading.
Anybody in need of working example ---
// Create a method to read lines (using buffreader) and should accept the batchsize as argument
private static List<String> readBatch(BufferedReader br, int batchSize) throws IOException {
// Create a List object which will contain your Batch Sized lines
List<String> result = new ArrayList<>();
for (int i = 1; i < batchSize; i++) { // loop thru all your lines
String line = br.readLine();
if (line != null) {
result.add(line); // add your lines to your (List) result
} else {
return result; // Return your (List) result
}
}
return result; // Return your (List) result
}
public static void main(String[] args) throws IOException {
//input file
BufferedReader br = new BufferedReader(new FileReader("c://ldap//buffreadstream2.csv"));
//output file
BufferedWriter bw = new BufferedWriter(new FileWriter("c://ldap//buffreadstream3.csv"));
// Your Batch size i.e. how many lines you want in your batch
int batchSize = 5; // Define your batchsize here
String line = null;
long batchNumber = 1;
try {
List<String> mylist = null;
while ((line = br.readLine()) != null) { // Do it for your all line in your csv file
bw.write("Batch Number # " + batchNumber + "\n");
System.out.println("Batch Number # " + batchNumber);
bw.write(line + "\n"); // Since br.readLine() reads the next line you have to catch your first line here itself
System.out.println(line); // else you will miss every batchsize number line
// process your First Line here...
mylist = readBatch(br, batchSize); // get/catch your (List) result here as returned from readBatch() method
for (int i = 0; i < mylist.size(); i++) {
System.out.println(mylist.get(i));
// process your lines here...
bw.write(mylist.get(i) + "\n"); // write/process your returned lines
}
batchNumber++;
}
System.out.println("Lines are Successfully copied!");
br.close(); // one you are done .. dont forget to close/flush
br = null; // all
bw.flush(); // your
bw.close(); // BR and
bw = null; // BWs..
} catch (Exception e) {
System.out.println("Exception caught: " + e.getMessage()); // Catch any exception here
}
}
I have been playing with java processes for a while and am stuck. What i want to do is run multiple system commands at the same time and print their output to console.
For example, ls -l ; cat someFile ; quit ; grep foo someOtherFile should all be running at the same time. I have read somewhere that the output of these commands should be intermixed. In addition, if there's a quit command anywhere in the string, continue executing other commands and then exit.
Right now, they are executing sequentially. How do I run them concurrently and print their output as it arrive.
String st = "ls -l ; cat someFile ; quit ; grep foo someOtherFile";
String[] rows = st.split(";");
String[][] strArray = new String[rows.length][];
int index = 0;
for(int i = 0; i < rows.length; i++) {
rows[index] = rows[index].trim();
strArray[index] = rows[index].split(" ");
index++;
}
for(int i = 0; i < strArray.length; i++) {
if(rows[i].equalsIgnoreCase("quit")) {
System.out.println("Abort");
break;
}
if(rows[i].equals("")) {
continue;
}
ProcessBuilder pb = new ProcessBuilder(strArray[i]);
pb.redirectErrorStream(true);
Process process = pb.start();
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
while ( (line = br.readLine()) != null) {
System.out.println(line);
}
br.close();
}
Just put the guts of your loop inside the run() function of a new thread, and each iteration of the loop will run in a separate thread:
new Thread() {
public void run() {
// loop guts go here
}
}.start();
You may have to declare a few variables as finals in order to access them inside this anonymous inner class.
You can try code similar to this:
// command list
String st = "ls -la; cat someFile";
String[] commands = st.split(";");
for (int i = 0; i < commands.length; i++) {
String currentCommand = commands[i].trim();
System.out.println("Command: " + currentCommand);
Thread thread = new Thread(() -> {
try {
ProcessBuilder command = new ProcessBuilder(currentCommand);
Process process = command.start();
InputStream is = process.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
String line;
while((line = br.readLine()) != null) {
System.out.println(line);
}
br.close();
} catch (IOException e) {
e.printStackTrace();
}
});
thread.start();
}
Disclaimer: not tested on a Linux machine. Windows machines will probably not work. See this link regarding Windows command line process execution.
You should look to some documentation about concurrency, threads an such http://docs.oracle.com/javase/tutorial/essential/concurrency/.
Here an edit to your code that may work, not tested.
String st = "ls -l ; cat someFile ; quit ; grep foo someOtherFile";
String[] rows = st.split(";");
String[][] strArray = new String[rows.length][];
int index = 0;
for(int i = 0; i < rows.length; i++) {
rows[index] = rows[index].trim();
strArray[index] = rows[index].split(" ");
index++;
}
List<Thread> threads = new ArrayList<Thread>();
for(int i = 0; i < strArray.length; i++) {
if(rows[i].equalsIgnoreCase("quit")) {
System.out.println("Abort");
break;
}
if(rows[i].equals("")) {
continue;
}
final int iForThread = i;
Thread thread = new Thread() {
public void run(){
try{
ProcessBuilder pb = new ProcessBuilder(strArray[iForThread]);
pb.redirectErrorStream(true);
Process process = pb.start();
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
while ( (line = br.readLine()) != null) {
System.out.println(line);
}
br.close();
}catch(IOException e){
//Log some awesome error
//Clean up
//Do whatever
}
}
};
threads.add(thread);
}
final CyclicBarrier gate = new CyclicBarrier(threads.size() + 1); //+1 is a tip from other post
for(Thread thread : threads){
thread.start();
}
try {
gate.await();
System.out.println("all threads started");
} catch (InterruptedException | BrokenBarrierException e) {
e.printStackTrace();
/* RONALDO OF ERROS
* MESSI OF HANDLERS*/
}
}
It creates an tread and executed it at the spot.
I if you are just messing around I think this enough.
Edit: Added start threads at "same time"
Based on: How to start two threads at "exactly" the same time
See:https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CyclicBarrier.html
My output is reflecting the file that I am needing to process into hex values but my hex values are not being reflected in the output. Why isn't my file being converted into hex values?
public class HexUtilityDump {
public static void main(String[] args) {
FileReader myFileReader = null;
try {
myFileReader = new FileReader("src/hexUtility/test.txt");
} catch(Exception ex) {
System.out.println("Error opening file: " + ex.getLocalizedMessage());
}
BufferedReader b = null;
b = new BufferedReader(myFileReader);
//Loop through all the records in the file and print them on the console
while (true){
String myLine;
try {
myLine = b.readLine();
//check for null returned from readLine() and exit loop if so.
if (myLine ==null){break;}
System.out.println(myLine);
} catch (IOException e) {
e.printStackTrace();
//it is time to exit the while loop
break;
}
}
}
Here is the code to pull the file through the conversion
public static void convertToHex(PrintStream out, File myFileReader) throws IOException {
InputStream is = new FileInputStream(myFileReader);
int bytesCounter =0;
int value = 0;
StringBuilder sbHex = new StringBuilder();
StringBuilder sbText = new StringBuilder();
StringBuilder sbResult = new StringBuilder();
while ((value = is.read()) != -1) {
//convert to hex value with "X" formatter
sbHex.append(String.format("%02X ", value));
//If the character is not convertible, just print a dot symbol "."
if (!Character.isISOControl(value)) {
sbText.append((char)value);
} else {
sbText.append(".");
}
//if 16 bytes are read, reset the counter,
//clear the StringBuilder for formatting purpose only.
if(bytesCounter==15) {
sbResult.append(sbHex).append(" ").append(sbText).append("\n");
sbHex.setLength(0);
sbText.setLength(0);
bytesCounter=0;
}else{
bytesCounter++;
}
}
//if still got content
if(bytesCounter!=0){
//add spaces more formatting purpose only
for(; bytesCounter<16; bytesCounter++){
//1 character 3 spaces
sbHex.append(" ");
}
sbResult.append(sbHex).append(" ").append(sbText).append("\n");
}
out.print(sbResult);
is.close();
}
You never call convertToHex, remove the file reading from your main() method. It appears you wanted to do something like,
File f = new File("src/hexUtility/test.txt");
convertToHex(System.out, f);
I am executing a command which returns me the Revision number of a file; 'fileName'. But if there is some problem executing the command, then the application hangs up. What can I do to avoid that condition? Please find below my code.
String cmd= "cmd /C si viewhistory --fields=revision --project="+fileName;
Process p = Runtime.getRuntime().exec(cmd) ;
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
} catch (Exception e) {
e.printStackTrace();
}
I guess the issue is that you are only reading InputStream and not reading ErrorStream. You also have to take care that both the streams are read in parallel. It may so happen that currently the data piped from the output stream fills up the OS buffer, your exec command will be automatically be suspended to give your reader a chance to empty the buffer. But the program will still be waiting for the output to process. Hence, the hang occurs.
You can create a separate class to handle both the Input and Error Stream as follows,
public class ReadStream implements Runnable {
String name;
InputStream is;
Thread thread;
public ReadStream(String name, InputStream is) {
this.name = name;
this.is = is;
}
public void start () {
thread = new Thread (this);
thread.start ();
}
public void run () {
try {
InputStreamReader isr = new InputStreamReader (is);
BufferedReader br = new BufferedReader (isr);
while (true) {
String s = br.readLine ();
if (s == null) break;
System.out.println ("[" + name + "] " + s);
}
is.close ();
} catch (Exception ex) {
System.out.println ("Problem reading stream " + name + "... :" + ex);
ex.printStackTrace ();
}
}
}
The way you use it is as follows,
String cmd= "cmd /C si viewhistory --fields=revision --project="+fileName;
Process p = Runtime.getRuntime().exec(cmd) ;
s1 = new ReadStream("stdin", p.getInputStream ());
s2 = new ReadStream("stderr", p.getErrorStream ());
s1.start ();
s2.start ();
p.waitFor();
} catch (Exception e) {
e.printStackTrace();
} finally {
if(p != null)
p.destroy();
}
This code is based on the same idea Arham's answer, but is implemented using a java 8 parallel stream, which makes it a little more concise.
public static String getOutputFromProgram(String program) throws IOException {
Process proc = Runtime.getRuntime().exec(program);
return Stream.of(proc.getErrorStream(), proc.getInputStream()).parallel().map((InputStream isForOutput) -> {
StringBuilder output = new StringBuilder();
try (BufferedReader br = new BufferedReader(new InputStreamReader(isForOutput))) {
String line;
while ((line = br.readLine()) != null) {
output.append(line);
output.append("\n");
}
} catch (IOException e) {
throw new RuntimeException(e);
}
return output;
}).collect(Collectors.joining());
}
You can call the method like this
getOutputFromProgram("cmd /C si viewhistory --fields=revision --project="+fileName);
Note that this method will hang if the program you are calling hangs, which will happen if it requires input.
In a java 6 webapp, I am attempting to retrieve a large amount of output from an executed command. I've "borrowed/stolen/based" it on the javaworld article. The problem I am facing is that the length appears to exceed a size limit since the output is lopped off. I've output the data to a file so I can see the size of what is returned, and that is exactly 32K (32768). I've experimented with changing the default size of the buffer (see BufferedReader constructor), but I have not observed any change to the length of the data returned no matter what value I have for the buffered-size (very small to very large).
Any advice would be very much appreciated!
public class StreamGobbler extends Thread {
private InputStream is;
private String type;
private List<String> output;
public StreamGobbler(InputStream is, String type) {
this.is = is;
this.type = type;
}
#Override
public void run() {
try {
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
this.output = new ArrayList<String>();
while ((line = br.readLine()) != null) {
this.getOutput().add(line + "\n");
System.out.println(type + ">" + line);
}
br.close();
} catch (IOException ioe) {
System.err.println("ERROR: " + ioe.getMessage());
}
}
/**
* #return the output
*/
public List<String> getOutput() {
return output;
}
}
public class JobClassAds {
private String CONDOR_HISTORY = "condor_history";
private String CONDOR_HISTORY_XML = CONDOR_HISTORY + " -xml";
private String CONDOR_HISTORY_LONG = CONDOR_HISTORY + " -long";
public String getHistory() {
try {
Runtime runtime = Runtime.getRuntime();
String exec = CONDOR_HISTORY_LONG;
Process process = runtime.exec(exec);
System.out.println("Running " + exec + " ...");
// Error message
StreamGobbler errGobbler = new StreamGobbler(process.getErrorStream(), "ERROR");
// Output
StreamGobbler outGobbler = new StreamGobbler(process.getInputStream(), "OUTPUT");
Thread outThread = new Thread(outGobbler);
Thread errThread = new Thread(errGobbler);
outThread.start();
errThread.start();
outThread.join();
errThread.join();
/*
String line = null;
while ((line = input.readLine()) != null) {
System.out.println(line);
content.append(line);
}
*
*/
int exitVal = process.waitFor();
List<String> output = outGobbler.getOutput();
String inputString = "";
for (String o : output) {
inputString += o;
}
System.out.println(exec + " Exited with error code " + exitVal);
BufferedWriter out = new BufferedWriter(new FileWriter("/tmp/history_result.xml"));
out.write(inputString);
out.close();
return inputString;
} catch (Exception e) {
System.err.println(e.getMessage());
return null;
}
}
The problem is not with the BufferedReader's buffer size.
I think that the real cause is something that the external command is doing. I suspect that it is bailing out without flushing its stdout stream. Note that you are "gobbling" but not outputting the command's stderr stream. That's where you may find the evidence pointing to the real cause of the problem.
By the way, you are using the StreamGobbler class in a suboptimal fashion. It extends Thread so the intended way to use is:
SteamGobbler sg = new StreamGobbler(...);
sg.start();
sg.join();
but you are effectively doing this:
SteamGobbler sg = new StreamGobbler(...);
Thread th = new Thread(sg);
th.start();
th.join();
It works ... but only because a Thread is-a Runnable.