Develop Reference

Correct Implementation of Multithreading

I've been thinking about this for a few days now, and I am under the belief that my thinking of how multithreading works is flawed. I've consulted the API for Concurrency, and still did not get an answer to my question.
If I have some process that I want to return a unique string, and I want 100 different threads to run this process (thereby giving me 100 threads with 100 different strings), I can utilize executorServices to accomplish this, correct?
Then, once every thread has a string, I want to send that string to a queue, thereby blasting the queue with messages (again, sounds like a usage for a executorService.submit() call).
Lastly, Once a thread sends the message to to the queue, I want this thread to immediately start checking another queue for a response (matching it's unique string), and if it matches, output some data and terminte.
Although I am under the impression that using Executorservice would be the answer to my issue, I am failing in implementation. Is there another solution that I am missing, or does multithreading using using this method suffice--and if so, how?
Code thus far. I stopped after the sendTextMessage after realizing my issue:
int arraySize = 750; //Number of hits to send out: MANIPULATE THIS IF YOU WANT TO SEND MORE
// String[] toSend = new String[arraySize];
String[] rfids = new String[arraySize];
double total = 0;
int count = 0;
//Insert Connection information and set up clients here//
clientSend.connectSend();
clientRec.connectRec(); //edit to ensure credientials are corrects
// System.out.println("Signed-in");
StringBuffer output = new StringBuffer(); //What holds out output
File infile = new File("infile.txt"); //Populating Rfids array
Scanner scan = new Scanner(infile);
for (int i = 0; i <= arraySize-1; i++)
{
if(scan.hasNextLine())
{
rfids[i]=scan.nextLine();
// System.out.println(rfids[i]);
}
}
scan.close();
count=0;
ExecutorService load = Executors.newFixedThreadPool(arraySize);
Callable<String> readAndSendPrep = () ->
{
StringBuffer fileBasedResponse = new StringBuffer();
String rfid = "";
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader("input.txt")); //This is the standard message that will be sent everytime, give or take
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
String line; //temp var
for(int x = 0; x<arraySize-1;x++)
{
try {
while ((line = reader.readLine ()) != null)
{
if (line.trim().startsWith("<MessageId>"))
{
// System.out.println(rf);
rfid = rfids[arraySize]; //not necessary i think
int endIndex = line.trim().indexOf("</MessageId>");
String messageId = line.trim().substring(11, endIndex);
line = "<MessageId>" + messageId + " - " + rfids[arraySize] + "</MessageId>"; //puts unique ID in thread details
}
else if (line.trim().startsWith("str"))
{
// System.out.println(allRFID[thisIndex]);
rfid = rfids[arraySize];
line = "str" + rfids[arraySize] + "str"; //Another unique ID
// System.out.println("BOOM");
}
fileBasedResponse.append(line); //put the whole response to the StringBuffer object
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Thread.sleep(1);
return fileBasedResponse.toString();
};
TimeUnit.SECONDS.sleep(10);
Future<String> fileBasedResponse = load.submit(readAndSendPrep);
while(!fileBasedResponse.isDone())
{
Thread.sleep(1);
}
String fileBasedResponseStr = fileBasedResponse.toString();
Runnable sender = () ->
{
try {
clientSend.sendTextMessage(fileBasedResponseStr);
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
};
clientSend.close(); //close connections
clientRec.close();
System.out.println(output.toString()); //output the results
System.out.println(count);

Java While Loop

I'm having some issues with a while loop. This while loop is run in a thread, and is the "main loop" of the thread, and I'm having issues with it not continuing if one of the if statements it loops over is false. This thread handles both sending and receiving IRC Commands/Messages for use in a minecraft client.
The code as it stands is below:
do {
System.out.println("IRC Thread Looped!");
if(tosend != "") {
writer.write("PRIVMSG " + channel + " :"+tosend+"\r\n");
System.out.println(tosend);
mc.thePlayer.addChatMessage("\247f[\247bIRC\247f] \2477"+nick+": \247f"+tosend);
tosend = "";
writer.flush();
}
if((line = reader.readLine()) != null) {
if(line.startsWith("PING ")) {
writer.write("PONG " + line.substring(5) + "\r\n");
writer.flush( );
}
else {
// we need to pretty this ugly sh*t up!
try {
String parsedline;
String args[] = line.split("!");
String args2[] = args[1].split(":");
args[0] = args[0].substring(1, args[0].length());
parsedline = "\2477"+ args[0] +": \247f"+ args2[1];
mc.thePlayer.addChatMessage("\247f[\247bIRC\247f] "+parsedline);
}
catch(Exception e) {
mc.thePlayer.addChatMessage("\247f[\247bIRC\247f]\247f "+line);
continue;
}
}
}
try {
Thread.sleep(90L);}catch(Exception e) { };
}
} while(true);

Note that java threads like to eat exceptions that get thrown out of the run() method.
Since you have IO such as if((line = reader.readLine()) != null){ not in a try/catch, you might be throwing an exception and exiting.
What I like to do with threads is to put the main thread code in runImpl() and have the run() look like this.
public void run()
{
try
{
runImpl();
}
catch( Throwable t )
{
// log the throwable
}
}
Then at a minimum you can discover that your thread threw something unexpected.

If the problem is caused by getLine() blocking behavior you could probably fix it this way. I haven't run that code so I don't guarantee it will work. Also make sure that reader is a BufferedReader.
StringBuilder sb = new StringBuilder();
String line = null;
int n = 0;
do {
System.out.println("IRC Thread Looped!");
if(tosend != ""){
writer.write("PRIVMSG " + channel + " :"+tosend+"\r\n");
System.out.println(tosend);
mc.thePlayer.addChatMessage("\247f[\247bIRC\247f] \2477"+nick+": \247f"+tosend);
tosend = "";
writer.flush();
}
while (reader.ready()) {
int c = reader.read();
if (c == -1) {
if (n > 0) {
line = sb.toString();
sb = new StringBuilder();
}
break;
}
if (c == '\n' || c == '\r') {
line = sb.toString();
sb = new StringBuilder();
break;
}
sb.append((char)c);
n++;
}
if(line != null){
if (line.startsWith("PING ")) {
writer.write("PONG " + line.substring(5) + "\r\n");
writer.flush( );
}
else
{
//we need to pretty this ugly shit up!
try{
String parsedline;
String args[] = line.split("!");
String args2[] = args[1].split(":");
args[0] = args[0].substring(1, args[0].length());
parsedline = "\2477"+args[0] +": \247f"+ args2[1];
mc.thePlayer.addChatMessage("\247f[\247bIRC\247f] "+parsedline);
}catch(Exception e){
mc.thePlayer.addChatMessage("\247f[\247bIRC\247f]\247f "+line);
continue;
}
}
line = null;
n = 0;
}
try{Thread.sleep(90L);}catch(Exception e){};
} while(true);

Termination Condition for threads in Java

I have written a multi-threaded Java application which reads a bunch of .jar files from a directory. This application spawns multiple threads and each threads reads bunch of jar files. I'm having trouble identifying the stopping condition for this application. How can i identify that all the files have been read?
The following is a snippet function which gets called from the run() method for each thread.
import java.io.*;
import java.util.Enumeration;
import java.util.jar.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipException;
import java.io.FilenameFilter;
public class ArchiveFileTest implements Runnable {
private static boolean stopAll = false;
private static int threadNumber = 0;
private int myNumber = 0;
public ArchiveFileTest () {
myNumber = threadNumber;
threadNumber++;
}
public static boolean setStopAll () {
return setStopAll(true);
}
public static boolean setStopAll (boolean b) {
stopAll = b;
return stopAll;
}
public static String[] listFiles (File parentDir,final String ext1,final String ext2,final String ext3,final String ext4) {
String allFiles[] = parentDir.list(new FilenameFilter() {
public boolean accept(File pDir, String fName) {
if (fName.endsWith("."+ext1) || fName.endsWith("."+ext2) || fName.endsWith("."+ext3) || fName.endsWith("."+ext4)) return true;
else return false;
}
});
for (int i=0; i<allFiles.length; i++)
allFiles[i] = parentDir.getAbsolutePath() + File.separator + allFiles[i];
return allFiles;
}
public ZipFile getMyZipFile (File parentDir) {
String fn[] = listFiles(parentDir, "jar", "zip", "war", "rar");
int fileNum = myNumber % fn.length;
ZipFile zFile = null;
for (int i=0; i<fn.length; i++) {
String jFile = fn[(fileNum + i)%fn.length];
try {
zFile = new ZipFile(jFile);
break;
} catch (IOException e) {
setStopAll();
}
}
return zFile;
}
public void doStuff() throws Exception {
File dName = new File("/home/sqatest/chander/sample-files");
final int N_TIMES = 15;
final int N_FILES = 500;
int counter = 0;
int fCount = 0;
if (!dName.isDirectory() || !dName.exists()) {
System.err.println("The parent directory given should point to an existing directory...");
setStopAll();
return;
}
while (counter < N_TIMES) {
ZipFile zipFile = getMyZipFile(dName);
if (zipFile == null) {
System.err.println("No zip file entry for the Thread-" + myNumber);
break;
}
try {
Enumeration <? extends ZipEntry> zipEntries = zipFile.entries();
fCount = 0;
ZipEntry ze = null;
while (zipEntries.hasMoreElements()) {
ze = zipEntries.nextElement();
if (ze.isDirectory()) continue; // if it is a directory go to next entry
InputStream is = zipFile.getInputStream(ze);
fCount++;
int readCount = 0;
try {
while(is.read((new byte[50])) != -1 && readCount != 200) readCount++;
System.out.println("Successfully Read " + zipFile.toString());
//is.close();
} catch (IOException e) {
e.printStackTrace();
}
if (fCount == N_FILES) break; // read maximum of N_FILES
}
if (stopAll) break;
} catch (Exception e) {
e.printStackTrace();
} finally {
counter++;
}
}
}
public void run () {
try {
doStuff();
} catch (IOException e) {
e.printStackTrace();
setStopAll();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main (String[] args) throws Exception {
final int MAX_THREADS = 500;
final int MAX_HOLDING_THREADS = 5;
int loopCount = 0;
Thread mainThread = Thread.currentThread();
for (int m=0; ; m++) {
Thread t[] = new Thread[MAX_HOLDING_THREADS];
for (int n=0; n<t.length; n++) {
t[n] = new Thread(new ArchiveFileTest());
t[n].start();
if ((m+1)*(n+1)==MAX_THREADS) {
System.out.println("\n" + MAX_THREADS + " reached... \nMain Sleeping for some mins...");
loopCount++;
try {
t[n].join();
System.out.println("\nMain is back... (" + loopCount + ")");
} catch (InterruptedException e) {
e.printStackTrace();
setStopAll();
}
m = 0;
}
}
}
}
}

I don't think your application will ever stop. You've got an infinite loop in the main method:
for (int m=0; ; m++) {
....
}
Note, setting m=0 inside the body won't break the loop, so I think you'll never end even if you have no file. It then continuously reads all zip/jar/war/rar files in the directory (choosing the file based on a rotating counter myNumber is not very maintainable), but never exits the loop.
If you're requirement is to read ZIP files using a number of threads, then I would go about it a different way.
Create a Set of files which you want to look at.
Create a ThreadPoolExecutor to create a fixed pool of 5 threads
Iterate over the set of files and create a new Runnable which does the Zip Extraction (though I'm not quite sure why you read the first 10000 bytes of a ZIP entry and then don't do anything with it), and call the execute method. That will use the thread pool to process 5 files at a time.
After submitting all the runnables Use the shutdown method, which will wait for all submitted tasks to finish, and the shutdown the thread pool.

If by stopping you mean terminating then the application will stop when all threads, that are not daemon special case, are finished.

In your class that launches the threads, have a volatile counter for your running threads.
In your thread constructor pass a reference to the launching class.
Have a synchronized method to let the threads notify the launching class that they are done.
After instancing and starting your threads wait for the counter to become 0;
while(getRunningThreads() > 0) // getRunningThreads must be synchronized too
Thread.sleep(500); // Check every half second.

Size limit on BufferedReader exceeded?

In a java 6 webapp, I am attempting to retrieve a large amount of output from an executed command. I've "borrowed/stolen/based" it on the javaworld article. The problem I am facing is that the length appears to exceed a size limit since the output is lopped off. I've output the data to a file so I can see the size of what is returned, and that is exactly 32K (32768). I've experimented with changing the default size of the buffer (see BufferedReader constructor), but I have not observed any change to the length of the data returned no matter what value I have for the buffered-size (very small to very large).
Any advice would be very much appreciated!
public class StreamGobbler extends Thread {
private InputStream is;
private String type;
private List<String> output;
public StreamGobbler(InputStream is, String type) {
this.is = is;
this.type = type;
}
#Override
public void run() {
try {
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
this.output = new ArrayList<String>();
while ((line = br.readLine()) != null) {
this.getOutput().add(line + "\n");
System.out.println(type + ">" + line);
}
br.close();
} catch (IOException ioe) {
System.err.println("ERROR: " + ioe.getMessage());
}
}
/**
* #return the output
*/
public List<String> getOutput() {
return output;
}
}
public class JobClassAds {
private String CONDOR_HISTORY = "condor_history";
private String CONDOR_HISTORY_XML = CONDOR_HISTORY + " -xml";
private String CONDOR_HISTORY_LONG = CONDOR_HISTORY + " -long";
public String getHistory() {
try {
Runtime runtime = Runtime.getRuntime();
String exec = CONDOR_HISTORY_LONG;
Process process = runtime.exec(exec);
System.out.println("Running " + exec + " ...");
// Error message
StreamGobbler errGobbler = new StreamGobbler(process.getErrorStream(), "ERROR");
// Output
StreamGobbler outGobbler = new StreamGobbler(process.getInputStream(), "OUTPUT");
Thread outThread = new Thread(outGobbler);
Thread errThread = new Thread(errGobbler);
outThread.start();
errThread.start();
outThread.join();
errThread.join();
/*
String line = null;
while ((line = input.readLine()) != null) {
System.out.println(line);
content.append(line);
}
*
*/
int exitVal = process.waitFor();
List<String> output = outGobbler.getOutput();
String inputString = "";
for (String o : output) {
inputString += o;
}
System.out.println(exec + " Exited with error code " + exitVal);
BufferedWriter out = new BufferedWriter(new FileWriter("/tmp/history_result.xml"));
out.write(inputString);
out.close();
return inputString;
} catch (Exception e) {
System.err.println(e.getMessage());
return null;
}
}

The problem is not with the BufferedReader's buffer size.
I think that the real cause is something that the external command is doing. I suspect that it is bailing out without flushing its stdout stream. Note that you are "gobbling" but not outputting the command's stderr stream. That's where you may find the evidence pointing to the real cause of the problem.
By the way, you are using the StreamGobbler class in a suboptimal fashion. It extends Thread so the intended way to use is:
SteamGobbler sg = new StreamGobbler(...);
sg.start();
sg.join();
but you are effectively doing this:
SteamGobbler sg = new StreamGobbler(...);
Thread th = new Thread(sg);
th.start();
th.join();
It works ... but only because a Thread is-a Runnable.

Runtime.getRuntime().exec(cmd); appending previous output with each new execution?

I am running into this problem.
my program invokes Runtime.getRuntime().exec(cmd); on a windows platform. I read the error and output stream and do something with this output. This method gets called in a loop after each 4-5 seconds and it goes on till the program is terminated.
Now what happens, each time i read the output, the previous output is appended to the new output and as such with each iteration the result grows bigger and bigger. Is there anyway to stop this thing. The command executed is "tasklist" with some filtering parameters.
I have made a method (which returns String output) for this Runtime.getTuntime().exec(cmd) in which i am also closing the process after execution but when it is called from within the loop, each time previous output is appended to the new one.
Here is the code:
class Track implements Runnable {
static int size = 0;
public void run() {
String cmd1 = "tasklist /fo list /fi \"imagename eq java.exe\"";
String cmd2 = "tasklist /fo list /fi \"imagename eq javaw.exe\"";
String text = "";
int i=1, j=0;
while(size < 100000){
try{
text = fList.pList(cmd2, 1);
if (text.indexOf("javaw.exe")== -1){
text = fList.pList(cmd1, 1);
}
if(j==22) System.out.println(text);
if (text.charAt(0)!= '0') continue;
i = text.lastIndexOf("Mem Usage: ")+14;
text = text.substring(i);
text = text.substring(0,text.lastIndexOf(" K"));
text = text.replaceFirst(",", "");
size = Integer.parseInt(text);
System.out.println(size);
Thread.sleep(3000);
j++;
} catch(Exception e){
System.out.println(e);
}
}
System.out.println("Memory utlization exceeded the permissible limit");
System.out.println("Now terminating the Program\n");
System.exit(1);
}
static void memoryCheck(int size) throws Exception{
(new Thread(new Track())).start();
}
}
in class fList is the method pList:
static String pList(String cmd, int eval) throws Exception{ //can execute external command
String out = "";
int val = 5; // should not be zero, to verify returned code zero for normal exec.
try
{
//String osName = System.getProperty("os.name" );
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd);
// any error message?
eProcList error = new eProcList(proc.getErrorStream());
// any output?
eProcList output = new eProcList(proc.getInputStream());
// kick them off
error.start();
output.start();
// any error???
int exitVal = proc.waitFor();
out = eProcList.procList();
val = exitVal;
proc.destroy();
proc.getInputStream().close();
proc.getErrorStream().close();
} catch (Throwable t)
{
t.printStackTrace();
}
if (eval==1) return val + out;
return out;
}
class eProcList extends Thread
{
InputStream iStream;
static String oPut = "";
eProcList(InputStream iStream)
{
this.iStream = iStream;
}
public void run()
{
try
{
InputStreamReader isr = new InputStreamReader(iStream);
BufferedReader br = new BufferedReader(isr);
String line=null;
while ( (line = br.readLine()) != null)
oPut = oPut + line+"\n";
} catch (IOException e)
{
e.printStackTrace();
}
}
static public String procList(){
return oPut;
}
}
you asked so iv'e copied all here.

You made oPut a static field - its initialized to "" once when the class is loaded and then shared between every new instance of eProcList, i.e. never cleared of the previous run. Either don't make it static (why is it static?) or clear it in the constructor.