I have written a multi-threaded Java application which reads a bunch of .jar files from a directory. This application spawns multiple threads and each threads reads bunch of jar files. I'm having trouble identifying the stopping condition for this application. How can i identify that all the files have been read?
The following is a snippet function which gets called from the run() method for each thread.
import java.io.*;
import java.util.Enumeration;
import java.util.jar.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipException;
import java.io.FilenameFilter;
public class ArchiveFileTest implements Runnable {
private static boolean stopAll = false;
private static int threadNumber = 0;
private int myNumber = 0;
public ArchiveFileTest () {
myNumber = threadNumber;
threadNumber++;
}
public static boolean setStopAll () {
return setStopAll(true);
}
public static boolean setStopAll (boolean b) {
stopAll = b;
return stopAll;
}
public static String[] listFiles (File parentDir,final String ext1,final String ext2,final String ext3,final String ext4) {
String allFiles[] = parentDir.list(new FilenameFilter() {
public boolean accept(File pDir, String fName) {
if (fName.endsWith("."+ext1) || fName.endsWith("."+ext2) || fName.endsWith("."+ext3) || fName.endsWith("."+ext4)) return true;
else return false;
}
});
for (int i=0; i<allFiles.length; i++)
allFiles[i] = parentDir.getAbsolutePath() + File.separator + allFiles[i];
return allFiles;
}
public ZipFile getMyZipFile (File parentDir) {
String fn[] = listFiles(parentDir, "jar", "zip", "war", "rar");
int fileNum = myNumber % fn.length;
ZipFile zFile = null;
for (int i=0; i<fn.length; i++) {
String jFile = fn[(fileNum + i)%fn.length];
try {
zFile = new ZipFile(jFile);
break;
} catch (IOException e) {
setStopAll();
}
}
return zFile;
}
public void doStuff() throws Exception {
File dName = new File("/home/sqatest/chander/sample-files");
final int N_TIMES = 15;
final int N_FILES = 500;
int counter = 0;
int fCount = 0;
if (!dName.isDirectory() || !dName.exists()) {
System.err.println("The parent directory given should point to an existing directory...");
setStopAll();
return;
}
while (counter < N_TIMES) {
ZipFile zipFile = getMyZipFile(dName);
if (zipFile == null) {
System.err.println("No zip file entry for the Thread-" + myNumber);
break;
}
try {
Enumeration <? extends ZipEntry> zipEntries = zipFile.entries();
fCount = 0;
ZipEntry ze = null;
while (zipEntries.hasMoreElements()) {
ze = zipEntries.nextElement();
if (ze.isDirectory()) continue; // if it is a directory go to next entry
InputStream is = zipFile.getInputStream(ze);
fCount++;
int readCount = 0;
try {
while(is.read((new byte[50])) != -1 && readCount != 200) readCount++;
System.out.println("Successfully Read " + zipFile.toString());
//is.close();
} catch (IOException e) {
e.printStackTrace();
}
if (fCount == N_FILES) break; // read maximum of N_FILES
}
if (stopAll) break;
} catch (Exception e) {
e.printStackTrace();
} finally {
counter++;
}
}
}
public void run () {
try {
doStuff();
} catch (IOException e) {
e.printStackTrace();
setStopAll();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main (String[] args) throws Exception {
final int MAX_THREADS = 500;
final int MAX_HOLDING_THREADS = 5;
int loopCount = 0;
Thread mainThread = Thread.currentThread();
for (int m=0; ; m++) {
Thread t[] = new Thread[MAX_HOLDING_THREADS];
for (int n=0; n<t.length; n++) {
t[n] = new Thread(new ArchiveFileTest());
t[n].start();
if ((m+1)*(n+1)==MAX_THREADS) {
System.out.println("\n" + MAX_THREADS + " reached... \nMain Sleeping for some mins...");
loopCount++;
try {
t[n].join();
System.out.println("\nMain is back... (" + loopCount + ")");
} catch (InterruptedException e) {
e.printStackTrace();
setStopAll();
}
m = 0;
}
}
}
}
}
I don't think your application will ever stop. You've got an infinite loop in the main method:
for (int m=0; ; m++) {
....
}
Note, setting m=0 inside the body won't break the loop, so I think you'll never end even if you have no file. It then continuously reads all zip/jar/war/rar files in the directory (choosing the file based on a rotating counter myNumber is not very maintainable), but never exits the loop.
If you're requirement is to read ZIP files using a number of threads, then I would go about it a different way.
Create a Set of files which you want to look at.
Create a ThreadPoolExecutor to create a fixed pool of 5 threads
Iterate over the set of files and create a new Runnable which does the Zip Extraction (though I'm not quite sure why you read the first 10000 bytes of a ZIP entry and then don't do anything with it), and call the execute method. That will use the thread pool to process 5 files at a time.
After submitting all the runnables Use the shutdown method, which will wait for all submitted tasks to finish, and the shutdown the thread pool.
If by stopping you mean terminating then the application will stop when all threads, that are not daemon special case, are finished.
In your class that launches the threads, have a volatile counter for your running threads.
In your thread constructor pass a reference to the launching class.
Have a synchronized method to let the threads notify the launching class that they are done.
After instancing and starting your threads wait for the counter to become 0;
while(getRunningThreads() > 0) // getRunningThreads must be synchronized too
Thread.sleep(500); // Check every half second.
Related
INSTRUCTIONS:
Write a program NumberCount that counts the numbers (including integers and floating point values) in
one or more text files. Note that only numbers separated by whitespace characters are counted, i.e., only
those numbers that can be read by either readInt() or readDouble() are considered.
So iv been trying to get this program to read text files and the title is pretty much the instructions but it does not want to read my textfiles that i have in the project folder (i tried moving it a bunch of times but anywhere i put it it didnt load up) This is my code
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
public class NumberCount implements Runnable {
private static int combinedCount = 0;
public static synchronized int getCombinedCount() {
return combinedCount;
}
public synchronized void setCombinedCount( int combinedCount) {
this.combinedCount = combinedCount;
}
String filename;
NumberCount(String filename) {
this.filename = filename;
}
NumberCount() {
}
#Override
public void run() {
String fileText = this.getTextFromFile(filename);
System.out.println(filename + ": " + countNumbers(fileText));
setCombinedCount(getCombinedCount() + countNumbers(fileText));
}
String getTextFromFile(String filename) {
try {
String data = "";
BufferedReader br = new BufferedReader(new FileReader(new File(filename)));
String st;
while ((st = br.readLine()) !=null) {
data += "\n" + st;
}
data = data.replaceAll("\n", " ");
return data;
} catch(Exception e) {
System.out.println("Unable to retrieve text from file: " + filename );
return "";
}
}
int countNumbers(String text) {
int count = 0;
String words[] = text.split(" ");
for (String word : words) {
try {
Integer.parseInt(word);
Float.parseFloat(word);
count++;
} catch (Exception e) {
}
}
return count;
}
String helpMessage() {
String data = "Please call as NumberCount <list of file names>\n";
data += "File names to be present in the same directory";
return data;
}
public static void main(String[] args) {
if (args.length == 0) {
System.out.println(new NumberCount().helpMessage());
} else {
Thread threads[] = new Thread[args.length];
int i = 0;
for (String filename : args) {
NumberCount nc = new NumberCount(filename);
threads[i] = new Thread(nc);
threads[i++].start();
}
try {
for(Thread t : threads) {
t.join();
}
}catch(Exception e) {
e.printStackTrace();
}
System.out.println("combined count: " + getCombinedCount());
}
}
}
I'm using this code to split and process a csv file, the problem is that the chunks are being set in an arbitrary location, maybe at the beginning, middle or end of the line!
How can I set the start_loc to be at the begging or end of the line, so the chunks are a complete CSV file without missing any data?
public static void main(String[] args) throws IOException {
long start = System.currentTimeMillis();
CSVReader reader = new CSVReader(new FileReader("x_tran.csv"));
String[] columnsNames = reader.readNext();
reader.close();
FileInputStream fileInputStream = new FileInputStream("x_tran.csv");
FileChannel channel = fileInputStream.getChannel();
long remaining_size = channel.size(); //get the total number of bytes in the file
long chunk_size = remaining_size / 4; //file_size/threads
//Max allocation size allowed is ~2GB
if (chunk_size > (Integer.MAX_VALUE - 5))
{
chunk_size = (Integer.MAX_VALUE - 5);
}
//thread pool
ExecutorService executor = Executors.newFixedThreadPool(4);
long start_loc = 0;//file pointer
int i = 0; //loop counter
boolean first = true;
while (remaining_size >= chunk_size)
{
//launches a new thread
executor.execute(new FileRead(start_loc, toIntExact(chunk_size), channel, i, String.join(",", columnsNames), first));
remaining_size = remaining_size - chunk_size;
start_loc = start_loc + chunk_size;
i++;
first = false;
}
//load the last remaining piece
executor.execute(new FileRead(start_loc, toIntExact(remaining_size), channel, i, String.join(",", columnsNames), first));
//Tear Down
executor.shutdown();
//Wait for all threads to finish
while (!executor.isTerminated())
{
//wait for infinity time
}
System.out.println("Finished all threads");
fileInputStream.close();
long finish = System.currentTimeMillis();
System.out.println( "Time elapsed: " + (finish - start) );
}
You can read the file once and then make each thread handles lines modulo the number of threads (for example first thread handles lines number 0, 4, 8, etc).
package ...;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class CsvParallelReader {
private static final int THREAD_NUMBER = 4;
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(THREAD_NUMBER);
try {
List<String> lines = Files.readAllLines(Path.of("yourfile.csv"));
for (int i = 0; i < THREAD_NUMBER; i++) {
Runnable readTask = new ReadTask(i, lines);
executor.submit(readTask);
}
} catch (IOException e) {
e.printStackTrace();
}
}
private static class ReadTask implements Runnable {
private final List<String> lines;
private int start;
public ReadTask(int start, List<String> lines) {
this.start = start;
this.lines = lines;
}
#Override
public void run() {
for (int i = start; i < lines.size(); i += THREAD_NUMBER) {
// do something with this line of data
}
}
}
}
Hey StackOverflow community,
I am currently trying to write a little tool, that reads a shapefiles geometries (Multipolygons / Polygons) and writes the WKT-representations of those into a text file.
To do so, I am using GeoTools and I managed to get it running fine, due to the fact that I am converting files with about 5000000 Polygons / Multipolygons, it takes pretty long to finish.
So my question is:
Is it possible to fasten up the file loading/writing?
As I am using a SimpleFeatureIterator I did not find out how to implement multithreading.
Is there a way to do so?
Or does anyone know, how to get the shapefiles geometries without using an iterator?
This is my code:
This method is just stating the File Chooser and starting the thread for each selected file.
protected static void printGeometriesToFile() {
JFileChooser chooser = new JFileChooser();
FileNameExtensionFilter filter = new FileNameExtensionFilter(
"shape-files", "shp");
chooser.setFileFilter(filter);
chooser.setDialogTitle("Choose the file to be converted.");
chooser.setMultiSelectionEnabled(true);
File[] files = null;
int returnVal = chooser.showOpenDialog(null);
if (returnVal == JFileChooser.APPROVE_OPTION) {
files = chooser.getSelectedFiles();
}
for (int i = 0; i < files.length; i++) {
MultiThreadWriter writer = new MultiThreadWriter(files[i]);
writer.start();
}
}
The class for multithreading:
class MultiThreadWriter extends Thread {
private File threadFile;
MultiThreadWriter(File file) {
threadFile = file;
System.out.println("Starting Thread for " + file.getName());
}
public void run() {
try {
File outputFolder = new File(threadFile.getAbsolutePath() + ".txt");
FileOutputStream fos = new FileOutputStream(outputFolder);
System.out.println("Now writing data to file: " + outputFolder.getName());
FileDataStore store = FileDataStoreFinder.getDataStore(threadFile);
SimpleFeatureSource featureSource = store.getFeatureSource();
SimpleFeatureCollection featureCollection = featureSource.getFeatures();
SimpleFeatureIterator featureIterator = featureCollection.features();
int pos = 0;
while (featureIterator.hasNext()) {
fos.write((geometryToByteArray((Polygonal) featureIterator.next().getAttribute("the_geom"))));
pos++;
System.out.println("The file " + threadFile.getName() + "'s current positon is: " + pos);
}
fos.close();
System.out.println("Finished writing.");
} catch (IOException e) {
e.printStackTrace();
}
}
}
This is just a helper function that converts the Multipolygons to polygons and returns its WKT-representation with a "|" as a seperator.
private byte[] geometryToByteArray(Polygonal polygonal) {
List<Polygon> polygonList;
String polygonString = "";
if (polygonal instanceof MultiPolygon) {
polygonList = GeometrieUtils.convertMultiPolygonToPolygonList((MultiPolygon) polygonal);
//The method above just converts a MultiPolygon into a list of Polygons
} else {
polygonList = new ArrayList<>(1);
polygonList.add((Polygon) polygonal);
}
for (int i = 0; i < polygonList.size(); i++) {
polygonString = polygonString + polygonList.get(i).toString() + "|";
}
return polygonString.getBytes();
}
}
I know my code is not pretty or good. I have just started learning Java and hope it will become better soon.
sincerely
ihavenoclue :)
You do not need create a new thread for every file, because creating new thread is an expensive operation. Instead, you can let MultiThreadWriter implements Runnable and use ThreadPoolExecuter manage all threads.
MultiThreadWriter
public class MultiThreadWriter implements Runnable {
#Override
public void run() {
//
}
}
Create thread pool matches your runtime processors.
ExecutorService service = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
for (int i = 0; i < files.length; i++) {
MultiThreadWriter writer = new MultiThreadWriter(files[i]);
service.submit(writer);
}
You can use BufferedWriter instead OutputStream, it is more
efficient when you repeatly write small pieces.
File outputFolder = new File(threadFile.getAbsolutePath() + ".txt");
FileOutputStream fos = new FileOutputStream(outputFolder);
BufferedWriter writer = new BufferedWriter(fos);
I would prefere to read files content as a list of objects, then split the list onto sublists, then create a thread to each list, example :
int nbrThreads = 10;
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(nbrThreads);
int count = myObjectsList != null ? myObjectsList.size() / nbrThreads : 0;
List<List<MyObject>> resultlists = choppeList(myObjectsList, count > 0 ? count : 1);
try
{
for (List<MyObject> list : resultlists)
{
// TODO : create your thread and passe the list of objects
}
executor.shutdown();
executor.awaitTermination(30, TimeUnit.MINUTESS); // chose time of termination
}
catch (Exception e)
{
LOG.error("Problem launching threads", e);
}
The choppeList method can be like that :
public <T> List<List<T>> choppeList(final List<T> list, final int L)
{
final List<List<T>> parts = new ArrayList<List<T>>();
final int N = list.size();
for (int i = 0; i < N; i += L)
{
parts.add(new ArrayList<T>(list.subList(i, Math.min(N, i + L))));
}
return parts;
}
I have problem. I need to create 9 files, each called from thread name. Each file will be called 1.txt, 2.txt, 3.txt etc. Each file will be filled with a symbol that corresponds to the name of the file (1.txt file is "1"). Each file should be 100 lines, each line length is 100 characters. This work must perform threads of execution and I\O. I need to read the contents of these files in the resulting file super.txt, when using several threads.
My code:
public class CustomThread extends Thread {
Thread t;
String threadName;
CustomThread(String threadName) {
this.threadName = threadName;
}
#Override
public void run() {
if (t == null) {
t = new Thread(this);
}
add(threadName);
}
public void add(String threadName) {
File f = new File(threadName + ".txt");
if (!f.exists()) {
try {
f.createNewFile();
} catch (IOException e) {
e.printStackTrace();
System.out.println("File does not exists!");
}
}
FileWriter fw = null;
try {
fw = new FileWriter(f);
for (int i = 0; i < 100; i++) {
for (int j = 0; j < 100; j++) {
fw.write(threadName);
}
fw.write('\n');
}
} catch (IOException e) {
e.printStackTrace();
System.out.println("File does not exists!");
}
}
}
My main:
public class Main {
public static void main(String[] args) {
CustomThread T1 = new CustomThread("1");
T1.start();
CustomThread T2 = new CustomThread("2");
T2.start();
}
}
First question. I need, to make threads in cycle. Look on my main: I create
CustomThread T1 = new CustomThread("1");
T1.start();
But, i want to create 9 files in cycle. How to do this ?
Second question. I need to write in every my file from multiple threads.
Third question. How to write from multiple threads in result file five contents of thats files ?
i want to create 9 files in cycle. How to do this ?
Use a loop
for (int i = 1; i <= 9; i++) {
new CustomThread("" + i).start();
}
I need to write in every my file from multiple threads.
How do you do this? Open the files before you start the threads and lock them when ever you use them.
How to write from multiple threads in result file five contents of thats files ?
Can you rephrase that question?
Am trying to write all the prime numbers less than 10000 into a PipedOutPutStream. When I execute the program it prints till 1619. But the the program is still running. If I comment the code for writing into the stream and execute the program then it prints the prime numbers correctly.Can anyone figure out why is this happening ?
public class PrimeThread implements Runnable {
private DataOutputStream os;
public PrimeThread(OutputStream out) {
os = new DataOutputStream(out);
}
#Override
public void run() {
try {
int flag = 0;
for (int i = 2; i < 10000; i++) {
flag = 0;
for (int j = 2; j < i / 2; j++) {
if (i % j == 0) {
flag = 1;
break;
}
}
if (flag == 0) {
System.out.println(i);
os.writeInt(i);//if i comment this line its printing all the primeno
os.flush();
}
}
os.writeInt(-1);
os.flush();
os.close();
} catch (IOException iOException) {
System.out.println(iOException.getMessage());
}
}
}
public class Main {
public static void main(String[] args) {
PipedOutputStream po1 = new PipedOutputStream();
//PipedOutputStream po2 = new PipedOutputStream();
PipedInputStream pi1 = null;
//PipedInputStream pi2 = null;
try {
pi1 = new PipedInputStream(po1);
//pi2 = new PipedInputStream(po2);
} catch (IOException iOException) {
System.out.println(iOException.getMessage());
}
Runnable prime = new PrimeThread(po1);
Thread t1 = new Thread(prime, "Prime");
t1.start();
//some code
}
}
PipedOutPutStream's internal buffer is overflowed and cannot accept more data until it is dumped. In the main thread, read from PipedInputStream pi1, and prime number calculation would go on.