Develop Reference

Multithreaded Geometry loading with GeoTools

Hey StackOverflow community,
I am currently trying to write a little tool, that reads a shapefiles geometries (Multipolygons / Polygons) and writes the WKT-representations of those into a text file.
To do so, I am using GeoTools and I managed to get it running fine, due to the fact that I am converting files with about 5000000 Polygons / Multipolygons, it takes pretty long to finish.
So my question is:
Is it possible to fasten up the file loading/writing?
As I am using a SimpleFeatureIterator I did not find out how to implement multithreading.
Is there a way to do so?
Or does anyone know, how to get the shapefiles geometries without using an iterator?
This is my code:
This method is just stating the File Chooser and starting the thread for each selected file.
protected static void printGeometriesToFile() {
JFileChooser chooser = new JFileChooser();
FileNameExtensionFilter filter = new FileNameExtensionFilter(
"shape-files", "shp");
chooser.setFileFilter(filter);
chooser.setDialogTitle("Choose the file to be converted.");
chooser.setMultiSelectionEnabled(true);
File[] files = null;
int returnVal = chooser.showOpenDialog(null);
if (returnVal == JFileChooser.APPROVE_OPTION) {
files = chooser.getSelectedFiles();
}
for (int i = 0; i < files.length; i++) {
MultiThreadWriter writer = new MultiThreadWriter(files[i]);
writer.start();
}
}
The class for multithreading:
class MultiThreadWriter extends Thread {
private File threadFile;
MultiThreadWriter(File file) {
threadFile = file;
System.out.println("Starting Thread for " + file.getName());
}
public void run() {
try {
File outputFolder = new File(threadFile.getAbsolutePath() + ".txt");
FileOutputStream fos = new FileOutputStream(outputFolder);
System.out.println("Now writing data to file: " + outputFolder.getName());
FileDataStore store = FileDataStoreFinder.getDataStore(threadFile);
SimpleFeatureSource featureSource = store.getFeatureSource();
SimpleFeatureCollection featureCollection = featureSource.getFeatures();
SimpleFeatureIterator featureIterator = featureCollection.features();
int pos = 0;
while (featureIterator.hasNext()) {
fos.write((geometryToByteArray((Polygonal) featureIterator.next().getAttribute("the_geom"))));
pos++;
System.out.println("The file " + threadFile.getName() + "'s current positon is: " + pos);
}
fos.close();
System.out.println("Finished writing.");
} catch (IOException e) {
e.printStackTrace();
}
}
}
This is just a helper function that converts the Multipolygons to polygons and returns its WKT-representation with a "|" as a seperator.
private byte[] geometryToByteArray(Polygonal polygonal) {
List<Polygon> polygonList;
String polygonString = "";
if (polygonal instanceof MultiPolygon) {
polygonList = GeometrieUtils.convertMultiPolygonToPolygonList((MultiPolygon) polygonal);
//The method above just converts a MultiPolygon into a list of Polygons
} else {
polygonList = new ArrayList<>(1);
polygonList.add((Polygon) polygonal);
}
for (int i = 0; i < polygonList.size(); i++) {
polygonString = polygonString + polygonList.get(i).toString() + "|";
}
return polygonString.getBytes();
}
}
I know my code is not pretty or good. I have just started learning Java and hope it will become better soon.
sincerely
ihavenoclue :)

You do not need create a new thread for every file, because creating new thread is an expensive operation. Instead, you can let MultiThreadWriter implements Runnable and use ThreadPoolExecuter manage all threads.
MultiThreadWriter
public class MultiThreadWriter implements Runnable {
#Override
public void run() {
//
}
}
Create thread pool matches your runtime processors.
ExecutorService service = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
for (int i = 0; i < files.length; i++) {
MultiThreadWriter writer = new MultiThreadWriter(files[i]);
service.submit(writer);
}
You can use BufferedWriter instead OutputStream, it is more
efficient when you repeatly write small pieces.
File outputFolder = new File(threadFile.getAbsolutePath() + ".txt");
FileOutputStream fos = new FileOutputStream(outputFolder);
BufferedWriter writer = new BufferedWriter(fos);

I would prefere to read files content as a list of objects, then split the list onto sublists, then create a thread to each list, example :
int nbrThreads = 10;
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(nbrThreads);
int count = myObjectsList != null ? myObjectsList.size() / nbrThreads : 0;
List<List<MyObject>> resultlists = choppeList(myObjectsList, count > 0 ? count : 1);
try
{
for (List<MyObject> list : resultlists)
{
// TODO : create your thread and passe the list of objects
}
executor.shutdown();
executor.awaitTermination(30, TimeUnit.MINUTESS); // chose time of termination
}
catch (Exception e)
{
LOG.error("Problem launching threads", e);
}
The choppeList method can be like that :
public <T> List<List<T>> choppeList(final List<T> list, final int L)
{
final List<List<T>> parts = new ArrayList<List<T>>();
final int N = list.size();
for (int i = 0; i < N; i += L)
{
parts.add(new ArrayList<T>(list.subList(i, Math.min(N, i + L))));
}
return parts;
}

How to interrupt thread for console reading?

I have such problem: program should read information from console in separate thread, and when l want to close this thread - input remains active. How to close input after interrupting thread? (programm runs at Eclipse IDE)
public static void main(final String[] args) {
for (int i = 0; i < 10; i++){
ChatRunner.reader = new BufferedReader(
new InputStreamReader(System.in, ChatRunner.charset));
//some treatment
final Thread input = new Thread(() -> {
try {
ChatRunner.answerFromConsole = ChatRunner.reader.readLine();
} catch (final IOException ioe) {
//
}
});
input.start();
// some treatment
input.interrupt();
}
}
If l simply close BufferedReader, i can't use it in next step in loop.

Reading multiple files - java.util.ConcurrentModificationException

Error "java.util.ConcurrentModificationException" sometimes appear, when I test more than three times the error will show up, I do not understand why it is always pointing to the "Word w = iter.next ();" and why it does not always show error.
//LoadFile.java
#Override
public void run() {
BufferedReader reader = null;
try {
reader = new BufferedReader(new InputStreamReader(
new FileInputStream(this.fileName), "UTF-8"));
String line = reader.readLine();
while (line != null) {
w = null;
String[] result1 = line.split(":");
w = new Word();
w.setId(result1[0]);
w.setWord(result1[1]);
w.setMean(result1[2]);
lw.add(w);
line = reader.readLine();
}
reader.close();
} catch (FileNotFoundException e) {
System.err.println(e.getMessage());
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("Thread " + fileName + " exiting.");
Gui.showWord(lw);
}
public void start() {
System.out.println("Starting " + fileName);
if (t == null) {
t = new Thread(this, fileName);
t.start();
}
}
//Gui.java
private void chooseFile() {
fileChooser = new JFileChooser();
fileChooser.setMultiSelectionEnabled(true);
int returnVal = fileChooser.showOpenDialog(Gui.this);
if (returnVal == JFileChooser.APPROVE_OPTION) {
file = fileChooser.getSelectedFiles();
for (int i = 0; i <= file.length - 1; i++) {
System.out.println(file[i].getName()
+ ".........................");
lf = new LoadFile(file[i].getName());
lf.start();
}
} else {
textAreaWord.append("Operation Canceled \n" + "new Line\n");
}
}
public static void showWord(List<Word> ls) {
Iterator<Word> iter = ls.iterator();
while (iter.hasNext()) {
Word w = iter.next();
textAreaWord.append(w.getWord() + "\n");
}
}
// error
Exception in thread "hello.txt" Thread hello.txt exiting.
Thread hi.txt exiting.
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(Unknown Source)
at java.util.ArrayList$Itr.next(Unknown Source)
at dictionary.Gui.showWord(Gui.java:138)
at dictionary.LoadFile.run(LoadFile.java:46)
at java.lang.Thread.run(Unknown Source)
Thanks!

Java Collection classes are fail-fast which means that if the Collection will be changed while some thread is traversing over it using iterator, the iterator.next() will throw a ConcurrentModificationException.
While one thread is iterating over lw (in showWord method), another thread is trying to add a Word in lw.

In multi Threaded environment while multiple threads are accessing same List it is safe to use CopyOnWriteArrayList.
Java 5 improves on the synchronized collections by providing several concurrent collection classes in java.util.concurrent package.
Also here is stackAnswer.

Is lw an ArrayList? if so when multiple threads access it at the same time it will throw the java.util.ConcurrentModificationException since it is not thread safe. You should use java.util.Vector instead as it is synchronized.

Termination Condition for threads in Java

I have written a multi-threaded Java application which reads a bunch of .jar files from a directory. This application spawns multiple threads and each threads reads bunch of jar files. I'm having trouble identifying the stopping condition for this application. How can i identify that all the files have been read?
The following is a snippet function which gets called from the run() method for each thread.
import java.io.*;
import java.util.Enumeration;
import java.util.jar.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipException;
import java.io.FilenameFilter;
public class ArchiveFileTest implements Runnable {
private static boolean stopAll = false;
private static int threadNumber = 0;
private int myNumber = 0;
public ArchiveFileTest () {
myNumber = threadNumber;
threadNumber++;
}
public static boolean setStopAll () {
return setStopAll(true);
}
public static boolean setStopAll (boolean b) {
stopAll = b;
return stopAll;
}
public static String[] listFiles (File parentDir,final String ext1,final String ext2,final String ext3,final String ext4) {
String allFiles[] = parentDir.list(new FilenameFilter() {
public boolean accept(File pDir, String fName) {
if (fName.endsWith("."+ext1) || fName.endsWith("."+ext2) || fName.endsWith("."+ext3) || fName.endsWith("."+ext4)) return true;
else return false;
}
});
for (int i=0; i<allFiles.length; i++)
allFiles[i] = parentDir.getAbsolutePath() + File.separator + allFiles[i];
return allFiles;
}
public ZipFile getMyZipFile (File parentDir) {
String fn[] = listFiles(parentDir, "jar", "zip", "war", "rar");
int fileNum = myNumber % fn.length;
ZipFile zFile = null;
for (int i=0; i<fn.length; i++) {
String jFile = fn[(fileNum + i)%fn.length];
try {
zFile = new ZipFile(jFile);
break;
} catch (IOException e) {
setStopAll();
}
}
return zFile;
}
public void doStuff() throws Exception {
File dName = new File("/home/sqatest/chander/sample-files");
final int N_TIMES = 15;
final int N_FILES = 500;
int counter = 0;
int fCount = 0;
if (!dName.isDirectory() || !dName.exists()) {
System.err.println("The parent directory given should point to an existing directory...");
setStopAll();
return;
}
while (counter < N_TIMES) {
ZipFile zipFile = getMyZipFile(dName);
if (zipFile == null) {
System.err.println("No zip file entry for the Thread-" + myNumber);
break;
}
try {
Enumeration <? extends ZipEntry> zipEntries = zipFile.entries();
fCount = 0;
ZipEntry ze = null;
while (zipEntries.hasMoreElements()) {
ze = zipEntries.nextElement();
if (ze.isDirectory()) continue; // if it is a directory go to next entry
InputStream is = zipFile.getInputStream(ze);
fCount++;
int readCount = 0;
try {
while(is.read((new byte[50])) != -1 && readCount != 200) readCount++;
System.out.println("Successfully Read " + zipFile.toString());
//is.close();
} catch (IOException e) {
e.printStackTrace();
}
if (fCount == N_FILES) break; // read maximum of N_FILES
}
if (stopAll) break;
} catch (Exception e) {
e.printStackTrace();
} finally {
counter++;
}
}
}
public void run () {
try {
doStuff();
} catch (IOException e) {
e.printStackTrace();
setStopAll();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main (String[] args) throws Exception {
final int MAX_THREADS = 500;
final int MAX_HOLDING_THREADS = 5;
int loopCount = 0;
Thread mainThread = Thread.currentThread();
for (int m=0; ; m++) {
Thread t[] = new Thread[MAX_HOLDING_THREADS];
for (int n=0; n<t.length; n++) {
t[n] = new Thread(new ArchiveFileTest());
t[n].start();
if ((m+1)*(n+1)==MAX_THREADS) {
System.out.println("\n" + MAX_THREADS + " reached... \nMain Sleeping for some mins...");
loopCount++;
try {
t[n].join();
System.out.println("\nMain is back... (" + loopCount + ")");
} catch (InterruptedException e) {
e.printStackTrace();
setStopAll();
}
m = 0;
}
}
}
}
}

I don't think your application will ever stop. You've got an infinite loop in the main method:
for (int m=0; ; m++) {
....
}
Note, setting m=0 inside the body won't break the loop, so I think you'll never end even if you have no file. It then continuously reads all zip/jar/war/rar files in the directory (choosing the file based on a rotating counter myNumber is not very maintainable), but never exits the loop.
If you're requirement is to read ZIP files using a number of threads, then I would go about it a different way.
Create a Set of files which you want to look at.
Create a ThreadPoolExecutor to create a fixed pool of 5 threads
Iterate over the set of files and create a new Runnable which does the Zip Extraction (though I'm not quite sure why you read the first 10000 bytes of a ZIP entry and then don't do anything with it), and call the execute method. That will use the thread pool to process 5 files at a time.
After submitting all the runnables Use the shutdown method, which will wait for all submitted tasks to finish, and the shutdown the thread pool.

If by stopping you mean terminating then the application will stop when all threads, that are not daemon special case, are finished.

In your class that launches the threads, have a volatile counter for your running threads.
In your thread constructor pass a reference to the launching class.
Have a synchronized method to let the threads notify the launching class that they are done.
After instancing and starting your threads wait for the counter to become 0;
while(getRunningThreads() > 0) // getRunningThreads must be synchronized too
Thread.sleep(500); // Check every half second.

Multiple 'if' statements into one outputStream

import java.io.*;
import java.io.File;
import java.io.FilenameFilter;
public class YDSearch{
public void listFiles(String dir) throws IOException{
File directory = new File(dir);
if (!directory.isDirectory()) {
System.out.println("No directory provided");
return;
}
//create a FilenameFilter and override its accept-method
FilenameFilter filefilter = new FilenameFilter() {
public boolean accept(File dir, String name) {
//if the file extension is .mp3 return true, else false
return name.endsWith(".mp3")||name.endsWith(".mp4")||name.endsWith(".3gp")
||name.endsWith(".mov")||name.endsWith(".avi")||name.endsWith(".wmv");
}
};
String[] filenames = directory.list(filefilter);
DataOutputStream output = new DataOutputStream(new FileOutputStream("C:/Users/Jonathan/Desktop/YouDetect/SearchByFileType/AllMediaFiles.dat"));
for (String name : filenames) {
output.writeUTF(dir + name);
}
output.close();
DataInputStream input = new DataInputStream(new FileInputStream("C:/Users/Jonathan/Desktop/YouDetect/SearchByFileType/AllMediaFiles.dat"));
DataOutputStream output2 = new DataOutputStream(new FileOutputStream("C:/Users/Jonathan/Desktop/ReadyForAnalysis.dat"));
for (String name : filenames) {
FileInputStream in = new FileInputStream(input.readUTF());
int byteCounter = 0;
int rowCounter = 0;
long bufferCounter = 0;
if(name.endsWith(".mp3")){
byte[] b = new byte[36];
int read = in.read(b, 0, 36);
if (byteCounter != 1000){
if (rowCounter == 1){
System.out.println("\n");
rowCounter = 0;
}
output2.writeUTF(org.apache.commons.codec.binary.Hex.encodeHexString(b)+ " " + dir + name);
bufferCounter ++;
rowCounter ++;
}else{
byteCounter = 0;
try{
Thread.sleep(200);
}catch(InterruptedException e) {
}
}
}
else if(name.endsWith(".mp4")){
byte[] b = new byte[29];
int read = in.read(b, 0, 29);
if (byteCounter != 1000){
if (rowCounter == 1){
System.out.println("\n");
rowCounter = 0;
}
output2.writeUTF(org.apache.commons.codec.binary.Hex.encodeHexString(b)+ " " + dir + name);
bufferCounter ++;
rowCounter ++;
}else{
byteCounter = 0;
try{
Thread.sleep(200);
}catch(InterruptedException e) {
}
}
}
//System.out.println("====================");
}
output2.close();
input.close();
DataInputStream input2 = new DataInputStream(new FileInputStream("C:/Users/Jonathan/Desktop/ReadyForAnalysis.dat"));
for (String name : filenames) {
System.out.println(input2.readUTF()+"\n");
}
}
public void checkHeaderSC(String allFiles)throws IOException{
}
public static void main(String[] args) throws IOException {
YDSearch YDSearch = new YDSearch();
YDSearch.listFiles("C:/Users/Jonathan/Desktop/YD Tests/1) High Quality/");
YDSearch.listFiles("C:/Users/Jonathan/Desktop/YD Tests/2) Medium Quality/");
YDSearch.listFiles("C:/Users/Jonathan/Desktop/YD Tests/3) Low Quality/");
YDSearch.checkHeaderSC("C:/Users/Jonathan/Desktop/YouDetect/SearchByFileType/ReadyForAnalysis.dat");
}
}
Hey there, having a little issue with the above coding and hoped someone here might be able to help. This is sort of a partial version of the code as the real one has 4 more if/else if statements involved.
The program compiles and begins to run fine. It produces several results back from the file that is being read into/then out of again in input2 but then stops, produces no more results and gives the error:
Exception in thread "main" java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
at java.io.DataInputStream.readUTF(DataInputStream.java:572)
at java.io.DataInputStream.readUTF(DataInputStream.java:547)
at YDSearch.listFiles(YDSearch.java:85)
at YDSearch.main(YDSearch.java:93)
Anybody know why this might be happening and have a solution they could share?
I've also tried making the variable 'b' to be inside of an if statement but that doesn't work because of scope. If b was defined by if's then there would only need to be one if statement to output to the file
Please let me know if you've got any ideas, I'd really appreciate it :)

As far as I can see, you don't always put out an output record for every name, only for when the name matches one of your patterns. However, you do try to read an input record for every name.
Ergo, if you have any filenames that don't match the patterns you try to read more than you write, and you will get the EOF.
EDIT:
In more detail, the problem is that you get a list of all the files that end with "mp3", "mp4", "3gp", "mov", "avi or "wmv". You then process that list, and write out something into C:/Users/Jonathan/Desktop/ReadyForAnalysis.dat for each "mp3" and "mp4" file. You then assume that for each entry in your list of files, that you will have an entry in ReadyForAnalysis.dat. However, if there are any files ending in "3gp", "mov", "avi or "wmv" then this will not hold true.