I am trying to edit a json file by 5 different threads and then copy it 5 times, as well, to the same destination path. As you see, there are concurrency problems.
I tried this for the copy method:
public static void copyFile(String originPath, String destinationPath, String file) throws IOException {
logger.debug("Starting copyFile of" + file);
FileChannel channel = null;
File lockFile = new File(originPath + file);
try {
logger.debug("Comienza el proceso de procesado del json del fichero:" + file);
channel = extracted(lockFile).getChannel();
FileLock fileLock = null;
try {
fileLock = channel.lock();
} catch(OverlappingFileLockException e) {
logger.error("3 "+e.getMessage());
Thread.sleep(200);
}
FileUtils.copyFile(FileUtils.getFile(originPath + file), FileUtils.getFile(destinationPath + file));
logger.debug("The copy of " + file + " ends.");
fileLock.release();
channel.close();
} catch (IOException e) {
logger.error("Failing to copy "+file " to " + destinationPath + e.getMessage());
}
}
I'm getting nulls and IOExceptions.
All I want is that when a file is processed the other threads just wait for it in a queue, one after each other.
If I understand correctly your problem is when one of your threads tries to copy one file, other threads wait in a queue and then start their work one by one.
you can achieve this by code below
public class FileLock {
static Semaphore semaphore = new Semaphore(1);
public static void copyFile(String originPath, String destinationPath, String file) throws IOException {
try {
semaphore.acquire();
logger.debug("Comienza el proceso de procesado del json del fichero:" + file);
FileUtils.copyFile(FileUtils.getFile(originPath + file), FileUtils.getFile(destinationPath + file));
logger.debug("The copy of " + file + " ends.");
} catch (IOException | InterruptedException e) {
logger.error("Failing to copy " + file" to " + destinationPath + e.getMessage());
} finally {
semaphore.release();
}
}
}
Related
I have this current implementation but I couldn't append the new data that were being passed on in processName, activity and numberOfRecords. It just keeps creating a separate .txt file for each transaction.
#Override
public void saveSummaryLogs(String processName, String activity,
int numberOfRecords) throws Exception {
log.info("Test if this method is being executed.");
log.info(INSERT_LOGS_TO_FILE);
log.info("Creating a folder directory is being executed!");
try {
File f = new File(summaryLogsDbDir);
f.mkdirs();
} catch (Exception e) {
throw new Exception("Permission denied to create parent folder.",
e);
}
writeToFile(processName, activity, numberOfRecords);
log.info("writeToFile has been called and executed!");
}
public void writeToFile(String processName, String activity,
int numberOfRecords) throws Exception {
log.info("writeToFile is being executed!");
Date date = new Date();
SimpleDateFormat dateFormat = new SimpleDateFormat("ddMMyyy_HHmmss");
try (BufferedWriter out = new BufferedWriter(new FileWriter(
summaryLogsDbDir + LOGS_FILENAME + dateFormat.format(date)
+ ".txt"))) {
out.append(dateFormat.format(date)+ ": " + processName + "\n");
out.append(dateFormat.format(date)+ ": " + activity + "\n");
out.append(dateFormat.format(date)+ ": " + numberOfRecords + "\n");
} catch (Exception e) {
throw new Exception("Failed a summary-logs.txt file.", e);
}
log.info("writeToFile has been executed!");
}
Any help of advise is greatly appreciated. Thanks!
I have Java project that we need to keep listening some path to detect if there is any new XML file and if true, we need to process it through few other rules.
The WatchService is doing very well to discover if I have a new file there to be processed, but I wasnt able to manage it so I could read the file, I just gives me back the file name, from the event.
Is there any way to that with WatchService? If not, whats another sugestion to reach?
Thanks.
Sure, you can easely to that:
Create the watch service:
WatchService watchService = null;
try {
watchService = FileSystems.getDefault().newWatchService();
Path path = get(pathToSee);
path.register(watchService, new WatchEvent.Kind[]{ENTRY_MODIFY, ENTRY_CREATE}, SensitivityWatchEventModifier.HIGH);
watchEvent(watchService, path);
log.info("Watch Service has ben created!");
} catch (IOException e) {
log.error("Exception has ben throw when the service have tried to createWatchService()", e);
}
Note: If you have a large amount of files to be added, you can put:
SensitivityWatchEventModifier.HIGH
To increase sensitivity.
Watch if have changes in you directory:
WatchKey key;
while (true) {
try {
if ((key = watchService.take()) == null) break;
for (WatchEvent<?> event : key.pollEvents()) {
log.info("Event kind:" + event.kind()
+ ". File affected: " + event.context() + ".");
String fileName = event.context().toString();
File directory = path.toFile();
yourService.readContent(directory, fileName);
}
key.reset();
} catch (InterruptedException | IOException e) {
log.error("InterruptedException when try watchEvent()" + e);
}
}
And finally, you can do what you want to do with that infos:
try (BufferedReader br = new BufferedReader(new FileReader(directory + "/" + fileName))) {
String strLine;
while ((strLine = br.readLine()) != null) {
}
}
}
}
Tips:
You can create a separete thread to do that, or use Spring #Async to create a separete thread to process this information and increase concurrency in your application.
You can use Apache Commons you to do that too!
e.g:
public void getAll() throws Exception {
FileAlterationObserver observer = new FileAlterationObserver(pathToSee);
observer.addListener(new FileAlterationListenerAdaptor() {
#SneakyThrows
#Override
public void onFileCreate(File file) {
}
#Override
public void onFileDelete(File file) {
}
});
}
So lately i ve been trying to wrap my head around concurrency. And currently I m trying to find a solution for the ReaderWriter Problem.
I got a class File, it counts the number of Readers/Writers and has two Semaphores.
When a Reader tries to read it has to wait as long as there is a Writer Thread writing. When it enters readCount gets incremented within the readerSemaphore
& decremented within the same Semaphore when it leaves.
When a Writer tries to enter it has to wait for as long as there is more than one reader. When it enters it aquires the writerSemaphore and increaese the writerCount. When it leaves it releases the Semaphore.
For some reason that I cant figure out the Writers are not editing the String file in class File.
Thanks in advance :)
public class Main {
public static void main(String[]args) {
File file = new File("1. Chapter: ");
Writer w1 = new Writer(file, " w1 ");
Writer w2 = new Writer(file, " w2 ");
Reader r1 = new Reader(file);
Reader r2 = new Reader(file);
Reader r3 = new Reader(file);
Reader r4 = new Reader(file);
Reader r5 = new Reader(file);
w1.start();
w2.start();
r1.start();
r2.start();
r3.start();
r4.start();
r5.start();
try {
w2.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("-> Final output: " + file.file);
}}
public class File {
public String file;
private int readCount;
private int writeCount;
private Semaphore semReader;
private Semaphore semWriter;
public File(String file) {
this.file = file;
readCount = 0;
writeCount = 0;
semReader = new Semaphore(1);
semWriter = new Semaphore(1);
}
public synchronized void startReading() {
try {
while(writeCount == 1) {
Thread.currentThread().wait();
}
semReader.acquire();
readCount++;
semReader.release();
System.out.println(" --- File was read");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public synchronized String endReading() {
String temp = file;
try {
semReader.acquire();
readCount--;
semReader.release();
} catch (InterruptedException e) {
e.printStackTrace();
}
return temp;
}
public synchronized void startWriting(String edit) {
try {
while(readCount > 0) {
Thread.currentThread().wait();
}
semWriter.acquire();
writeCount++;
System.out.println(" --- File got edited");
file = file + "hi";
}
catch (Exception e) {
}
}
public synchronized void endWriting() {
writeCount--;
semWriter.release();
}}
public class Writer extends Thread {
private File file;
private String edit;
public Writer(File file, String edit) {
this.file = file;
this.edit = edit;
}
#Override
public void run() {
Random rand = new Random();
try {
sleep(1000);
System.out.println(">W: " + Thread.currentThread().getName() + " started first write.");
file.startWriting(" first" + edit);
sleep(3000);
System.out.println(">W: " + Thread.currentThread().getName() + " ended first write.");
file.endWriting();
sleep(2000);
System.out.println(">W: " + Thread.currentThread().getName() + " started second write.");
file.startWriting(" second" + edit);
sleep(3000);
System.out.println(">W: " + Thread.currentThread().getName() + " ended second write.");
file.endWriting();
System.out.println(">W: " + Thread.currentThread().getName() + " finished");
} catch (InterruptedException e) {
e.printStackTrace();
}
}}
public class Reader extends Thread {
private File file;
public Reader(File file) {
this.file = file;
}
#Override
public void run() {
Random rand = new Random();
try {
sleep(rand.nextInt(2000));
System.out.println(">R: " + Thread.currentThread().getName() + " startet first read.");
file.startReading();
sleep(3000);
System.out.print(">R: " + Thread.currentThread().getName() + " ended first read: ");
System.out.println(file.endReading());
sleep(rand.nextInt(2000));
System.out.println(">R: " + Thread.currentThread().getName() + " startet second read.");
file.startReading();
sleep(3000);
System.out.print(">R: " + Thread.currentThread().getName() + " ended second read: ");
System.out.println(file.endReading());
System.out.println(">R: " + Thread.currentThread().getName() + " finished");
} catch (InterruptedException e) {
e.printStackTrace();
}
}}
Edit:
Thread.currentThread().wait() was wrong.
Waiting on threads is discouraged in the java docs.
Thanks, #JB Nizet for helping me.
I have a Python app and a Java app running simultaneously.
I want Java to start the Python process and then communicate using the normal STDIN/STDOUT streams with Python.
I have started the process correctly and have two threads to handle the two I/O streams.
OUTPUT THREAD:
class output2 extends Thread {
Process process;
OutputStream stdin;
BufferedWriter writer;
Scanner in = new Scanner(System.in);
output2(Process p) {
try {
process = p;
stdin = process.getOutputStream();
writer = new BufferedWriter(new OutputStreamWriter(stdin));
} catch (Exception e) {
System.out.println("ERROR output2(): " + e);
}
}
#Override
public void run() {
System.out.println("Starting OUTPUT THREAD");
try {
while (true) {
String input = in.nextLine();
writer.write(input);
writer.flush();
}
} catch (Exception e) {
System.out.println("ERROR output2_run(): " + e);
}
System.out.println("Ending OUTPUT THREAD");
}
}
INPUT THREAD :
class input2 extends Thread {
Process process;
InputStream stdout;
BufferedReader reader;
input2(Process p) {
try {
process = p;
stdout = process.getInputStream();
reader = new BufferedReader(new InputStreamReader(stdout));
} catch (Exception e) {
System.out.println("ERROR input2(): " + e);
}
}
#Override
public void run() {
System.out.println("Started INPUT THREAD");
try {
while (true) {
System.out.println(Thread.currentThread().getName() + " is executing");
if (reader.readLine() != null) {
System.out.println("Stdout: " + reader.readLine());
}
Thread.sleep(1000);
System.out.println(Thread.currentThread().getName() + " stopped executing");
}
} catch (Exception e) {
System.out.println("ERROR input2_run(): " + e);
}
System.out.println("Ending INPUT THREAD");
}
}
MAIN :
public class My_Java {
public static void main(String args[]) {
File file = new File("C:\\Location\\");
try {
Process process = Runtime.getRuntime().exec("C:\\Python27\\python.exe chat_from_file.py", null, file);
input2 input = new input2(process);
output2 output = new output2(process);
input.setName("INPUT THREAD");
output.setName("OUTPUT THREAD");
input.start();
output.start();
} catch (Exception e) {
System.out.println("ERROR main(): " + e);
}
}
}
This doesn't seem to give any response at all.
It starts both threads, says INPUT THREAD is executing but nothing after that.
Where am I going wrong?
First of all, after calling if (reader.readLine() != null) { in your input class, you effectively have read the line and the next call will return null.
Use ready to check for non-blocking read possibility. Don't read upfront.
However, I'm pretty sure that you process exists abnormally, with something like python: can't open file 'chat_from_file.py': [Errno 2] No such file or directory or, throws a stacktrace and exits.
Use getErrorStream to check what the process is outputting if an error exists. This will put you on the correct path to solve your issue.
Also, just in case, make sure there's actually something to be read. Make sure your Python application is outputting enough data for buffers to be flushed (or flushing its writes).
And don't forget to join and exit cleanly and correctly. Good luck.
I am working on an application that retrieves files from different URL's.
There is a TreeSet that contains the target to download. This is processed in a loop with each item being called with an ExecutorService. Here's some code:
private void retrieveDataFiles() {
if (this.urlsToRetrieve.size() > 0) {
System.out.println("Target URLs to retrieve: " + this.urlsToRetrieve.size());
ExecutorService executorProcessUrls = Executors.newFixedThreadPool(this.urlsToRetrieve.size());//could use fixed pool based on size of urls to retrieve
for (Entry target : this.urlsToRetrieve.entrySet()) {
final String fileName = (String) target.getKey();
final String url = (String) target.getValue();
String localFile = localDirectory + File.separator + fileName;
System.out.println(localFile);
executorProcessUrls.submit(new WikiDumpRetriever(url, localFile));
dumpFiles.add(localFile);
//TODO: figure out why only 2 files download
}
executorProcessUrls.shutdown();
try {
executorProcessUrls.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException ex) {
System.out.println("retrieveDataFiles InterruptedException: " + ex.getMessage());
}
} else {
System.out.println("No target URL's were retrieved");
}
}
Then the WikiDumpRetriever:
private static class WikiDumpRetriever implements Runnable {
private String wikiUrl;
private String downloadTo;
public WikiDumpRetriever(String targetUrl, String localDirectory) {
this.downloadTo = localDirectory;
this.wikiUrl = targetUrl;
}
public void downloadFile() throws FileNotFoundException, IOException, URISyntaxException {
HTTPCommunicationGet httpGet = new HTTPCommunicationGet(wikiUrl, "");
httpGet.downloadFiles(downloadTo);
}
#Override
public void run() {
try {
downloadFile();
} catch (FileNotFoundException ex) {
System.out.println("WDR: FileNotFound " + ex.getMessage());
} catch (IOException ex) {
System.out.println("WDR: IOException " + ex.getMessage());
} catch (URISyntaxException ex) {
System.out.println("WDR: URISyntaxException " + ex.getMessage());
}
}
}
As you can see this is an inner class. The TreeSet contains:
Key : Value
enwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
elwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/elwiki-latest-pages-articles.xml.bz2
zhwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/zhwiki-latest-pages-articles.xml.bz2
hewiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/hewiki-latest-pages-articles.xml.bz2
The problem is that this process downloads 2 of the four files. I know that all four are available and I know that they can be downloaded. However, only 2 of them process at any time.
Can anyone shed any light on this for me please - what am I missing or what am I getting wrong?
Thanks
nathj07
Thanks to ppeterka - it was a limit from the source. So, to overcome this I set the fixed thread pool size to 2. This means that only 2 files are downloaded simultaneously.
The answer then was to find the vendor imposed limit and set the thread pool:
ExecutorService executorProcessUrls = Executors.newFixedThreadPool(2);
I wanted to accept an answer but couldn't seem to do it with the comments. Sorry if this was the wrong way to do it.
Thanks for all the pointers - the 'group think' really helped solve this for me.