WHAT?
I am trying to build a tool that will reads a text file and publishes the text, after doing some string transformation.
HOW?
The tool reads the file line by line and populates a LinkedBlockingQueue. At the same time I initiate multiple threads that will then take a message each from the LBQ, do some processing and publish them.
Main
private static LinkedBlockingQueue<String> lbQueue = new LinkedBlockingQueue<>();
private static Boolean keepPublisherActive = Boolean.TRUE;
public static void main(String[] args) {
try {
tool.initMessagePublish();
tool.searchUsingScanner();
} catch (Exception ex) {
logger.error("Exception in Tool Main() " + ex.toString());
throw ex;
}
}
File Reader
private void searchUsingScanner() {
Scanner scanner = null;
try {
scanner = new Scanner(new File(LOG_FILE_PATH));
while (scanner.hasNextLine()) {
String line = scanner.nextLine().trim();
if (StringUtils.isNotBlank(line)) {
lbQueue.offer(line);
}
}
} catch (Exception e) {
logger.error("Error while processing file: " + e.toString());
} finally {
try {
if (scanner != null) {
scanner.close();
}
// end thread execution
keepPublisherActive = false;
} catch (Exception e) {
logger.error("Exception while closing file scanner " + e.toString());
throw e;
}
}
}
Multi-threaded Publisher
private void initMessagePublish() throws InterruptedException {
ExecutorService service = Executors.newFixedThreadPool(6);
try {
while (keepPublisherActive || lbQueue.getSize() > 0) {
service.execute(messagePublisher); // messagePublisher implements Runnable
}
} catch (Exception ex) {
logger.error("Multi threaded message publish failed " + ex.toString());
throw ex;
} finally {
service.shutdown();
}
}
THE PROBLEM
The intention behind calling initMessagePublish() fist is that the publisher need not wait for all lines to be read from the file before starting to publish. It should start publishing as soon as something becomes available in the LBQ.
But with the current implementation, the control never comes out of the initMessagePublish and start searchUsingScanner. How do I solve this? Basically, the two methods should execute parallely.
Just start messagePublisher in a new Thread (Line no #5 in Main class):
new Thread(()->tool.initMessagePublish()).start();
It should solve your problem.
Related
I am writing an application that searches for Java files in a given directory and its subdirectories and writes all the strings from those files in reverse order to a new folder. Each directory and file is handled in a separate thread.
At the moment my program works correctly, but I want to change its behavior.
Right now, the program overwrites the files correctly and outputs the number of overwritten files to the console at the end. I want my program to just overwrite the files and display the line "All files overwritten" at the end. But I don't quite understand how I can change my code and replace Future (I think that's my problem). Here is part of the code from the Main class:
ExecutorService pool = Executors.newCachedThreadPool();
ReverseWritter reverseWritter = new ReverseWritter(dirToSearch, dirToStorePath + "//" + dirToStoreName, pool);
Future<Integer> res = pool.submit(reverseWritter);
try {
System.out.println(res.get() + " files reversed");
} catch (ExecutionException | InterruptedException e) {
e.printStackTrace();
}
pool.shutdown();
Here's the method that overwrites the file:
public boolean reverseWrite(File file) {
if (file.isFile() && file.toString().endsWith(".java")) {
String whereTo = dirToStorePathName + "\\" + file.getName().substring(0, file.getName().indexOf(".java")) + "Reversed" + ".java";
try ( Scanner myReader = new Scanner(file); FileWriter myWriter = new FileWriter(whereTo);) {
while (myReader.hasNextLine()) {
String data = myReader.nextLine();
myWriter.write(new StringBuffer(data).reverse().toString());
myWriter.write(System.getProperty("line.separator"));
}
} catch (FileNotFoundException e) {
System.out.println("An error occurred.");
e.printStackTrace();
return false;
} catch (IOException e) {
System.out.println("An error occurred.");
e.printStackTrace();
return false;
}
}
return true;
}
And this is the call method (my class implements the Callable interface):
#Override
public Integer call() {
int count = 0;
try {
File[] files = dirToSearch.listFiles();
ArrayList<Future<Integer>> result = new ArrayList<>();
for (File f : files) {
if (f.isDirectory()) {
ReverseWritter reverseWritter = new ReverseWritter(f, dirToStorePathName, pool);
Future<Integer> rez = pool.submit(reverseWritter);
result.add(rez);
} else if (reverseWrite(f)) {
count++;
}
for (Future<Integer> rez : result) {
count += rez.get();
}
}
} catch (ExecutionException | InterruptedException e) {
e.printStackTrace();
}
return count;
}
You just need to change the class to implement Callable<Void> and remove the operations which do the counting. Change the return type of call from Integer to Void.
public class ReverseWriterCallable implements Callable<Void> {
#Override
public Void call() throws Exception {
//do stuff
//don't do the counting operations
//when return type is Void you can only return null
return null;
}
}
Or implement Runnable and submit it to the executor service.
public class ReverseWriterRunnable implements Runnable {
#Override
public void run() {
//do stuff
//don't do the counting operations
}
}
Then just don't care about the result of the Future:
try {
res.get();
System.out.println("All files reversed");
} catch (ExecutionException | InterruptedException e) {
e.printStackTrace();
}
pool.shutdown();
Edit: I know what Thread.interrupt() does.
while (!Thread.currentThread().isInterrupted()) does not exit when I interrupt the thread.
I also tried to catch an exception from url.openStream(); when the
thread is interrupted (desperation, maybe it was a blocking method,
which is not) and exit the loop, without any success.
The application creates a Thread that continuously reads a URL. After 3 seconds that Thread gets interrupted but unfortunately continues to execute.
How to stop the thread from executing?
Code (Main.java, MyRunnable.java):
public class Main {
public static void main(String[] args) {
MyRunnable runnable = new MyRunnable("http://ninjaflex.com/");
Thread thread = new Thread(runnable);
thread.start();
sleep(3000);
thread.interrupt();
System.out.println("Thread.interrupt() invoked.");
}
private static void sleep(long timeMilli) {
try {
Thread.sleep(timeMilli);
} catch (Exception e) {
}
}
}
public class MyRunnable implements Runnable {
private String website;
MyRunnable(String website) {
this.website = website;
}
#Override
public void run() {
URL url = createUrl();
if (url != null) {
while (!Thread.currentThread().isInterrupted()) {
sleepOneSec();
readFromUrl(url);
System.out.println("Read from " + website);
}
System.out.println("Script: Interrupted, exiting.");
}
}
private URL createUrl() {
URL url = null;
try {
url = new URL(website);
} catch (MalformedURLException e) {
System.out.println("Wrong URL?");
}
return url;
}
private void sleepOneSec() {
try {
Thread.sleep(1000);
} catch (Exception e) {
System.out.println("Error sleeping");
}
}
private void readFromUrl(URL url) {
InputStream in = null;
try {
in = url.openStream();
} catch (Exception e) {
System.out.println("Exception while url.openStream().");
e.printStackTrace();
} finally {
closeInputStream(in);
}
}
private void closeInputStream(InputStream in) {
try {
in.close();
} catch (IOException e) {
System.out.println("Error while closing the input stream.");
}
}
}
Basically, your MyRunnable thread is interrupted during sleep. InterreuptedException is thrown but catched. By the way, it's a bad habit to catch Exception and you should not do that.
From the javadoc: "The interrupted status of the current thread is cleared when this exception is thrown".
Therefore, your while loop will never see the flag.
Replace the call to the sleepOneSec method with a simple Thread.sleep call. Catch InterruptedException outside your while loop. This will cause the loop to exit naturally:
try {
while (true) {
Thread.sleep(1000);
readFromUrl(url);
System.out.println("Read from " + website);
}
} catch (InterruptedException e) {
System.out.println("Script: Interrupted, exiting.");
}
I removed the MyRunnable.sleepOneSec and your code started to work.
I am writing a java program that will need to run a python script.
The script will print output which will java need to read to know the progress of the script.
To be able to pause the script while running I want it to ask for input once in a while, only when java give it input the script will keep going.
Here is my Java method:
private static void sevenTry(String[] strCommands) throws IOException {
Object oLock1 = new Object();
Object oLock2 = new Object();
ProcessBuilder pBuilder = new ProcessBuilder(strCommands);
pBuilder.redirectErrorStream(true);
Process proc = pBuilder.start();
Thread tReader = new Thread() {
#Override
public void run() {
System.out.println("~~tReader starting~~");
BufferedReader reader = new BufferedReader(new InputStreamReader(proc.getInputStream()));
synchronized (oLock1) {
try {
String line = reader.readLine();
while (line != null && !line.trim().equals("--EOF--")) {
System.out.println("Stdout: " + line);
if (line.trim().equals("--INPUT--")) {
synchronized (oLock2) {
oLock2.notify();
}
oLock1.wait();
}
line = reader.readLine();
}
} catch (IOException e) {
System.out.println("tReader: " + e.getMessage());
} catch (InterruptedException e) {
System.out.println("tReader: " + e.getMessage());
} catch (Exception e) {
System.out.println("tReader: " + e.getMessage());
}
}
System.out.println("~~tReader end~~");
synchronized (oLock2) {
oLock2.notify();
}
}
};
Thread tWriter = new Thread() {
#Override
public void run() {
System.out.println("~~tWriter starting~~");
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(proc.getOutputStream()));
String line, input;
Scanner scan = new Scanner(System.in);
synchronized (oLock2) {
try {
oLock2.wait();
} catch (InterruptedException e1) {
System.out.println("tWriter: " + e1.getMessage());
}
}
while (tReader.isAlive()) {
synchronized (oLock1) {
System.out.println("Java: insert input");
scan.hasNext();
input = scan.nextLine();
try {
writer.write(input + "\n");
writer.flush();
} catch (IOException e) {
System.out.println("tWriter: " + e.getMessage());
}
oLock1.notify();
}
try {
Thread.sleep(2000);
} catch (InterruptedException e1) {
System.out.println("tWriter: " + e1.getMessage());
}
}
System.out.println("~~tWriter end~~");
}
};
tReader.start();
tWriter.start();
System.out.println("~~everything submitted~~");
try {
tReader.join();
tWriter.join();
System.out.println("~~finish~~");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
This is my python script:
# coding=utf-8
import sys
print '1'
print '--INPUT--'
inum = sys.stdin.readline()
print '2'
print '--EOF--'
I tried running my code
sevenTry("python", "C:\\Testing.py");
but on java side it get stuck inside tReader at line:
String line = reader.readLine();
The program does work if i take out the input line from the python file
inum = sys.stdin.readline()
Using
inum = raw_input()
still bring up the same problem (im using python 2.7)
The most confusing part here that i even tried to test this with a java file (instead of python)
sevenTry("java", "-classpath", "C:\\class", "CheckCMD");
and it worked even with the input lines
import java.util.Scanner;
public class CheckCMD {
public static void main(String[] args) {
Scanner in = new Scanner(System.in);
String line;
System.out.println("1");
System.out.println("--INPUT--");
in.hasNext();
line = in.nextLine();
System.out.println("2");
System.out.println("--EOF--");
}
}
As you may have noticed, this is a problem related to Python.
As described in https://unix.stackexchange.com/questions/182537/write-python-stdout-to-file-immediately,
" when process STDOUT is redirected to something other than a terminal, then the output is buffered into some OS-specific-sized buffer (perhaps 4k or 8k in many cases)."
So, you need to call sys.stdout.flush() after each invoke to print.
Or, as a better option, you can change the default behaviour for the process, using the -u param, to get unbuffered output.
I am working on a java application in which I am facing a problem. When I send a file to a server and an exception is thrown, the file is not sent. How can I retry sending the file?
public void uploadtxtFile(String localFileFullName, String fileName, String hostDir)
throws Exception {
File file = new File(localFileFullName);
if (!(file.isDirectory())) {
if (file.exists()) {
FileInputStream input = null;
try {
input = new FileInputStream(new File(localFileFullName));
if (input != null) {
hostDir = hostDir.replaceAll("//", "/");
logger.info("uploading host dir : " + hostDir);
//new
// TestThread testThread=new TestThread(hostDir,input);
// Thread t=new Thread(testThread);
//
// try{
// t.start();
//
// }catch(Exception ex){
// logger.error("UPLOADE start thread create exception new:" + ex);
// }
// // new end
DBConnection.getFTPConnection().enterLocalPassiveMode();
// the below line exeption is come
boolean bool = DBConnection.getFTPConnection().storeFile(hostDir, input);
//input.close();//new comment
if (bool) {
logger.info("Success uploading file on host dir :"+hostDir);
} else {
logger.error("file not uploaded.");
}
} else {
logger.error("uploading file input null.");
}
}catch(CopyStreamException cs)
{ logger.error("Copy StreamExeption is come "+cs);
} catch(Exception ex)
{
logger.error("Error in connection ="+ex);//this is catch where I handle the exeption
}finally {
// boolean disconnect= DBConnection.disConnect();
input.close();
}
} else {
logger.info("uploading file is not exists.");
}
}
}
This is the code and I want to restart the file uploading but I don't have any idea. I tried it using the thread but the exception is thrown again. I also tried to use a while loop, but it loops infinitely and also shows the exception as well as another exception.
Below is the thread code that I use:
public void run() {
System.out.println("Enter Thread TestThread");
DBConnection.getFTPConnection().enterLocalPassiveMode();
// System.out.println("Error in DBConnection ");
//here server timeout error is get
boolean bool1=false;
boolean bool=true;
try {
bool = DBConnection.getFTPConnection().storeFile(hostDir1, input1);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}finally {
//disconnect();
try {
input1.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
if (bool) {
System.out.println("File is Uploded");
} else {
while(bool!=true){
try {
DBConnection.getFTPConnection().enterLocalPassiveMode();
bool1=DBConnection.getFTPConnection().storeFile(hostDir1, input1);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}finally {
//disconnect();
try {
input1.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
System.out.println("file not uploaded."+bool1);
bool=bool1;
}
}
}
}
}
Can any one have a solution to how to upload the file to the server?
The exception is shown below:
Software caused connection abort: recv failed
Software caused connection abort: socket write error
org.apache.commons.net.io.CopyStreamException: IOException caught while copying.
Add a static class as below in a class from where you are calling the method which need to be retried:
static class RetryOnExceptionStrategy {
public static final int DEFAULT_RETRIES = 3;
public static final long DEFAULT_WAIT_TIME_IN_MILLI = 2000;
private int numberOfRetries;
private int numberOfTriesLeft;
private long timeToWait;
public RetryOnExceptionStrategy() {
this(DEFAULT_RETRIES, DEFAULT_WAIT_TIME_IN_MILLI);
}
public RetryOnExceptionStrategy(int numberOfRetries,
long timeToWait) {
this.numberOfRetries = numberOfRetries;
numberOfTriesLeft = numberOfRetries;
this.timeToWait = timeToWait;
}
/**
* #return true if there are tries left
*/
public boolean shouldRetry() {
return numberOfTriesLeft > 0;
}
public void errorOccured() throws Exception {
numberOfTriesLeft--;
if (!shouldRetry()) {
throw new Exception("Retry Failed: Total " + numberOfRetries
+ " attempts made at interval " + getTimeToWait()
+ "ms");
}
waitUntilNextTry();
}
public long getTimeToWait() {
return timeToWait;
}
private void waitUntilNextTry() {
try {
Thread.sleep(getTimeToWait());
} catch (InterruptedException ignored) {
}
}
}
Now wrap your method call as below in a while loop :
RetryOnExceptionStrategy errorStrategy=new RetryOnExceptionStrategy();
while(errorStrategy.shouldRetry()){
try{
//Method Call
}
catch(Exception excep){
errorStrategy.errorOccured();
}
}
Basically you are just wrapping you method call in while loop which will
keep returnig true till your retry count is reached to zero say you started with 3.
Every time an exception occurred, the exception is caught and a method is called
which will decrement your retryCount and method call is again executed with some delay.
A general way of working with such application is:
Create a class, say, UploadWorker which extends Callable as the wrapper. Make the wrapper return any error and detail information you need when it fails.
Create a ExecutorService (basically a thread pool) for this wrapper to run in threads.
Submit your UploadWorker instance and then you get a Future. Call get() on the future to wait in blocking way or simply wait some time for the result.
In case the get() returns you the error message, submit your worker again to the thread pool.
I have a Python app and a Java app running simultaneously.
I want Java to start the Python process and then communicate using the normal STDIN/STDOUT streams with Python.
I have started the process correctly and have two threads to handle the two I/O streams.
OUTPUT THREAD:
class output2 extends Thread {
Process process;
OutputStream stdin;
BufferedWriter writer;
Scanner in = new Scanner(System.in);
output2(Process p) {
try {
process = p;
stdin = process.getOutputStream();
writer = new BufferedWriter(new OutputStreamWriter(stdin));
} catch (Exception e) {
System.out.println("ERROR output2(): " + e);
}
}
#Override
public void run() {
System.out.println("Starting OUTPUT THREAD");
try {
while (true) {
String input = in.nextLine();
writer.write(input);
writer.flush();
}
} catch (Exception e) {
System.out.println("ERROR output2_run(): " + e);
}
System.out.println("Ending OUTPUT THREAD");
}
}
INPUT THREAD :
class input2 extends Thread {
Process process;
InputStream stdout;
BufferedReader reader;
input2(Process p) {
try {
process = p;
stdout = process.getInputStream();
reader = new BufferedReader(new InputStreamReader(stdout));
} catch (Exception e) {
System.out.println("ERROR input2(): " + e);
}
}
#Override
public void run() {
System.out.println("Started INPUT THREAD");
try {
while (true) {
System.out.println(Thread.currentThread().getName() + " is executing");
if (reader.readLine() != null) {
System.out.println("Stdout: " + reader.readLine());
}
Thread.sleep(1000);
System.out.println(Thread.currentThread().getName() + " stopped executing");
}
} catch (Exception e) {
System.out.println("ERROR input2_run(): " + e);
}
System.out.println("Ending INPUT THREAD");
}
}
MAIN :
public class My_Java {
public static void main(String args[]) {
File file = new File("C:\\Location\\");
try {
Process process = Runtime.getRuntime().exec("C:\\Python27\\python.exe chat_from_file.py", null, file);
input2 input = new input2(process);
output2 output = new output2(process);
input.setName("INPUT THREAD");
output.setName("OUTPUT THREAD");
input.start();
output.start();
} catch (Exception e) {
System.out.println("ERROR main(): " + e);
}
}
}
This doesn't seem to give any response at all.
It starts both threads, says INPUT THREAD is executing but nothing after that.
Where am I going wrong?
First of all, after calling if (reader.readLine() != null) { in your input class, you effectively have read the line and the next call will return null.
Use ready to check for non-blocking read possibility. Don't read upfront.
However, I'm pretty sure that you process exists abnormally, with something like python: can't open file 'chat_from_file.py': [Errno 2] No such file or directory or, throws a stacktrace and exits.
Use getErrorStream to check what the process is outputting if an error exists. This will put you on the correct path to solve your issue.
Also, just in case, make sure there's actually something to be read. Make sure your Python application is outputting enough data for buffers to be flushed (or flushing its writes).
And don't forget to join and exit cleanly and correctly. Good luck.