I have a system that, when files of a certain type are found, I download, encode, and upload them in a separate thread.
while(true) {
for(SftpClient c : clients) {
try {
filenames = c.list("*.wav", "_rdy_");
} catch (SftpException e) {
e.printStackTrace();
}
if(filenames.size() > 0) {
//AudioThread run() method handles the download, encode, and upload
AudioThread at = new AudioThread(filenames);
at.setNode(c.getNode());
Thread t = new Thread(at);
t.start();
}
}
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
The run method from AudioThread
public void run() {
System.out.println("Running...");
this.buildAsteriskMapping();
this.connectToSFTP();
ac = new AudioConvert();
this.connectToS3();
String downloadDir = "_rough/" + getNode() + "/" + Time.getYYYYMMDDDate() + "/";
String encodeDir = "_completed" + getNode() + "/" + Time.getYYYYMMDDDate() + "/";
String uploadDir = getNode() + "/" + Time.getYYYYMMDDDate() + "/";
System.out.println("Downloading...");
try {
sftp.get(filenames, downloadDir);
} catch (SftpException e) {
//download failed
System.out.println("DL Failed...");
e.printStackTrace();
}
System.out.println("Encoding...");
try {
ac.encodeWavToMP3(filenames, downloadDir, encodeDir);
} catch (IllegalArgumentException | EncoderException e) {
System.out.println("En Failed...");
e.printStackTrace();
}
System.out.println("Uploading...");
try {
s3.upload(filenames, encodeDir, uploadDir);
} catch (AmazonClientException e) {
System.out.println("Up Failed...");
e.printStackTrace();
}
}
The download method:
public void get(ArrayList<String> src, String dest) throws SftpException {
for(String file : src) {
System.out.println(dest + file);
channel.get(file, dest + file);
}
}
The encode method:
public void encodeWavToMP3(ArrayList<String> filenames, String downloadDir, String encodeDir) throws IllegalArgumentException, EncoderException {
for(String f : filenames) {
File wav = new File(downloadDir + f);
File mp3 = new File(encodeDir + wav.getName().replace(".wav", ".mp3"));
encoder.encode(wav, mp3, attrs);
}
}
The upload method:
public void upload(ArrayList<String> filenames, String encodeDir, String uploadDir) throws AmazonClientException, AmazonServiceException {
for(String f : filenames) {
s3.putObject(new PutObjectRequest(bucketName, uploadDir, new File(encodeDir + f)));
}
}
The issue is I keep downloading the same files (or about the same files) for every thread. I want to add a variable for each client that holds the files that are being downloaded but I don't know how to remove the lists/filenames from this variable. What would be a solution? My boss would also like to only allow x amount of threads to run.
It's kind of hard to see the problem, as the code that actually does the download is missing :P
However, I would use some kind of ExecutorService instead.
Basically, I would add each download request to the service (wrapped in a "DownloadTask" with a reference to the file to be downloaded and any other relevant information it might need to get the file) and let the service take care of the rest.
The download tasks could be coded to take into account existing files as you see fit.
Depending on your requirements, this could be a single thread or multi-threaded service. It could also allow you to place upload quests in it as well.
Check out the Executors trail for more info
The general idea is to use a kind of producer/consumer pattern. You would have (at least) a thread that would look up all the files to be downloaded and for each file, you would add it to the executor service. After the file has been downloaded, I would queue and upload request into the same service.
This way, you avoid all the mess with synchronization and thread management :D
You could use the same idea with the scan tasks, for each client, you could a task to a separate service
There is a problem in your code where you instantiate AudioThread in a while loop.
Note that after you create a thread and do a t.start(), all downloading, encoding and uploading happens asynchronously. Therefore, after you start the thread the loop continuous to do another call to c.list(...) while the first thread you created is still processing the first set of files. Most probably the same set of files is returned in the succeeding c.list() calls since you specified a file pattern in the call and there is no code which marks which files are currently being processed.
My suggestion:
Use Executors.newFixedThreadPool(int nThreads) as mentioned in previous post. And specify the number of threads to the number of processors in your machine. Do this before your while loop.
For each filename you retrieved from ftp s.list(), create a Callable class and call ExecutorService.invokeAll(Collection<Callable<T>> tasks). The code in the Callable you will create is your AudioThread code. Modify AudioThread code to only process one file at at time (if possible), this way you are doing downloads,uploads, encoding in parallel for each file.
Add code which marks which files were already processed. I would suggest adding a code which renames the files you have processed to a different name to avoid getting returned in the next c.list() call.
Call ExecutorService.shutdown(...) after your while loop block
Related
I'm a small java developer currently working on a discord bot that I made in Java. one of the features of my bot is to simply have a leveling system whenever anyone sends a message (and other conditions but this is irrelevant for the problem I'm encountering).
Whenever someone sends a message an event is fired and a thread is created to compute how much exp the user should gain. and eventually, the function to edit the storage file is called.
which works fine when called sparsely. but if two threads try to write on the file at once, the file usually gets deleted or truncated. either of these two cases being undesired behavior
I then tried to make a queuing system that worked for over 24h but still failed once so it is more stable in a way. I only know the basics of how threads work so I may've skipped over an important thing that causes the problem
the function looks like this
Thread editingThread = null;
public boolean editThreadStarted = false;
HashMap<String, String> queue = new HashMap<>();
public final boolean editParameter(String key, String value) {
queue.put(key, value);
if(!editThreadStarted) {
editingThread = new Thread(new Runnable() {
#Override
public void run() {
while(queue.keySet().size() > 0) {
String key = (String) queue.keySet().toArray()[0];
String value = queue.get(key);
File inputFile = getFile();
File tempFile = new File(getFile().getName() + ".temp");
try {
tempFile.createNewFile();
} catch (IOException e) {
DemiConsole.error("Failed to create temp file");
handleTrace(e);
continue;
}
//System.out.println("tempFile.isFile = " + tempFile.isFile());
try (BufferedReader reader = new BufferedReader(new FileReader(inputFile)); BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile))){
String currentLine;
while((currentLine = reader.readLine()) != null) {
String trimmedLine = currentLine.trim();
if(trimmedLine.startsWith(key)) {
writer.write(key + ":" + value + System.getProperty("line.separator"));
continue;
}
writer.write(currentLine + System.getProperty("line.separator"));
}
writer.close();
reader.close();
inputFile.delete();
tempFile.renameTo(inputFile);
} catch (IOException e) {
DemiConsole.error("Caught an IO exception while attempting to edit parameter ("+key+") in file ("+getFile().getName()+"), returning false");
handleTrace(e);
continue;
}
queue.remove(key);
}
editThreadStarted = false;
}
});
editThreadStarted = true;
editingThread.start();
}
return true;
}
getFile() returns the file the function is meant to write to
the file format is
memberid1:expamount
memberid2:expamount
memberid3:expamount
memberid4:expamount
the way the editing works is by creating a temporary file to which i will write all of the original file's data line by line, checking if the memberid matches with what i want to edit, if it does, then instead of writing the original file's line, i will write the new edited line with the new expamount instead, before continuing on with the rest of the lines. Once that is done, the original file is deleted and the temporary file is renamed to the original file, replacing it.
This function will always be called asynchronously so making the whole thing synchronous is not an option.
Thanks in advance
Edit(1) :
I've been suggested to use semaphores and after digging a little into it (i never heard of semaphores before) it seems to be a really good option and would remove the need for a queue, simply aquire in the beginning and release at the end, nothing more required!
I ended up using semaphores as per user207421's suggestions and it seems to work perfectly
I simply put delays between each line write to artificially make the task longer and make it easier to have multiple threads trying to write at once, and they all wait for their turns!
Thanks
**UPDATE---The issue was not the file writer not closing but incorrect termination of the Java application. I have updated the question.
I have the following classes launching a JAVAFX web view and exposing some java objects to the web view's html.
public class FileSystemBridge {
private void writeToFile(String[] fileContents){
if (content!=null){
String fileName ="pathToFile";
BufferedWriter fileWriter;
for (int i =0; i<fileContents.length(); i++ ){
String fileContent fileContents[i]);
try {
fileName = fileName+Integer.toString(i)+".txt";
fileName = fileName.replaceAll("\\s","");
System.out.println(fileName);
File f= new File(filesDir+"/"+fileName);
f.createNewFile();
fileWriter = new BufferedWriter(new FileWriter(f));
fileWriter.write("");
fileWriter.write(fileContent);
fileWriter.flush();
fileWriter.close();
} catch (IOException e) {
System.out.println("in the exception!");
e.printStackTrace();
}
}
}
else {
System.out.println("no content");
}
System.out.println("done writing, exit app now");
}
public void exit(){
System.out.println("EXITING!");
Platform.exit();
System.exit(0);
}
}
The above class also has additional member classes who serve as POJOS to expose the structure of files being read/written to the "front-end" html.
I pass an instance of FileSystemBridge to the web view by overriding the default Browser class constructor and adding the following code.
webEngine.getLoadWorker().stateProperty().addListener(
(ObservableValue<? extends State> ov, State oldState,
State newState) -> {
if (newState == State.SUCCEEDED) {
JSObject context= (JSObject) webEngine.executeScript("window");
context.setMember("fsBridge", new FileSystemBridge());
webEngine.executeScript("init('desktop')");//the hook into our app essentially
}
});
The webEngine.executeScrit("init) essentially performs some initialization on our front end. Then on the javascript executing on the webview on user interaction we invoke our FileSystemBridge write method with a callback to invoke the FileSystemBridge's exit method, which is essentially a call to Platform.exit().
On user click
App.handleWrite(contentToBeWritten, function(success){
if (success){
console.log("inside success!");
App.handleExit();
}
});
then our handleWrite javascript function
handleWrite: function(content, callback){
fsBridge.callWrite(content);
retVal = true;//more logic to this but simplified for demo
callBack(retVal);
}
Now in the FileSystemBridge.exit() method I have added System.exit(0), which succesfully terminates my java instance which was the original problem.However I would like to know if this is the correct approach to handling the exiting of a java app which uses a JAVAFX webview. Are there unforeseen consequences to using System.exit(0) in this manner?
Java applications that create a UI don't terminate until certain conditions are met. For a JavaFX application, make sure you are closing all Stage instances.
If you can't manpulate it into a finally block per the previous answer, how about a try-with-resources? Like so (you'll need to remove the previous declaration of fileWriter):
'
for (int i =0; i<fileContents.length(); i++ ){
String fileContent fileContents[i]);
try {
fileName = fileName+Integer.toString(i)+".txt";
fileName = fileName.replaceAll("\\s","");
System.out.println(fileName);
File f= new File(filesDir+"/"+fileName);
f.createNewFile();
try(BufferedWriter fileWriter = new BufferedWriter(new FileWriter(f));)
{
fileWriter.write("");
fileWriter.write(fileContent);
}
} catch (IOException e) {
System.out.println("in the exception!");
e.printStackTrace();
}
}
'
Try putting the close in a finally block like below and see if it makes any difference.
try {
} catch (Exception e) {
} finally {
fileWriter.close();
}
I did see some pitfalls, but no real problem, especially as you handle the exceptions. The length() is an indication that you simplified the code here.
Try it in the newer style Path/Files as it is more "atomic" - even more simple.
String fileName ="pathToFile";
fileName = fileName.replaceAll("[\\s/\\\\]", "");
for (int i = 0; i < fileContents.length; i++) {
Path path = Paths.get(fileName + i + ".txt";
byte[] bytes = ("\ufeff" + fileContents).getBytes("UTF-8");
//Or: byte[] bytes = fileContents.getBytes();
try {
Files.write(path, bytes);
} catch (IOException e) {
System.out.println("Could not write " + path.getFileName());
}
}
This version writes in UTF-8 with a starting BOM char. The BOM is ugly, but allows Windows to recognize UTF-8. That allows special characters.
Files.write can have extra parameters list StandardOpenOptions.REPLACE_EXISTING.
I was successful in reading a file while using multi-process environment using file locking
and in case of multithreaded(singleprocess) i used a queue filled it with file names, opened a thread separately, read from it and then waited till the entire reading was over, after which i used to rename them. In this way i used to read files in multithreaded(in a batch).
Now, i want to read the files in a directory using both multiprocess and multithreads. I tried merging my two approaches but that didn't fare well. log showed a lot of files were showing FileNotFound exception(because their names were changed), some were never read (because thread died), sometimes locks were not released.
///////////////////////////////////////////////////////////////////////
//file filter inner class
class myfilter implements FileFilter{
#Override
public boolean accept(File pathname) {
// TODO Auto-generated method stub
Pattern pat = Pattern.compile("email[0-9]+$");
Matcher mat = pat.matcher(pathname.toString());
if(mat.find()) {
return true;
}
return false;
}
}
/////////////////////////////////////////////////////////////////////////
myfilter filter = new myfilter();
File alreadyread[] = new File[5];
Thread t[] = new Thread[5];
fileread filer[] = new fileread[5];
File file[] = directory.listFiles(filter);
FileChannel filechannel[] = new FileChannel[5];
FileLock lock[] = new FileLock[5];
tuple_json = new ArrayList();
//System.out.println("ayush");
while(true) {
//declare a queue
ConcurrentLinkedQueue filequeue = new ConcurrentLinkedQueue();
//addfilenames to queue and their renamed file names
try{
if(file.length!=0) {
//System.out.println(file.length);
for(int i=0;i<5 && i<file.length;i++) {
System.out.println("acquiring lock on file " + file[i].toString());
try{
filechannel[i] = new RandomAccessFile(file[i], "rw").getChannel();
lock[i] = filechannel[i].tryLock();
}
catch(Exception e) {
file[i] = null;
lock[i] = null;
System.out.println("cannot acquire lock");
}
if(lock[i]!=null){
System.out.println("lock acquired on file " + file[i].toString());
filequeue.add(file[i]);
alreadyread[i] = new File(file[i].toString() + "read");
System.out.println(file[i].toString() + "-----" + times);
}
else{
System.out.println("else condition of acquiring lock");
file[i] = null;
}
System.out.println("-----------------------------------");
}
//starting the thread to read the files
for(int i=0;i<5 && i<file.length && lock[i]!=null && file[i]!=null;i++){
filer[i] = new fileread(filequeue.toArray()[i].toString());
t[i] = new Thread(filer[i]);
System.out.println("starting a thread to read file" + file[i].toString());
t[i].start();
}
//read the text
for(int i=0;i<5 && i<file.length && lock[i]!=null && file[i]!=null;i++) {
try {
System.out.println("waiting to read " + file[i].toString() + " to be read completely");
t[i].join();
System.out.println(file[i] + " was read completetly");
//System.out.println(filer[i].getText());
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
//file has been read Now rename the file
for(int i=0;i<5 && i<file.length && lock[i]!=null && file[i]!=null;i++){
if(lock[i]!=null){
System.out.println("renaming file " + file[i].toString());
file[i].renameTo(alreadyread[i]);
System.out.println("releasing lock on file " + file[i].toString());
lock[i].release();
}
}
//rest of the processing
/////////////////////////////////////////////////////////////////////////////////////////////////////
Fileread class
class fileread implements Runnable{
//String loc = "/home/ayusun/workspace/Eclipse/fileread/bin";
String fileloc;
BufferedReader br;
String text = "";
public fileread(String filename) {
this.fileloc = filename;
}
#Override
public void run() {
try {
br = new BufferedReader(new FileReader(fileloc));
System.out.println("started reading file" + fileloc);
String currline;
while((( currline = br.readLine())!=null)){
if(text == "")
text += currline;
else
text += "\n" + currline;
}
System.out.println("Read" + fileloc + " completely");
br.close();
} catch ( IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public String getText() {
return text;
}
}
I would like to know, if there is nay other approach that i can adopt.
If you want to create exclusive access to a file, you cannot use file locking, as on most OSes file locking is advisory, not mandatory.
I'd suggest creating a common lock directory for all your processes; in this lock directory, you would create a directory per file you want to lock, right before you open a file.
The big advantage is that directory creation, unlike file creation, is atomic; as such, you can use Files.createDirectory() (or File's .mkdir() if you still use Java6 but then don't forget to check the return code) to grab a lock on the files you read. If this fails, you know someone else is using the file.
Of course, when you're done with a file, don't forget to remove the lock directory matching this file... (in a finally block)
(note: with Java 7 you can use Files.newBufferedReader(); there is even Files.readAllLines())
If you need to process a large number of files using multiple threads, you should probably first distribute the specific files to each thread before it starts.
For example, if you only want to process files whose names start with email and are followed by some digits, you could create 10 threads. The first thread would look for files with names starting with email0, the second thread could handle email1, etc.
This of course would be efficient only if the numbers are evenly distributed.
Another way could be do have the main thread run through and collect all filenames to deal with. It could then divide the files across the number of available threads, and pass each thread an array of those file names.
There could be other ways of dividing the system load which are relevant to your situation.
I am trying to write a file on a C:\ drive, but I get an exception.
java.io.IOException: Access denied.
Code:
public class Test {
public static void main(String[] args) {
try {
StringBuilder sb = new StringBuilder(File.separator);
sb.append("index.txt");
// sb is "\\index.txt"
File f = new File(sb.toString());
boolean isCreated = f.createNewFile();
System.out.println(isCreated);
} catch (IOException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
Actually, I get it, I don't have permission to write a file there, but I am quite sure it can be done somehow. If I had an applet, I'd just obtain a permission, but here, I don't know how to do it.
The probable solution may be checking if I can write a file there, but to check it I might try to write a file first adn then delete it in order to check if it is possible to write a file there, but I don't find this solution an optimal way.
The easiest way to check is to use File.canWrite().
Having said that, it looks like you're writing into the root of the drive. On Windows that's probably not a good idea, and you may want to consider writing elsewhere - e.g. a temp dir.
I have written a method, that takes a String to a directory, and checks, whether you can write a file out there:
static boolean canWrite(String folderPath) {
File file = new File(folderPath);
String new_file = "HastaLaVistaBaby";
if (file.isDirectory()) {
try {
new File(file + "\\" + new_file).createNewFile();
} catch (IOException e) {
return false;
}
new File(file + "\\" + new_file).delete();
return true;
} else {
return false;
}
}
To improve it, you may check, whether file.isFile() and get a parent directory and call this method.
This line should be:
sb.append("C:\\index.txt");
The extra backslash escapes a backslash.
Whether you hard-code a file name, like I did, or you get a file name from the user, you need the full path and file name.
I have the following code in a java Web Service:
public boolean makeFile(String fileName, String audio)
{
if (makeUserFolder())
{
File file = new File(getUserFolderPath() + fileName + amr);
FileOutputStream fileOutputStream = null;
try
{
file.createNewFile();
fileOutputStream = new FileOutputStream(file);
fileOutputStream.write(Base64.decode(audio));
return true;
}
catch(FileNotFoundException ex)
{
return false;
}
catch(IOException ex)
{
return false;
}
finally{
try {
fileOutputStream.close();
convertFile(fileName);
} catch (IOException ex) {
Logger.getLogger(FileUtils.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
else
return false;
}
public boolean convertFile(String fileName)
{
Process ffmpeg;
String filePath = this.userFolderPath + fileName;
try {
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",filePath + amr,filePath + mp3);
pb.redirectErrorStream();
ffmpeg = pb.start();
} catch (IOException ex) {
return false;
}
return true;
}
It used to work and now it simply won't execute the ffmpeg conversion for some reason. I thought it was a problem with my file but after running the command from terminal no errors are thrown or anything, thought it was maybe permissions issue but all the permissions have been granted in the folder I'm saving the files. I noticed that the input BufferedReader ins being set to null after running the process, any idea what's happening?
First of all, a small nitpick with your code...when you create the FileOutputStream you create it using a string rather than a File, when you have already created the File before, so you might as well recycle that rather than force the FileOutputStream to instantiate the File itself.
Another small nitpick is the fact that when you are writing out the audio file, you should enclose that in a try block and close the output stream in a finally block. If you are allowed to add a new library to your project, you might use Guava which has a method Files.write(byte[],File), which will take care of all the dirty resource management for you.
The only thing that I can see that looks like a definite bug is the fact that you are ignoring the error stream of ffmpeg. If you are blocking waiting for input on the stdout of ffmpeg, then it will not work.
The easiest way to take care of this bug is to use ProcessBuilder instead of Runtime.
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",filePath+amr,filePath+mp3);
pb.redirectErrorStream(); // This will make both stdout and stderr be redirected to process.getInputStream();
ffmpeg = pb.start();
If you start it this way, then your current code will be able to read both input streams fully. It is possible that the stderr was hiding some error that you were not able to see due to not reading it.
If that was not your problem, I would recommend using absolute paths with ffmpeg...in other words:
String lastdot = file.getName().lastIndexOf('.');
File mp3file = new File(file.getParentFile(),file.getName().substring(0,lastdot)+".mp3");
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",file.getAbsolutePath(),mp3file.getAbsolutePath());
// ...
If that doesn't work, I would change ffmpeg to be an absolute path as well (in order to rule out path issues).
Edit: Further suggestions.
I would personally refactor the writing code into its own method, so that you can use it elsewhere necessary. In other other words:
public static boolean write(byte[] content, File to) {
FileOutputStream fos = new FileOutputStream(to);
try {
fos.write(content);
} catch (IOException io) {
// logging code here
return false;
} finally {
closeQuietly(fos);
}
return true;
}
public static void closeQuietly(Closeable toClose) {
if ( toClose == null ) { return; }
try {
toClose.close();
} catch (IOException e) {
// logging code here
}
}
The reason that I made the closeQuietly(Closeable) method is due to the fact that if you do not close it in that way, there is a possibility that an exception will be thrown by the close() method, and that exception will obscure the exception that was thrown originally. If you put these in a utility class (although looking at your code, I assume that the class that it is currently in is named FileUtils), then you will be able to use them throughout your application whenever you need to deal with file output.
This will allow you to rewrite the block as:
File file = new File(getUserFolderPath() + fileName + amr);
file.createNewFile()
write(Base64.decode(audio),file);
convertFile(fileName);
I don't know whether or not you should do this, however if you want to be sure that the ffmpeg process has completed, then you should say ffmpeg.waitFor(); to be sure that it has completed. If you do that, then you should examine ffmpeg.exitValue(); to make sure that it completed successfully.
Another thing that you might want to do is once it has completed, write what it output to a log file so you have a record of what happened, just in case something happens.