I'm a small java developer currently working on a discord bot that I made in Java. one of the features of my bot is to simply have a leveling system whenever anyone sends a message (and other conditions but this is irrelevant for the problem I'm encountering).
Whenever someone sends a message an event is fired and a thread is created to compute how much exp the user should gain. and eventually, the function to edit the storage file is called.
which works fine when called sparsely. but if two threads try to write on the file at once, the file usually gets deleted or truncated. either of these two cases being undesired behavior
I then tried to make a queuing system that worked for over 24h but still failed once so it is more stable in a way. I only know the basics of how threads work so I may've skipped over an important thing that causes the problem
the function looks like this
Thread editingThread = null;
public boolean editThreadStarted = false;
HashMap<String, String> queue = new HashMap<>();
public final boolean editParameter(String key, String value) {
queue.put(key, value);
if(!editThreadStarted) {
editingThread = new Thread(new Runnable() {
#Override
public void run() {
while(queue.keySet().size() > 0) {
String key = (String) queue.keySet().toArray()[0];
String value = queue.get(key);
File inputFile = getFile();
File tempFile = new File(getFile().getName() + ".temp");
try {
tempFile.createNewFile();
} catch (IOException e) {
DemiConsole.error("Failed to create temp file");
handleTrace(e);
continue;
}
//System.out.println("tempFile.isFile = " + tempFile.isFile());
try (BufferedReader reader = new BufferedReader(new FileReader(inputFile)); BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile))){
String currentLine;
while((currentLine = reader.readLine()) != null) {
String trimmedLine = currentLine.trim();
if(trimmedLine.startsWith(key)) {
writer.write(key + ":" + value + System.getProperty("line.separator"));
continue;
}
writer.write(currentLine + System.getProperty("line.separator"));
}
writer.close();
reader.close();
inputFile.delete();
tempFile.renameTo(inputFile);
} catch (IOException e) {
DemiConsole.error("Caught an IO exception while attempting to edit parameter ("+key+") in file ("+getFile().getName()+"), returning false");
handleTrace(e);
continue;
}
queue.remove(key);
}
editThreadStarted = false;
}
});
editThreadStarted = true;
editingThread.start();
}
return true;
}
getFile() returns the file the function is meant to write to
the file format is
memberid1:expamount
memberid2:expamount
memberid3:expamount
memberid4:expamount
the way the editing works is by creating a temporary file to which i will write all of the original file's data line by line, checking if the memberid matches with what i want to edit, if it does, then instead of writing the original file's line, i will write the new edited line with the new expamount instead, before continuing on with the rest of the lines. Once that is done, the original file is deleted and the temporary file is renamed to the original file, replacing it.
This function will always be called asynchronously so making the whole thing synchronous is not an option.
Thanks in advance
Edit(1) :
I've been suggested to use semaphores and after digging a little into it (i never heard of semaphores before) it seems to be a really good option and would remove the need for a queue, simply aquire in the beginning and release at the end, nothing more required!
I ended up using semaphores as per user207421's suggestions and it seems to work perfectly
I simply put delays between each line write to artificially make the task longer and make it easier to have multiple threads trying to write at once, and they all wait for their turns!
Thanks
Related
Now this may sound like a question that has been repeated many times before but I've done a day of research with people that has other reasons for this Issue.
I have a function that reads a part of the save file and its been shown that it does receive the correct data. So the error is that the integer variable completely ignores the new variable and shows no change in the live debugger so like many other post it is not just a duplicate object error. I cant seem to pinpoint what is the main issue is here and it's the last major thing holding me back. Any help would be great and I'm very extremely sorry if I did manage to miss a topic about this on the internet.
Code that fails:
#Override
public void read(List<String> data) {
//world positions are not being changed at all
System.out.println(data.get(1));
int test = Integer.valueOf(data.get(1).replaceAll("[^\\d.]", ""));
worldXPos = Integer.valueOf(data.get(0).replaceAll("[^\\d.]", ""));
worldZPos = test;
}
Another class that gives the data:
public void readSaveFunctions(){
if(!gameSaves.exists()){
gameSaves.mkdir();
}
String currentLine;
try {
List<String> data = new ArrayList<String>();
FileReader read = new FileReader(currentFile);
BufferedReader reader = new BufferedReader(read);
String key = "";
while((currentLine = reader.readLine()) != null){
if(currentLine.contains("#")){
key = currentLine;
data = new ArrayList<String>();
}else if(currentLine.contains("*end")){
for(int i = 0; i < saves.length; i++){
String tryKey = "#" + saves[i].IDName();
if(tryKey.equals(key)){
key = "";
saves[i].read(data);
}
}
}else data.add(currentLine);
}
reader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Another way of explaining it is this:
Debugger is set to step - to - step mode so I see each line getting executed at human speed then I get to a line like this but all of the ones setting the variables have the same effect:
worldXPos = Integer.valueOf(data.get(0).replaceAll("[^\\d.]", ""));
and the debugger shows the two integers having different numbers but the instant class variable stays exactly the same with no effect in the debugger after the line goes through.
Update:
I forgot to mention the method has a #override method and it seems that this #override may be causing this issue, now finally I may have a path to follow again
So I found my answer: The AWT thread manage to activate calling a method from another class that changed the integer before it could be read. It really though me off at first because the debugger only showed one of the threads and with no way to know the other one was actively changing it to early. Thanks for all the help :P.
This question already has an answer here:
JavaFX 2 StringProperty does not update field until enclosing method returns
(1 answer)
Closed 7 years ago.
I am currently learning JavaFX. I am building an app that takes a file as input, connects to an external server, and returns the parsed results to a MySQL database. The core functionality is working fine, but I would like to have JavaFX update to the user, so that they know they are waiting to connect to the server etc. However, when the method is invoked, the text updates are not visible until the method has completed execution - I would like the updates to occur in real time. The offending method is below:
public void doBlast(){
//this line is not updated until completion of the program - not straight away
status.setText("Connecting to NCBI servers..NCBI" + "\n");
//Biojava API details extracted as it is not relevant
BufferedReader bufferedFileReader = null;
Scanner bufferedScanner = null;
BufferedWriter bufferedWriterXML = null;
String rid = null;
int count = 0;
try {
//this.getFasta() represents the input file
bufferedScanner
= new Scanner(new BufferedReader(new FileReader (this.getFasta())));
//this.getTempXML() is a temporary file for temporary storage of xml output from the server
bufferedWriterXML
= new BufferedWriter (new FileWriter (this.getTempXML(), true));
String line, nextline, query, xmlLine;
//Loops for a specified number of times
while (count < alignmentLimit) {
this.setSequence_id(bufferedScanner.nextLine());
//newline here
bufferedScanner.nextLine();
//this line from the file is used to search a database in the server
query = bufferedScanner.nextLine();
//this line is not updated until completion of the program - not straight away
status.setText("Connecting to NCBI servers.." + "\n");
//below service connects to server
rid = service.sendAlignmentRequest(query, props);
//results
InputStream in = service.getAlignmentResults(rid, outputProps);
//this line is not updated until completion of the program - not straight away
status.setText("Connected to NCBI." + "\n");
bufferedFileReader = new BufferedReader(new InputStreamReader(in));
xmlLine = bufferedFileReader.readLine();
while (xmlLine != null) {
bufferedWriterXML.write("\n" + xmlLine);
xmlLine = bufferedFileReader.readLine();
// have to flush here so that it prints to file;
// the file can then be deleted each loop
bufferedWriterXML.flush();
}
count++;
//this method removes specific strings
this.removeAll("<?xml", "<!DOCTYPE", this.getTempXML(), this.getXML());
//this method fills a database with the data from server
this.xml2mysql();
//file cleaning
this.eraseData(this.getTempXML());
this.eraseData(this.getXML());
bufferedScanner.nextLine();
//this line is not updated until completion of the program - not straight away
numberOfRecords.setText(Integer.toString(count));
}
}
catch (Exception anException){
anException.printStackTrace();
}
finally {
try {
bufferedWriterXML.close();
bufferedScanner.close();
bufferedFileReader.close();
}
catch (Exception anException){
System.out.println("Error: " + anException);
}
}
}
The setText() methods use a FXML-defined fx:id variable and the above method is in a controller.
Much appreciated if anyone can help me out. I have searched everywhere for a solution.
Thanks.
You should look at using a task or service which will place the loading of the file on a background thread.
You can hook into the methods called during the execution:
From Oracle documentation:
Inside the call method, you can use the updateProgress, updateMessage,
updateTitle methods, which update the values of the corresponding
properties on the JavaFX Application thread. However, if the task was
canceled, a return value from the call method is ignored
If you perform the long process on the JavaFX Thread, the UI will be blocked and appear frozen.
public class Parser {
public static void main(String[] args) {
Parser p = new Parser();
p.matchString();
}
parserObject courseObject = new parserObject();
ArrayList<parserObject> courseObjects = new ArrayList<parserObject>();
ArrayList<String> courseNames = new ArrayList<String>();
String theWebPage = " ";
{
try {
URL theUrl = new URL("http://ocw.mit.edu/courses/");
BufferedReader reader =
new BufferedReader(new InputStreamReader(theUrl.openStream()));
String str = null;
while((str = reader.readLine()) != null) {
theWebPage = theWebPage + " " + str;
}
reader.close();
} catch (MalformedURLException e) {
// do nothing
} catch (IOException e) {
// do nothing
}
}
public void matchString() {
// this is my regex that I am using to compare strings on input page
String matchRegex = "#\\w+(-\\w+)+";
Pattern p = Pattern.compile(matchRegex);
Matcher m = p.matcher(theWebPage);
int i = 0;
while (!m.hitEnd()) {
try {
System.out.println(m.group());
courseNames.add(i, m.group());
i++;
} catch (IllegalStateException e) {
// do nothing
}
}
}
}
What I am trying to achieve with the above code is to get the list of departments on the MIT OpencourseWare website. I am using a regular expression that matches the pattern of the department names as in the page source. And I am using a Pattern object and a Matcher object and trying to find() and print these department names that match the regular expression. But the code is taking forever to run and I don't think reading in a webpage using bufferedReader takes that long. So I think I am either doing something horribly wrong or parsing websites takes a ridiculously long time. so I would appreciate any input on how to improve performance or correct a mistake in my code if any. I apologize for the badly written code.
The problem is with the code
while ((str = reader.readLine()) != null)
theWebPage = theWebPage + " " +str;
The variable theWebPage is a String, which is immutable. For each line read, this code creates a new String with a copy of everything that's been read so far, with a space and the just-read line appended. This is an extraordinary amount of unnecessary copying, which is why the program is running so slow.
I downloaded the web page in question. It has 55,000 lines and is about 3.25MB in size. Not too big. But because of the copying in the loop, the first line ends up being copied about 1.5 billion times (1/2 of 55,000 squared). The program is spending all its time copying and garbage collecting. I ran this on my laptop (2.66GHz Core2Duo, 1GB heap) and it took 15 minutes to run when reading from a local file (no network latency or web crawling countermeasures).
To fix this, make theWebPage into a StringBuilder instead, and change the line in the loop to be
theWebPage.append(" ").append(str);
You can convert theWebPage to a String using toString() after the loop if you wish. When I ran the modified version, it took a fraction of a second.
BTW your code is using a bare code block within { } inside a class. This is an instance initializer (as opposed to a static initializer). It gets run at object construction time. This is legal, but it's quite unusual. Notice that it misled other commenters. I'd suggest converting this code block into a named method.
Is this your whole program? Where is the declaration of parserObject?
Also, shouldn't all of this code be in your main() prior to calling matchString()?
parserObject courseObject = new parserObject();
ArrayList<parserObject> courseObjects = new ArrayList<parserObject>();
ArrayList<String> courseNames = new ArrayList<String>();
String theWebPage=" ";
{
try {
URL theUrl = new URL("http://ocw.mit.edu/courses/");
BufferedReader reader = new BufferedReader(new InputStreamReader(theUrl.openStream()));
String str = null;
while((str = reader.readLine())!=null)
{
theWebPage = theWebPage+" "+str;
}
reader.close();
} catch (MalformedURLException e) {
} catch (IOException e) {
}
}
You are also catching exceptions and not displaying any error messages. You should always display an error message and do something when you encounter an exception. For example, if you can't download the page, there is no reason to try to parse a empty string.
From you comment I learned about static blocks in classes (thank you, didn't know about them). However, from what I've read you need to put the keyword static before the start of the block {. Also, it might just be better to put the code into your main, that way you can exit if you get a MalformedURLException or IOException.
You can, of course, solve this assignment with the limited JDK 1.0 API, and run into the issue that Stuart Marks helped you solve in his excellent answer.
Or, you just use a popular de-facto standard library, like for instance, Apache Commons IO, and read your website into a String using a no-brainer like this:
// using this...
import org.apache.commons.io.IOUtils;
// run this...
try (InputStream is = new URL("http://ocw.mit.edu/courses/").openStream()) {
theWebPage = IOUtils.toString(is);
}
byte[] bytes = value.getBytes();
Process q = new ProcessBuilder("process","arg1", "arg2").start();
q.getOutputStream().write(bytes);
q.getOutputStream().flush();
System.out.println(q.getInputStream().available());
I'm trying to stream file contents to an executable and capture the output but the output(InputStream) is always empty. I can capture the output if i specify the the file location but not with streamed input.
How might I overcome this?
Try wrapping your streams with BufferedInputStream() and BufferedOutputStream():
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getOutputStream%28%29
Implementation note: It is a good idea for the output stream to be buffered.
Implementation note: It is a good idea for the input stream to be buffered.
Even with buffered streams, it is still possible for the buffer to fill if you're dealing with large amounts of data, you can deal with this by starting a separate thread to read from q.getInputStream(), so you can still be reading from the process while writing to the process.
Perhaps the program you execute only starts its work when it detects the end of its input data. This is normally done by waiting for an EOF (end-of-file) symbol. You can send this by closing the output stream to the process:
q.getOutputStream().write(bytes);
q.getOutputStream().close();
Try this together with waiting for the process.
I dont know if something else may also be wrong here, but the other process ("process") does not even have time to respond, you are not waiting for it (the method available() does not block). To try this out you can first insert a sleep(2000) after the flush(), and if that works you should switch to query'ing q.getInputStream().available() multiple times with short pauses in between.
I think, you have to wait, until the process finished.
I implemented something like this this way:
public class ProcessReader {
private static final int PROCESS_LOOP_SLEEP_MILLIS = 100;
private String result;
public ProcessReader(Process process) {
BufferedReader resultReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder resultOutput = new StringBuilder();
try {
while (!checkProcessTerminated(process, resultReader, resultOutput)) {
}
} catch (Exception ex1) {
throw new RuntimeException(ex1);
}
result = resultOutput.toString();
}
public String getResult(){
return result;
}
private boolean checkProcessTerminated(Process process, BufferedReader resultReader, StringBuilder resultOutput) throws Exception {
try {
int exit = process.exitValue();
return true;
} catch (IllegalThreadStateException ex) {
Thread.sleep(PROCESS_LOOP_SLEEP_MILLIS);
} finally {
while (resultReader.ready()) {
String out = resultReader.readLine();
resultOutput.append(out).append("\n");
}
}
return false;
}
}
I just removed now some specific code, that you dont need, but it should work, try it.
Regards
I have the following code in a java Web Service:
public boolean makeFile(String fileName, String audio)
{
if (makeUserFolder())
{
File file = new File(getUserFolderPath() + fileName + amr);
FileOutputStream fileOutputStream = null;
try
{
file.createNewFile();
fileOutputStream = new FileOutputStream(file);
fileOutputStream.write(Base64.decode(audio));
return true;
}
catch(FileNotFoundException ex)
{
return false;
}
catch(IOException ex)
{
return false;
}
finally{
try {
fileOutputStream.close();
convertFile(fileName);
} catch (IOException ex) {
Logger.getLogger(FileUtils.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
else
return false;
}
public boolean convertFile(String fileName)
{
Process ffmpeg;
String filePath = this.userFolderPath + fileName;
try {
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",filePath + amr,filePath + mp3);
pb.redirectErrorStream();
ffmpeg = pb.start();
} catch (IOException ex) {
return false;
}
return true;
}
It used to work and now it simply won't execute the ffmpeg conversion for some reason. I thought it was a problem with my file but after running the command from terminal no errors are thrown or anything, thought it was maybe permissions issue but all the permissions have been granted in the folder I'm saving the files. I noticed that the input BufferedReader ins being set to null after running the process, any idea what's happening?
First of all, a small nitpick with your code...when you create the FileOutputStream you create it using a string rather than a File, when you have already created the File before, so you might as well recycle that rather than force the FileOutputStream to instantiate the File itself.
Another small nitpick is the fact that when you are writing out the audio file, you should enclose that in a try block and close the output stream in a finally block. If you are allowed to add a new library to your project, you might use Guava which has a method Files.write(byte[],File), which will take care of all the dirty resource management for you.
The only thing that I can see that looks like a definite bug is the fact that you are ignoring the error stream of ffmpeg. If you are blocking waiting for input on the stdout of ffmpeg, then it will not work.
The easiest way to take care of this bug is to use ProcessBuilder instead of Runtime.
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",filePath+amr,filePath+mp3);
pb.redirectErrorStream(); // This will make both stdout and stderr be redirected to process.getInputStream();
ffmpeg = pb.start();
If you start it this way, then your current code will be able to read both input streams fully. It is possible that the stderr was hiding some error that you were not able to see due to not reading it.
If that was not your problem, I would recommend using absolute paths with ffmpeg...in other words:
String lastdot = file.getName().lastIndexOf('.');
File mp3file = new File(file.getParentFile(),file.getName().substring(0,lastdot)+".mp3");
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",file.getAbsolutePath(),mp3file.getAbsolutePath());
// ...
If that doesn't work, I would change ffmpeg to be an absolute path as well (in order to rule out path issues).
Edit: Further suggestions.
I would personally refactor the writing code into its own method, so that you can use it elsewhere necessary. In other other words:
public static boolean write(byte[] content, File to) {
FileOutputStream fos = new FileOutputStream(to);
try {
fos.write(content);
} catch (IOException io) {
// logging code here
return false;
} finally {
closeQuietly(fos);
}
return true;
}
public static void closeQuietly(Closeable toClose) {
if ( toClose == null ) { return; }
try {
toClose.close();
} catch (IOException e) {
// logging code here
}
}
The reason that I made the closeQuietly(Closeable) method is due to the fact that if you do not close it in that way, there is a possibility that an exception will be thrown by the close() method, and that exception will obscure the exception that was thrown originally. If you put these in a utility class (although looking at your code, I assume that the class that it is currently in is named FileUtils), then you will be able to use them throughout your application whenever you need to deal with file output.
This will allow you to rewrite the block as:
File file = new File(getUserFolderPath() + fileName + amr);
file.createNewFile()
write(Base64.decode(audio),file);
convertFile(fileName);
I don't know whether or not you should do this, however if you want to be sure that the ffmpeg process has completed, then you should say ffmpeg.waitFor(); to be sure that it has completed. If you do that, then you should examine ffmpeg.exitValue(); to make sure that it completed successfully.
Another thing that you might want to do is once it has completed, write what it output to a log file so you have a record of what happened, just in case something happens.