Develop Reference

Java concurrency for periodic database batch insert

Scenario: One thread is being called up to thousands of times per second to do inserts to the same table and is currently doing them one-by-one.
Goal: Do periodic batch inserts instead to improve performance.
Trying to use a TimerTask to instead add objects being saved to a list as the thread's saveItem method gets called, then combine them for a batch insert every 2 seconds or so.
First thought was to have two Lists, call them toSave and toSaveBackup. When the thread's saveItem method is called to save something it will be added to the toSave list, but once the TimerTask kicks off and needs to save everything to the database, it will set an AtomicBoolean flag saveInProgress to true. This flag is checked by saveItem and it will add to toSaveBackup instead of toSave if saveInProgress is true. When the batch save is complete, all items will in toSaveBackup will be moved to the toSave list, probably with a synchronized block on the lists.
Is this a reasonable approach? Or is there a better best practice? My googling skills have failed me so any help is welcome.
Misc info:
All these inserts are to the same table
Inserts are driven by receipt of MQTT messages, so I can't combine them in a batch before this point
Update: A tweak on CKing's answer below achieved the desired approach: A TimerTask runs every 100 ms and checks the size of the saveQueue and how long it's been since a batch was saved. If either of these values exceed the configured limit (save every 2 seconds or every 1000 records etc) then we save. A LinkedBlockingQueue is used to simplify sychronization.
Thanks again to everyone for their help!

It looks like your primary objective is to wait for a predefined amount of time and then trigger an insert. When an insert is in progress, you wan't other insert requests to wait till the insert is complete. After the insert is complete, you want to repeat the same process again for the next insert requests.
I would propose the following solution with the above understanding in mind. You don't need to have two separate lists to achieve your goal. Also note that I am proposing an old fashioned solution for the sake of explanation. I cover some other APIs you can use at the end of my explanation. Here goes :
Define a Timer and a TimerTask that will run every N seconds.
Define an ArrayList that will be used for queuing up insert requests sent to saveItem method.
The saveItem method can define a sycnrhonized block around this ArrayList. You can add items to the ArrayList within this synchronized block as and when saveItem is called.
On the other side of the equation, TimerTask should have a synchronized block on the same ArrayList as well inside its run method. It should insert all the records present in the ArrayList at that given moment into the database. Once the insert is complete, the TimerTask should clear the ArrayList and finally come out of the synchronized block.
You will no longer need to explicitly monitor if an insert is in progress or create a copy of your ArrayList when an insert is in progress. Your ArrayList becomes the shared resource in this case.
If you also want size to be a deciding factor for proceeding with inserts, you can do this :
Define an int called waitAttempts in TimerTask. This field indicates the number of consecutive wake ups for which the TimerTask should do nothing if the size of the list is not big enough.
Everytime the TimerTask wakes up, it can do something like if(waitAttempts%3==0 || list.size > 10) { insert data } else { increment waitAttempts and do nothing. Exit the synchronized block and the run method }. You can change 3 and 10 to whatever number suits your throughput requirements.
Note Intrinsic locking was used as a means of explaining the approach. One can always take this approach and implement it using modern constructs such as a BlockingQueue that would eliminate the need to synchronize manually on the ArrayList. I would also recommend the use of Executors.newSingleThreadScheduledExecutor() instead of a TimerTask as it ensures that there will only be one thread running at any given time and there wont be an overlap of threads. Also, the logic for waitAttempts is indicative and will need to be adjusted to work correctly.

How to run progress bar while executing the mysql query in java?

I want to delete duplicate record in my database table, and I do it in java by using this query
String sql = "DELETE e1 FROM tweet_after_preprocessing e1, tweet_after_preprocessing e2 WHERE e1.tweet = e2.tweet AND e1.tweet_after_preprocessing_id > e2.tweet_after_preprocessing_id"
The problem is when there are so many records in my database table, the process will take so long, and make my program look not curently running.
and I want to use progress bar to show progress of the executing, how can I do that?. I don't now the maximum and the minimum value, so how can i accessing the progress bar?.

You can create an indeterminate progress bar by setting the property indeterminate to true: JProgressBar.html#setIndeterminate().
Also it is wise to not execute long lasting work in the EDT but use a different thread for this.

The problem is that you're doing all your work on the EDT, which is blocking your GUI. You need to do the loading on another Thread, so your GUI can still update, display, and respond to user input.
Once you have the work on another thread, then from that thread, you can post updates to the EDT using the SwingUtilities.invokeLater() method.
You could also look into the SwingWorker class, which handles some of that for you.
Recommended reading: http://docs.oracle.com/javase/tutorial/uiswing/concurrency/

Excellent question.
Jenkins uses the time of the last build to guess how long the current build will take to run. So for example if the last time a build ran it took 10 minutes, and it is 5 minutes into a current build it will show that it is 50% complete.
You can do something similar. Maybe query the DB first to see how many items need to be deleted and have a table of how long it will take for the amount of items being deleted and how many items are in the table.

How to Display a Database data in jtextarea by using time delay?

int time = 0;
time++; // set by 1second
int countdata = 0; // countdata queried by table shows the numbers of table values
rs = stmt.executeQuery("SELECT ID,test From Table");
StringBuilder strBuilder = new StringBuilder();
while (rs.next()) {
strBuilder.append(rs.getString(0)).append(" ").append(rs.getString(1));
strBuilder.append("\n");
if(time <= countdata){
// this is my idea but i dont know how to fetch the data each row
}
jtextarea.setText(strBuilder.toString());
}
the output is like this:
// time 1 seconds
test1
// time 2 seconds
test2
// time 3 reconds
test3
//until it reach the last data of table
please help
i want to fetch data by every 1 second until it reach the last value. the time is fix by 1 second delay. every 1 second shows the row of data.

You have two basic choices (there are others, but lets keep it simple...)
Basically, you want to perform the query in some kind of background thread but ensure that the updates to the UI are carried out within the context of the Event Dispatching Thread...
This ensures that while the query is taking place, the UI remains responsive to the user and to won't make it look like your application has crashed...
You Could...
Use a Swing Timer to schedule a call back every n milliseconds. The benefit of this is that it the callback is triggered within the context of the Event Dispatching Thread.
The drawback is that the callback is triggered within the context of the Event Dispatching Thread.
The main problem I see is you don't want to be querying the database from the context of the EDT as this will prevent it from performing updates to the UI and remaining responsive to the user while the processing is occurring.
Now you could spawn another Thread at this stage, but the problem is, the time it takes to complete the query may be more than the time allocated between updates, meaning that it would be possible for last query to still be running when the next one is triggered. You also run into issues with having to synchronise updates to the UI manually, while not especially difficult, why would you want to make life more difficult ;)
Take a look at How to Use Swing Timers and Concurrency in Swing for more details
You Could...
Use a SwingWorker. This will allow you to perform the query in a background thread while providing you with functionality to send updates back to the Event Dispatching Thread so the UI can be updated properly.
The drawback with this is that the SwingWorker doesn't have timer concept of it's own, but, you could simply use Thread.sleep.
Now, depending on whether you want regular updates; ie they MUST occur every second, exactly, or delayed updates; ie they must occur every second AFTER the last update will depend on how you use this, but what this means is you gain control.
Basically what this means is, you can ensure that only one query is been executed at a time and control the amount of time before the next one.
Take a look at Worker Threads and SwingWorker for more details

Start Java event on date/time based on database row

I'm trying to make a Listener (or something like that?) that will start a specific event when a date field from a database row is the same as the current time. Of course I can trigger every second to check if the date/time is the same as the current, but I think that is quite expansive. There should be a better alternative..
What I trying to do is the following:
I have several (for example football) matches scheduled in my database. At the specific time when the match should start, I will start a event in my Java app. This could be 1 or more matches at that time.

I understand that you are trying to schedule execution of future events in java app not in database.
You should consider using ScheduledExecutorService method schedule to delay execution of task to specific point in time.
The only problem you have to solve is how you synchronize task in database with this in schedule.
EDIT:
If you keep map with taskID->ScheduledFuture object returned you can easily call cancel on the object to remove task. But you have to add some kind of last-modification column to detect new and updated tasks and query database to check if there are not any new tasks.

Implementing java FixedTreadPool status listener

It's about an application which is supposed to process (VAD, Loudness, Clipping) a lot of soundfiles (e.g. 100k). At this time, I create as many worker threads (callables) as I can put into memory, and then run all with a threadPool.invokeAll(), write results to file system, unload processed files and continue at step 1. Due to the fact it's an app with a GUI, i don't want to user to feel like the app "is not responding" while processing all soundfiles. (which it does at this time cause invokeAll is blocking). I'm not sure what is a "good" way to fix this. It shall not be possible for the user to do other things while processing, but I'd like to show a progress bar like "10 of 100000 soundfiles are done". So how do I get there? Do I have to create a "watcher thread", so that every worker hold a callback on it? I'm quite new to multi threading, and don't get the idea of such a mechanism.
If you need to know: I'm using SWT/JFace.

You could use an ExecutorCompletionService for this purpose; if you submit each of the Callable tasks in a loop, you can then call the take method of the completion service - receiving tasks one at a time as they finish. Every time you take a task, you can update your GUI.
As another option, you could implement your own ExecutorService that is also an Observable, allowing the publication of updates to subscribing Observers whenever a task is completed.

You should have a look at SwingWorker. It's a good class for doing lengthy operations whilst reporting back progress to the gui and maintaining a responsive gui.
Using a Swing Worker Thread provides some good information.