I'm using JDBC, need to constantly check the database against changing values.
What I have currently is an infinite loop running, inner loop iterating over a changing values, and each iteration checking against the database.
public void runInBG() { //this method called from another thread
while(true) {
while(els.hasElements()) {
Test el = (Test)els.next();
String sql = "SELECT * FROM Test WHERE id = '" + el.getId() + "'";
Record r = db.getTestRecord(sql);//this function makes connection, executeQuery etc...and return Record object with values
if(r != null) {
//do something
}
}
}
}
I'm think this isn't the best way.
The other way I'm thinking is the reverse, to keep iterating over the database.
UPDATE
Thank you for the feedback regarding timers, but I don't think it will solve my problem.
Once a change occurs in the database I need to process the results almost instantaneously against the changing values ("els" from the example code).
Even if the database does not change it still has to check constantly against the changing values.
UPDATE 2
OK, to anyone interested in the answer I believe I have the solution now. Basically the solution is NOT to use the database for this. Load in, update, add, etc... only whats needed from the database to memory.
That way you don't have to open and close the database constantly, you only deal with the database when you make a change to it, and reflect those changes back into memory and only deal with whatever is in memory at the time.
Sure this is more memory intensive but performance is absolute key here.
As to the periodic "timer" answers, I'm sorry but this is not right at all. Nobody has responded with a reason how the use of timers would solve this particular situation.
But thank you again for the feedback, it was still helpful nevertheless.
Another possibility would be using ScheduledThreadPoolExecutor.
You could implement a Runnable containing your logic and register it to the ScheduledExecutorService as follows:
ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(10);
executor.scheduleAtFixedRate(myRunnable, 0, 5, TimeUnit.SECONDS);
The code above, creates a ScheduledThreadPoolExecutor with 10 Threads in its pool, and would have a Runnable registered to it that will run in a 5 seconds period starting immediately.
To schedule your runnable you could use:
scheduleAtFixedRate
Creates and executes a periodic action that becomes enabled first after the given initial delay, and subsequently with the given period; that is executions will commence after initialDelay then initialDelay+period, then initialDelay + 2 * period, and so on.
scheduleWithFixedDelay
Creates and executes a periodic action that becomes enabled first after the given initial delay, and subsequently with the given delay between the termination of one execution and the commencement of the next.
And here you can see the advantages of ThreadPoolExecutor, in order to see if it fits to your requirements. I advise this question: Java Timer vs ExecutorService? too in order to make a good decision.
Keeping the while(true) in the runInBG() is a bad idea. You better remove that. Instead you can have a Scheduler/Timer(use Timer & TimerTask) which would call the runInBG() periodically and check for the updates in the DB.
u could use a timer--->
Timer timer = new Timer("runInBG");
//Taking an instance of class contains your repeated method.
MyClass t = new MyClass();
timer.schedule(t, 0, 2000);
As you said in the comment above, if application controls the updates and inserts then you can create a framework which notifies for 'BG' thread or process about change in database. Notification can be over network via JMS or intra VM using observer pattern or both local and remote notifications.
You can have generic notification message like (it can be class for local notification or text message for remote notifications)
<Notification>
<Type>update/insert</Type>
<Entity>
<Name>Account/Customer</Name>
<Id>id</Id>
<Entity>
</Notification>
To avoid a 'busy loop', I would try to use triggers. H2 also supports a DatabaseEventListener API, that way you wouldn't have to create a trigger for each table.
This may not always work, for example if you use a remote connection.
UPDATE 2
OK, to anyone interested in the answer I believe I have the solution now. Basically the solution is NOT to use the database for this. Load in, update, add, etc... only whats needed from the database to memory. That way you don't have to open and close the database constantly, you only deal with the database when you make a change to it, and reflect those changes back into memory and only deal with whatever is in memory at the time. Sure this is more memory intensive but performance is absolute key here.
Related
How is Apache NIO HttpAsyncClient able to wait for a remote response without blocking any thread? Does it have a way to setup a callback with the OS (I doubt so?). Otherwise does it perform some sort of polling?
EDIT - THIS ANSWER IS WRONG. PLEASE IGNORE AS IT IS INCORRECT.
You did not specify a version, so I can not point you to source code. But to answer your question, the way that Apache does it is by returning a Future<T>.
Take a look at this link -- https://hc.apache.org/httpcomponents-asyncclient-4.1.x/current/httpasyncclient/apidocs/org/apache/http/nio/client/HttpAsyncClient.html
Notice how the link says nio in the package. That stands for "non-blocking IO". And 9 times out of 10, that is done by doing some work with a new thread.
This operates almost exactly like a CompletableFuture<T> from your first question. Long story short, the library kicks off the process in a new thread (just like CompletableFuture<T>), stores that thread into the Future<T>, then allows you to use that Future<T> to manage that newly created thread containing your non-blocking task. By doing this, you get to decide exactly when and where the code blocks, potentially giving you the chance to make some significant performance optimizations.
To be more explicit, let's give a pseudocode example. Let's say I have a method attached to an endpoint. Whenever the endpoint is hit, the method is executed. The method takes in a single parameter --- userID. I then use that userID to perform 2 operations --- fetch the user's personal info, and fetch the user's suggested content. I need both pieces, and neither request needs to wait for the other to finish before starting. So, what I do is something like the following.
public StoreFrontPage visitStorePage(int userID)
{
final Future<UserInfo> userInfoFuture = this.fetchUserInfo(userID);
final Future<PageSuggestion> recommendedContentFuture = this.fetchRecommendedContent(userId);
final UserInfo userInfo = userInfoFuture.get();
final PageSuggestion recommendedContent = recommendedContentFuture.get();
return new StoreFrontPage(userInfo, recommendedContent);
}
When I call this.fetchUserInfo(userID), my code creates a new thread, starts fetching user info on that new thread, but let's my main thread continue and kick off this.fetchRecommendedContent(userID) in the meantime. The 2 fetches are occurring in parallel.
However, I need both results in order to create my StoreFrontPage. So, when I decided that I cannot continue any further until I have the results from both fetches, I call Future::get on each of my fetches. What this method does is merge the new thread back into my original one. In short, it says "wait for that one thread you created to finish doing what it was doing, then output the result as a return value".
And to more explicitly answer your question, no, this tool does not require you to do anything involving callbacks or polling. All it does is give you a Future<T> and lets you decide when you need to block the thread to wait on that Future<T> to finish.
EDIT - THIS ANSWER IS WRONG. PLEASE IGNORE AS IT IS INCORRECT.
Scenario: One thread is being called up to thousands of times per second to do inserts to the same table and is currently doing them one-by-one.
Goal: Do periodic batch inserts instead to improve performance.
Trying to use a TimerTask to instead add objects being saved to a list as the thread's saveItem method gets called, then combine them for a batch insert every 2 seconds or so.
First thought was to have two Lists, call them toSave and toSaveBackup. When the thread's saveItem method is called to save something it will be added to the toSave list, but once the TimerTask kicks off and needs to save everything to the database, it will set an AtomicBoolean flag saveInProgress to true. This flag is checked by saveItem and it will add to toSaveBackup instead of toSave if saveInProgress is true. When the batch save is complete, all items will in toSaveBackup will be moved to the toSave list, probably with a synchronized block on the lists.
Is this a reasonable approach? Or is there a better best practice? My googling skills have failed me so any help is welcome.
Misc info:
All these inserts are to the same table
Inserts are driven by receipt of MQTT messages, so I can't combine them in a batch before this point
Update: A tweak on CKing's answer below achieved the desired approach: A TimerTask runs every 100 ms and checks the size of the saveQueue and how long it's been since a batch was saved. If either of these values exceed the configured limit (save every 2 seconds or every 1000 records etc) then we save. A LinkedBlockingQueue is used to simplify sychronization.
Thanks again to everyone for their help!
It looks like your primary objective is to wait for a predefined amount of time and then trigger an insert. When an insert is in progress, you wan't other insert requests to wait till the insert is complete. After the insert is complete, you want to repeat the same process again for the next insert requests.
I would propose the following solution with the above understanding in mind. You don't need to have two separate lists to achieve your goal. Also note that I am proposing an old fashioned solution for the sake of explanation. I cover some other APIs you can use at the end of my explanation. Here goes :
Define a Timer and a TimerTask that will run every N seconds.
Define an ArrayList that will be used for queuing up insert requests sent to saveItem method.
The saveItem method can define a sycnrhonized block around this ArrayList. You can add items to the ArrayList within this synchronized block as and when saveItem is called.
On the other side of the equation, TimerTask should have a synchronized block on the same ArrayList as well inside its run method. It should insert all the records present in the ArrayList at that given moment into the database. Once the insert is complete, the TimerTask should clear the ArrayList and finally come out of the synchronized block.
You will no longer need to explicitly monitor if an insert is in progress or create a copy of your ArrayList when an insert is in progress. Your ArrayList becomes the shared resource in this case.
If you also want size to be a deciding factor for proceeding with inserts, you can do this :
Define an int called waitAttempts in TimerTask. This field indicates the number of consecutive wake ups for which the TimerTask should do nothing if the size of the list is not big enough.
Everytime the TimerTask wakes up, it can do something like if(waitAttempts%3==0 || list.size > 10) { insert data } else { increment waitAttempts and do nothing. Exit the synchronized block and the run method }. You can change 3 and 10 to whatever number suits your throughput requirements.
Note Intrinsic locking was used as a means of explaining the approach. One can always take this approach and implement it using modern constructs such as a BlockingQueue that would eliminate the need to synchronize manually on the ArrayList. I would also recommend the use of Executors.newSingleThreadScheduledExecutor() instead of a TimerTask as it ensures that there will only be one thread running at any given time and there wont be an overlap of threads. Also, the logic for waitAttempts is indicative and will need to be adjusted to work correctly.
I am building a fitness app which continually logs activity on the device. I need to log quite often, but I also don't want to unnecessarily drain the battery of my users which is why I am thinking about batching network calls together and transmitting them all at once as soon as the radio is active, the device is connected to a WiFi or it is charging.
I am using a filesystem based approach to implement that. I persist the data first to a File - eventually I might use Tape from Square to do that - but here is where I encounter the first issues.
I am continually writing new log data to the File, but I also need to periodically send all the logged data to my backend. When that happens I delete the contents of the File. The problem now is how can I prevent both of those operations from happening at the same time? Of course it will cause problems if I try to write log data to the File at the same time as some other process is reading from the File and trying to delete its contents.
I am thinking about using an IntentService essentially act as a queue for all those operations. And since - at least I have read as much - an IntentServices handles Intents sequentially in single worker Thread it shouldn't be possible for two of those operations to happen at the same time, right?
Currently I want to schedule a PeriodicTask with the GcmNetworkManager which would take care of sending the data to the server. Is there any better way to do all this?
1) You are overthinking this whole thing!
Your approach is way more complicated than it has to be! And for some reason none of the other answers point this out, but GcmNetworkManager already does everything you are trying to implement! You don't need to implement anything yourself.
2) Optimal way to implement what you are trying to do.
You don't seem to be aware that GcmNetworkManager already batches calls in the most battery efficient way with automatic retries etc and it also persists the tasks across device boots and can ensure their execution as soon as is battery efficient and required by your app.
Just whenever you have data to save schedule a OneOffTask like this:
final OneoffTask task = new OneoffTask.Builder()
// The Service which executes the task.
.setService(MyTaskService.class)
// A tag which identifies the task
.setTag(TASK_TAG)
// Sets a time frame for the execution of this task in seconds.
// This specifically means that the task can either be
// executed right now, or must have executed at the lastest in one hour.
.setExecutionWindow(0L, 3600L)
// Task is persisted on the disk, even across boots
.setPersisted(true)
// Unmetered connection required for task
.setRequiredNetwork(Task.NETWORK_STATE_UNMETERED)
// Attach data to the task in the form of a Bundle
.setExtras(dataBundle)
// If you set this to true and this task already exists
// (just depends on the tag set above) then the old task
// will be overwritten with this one.
.setUpdateCurrent(true)
// Sets if this task should only be executed when the device is charging
.setRequiresCharging(false)
.build();
mGcmNetworkManager.schedule(task);
This will do everything you want:
The Task will be persisted on the disk
The Task will be executed in a batched and battery efficient way, preferably over Wifi
You will have configurable automatic retries with a battery efficient backoff pattern
The Task will be executed within a time window you can specify.
I suggest for starters you read this to learn more about the GcmNetworkManager.
So to summarize:
All you really need to do is implement your network calls in a Service extending GcmTaskService and later whenever you need to perform such a network call you schedule a OneOffTask and everything else will be taken care of for you!
Of course you don't need to call each and every setter of the OneOffTask.Builder like I do above - I just did that to show you all the options you have. In most cases scheduling a task would just look like this:
mGcmNetworkManager.schedule(new OneoffTask.Builder()
.setService(MyTaskService.class)
.setTag(TASK_TAG)
.setExecutionWindow(0L, 300L)
.setPersisted(true)
.setExtras(bundle)
.build());
And if you put that in a helper method or even better create factory methods for all the different tasks you need to do than everything you were trying to do should just boil down to a few lines of code!
And by the way: Yes, an IntentService handles every Intent one after another sequentially in a single worker Thread. You can look at the relevant implementation here. It's actually very simple and quite straight forward.
All UI and Service methods are by default invoked on the same main thread. Unless you explicitly create threads or use AsyncTask there is no concurrency in an Android application per se.
This means that all intents, alarms, broad-casts are by default handled on the main thread.
Also note that doing I/O and/or network requests may be forbidden on the main thread (depending on Android version, see e.g. How to fix android.os.NetworkOnMainThreadException?).
Using AsyncTask or creating your own threads will bring you to concurrency problems but they are the same as with any multi-threaded programming, there is nothing special to Android there.
One more point to consider when doing concurrency is that background threads need to hold a WakeLock or the CPU may go to sleep.
Just some idea.
You may try to make use of serial executor for your file, therefore, only one thread can be execute at a time.
http://developer.android.com/reference/android/os/AsyncTask.html#SERIAL_EXECUTOR
Currently I use while(true) and Thread.sleep() for checking for new records in the db and execute java code.
Here is an example:
public class StartCommands implements Runnable{
private Active_Job activeJob;
Runnable execute_command;
public StartCommands(){
activeJobs = new Active_Job();
}
#Override
public void run(){
int jobId = 0;
while(true){
//access the db and get one row from the table by the status
jobId = activeJobs.get(Status.NEW);
if (jobId > 0){
activeJob.updateStatus(Status.INIT);
execute_command = activeJob.getCommand();
new Thread(execute_command).start();
activeJob = new Active_Job();
jobId = 0;
}
Thread.sleep(10*1000);
}
}
}
I've few places in the code that I use this method. But I dont like the endless loop and check every 10 seconds for new row.
So what I'm looking for is some kind of listener: once new record has been entered - execute java code. Some of the inserts executed from the application and some are not.
The technique you are using is called polling. You are checking for new records, waiting a set amount of time, then checking again for new records. One good way to respond to new records might be to create a controller that handles inserting new records into the database and force all clients (who update database records) to use the controller to do so. Then the controller can alert you when there is a new record. To facilitate the controller's alerts, you can set up a web service where the controller can contact you.
I say that this "might" be a good way to do it because creating a controller and a web service is obviously extra work. However, it would make polling unnecessary. If you want to continue using your polling technique, you could make a service (producer) that does the polling and fills a queue with the new results. Your other program (consumer) can then retrieve items from the queue and do something with them.
There is no builtin "update listener" in MySQL (or any SQL database I'm aware of), so you have to build your own.
Notice that in your implementation, if two new rows are added you will handle one, wait 10 seconds, then handle the next one. Your code cannot handle more than one event every 10 seconds.
What you want to do is separate the polling of the database from the dispatching of the worker threads. Have the polling loop wake up every n seconds, read ALL new records from the database, and add them to a work queue. Have a consumer thread that is waiting on the queue and launches processors as messages appear on the queue. using a thread pool implementation.
#nir, Since there is no mysql database update listener in java so far, so what you can do is, create a database update trigger against the table, the change of which you want to listen. Within the trigger statement or code construct a function.Now from within that function call java function. Java function should be such that it modify some text, say "a". Now register the listener against the change in "a". And within the class implementing the text change listener of "a",put the code you want to execute.
The Condition Interface would work nicely for your needs. It will give you the granular control you are looking for, and it will avoid the problem of spinning the thread constantly.
http://docs.oracle.com/javase/tutorial/essential/concurrency/newlocks.html
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/Condition.html
Use a trigger, call a User Defined Function that uses sys_exec() to run an external app that signals an inter-process semaphore. Your listener thread can wait on that and, when signaled, process the new records.
In oracle exists something called database change notification http://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm and I just implement a component like yours is there something like that in mysql or what approach you arrived?
This question already has answers here:
How can I abort a running JDBC transaction?
(4 answers)
Closed 5 years ago.
I have a program that continually polls the database for change in value of some field. It runs in the background and currently uses a while(true) and a sleep() method to set the interval. I am wondering if this is a good practice? And, what could be a more efficient way to implement this? The program is meant to run at all times.
Consequently, the only way to stop the program is by issuing a kill on the process ID. The program could be in the middle of a JDBC call. How could I go about terminating it more gracefully? I understand that the best option would be to devise some kind of exit strategy by using a flag that will be periodically checked by the thread. But, I am unable to think of a way/condition of changing the value of this flag. Any ideas?
I am wondering if this is a good practice?
No. It's not good. Sometimes, it's all you've got, but it's not good.
And, what could be a more efficient way to implement this?
How do things get into the database in the first place?
The best change is to fix programs that insert/update the database to make requests which go to the database and to your program. A JMS topic is good for this kind of thing.
The next best change is to add a trigger to the database to enqueue each insert/update event into a queue. The queue could feed a JMS topic (or queue) for processing by your program.
The fall-back plan is your polling loop.
Your polling loop, however, should not trivially do work. It should drop a message into a queue for some other JDBC process to work on. A termination request is another message that can be dropped into the JMS queue. When your program gets the termination message, it absolutely must be finished with the prior JDBC request and can stop gracefully.
Before doing any of this, look at ESB solutions. Sun's JCAPS or TIBCO already have this. An open source ESB like Mulesource or Jitterbit may already have this functionality already built and tested.
This is really too big an issue to answer completely in this format. Do yourself a favour and go buy Java Concurrency in Practice. There is no better resource for concurrency on the Java 5+ platform out there. There are whole chapters devoted to this subject.
On the subject of killing your process during a JDBC call, that should be fine. I believe there are issues with interrupting a JDBC call (in that you can't?) but that's a different issue.
As others have said, the fact that you have to poll is probably indicative of a deeper problem with the design of your system... but sometimes that's the way it goes, so...
If you'd like to handle "killing" the process a little more gracefully, you could install a shutdown hook which is called when you hit Ctrl+C:
volatile boolean stop = false;
Runtime.getRuntime().addShutdownHook(new Thread("shutdown thread") {
public void run() {
stop = true;
}
});
then periodically check the stop variable.
A more elegant solution is to wait on an event:
boolean stop = false;
final Object event = new Object();
Runtime.getRuntime().addShutdownHook(new Thread("shutdown thread") {
public void run() {
synchronized(event) {
stop = true;
event.notifyAll();
}
}
});
// ... and in your polling loop ...
synchronized(event) {
while(!stop) {
// ... do JDBC access ...
try {
// Wait 30 seconds, but break out as soon as the event is fired.
event.wait(30000);
}
catch(InterruptedException e) {
// Log a message and exit. Never ignore interrupted exception.
break;
}
}
}
Or something like that.
Note that a Timer (or similar) would be better in that you could at least reuse it and let it do with all of the details of sleeping, scheduling, exception handling, etc...
There are many reasons your app could die. Don't focus on just the one.
If it's even theoretically possible for your JDBC work to leave things in a half-correct state, then you have a bug you should fix. All of your DB work should be in a transaction. It should go or not go.
This is Java. Move your processing to a second thread. Now you can
Read from stdin in a loop. If someone types "QUIT", set the while flag to false and exit.
Create a AWT or Swing frame with a STOP button.
Pretend you are a Unix daemon and create a server socket. Wait for someone to open the socket and send "QUIT". (This has the added bonus that you can change the sleep to a select with timeout.)
There must be hundreds of variants on this.
Set up a signal handler for SIGTERM that sets a flag telling your loop to exit its next time through.
Regarding the question "The program could be in the middle of a JDBC call. How could I go about terminating it more gracefully?" - see How can I abort a running jdbc transaction?
Note that using a poll with sleep() is rarely the correct solution - implemented improperly, it can end up hogging CPU resources (the JVM thread-scheduler ends up spending inordinate amount of time sleeping and waking up the thread).
I‘ve created a Service class in my current company’s utility library for these kinds of problems:
public class Service implements Runnable {
private boolean shouldStop = false;
public synchronized stop() {
shouldStop = true;
notify();
}
private synchronized shouldStop() {
return shouldStop;
}
public void run() {
setUp();
while (!shouldStop()) {
doStuff();
sleep(60 * 1000);
}
}
private synchronized sleep(long delay) {
try {
wait(delay);
} catch (InterruptedException ie1) {
/* ignore. */
}
}
}
Of course this is far from complete but you should get the gist. This will enable you to simply call the stop() method when you want the program to stop and it will exit cleanly.
If that's your application and you can modify it, you can:
Make it read a file
Read for the value of a flag.
When you want to kill it, you just modify the file and the application will exit gracefully.
Not need to work it that harder that that.
You could make the field a compound value that includes (conceptually) a process-ID and a timestamp. [Better yet, use two or more fields.] Start a thread in the process that owns access to the field, and have it loop, sleeping and updating the timestamp. Then a polling process that is waiting to own access to the field can observe that the timestamp has not updated in some time T (which is much greater than the time of the updating loop's sleep interval) and assume that the previously-owning process has died.
But this is still prone to failure.
In other languages, I always try to use flock() calls to synchronize on a file. Not sure what the Java equivalent is. Get real concurrency if you at all possibly can.
I'm surprised nobody mentioned the interrupt mechanism implemented in Java. It's supposed to be a solution to the problem of stopping a thread. All other solutions have at least one flaw, that's why this mechanism is needed to be implemented in the Java concurrency library.
You can stop a thread by sending it an interrupt() message, but there are others ways that threads get interrupted. When this happens an InterruptedException is thrown. That's why you have to handle it when calling sleep() for example. That's where you can do cleanup and end gracefully, like closing the database connection.
Java9 has another "potential" answer to this: Thread.onSpinWait():
Indicates that the caller is momentarily unable to progress, until the occurrence of one or more actions on the part of other activities. By invoking this method within each iteration of a spin-wait loop construct, the calling thread indicates to the runtime that it is busy-waiting. The runtime may take action to improve the performance of invoking spin-wait loop constructions.
See JEP 285 for more details.
I think you should poll it with timertask instead.
My computer is running a while loop 1075566 times in 10 seconds.
Thats 107557 times in one second.
How often is it truly needed to poll it? A TimerTask runs at its fastest 1000 times in 1 second. You give it a parameter in int (miliseconds) as parameters. If you are content with that - that means you strain your cpu 108 times less with that task.
If you would be happy with polling once each second that is (108 * 1000). 108 000 times less straining. That also mean that you could check 108 000 values with the same cpu strain that you had with your one while loop - beause the you dont assign your cpu to check as often. Remember the cpu has a clock cycle. Mine is 3 600 000 000 hertz (cycles per second).
If your goal is to have it updated for a user - you can run a check each time the user logs in (or manually let him ask for an update) - that would practically not strain the cpu whatsoever.
You can also use thread.sleep(miliseconds); to lower the strain of your polling thread (as it wont be polling as often) you where doing.