Develop Reference

How to prevent next #Scheduler call until current #scheduler have finished in Spring Boot?

I am trying to create a Spring boot application, which asynchronously process thousand of records using #Async multithreading. For this I am creating 8 threads and 8 sub list from main list so that 8 thread process 8 sub list asynchronously. I am also using #scheduler so that method calls in every 2 seconds.
But problem is that due to scheduler, sometimes this application process duplicate records because this method get called in every 2 seconds and retrieve data from database in every 2 seconds. for example first time method get called and retrieved 72000 records from database whose flag 0 and then #Async method process all these records and change processed records flag from 0 to 1. And again in 2 seconds method get called and retrieve new records wh0se flag is 0.
In log attached pic you can see that first time scheduler get called and retrieved 72000 records and multiple threads started processing in between next scheduler started and retrieved 16000 records which contain those records also which are present in current scheduler.
I am looking solution that next scheduler should not call until first scheduler get completed. Because sometime first scheduler processing records in between if next scheduler call in 2 seconds then may be it retrieve those records again which are already present in first scheduler call.
I can't increase scheduler call time. Because maximum time we get the records around 400-500 and sometime we get records in thousands.
Code as below.
#SpringBootApplication
#EnableScheduling
#EnableAsync
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder.build();
}
#Bean("ThreadPoolTaskExecutor")
public TaskExecutor getAsyncExecutor() {
final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(8);
executor.setMaxPoolSize(8);
executor.setQueueCapacity(8);
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.setThreadNamePrefix("async-");
executor.setRejectedExecutionHandler(new RejectedExecutionHandlerImpl());
executor.initialize();
return executor;
}
}
#Service
public class Main {
#Autowired
private DemoDao dao;
#Autowired
private Asyn asyn;
static int schedulerCount = 0;
#Scheduled(cron = "0/2 * * * * *")
public void schedule() {
System.out.println("++++++++++++++++++++++++++++++++++++Scheduler started schedulerCount : "+schedulerCount+"+++++++++++++++++++++++++++++++++++++"+ LocalDateTime.now());
List<Json> jsonList = new ArrayList<Json>();
List<List<Json>> smallLi = new ArrayList<List<Json>>();
try {
jsonList = dao.getJsonList();
System.out.println("jsonList size : " + jsonList.size());
int count = jsonList.size();
//Creating 8 sublist (8 sublist because thread pool size is 8) from main list(jsonList)
int limit = Math.round(count / 8) + 1;
for (int j = 0; j < count; j += limit) {
smallLi.add(new ArrayList<Json>(jsonList.subList(j, Math.min(count, j + limit))));
}
System.out.println("smallLi : " + smallLi.size());
//After creating 8 sublist, sending sublists with Async method so that 8 threads create and each thread process one sublist asynchronously.
for (int i = 0; i < smallLi.size(); i++) {
asyn.withAsyn(smallLi.get(i), schedulerCount);
}
schedulerCount++;
} catch (Exception e) {
e.printStackTrace();
}
}
#Async("ThreadPoolTaskExecutor")
public void withAsyn(List<Json> li, int schedulerCount) throws Exception {
System.out.println("with start+++++++++++++ schedulerCount " + schedulerCount + ", name : "
+ Thread.currentThread().getName() + ", time : " + LocalDateTime.now() + ", start index : "
+ li.get(0).getId() + ", end index : " + li.get(li.size() - 1).getId());
try {
XSSFWorkbook workbook = new XSSFWorkbook();
XSSFSheet spreadsheet = workbook.createSheet("Data");
XSSFRow row;
for (int i = 0; i < li.size(); i++) {
row = spreadsheet.createRow(i);
Cell cell9 = row.createCell(0);
cell9.setCellValue(li.get(i).getId());
Cell cell = row.createCell(1);
cell.setCellValue(li.get(i).getName());
Cell cell1 = row.createCell(2);
cell1.setCellValue(li.get(i).getPhone());
Cell cell2 = row.createCell(3);
cell2.setCellValue(li.get(i).getEmail());
Cell cell3 = row.createCell(4);
cell3.setCellValue(li.get(i).getAddress());
Cell cell4 = row.createCell(5);
cell4.setCellValue(li.get(i).getPostalZip());
}
FileOutputStream out = new FileOutputStream(new File("C:\\Users\\RK658\\Desktop\\logs\\generated\\"
+ Thread.currentThread().getName() + "_" + schedulerCount + ".xlsx"));
workbook.write(out);
out.close();
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("with end+++++++++++++ schedulerCount " + schedulerCount + ", name : "
+ Thread.currentThread().getName() + ", time : " + LocalDateTime.now() + ", start index : "
+ li.get(0).getId() + ", end index : " + li.get(li.size() - 1).getId());
}

I am looking solution that next scheduler should not call until first
scheduler get completed.
You have defined #Service public class Main {...} and by default #Service is an indication for spring to build a singleton. If you have not done anything to modify this, then the expected behavior is for spring to create only a single instance of your class Main in heap memory.
So in this case if you add the synchronized keyword in your method you will allow only 1 thread to be able to execute your method at some time. Every other thread requesting to execute your scheduled method will need to wait until the thread that is already running finishes.
#Scheduled(cron = "0/2 * * * * *")
public synchronized void schedule() { .... }
Example of above solution scenario.
Thread A starts executing method schedule.
2 seconds later while Thread A still executes, Thread B starts and wants to execute the same method.
Thread B will wait outside and will start executing the method only after Thread A has finished execution.
So in the above scenario if 3 schedulers are spawn at close 3 time intervals all of them will wait for execution and maybe this will lead to some bottle neck of your scheduling and might put some heavy load on your resources.
If that is the case another alternative solution would be to cancel Thread B method execution if it arrives while Thread A already executes the same method. This way you will be sure that no bottle neck is built up.
To implement this you can use a ReentrantLock.
#Service
public class Main {
private final ReentrantLock reentrantLock = new ReentrantLock();
....
#Scheduled(cron = "0/2 * * * * *")
public void schedule() {
if (reentrantLock.tryLock()) {
try {
//... all your existing method code
} finally {
reentrantLock.unlock();
}
} else {
//log that the `Thread B` was cancelled if you want, as it requested to execute the method while `Thread A` was already executing.
}
}
}

I believe you can use ShedLock to solve this issue. https://www.baeldung.com/shedlock-spring
https://github.com/lukas-krecan/ShedLock

Why the Spring Boot controller can't handle other requests if Thread.sleep is used?

When we use the Thread.sleep(...) method in the Spring Boot controller, we cannot accept requests concurrently. All requests can only be processed one by one. Why not every request is processed in parallel？
For example, there is a Thread.sleep(...) in the sleep method. In order to simulate the parallel access to the sleep method, I used the Chrome browser to open multiple tabs to visit http://localhost:8080/async. Observe that multiple tabs are executed in series. Why is it not executed in parallel here?
#RestController
public class AsyncController {
#RequestMapping("async")
public String sleep() throws InterruptedException {
long startTime = System.currentTimeMillis();
System.out.println("[before]name is " + Thread.currentThread().getName() + " time is " + startTime);
Thread.sleep(10000);
long endTime = System.currentTimeMillis();
System.out.println("[after]name is " + Thread.currentThread().getName() + " time is " + endTime + " cos " + (endTime-startTime));
return Thread.currentThread().getName();
}
}
Console printing
[before]name is http-nio-8080-exec-2 time is 1602751326997
[after]name is http-nio-8080-exec-2 time is 1602751337000 cos 10003
[before]name is http-nio-8080-exec-3 time is 1602751337004
[after]name is http-nio-8080-exec-3 time is 1602751347008 cos 10004

Using a Commonj Work Manager to send Asynchronous HTTP calls

I switched from making sequential HTTP calls to 4 REST services, to making 4 simultaneous calls using a commonj4 work manager task executor. I'm using WebLogic 12c. This new code works on my development environment, but in our test environment under load conditions, and occasionally while not under load, the results map is not populated with all of the results. The logging suggests that each work item did receive back the results though. Could this be a problem with the ConcurrentHashMap? In this example from IBM, they use their own version of Work and there's a getData() method, although it doesn't like that method really exists in their class definition. I had followed a different example that just used the Work class but didn't demonstrate how to get the data out of those threads into the main thread. Should I be using execute() instead of schedule()? The API doesn't appear to be well documented. The stuckthreadtimeout is sufficiently high. component.processInbound() actually contains the code for the HTTP call, but I the problem isn't there because I can switch back to the synchronous version of the class below and not have any issues.
http://publib.boulder.ibm.com/infocenter/wsdoc400/v6r0/index.jsp?topic=/com.ibm.websphere.iseries.doc/info/ae/asyncbns/concepts/casb_workmgr.html
My code:
public class WorkManagerAsyncLinkedComponentRouter implements
MessageDispatcher<Object, Object> {
private List<Component<Object, Object>> components;
protected ConcurrentHashMap<String, Object> workItemsResultsMap;
protected ConcurrentHashMap<String, Exception> componentExceptionsInThreads;
...
//components is populated at this point with one component for each REST call to be made.
public Object route(final Object message) throws RouterException {
...
try {
workItemsResultsMap = new ConcurrentHashMap<String, Object>();
componentExceptionsInThreads = new ConcurrentHashMap<String, Exception>();
final String parentThreadID = Thread.currentThread().getName();
List<WorkItem> producerWorkItems = new ArrayList<WorkItem>();
for (final Component<Object, Object> component : this.components) {
producerWorkItems.add(workManagerTaskExecutor.schedule(new Work() {
public void run() {
//ExecuteThread th = (ExecuteThread) Thread.currentThread();
//th.setName(component.getName());
LOG.info("Child thread " + Thread.currentThread().getName() +" Parent thread: " + parentThreadID + " Executing work item for: " + component.getName());
try {
Object returnObj = component.processInbound(message);
if (returnObj == null)
LOG.info("Object returned to work item is null, not adding to producer components results map, for this producer: "
+ component.getName());
else {
LOG.info("Added producer component thread result for: "
+ component.getName());
workItemsResultsMap.put(component.getName(), returnObj);
}
LOG.info("Finished executing work item for: " + component.getName());
} catch (Exception e) {
componentExceptionsInThreads.put(component.getName(), e);
}
}
...
}));
} // end loop over producer components
// Block until all items are done
workManagerTaskExecutor.waitForAll(producerWorkItems, stuckThreadTimeout);
LOG.info("Finished waiting for all producer component threads.");
if (componentExceptionsInThreads != null
&& componentExceptionsInThreads.size() > 0) {
...
}
List<Object> resultsList = new ArrayList<Object>(workItemsResultsMap.values());
if (resultsList.size() == 0)
throw new RouterException(
"The producer thread results are all empty. The threads were likely not created. In testing this was observed when either 1)the system was almost out of memory (Perhaps the there is not enough memory to create a new thread for each producer, for this REST request), or 2)Timeouts were reached for all producers.");
//** The problem is identified here. The results in the ConcurrentHashMap aren't the number expected .
if (workItemsResultsMap.size() != this.components.size()) {
StringBuilder sb = new StringBuilder();
for (String str : workItemsResultsMap.keySet()) {
sb.append(str + " ");
}
throw new RouterException(
"Did not receive results from all threads within the thread timeout period. Only retrieved:"
+ sb.toString());
}
LOG.info("Returning " + String.valueOf(resultsList.size()) + " results.");
LOG.debug("List of returned feeds: " + String.valueOf(resultsList));
return resultsList;
}
...
}
}

I ended up cloning the DOM document used as a parameter. There must be some downstream code that has side effects on the parameter.

Quartz How can i stop/pause/start/interrupt the Scheduler from Jsp i.e on click of particular button

Please can any one forward me the sample code related. Cause i tried a lot and on internet no useful info or links i can found related to it.
Thanks in Advance

This might be a workaround. But it works!
In your scheduler, have a default thread running every 1 minute (Or interval of your choice) that pings a file or DB for any changes.
The scheduler should be refreshed if the scheduler finds an entry in the DB.
From your JSP, on click of a button, create a relevant entry in the DB.
While pinging the DB, if the scheduler finds an entry, then it will do the necessary action.
Code snippet
// Default constructor.
public Scheduler()throws SchedulerException, Exception
{
try
{
SchedulerFactory sf = new StdSchedulerFactory();
sche = sf.getScheduler();
sche.start();
if(sche.isShutdown())
{
SendAlerts.sendMsgToGroup("Scheduler Failed To Start at "+sdtf3.format(new Date())+" hrs.",defaultMsgGroup);
logger.fatal("Scheduler Failed To Start At = " + sdtf1.format(new Date()) );
}
else
{
SendAlerts.sendMsgToGroup("Scheduler started at "+sdtf3.format(new Date())+" hrs.",SchStartAlertGroup);
logger.fatal("Scheduler Started At = " + sdtf1.format(new Date()) );
}
sysdate = new Date();
readFromDBAndConfigureSchedules();
while (true)
{
if(sche.isShutdown())
{
SendAlerts.sendMsgToGroup("Scheduler Failed To Start at "+sdtf3.format(new Date())+" hrs.",defaultMsgGroup);
logger.fatal("Scheduler Failed To Start At = " + sdtf1.format(new Date()) );
}
else
{
logger.info("Scheduler is Running. Table Last Pinged at : "+sdtf1.format(sysdate));
}
/*
-----------------
IN THE CHECK DB TABLE METHOD, HANDLE REQUESTS FOR STOP, PAUSE, RE-SCHEDULE ETC
------------------
*/
SchRunJob.checkDBTable();
// Loop will repeat every 1 hour = 60 minutes * 60 seconds = 3600 seconds
Thread.sleep (3600 * 1000);
} // End of while Start Flag is Y
} // End of try block
catch (Exception e)
{
SendAlerts.sendMsgToGroup( "Fatal Exception Caught.Scheduler Shut Down at " + sdtf1.format(new Date()),defaultMsgGroup);
logger.fatal("Fatal Exception Caught.Scheduler Shut Down at " + sdtf1.format(new Date()));
e.printStackTrace();
System.exit(0);
}
} // End of default constructor**

Java Concurrency issues when updating video control UI while JMF video is playing

I'm building a video player in pure Java around JMF, with completely custom UI controls. Everything was working great until I put in a JLabel which updates the current play time in hh:mm:ss.ss format. The label updates acceptably, but occasionally halts for 10+ seconds at at a time, which is unacceptable.
The JMF Player is created in a SwingUtilities.invokeLater(new Runnable()... block. Here is the UI updating code:
protected void createUIUpdater() {
System.out.println("Creating UI updating worker thread");
new Thread() {
#Override
public void run() {
while(mediaPlayer.getState() == Controller.Started) {
updatePlayPosition();
try {
Thread.sleep(100);
} catch (InterruptedException e) {
System.err.println("Sleep interrupted!");
}
}
System.out.println("UI Updater thread finished.");
}
}.start();
}
protected void updatePlayPosition() {
Movie movie = timelineEditor.getMovie();
movie.setInsertionPoint(Rational.valueOf(mediaPlayer.getMediaTime().getSeconds()));
updateTimeLabel();
}
protected void updateTimeLabel() {
Movie movie = timelineEditor.getMovie();
Rational time = movie == null ? new Rational(0,1) : movie.getInsertionPoint();
// ... hours/minutes/seconds calculated
timeLabel.setText((hours < 10 ? "0" + hours : hours) + ":" + (minutes < 10 ? "0" + minutes : minutes)
+ ":" + (seconds < 10 ? "0" + seconds : seconds)
+ "." + (frame < 10 ? "0" + frame : frame));
}
which is called in a ControllerListener on a StartEvent. During this time the audio/video are playing.
I'm admittedly a bit of a novice when it comes to concurrency in Java, and I'm sure there's a better way of handling this separate thread. Any suggestions?

you could replace your Thread/sleep with ScheduledExecutorService:
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ScheduledExecutorService.html
because when using sleep there is no guarantee as to how long the thread will actually sleep , it may take more the the time you specify to return to the running status.

After closer inspection, it seems that my thread is executing fine, it is just getting blocked occasionally on mediaPlayer.getMediaTime() (where mediaPlayer is a javax.media.Player).
Turns out, calling mediaPlayer.getMediaNanoseconds() does not block.