I have researched for this subject a lot but couldn't find any useful information. And I have decided to ask my first question on this platform. So, I am using a scheduled executor to repeat task in a specific period. Everything is fine. But there is a misunderstanding.... My code executes task but if task takes longer than schedule time then it waits to finish task and later starts execute new task. I want it to do that execute task when schedule time arrives and don't wait previous task to finish. How can I achieve this? I used SwingWorker on a swing project but this project is not a swing project. Thanks for reading.
Main method
LogFactory.log(LogFactory.INFO_LEVEL, Config.MODULE_NAME + " - Available processors for Thread Pool: " + AVAILABLE_PROCESSORS);
ScheduledExecutorService executor = Executors.newScheduledThreadPool(AVAILABLE_PROCESSORS);
LogFactory.log(LogFactory.INFO_LEVEL, Config.MODULE_NAME + " - [ScheduledExecutorService] instance created.");
MainWorker task = new MainWorker();
LogFactory.log(LogFactory.INFO_LEVEL, Config.MODULE_NAME + " - [Main worker] created...");
executor.scheduleWithFixedDelay(task, 0, Config.CHECK_INTERVAL, TimeUnit.SECONDS);
Main Worker
public class MainWorker implements Runnable {
private final NIFIncomingController controller = new NIFIncomingController();
#Override
public void run() {
LogFactory.log(LogFactory.INFO_LEVEL, Config.MODULE_NAME + " - [Task] executed - [" + Thread.currentThread().getName() + "]");
controller.run();
}
}
You can try to combine several executors in order to achieve the desired behavior. Please, find an example code below:
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class CombinedThreadPoolsExample {
private static final DateFormat DATE_FORMAT = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
private static final int INITIAL_DELAY = 0;
private static final int FIXED_DELAY_IN_MILLISECONDS = 1000;
private static final int TASK_EXECUTION_IN_MILLISECONDS = FIXED_DELAY_IN_MILLISECONDS * 2;
public static void main(String[] args) {
int availableProcessors = Runtime.getRuntime().availableProcessors();
System.out.println("Available processors: [" + availableProcessors + "].");
ExecutorService fixedThreadPool = Executors.newFixedThreadPool(availableProcessors);
Runnable runnableThatTakesMoreTimeThanSpecifiedDelay = new Runnable() {
#Override
public void run() {
System.out.println("Thread name: [" + Thread.currentThread().getName() + "], time: [" + DATE_FORMAT.format(new Date()) + "].");
try {
Thread.sleep(TASK_EXECUTION_IN_MILLISECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
};
ScheduledExecutorService singleThreadScheduledExecutor = Executors.newSingleThreadScheduledExecutor();
singleThreadScheduledExecutor.scheduleWithFixedDelay(new Runnable() {
#Override
public void run() {
fixedThreadPool.execute(runnableThatTakesMoreTimeThanSpecifiedDelay);
}
}, INITIAL_DELAY, FIXED_DELAY_IN_MILLISECONDS, TimeUnit.MILLISECONDS);
}
}
The first lines of the output are the following for my machine:
Available processors: [8].
Thread name: [pool-1-thread-1], time: [2017-12-25 11:22:00.103].
Thread name: [pool-1-thread-2], time: [2017-12-25 11:22:01.104].
Thread name: [pool-1-thread-3], time: [2017-12-25 11:22:02.105].
Thread name: [pool-1-thread-4], time: [2017-12-25 11:22:03.105].
Thread name: [pool-1-thread-5], time: [2017-12-25 11:22:04.106].
Thread name: [pool-1-thread-6], time: [2017-12-25 11:22:05.107].
Thread name: [pool-1-thread-7], time: [2017-12-25 11:22:06.107].
Thread name: [pool-1-thread-8], time: [2017-12-25 11:22:07.107].
Thread name: [pool-1-thread-1], time: [2017-12-25 11:22:08.108].
Thread name: [pool-1-thread-2], time: [2017-12-25 11:22:09.108].
Thread name: [pool-1-thread-3], time: [2017-12-25 11:22:10.108].
Thread name: [pool-1-thread-4], time: [2017-12-25 11:22:11.109].
However, beware of relying on such solution when a task execution can take a long time, compare to a pool size. Let's say we increase the time necessary for a task execution:
private static final int TASK_EXECUTION_IN_MILLISECONDS = FIXED_DELAY_IN_MILLISECONDS * 10;
This won't cause any exceptions during execution, but cannot enforce the specified delay between executions. It can be observed in the execution output after the aforementioned delay alteration:
Available processors: [8].
Thread name: [pool-1-thread-1], time: [2017-12-25 11:31:23.258].
Thread name: [pool-1-thread-2], time: [2017-12-25 11:31:24.260].
Thread name: [pool-1-thread-3], time: [2017-12-25 11:31:25.261].
Thread name: [pool-1-thread-4], time: [2017-12-25 11:31:26.262].
Thread name: [pool-1-thread-5], time: [2017-12-25 11:31:27.262].
Thread name: [pool-1-thread-6], time: [2017-12-25 11:31:28.263].
Thread name: [pool-1-thread-7], time: [2017-12-25 11:31:29.264].
Thread name: [pool-1-thread-8], time: [2017-12-25 11:31:30.264].
Thread name: [pool-1-thread-1], time: [2017-12-25 11:31:33.260].
Thread name: [pool-1-thread-2], time: [2017-12-25 11:31:34.261].
Thread name: [pool-1-thread-3], time: [2017-12-25 11:31:35.262].
This can be achieved by using a single executor service. Say your schedule time is Y
Identify the time that can be taken by the task in worst case. Say X
If it is not in control, then control it in the task implementation to identify a timeout.
if X > Y, then create another task object and double up the schedule time
So the code may look like
executorService.schedule(taskObject1, 0/*Initial Delay*/, 2Y);
executorService.schedule(taskObject2, Y/*Initial Delay*/, 2Y);
If X > n(Y) then create n+1 tasks with scheduled time as (n+1)Y and initial delay as 0, 2Y, 3Y......(n-1)Y respectively.
Related
My idea is to share data from the main to the child thread (from the thread pool).
I found InheritableThreadLocal
Below is my example:
public static void main(String[] args) {
InheritableThreadLocal<String> inheritableThreadLocal =
new InheritableThreadLocal<>();
System.out.println("Thread name: " + Thread.currentThread().getName() + " - Set data into inheritableThreadLocal: " + "ABCXYZ");
inheritableThreadLocal.set("ABCXYZ");
System.out.println();
List<Integer> listIndex = List.of(1, 2);
listIndex
.stream()
.parallel()
.map(index -> {
System.out.println("Thread name: " + Thread.currentThread().getName() + " - Get data from inheritableThreadLocal : " + inheritableThreadLocal.get());
return inheritableThreadLocal.get();
})
.collect(Collectors.toList());
}
When tested I have strange behavior when running this example in different Java versions.
Java 11.0.12:
Thread name: main - Set data into inheritableThreadLocal: ABCXYZ
Thread name: main - Get data from inheritableThreadLocal : ABCXYZ
Thread name: ForkJoinPool.commonPool-worker-3 - Get data from inheritableThreadLocal : ABCXYZ
Java 17.0.1:
Thread name: main - Set data into inheritableThreadLocal: ABCXYZ
Thread name: ForkJoinPool.commonPool-worker-1 - Get data from inheritableThreadLocal : null
Thread name: main - Get data from inheritableThreadLocal : ABCXYZ
I do not know why have different output on different Java version like this (The thread on the pool can not find any data from inheritableThreadLocal area).
Could anyone give me the answer?
I have Java web application with REST calls using SPRING.
I want to control the number of threads the application is opening for the requests.
So I added Thread config:
package myPackage;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.task.TaskExecutor;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
#Configuration
public class ThreadConfig {
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(1);
executor.setMaxPoolSize(1);
executor.initialize();
return executor;
}
}
I'm using Sync service not Async, I tested it and it doesn't limit the threads handling the requests, it handles them all at the same time.
What I was expecting is when I send 2 requests at a time - either the 2nd request will be thrown or it will wait until the 1st request will finish.
I'm not implementing Thread in my application at all.
This is the relevant code from my controller:
#RestController
public class Module1Controller {
#RequestMapping(method = RequestMethod.GET, path = "/module1")
InterruptedException {
public Module1 Module1() throws InterruptedException {
Date startDate = new Date();
System.out.println("Thread #: " + Thread.currentThread().getId() + " Request received at: " + startDate);
Thread.sleep(10000);
Date endDate = new Date();
long diff = endDate.getTime() - startDate.getTime();
long seconds = TimeUnit.MILLISECONDS.toSeconds(diff);
System.out.println("Thread #: " + Thread.currentThread().getId() + " thread released at: " + endDate + ", total seconds: " + seconds);
return new Module1(new Clock());
}
This is the console result:
Thread #: 34 Request received at: Sun Dec 17 10:16:20 IST 2017
Thread #: 35 Request received at: Sun Dec 17 10:16:21 IST 2017
Thread #: 34 thread released at: Sun Dec 17 10:16:30 IST 2017, total seconds: 10
Thread #: 35 thread released at: Sun Dec 17 10:16:31 IST 2017, total seconds: 10
What am I missing here?
The problem is that the creation of a TaskExecutor in a configuration bean has no effect on your RestController.
The easiest way to make your RestController process only 1 request at a time is to make the handling method synchronized, e.g. like this:
#RequestMapping(method = RequestMethod.GET, path = "/module1")
public synchronized Module1 getModule1() throws InterruptedException {
If you want a certain maximum number of requests to be processed simultaneously you can use a FixedThreadPool, e.g. like this:
// allow only 2 requests at a time, more requests are automatically placed in a queue
private final ExecutorService es = Executors.newFixedThreadPool(2);
#RequestMapping(method = RequestMethod.GET, path = "/module1")
public Module1 getModule1() throws ExecutionException, InterruptedException {
Future<Module1> result = es.submit(new Callable<Module1>() {
#Override
public String call() throws Exception {
try {
//.... do your work here....
return Module1()
} catch (InterruptedException e) {
return null;
}
}
});
return result.get();
}
I'm not sure why you would want to do this. Limiting the number of requests will result in bad performance and users are not going to like this.
You can not control the threads of request in application instead in container. Maybe you want to run some tasks in limited threads in application. You can do like this:
#RestController
public class ThreadController {
#Autowired
private TaskExecutor taskExecutor;
#RequestMapping(method = RequestMethod.GET, path = "/thread")
public void Module1() {
taskExecutor.execute(new Runnable() {
#Override
public void run() {
Date startDate = new Date();
System.out.println("Thread #: " + Thread.currentThread().getId() +
" Request received at: " + startDate);
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
Date endDate = new Date();
long diff = endDate.getTime() - startDate.getTime();
long seconds = TimeUnit.MILLISECONDS.toSeconds(diff);
System.out.println("Thread #: " + Thread.currentThread().getId() +
" thread released at: " + endDate + ", total seconds: " + seconds);
}
});
}
}
The result:
Thread #: 55 Request received at: Sun Dec 17 22:40:57 CST 2017
Thread #: 55 thread released at: Sun Dec 17 22:41:07 CST 2017, total seconds: 10
Thread #: 55 Request received at: Sun Dec 17 22:41:16 CST 2017
Thread #: 55 thread released at: Sun Dec 17 22:41:26 CST 2017, total seconds: 10
Thread #: 55 Request received at: Sun Dec 17 22:41:32 CST 2017
Thread #: 55 thread released at: Sun Dec 17 22:41:42 CST 2017, total seconds: 10
In my application we have used Thread pool, we have specified a timeout to the thread pool but it seems that the timeout is not called, below is the code :
import java.sql.Timestamp;
import java.util.ArrayList;
import java.util.Date;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public class ThreadPoolDemo extends Thread{
public void run(){
System.out.println("Starting---" + new Timestamp((new Date()).getTime()) + "--" + Thread.currentThread().getName());
try {
Thread.sleep(30000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Finishing---" + new Timestamp((new Date()).getTime()) + "--" +Thread.currentThread().getName());
}
public static void main (String[] args){
ArrayBlockingQueue<Runnable> threadQueue = new ArrayBlockingQueue<Runnable>(5);
ThreadPoolExecutor thumbnailGeneratorThreadPool = new ThreadPoolExecutor(1, 3,
5, TimeUnit.SECONDS, threadQueue);
thumbnailGeneratorThreadPool.allowCoreThreadTimeOut(true);
ArrayList fTasks = new ArrayList();
for (int i = 0; i < 15; i++) {
System.out.println("Submitting Thread : " + (i+1) + "- Current Queue size is : " + threadQueue.size());
ThreadPoolDemo tpd = new ThreadPoolDemo();
Future future = thumbnailGeneratorThreadPool.submit(tpd);
}
}
}
The out put of the code is :
Submitting Thread : 1- Current Queue size is : 0
Submitting Thread : 2- Current Queue size is : 0
Submitting Thread : 3- Current Queue size is : 1
Submitting Thread : 4- Current Queue size is : 2
Submitting Thread : 5- Current Queue size is : 3
Submitting Thread : 6- Current Queue size is : 4
Submitting Thread : 7- Current Queue size is : 5
Submitting Thread : 8- Current Queue size is : 5
Submitting Thread : 9- Current Queue size is : 5
Exception in thread "main" java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1774)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:768)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:656)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)
at ThreadPoolDemo.main(ThreadPoolDemo.java:33)
Starting---2016-10-26 14:20:16.254--pool-1-thread-2
Starting---2016-10-26 14:20:16.254--pool-1-thread-3
Starting---2016-10-26 14:20:16.254--pool-1-thread-1
Finishing---2016-10-26 14:20:46.261--pool-1-thread-1
Finishing---2016-10-26 14:20:46.261--pool-1-thread-2
Finishing---2016-10-26 14:20:46.261--pool-1-thread-3
Starting---2016-10-26 14:20:46.261--pool-1-thread-2
Starting---2016-10-26 14:20:46.261--pool-1-thread-3
Starting---2016-10-26 14:20:46.261--pool-1-thread-1
Finishing---2016-10-26 14:21:16.265--pool-1-thread-1
Starting---2016-10-26 14:21:16.265--pool-1-thread-1
Finishing---2016-10-26 14:21:16.265--pool-1-thread-3
Starting---2016-10-26 14:21:16.265--pool-1-thread-3
Finishing---2016-10-26 14:21:16.265--pool-1-thread-2
Finishing---2016-10-26 14:21:46.277--pool-1-thread-1
Finishing---2016-10-26 14:21:46.277--pool-1-thread-3
Now in the ThreadPoolExecutor the keepAliveTime is set to 5 seconds.
However if we see the output the thread takes 30 seconds to complete. I'm not sure why the InterruptedException is not being called by the ThreadPoolExecutor on the thread.
I would like a mechanism to stop the threads if the thread is still active beyond the timeout specified.
As already indicated in the comments, you did not specify a time out. You only specified the keep-alive time of the ThreadPoolExecutor.
The keep-alive does not terminate or interrupt running threads, but only releases idle threads of the Executor (see getKeepAliveTime).
If you want to set a time out for your tasks, you have to use the invokeAll or invokeAny methods instead of submit.
See also
Killing thread after some specified time limit in Java and
How to timeout a thread
We've performed a performance test with Oracle Advanced Queue on our Oracle DB environment. We've created the queue and the queue table with the following script:
BEGIN
DBMS_AQADM.create_queue_table(
queue_table => 'verisoft.qt_test',
queue_payload_type => 'SYS.AQ$_JMS_MESSAGE',
sort_list => 'ENQ_TIME',
multiple_consumers => false,
message_grouping => 0,
comment => 'POC Authorizations Queue Table - KK',
compatible => '10.0',
secure => true);
DBMS_AQADM.create_queue(
queue_name => 'verisoft.q_test',
queue_table => 'verisoft.qt_test',
queue_type => dbms_aqadm.NORMAL_QUEUE,
max_retries => 10,
retry_delay => 0,
retention_time => 0,
comment => 'POC Authorizations Queue - KK');
DBMS_AQADM.start_queue('q_test');
END;
/
We've published 1000000 messages with 2380 TPS using a PL/SQL client. And we've consumed 1000000 messages with 292 TPS, using Oracle JMS API Client.
The consumer rate is almost 10 times slower than the publisher and that speed does not meet our requirements.
Below, is the piece of Java code that we use to consume messages:
if (q == null) initializeQueue();
System.out.println(listenerID + ": Listening on queue " + q.getQueueName() + "...");
MessageConsumer consumer = sess.createConsumer(q);
for (Message m; (m = consumer.receive()) != null;) {
new Timer().schedule(new QueueExample(m), 0);
}
sess.close();
con.close();
Do you have any suggestion on, how we can improve the performance at the consumer side?
Your use of Timer may be your primary issue. The Timer definition reads:
Corresponding to each Timer object is a single background thread that is used to execute all of the timer's tasks, sequentially. Timer tasks should complete quickly. If a timer task takes excessive time to complete, it "hogs" the timer's task execution thread. This can, in turn, delay the execution of subsequent tasks, which may "bunch up" and execute in rapid succession when (and if) the offending task finally completes.
I would suggest you use a ThreadPool.
// My executor.
ExecutorService executor = Executors.newCachedThreadPool();
public void test() throws InterruptedException {
for (int i = 0; i < 1000; i++) {
final int n = i;
// Instead of using Timer, create a Runnable and pass it to the Executor.
executor.submit(new Runnable() {
#Override
public void run() {
System.out.println("Run " + n);
}
});
}
executor.shutdown();
executor.awaitTermination(1, TimeUnit.DAYS);
}
When we run data intensive job over Hadoop. Hadoop executes the job.
Now what i want is when the job is completed. it will give me the statistics regarding
executed job i.e; time consumed, mapper quantity, reducer quantity and other useful information.
The information displayed in browser like job tracker, data node during the job execution.
But how can i get the statistics in my application which runs the job over Hadoop and gives me results like a report at the end of job completion. My application is in JAVA
Any API which can help me.
Suggestions will be appreciated.
Look into the following methods of JobClient:
getMapTaskReports(JobID)
getReduceTaskReports(JobID)
Both these calls return arrays of TaskReport object, from which you can pull start / finish times, and individual counters for each task
Chirs is correct. The documentation of TaskReport states that org.apache.hadoop.mapred.TaskReport inherits those methods from org.apache.hadoop.mapreduce.TaskReport. So, one could get such values.
Here are the codes to get the start and end time of a job, grouped for each Map and Reduce tasks.
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobStatus;
import org.apache.hadoop.conf.Configuration;
import java.net.InetSocketAddress;
import java.util.*;
import org.apache.hadoop.mapred.TaskReport;
import org.apache.hadoop.mapred.RunningJob;
import org.apache.hadoop.util.StringUtils;
import java.text.SimpleDateFormat;
public class mini{
public static void main(String args[]){
String jobTrackerHost = "192.168.151.14";
int jobTrackerPort = 54311;
try{
Configuration conf = new Configuration();
JobClient jobClient = new JobClient(new InetSocketAddress(jobTrackerHost, jobTrackerPort), conf);
JobStatus[] activeJobs = jobClient.jobsToComplete();
SimpleDateFormat dateFormat = new SimpleDateFormat("d-MMM-yyyy HH:mm:ss");
for(JobStatus js: activeJobs){
System.out.println(js.getJobID());
RunningJob runningjob = jobClient.getJob(js.getJobID());
while(runningjob.isComplete() == false){ /*Wait till the job completes.*/}
TaskReport[] maptaskreports = jobClient.getMapTaskReports(js.getJobID());
for(TaskReport tr: maptaskreports){
System.out.println("Task ID: "+tr.getTaskID()+" Start TIme: "+StringUtils.getFormattedTimeWithDiff(dateFormat, tr.getStartTime(), 0)+" Finish Time: "+StringUtils.getFormattedTimeWithDiff(dateFormat, tr.getFinishTime(), tr.getStartTime()));
}
TaskReport[] reducetaskreports = jobClient.getReduceTaskReports(js.getJobID());
for(TaskReport tr: reducetaskreports){
System.out.println("Task ID: "+tr.getTaskID()+" Start TIme: "+StringUtils.getFormattedTimeWithDiff(dateFormat, tr.getStartTime(), 0)+" Finish Time: "+StringUtils.getFormattedTimeWithDiff(dateFormat, tr.getFinishTime(), tr.getStartTime()));
}
}
}catch(Exception ex){
ex.printStackTrace();
}
}
}
This is a simple example to get the Start and Finish time of a running job. You can in the way you want.
And here is the run of this program for a "Word Count" MapReduce job.
[root#dev1-slave1 ~]# java -classpath /usr/lib/hadoop/hadoop-core.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-lang-2.4.jar:. mini
job_201501151144_0042
Task ID: task_201501151144_0042_m_000000 Start TIme: 16-Jan-2015 17:07:35 Finish Time: 16-Jan-2015 17:07:43 (7sec)
Task ID: task_201501151144_0042_m_000001 Start TIme: 16-Jan-2015 17:07:35 Finish Time: 16-Jan-2015 17:07:56 (20sec)
Task ID: task_201501151144_0042_m_000002 Start TIme: 16-Jan-2015 17:07:35 Finish Time: 16-Jan-2015 17:07:43 (7sec)
Task ID: task_201501151144_0042_m_000003 Start TIme: 16-Jan-2015 17:07:43 Finish Time: 16-Jan-2015 17:07:53 (10sec)
Task ID: task_201501151144_0042_m_000004 Start TIme: 16-Jan-2015 17:07:43 Finish Time: 16-Jan-2015 17:07:53 (10sec)
Task ID: task_201501151144_0042_r_000000 Start TIme: 16-Jan-2015 17:07:43 Finish Time: 16-Jan-2015 17:08:00 (17sec)
Task ID: task_201501151144_0042_r_000001 Start TIme: 16-Jan-2015 17:07:43 Finish Time: 16-Jan-2015 17:08:05 (22sec)
Task ID: task_201501151144_0042_r_000002 Start TIme: 16-Jan-2015 17:07:43 Finish Time: 16-Jan-2015 17:08:05 (21sec)
Its good to open the desired jsp files of hadoop in its mapreduce/src/webapps/job/ directory and figure out how JOBTRACKER Web UI is displaying information.
I have derived above codes from jobtasks.jsp.
Hope it helps. :)