EDIT: It turns out that my docker container was closing because of an errant health check failing. This caused the kubernetes probe to fail too many times and killed the pod. Since the command came from outside the JVM, it is likely not possible for the code to know when the container around the JVM was killed. Leaving this question up in case someone finds an answer.
I have a Java SE project that reads in arguments to process files. I am using Weld-SE for injection. After a few minutes of running, the weld container shuts down mid progressing without explanation. The following is a skeleton of my code. Here in my Main Class
#ApplicationScoped
public class Main {
#Setter //Lombok annotation
private Scheduler scheduler;
public void schedule(String[] filenames) throws InterruptedException {
scheduler.schedule(filenames);
}
#PreDestroy
public void cleanUp() {
log.info("PreDestroy - Main");
//clean up omitted
}
public static void main(String[] args) {
Weld weld = new Weld();
Main main = null;
try (WeldContainer container = weld.initialize()) {
main = container.select(Main.class).get();
Scheduler scheduler = CDI.current().select(Scheduler.class).get();
main.setScheduler(scheduler);
main.schedule(args);
log.info("Main - completed processing");
} catch (Exception e) {
log.info("Exception occurred: " + e.getMessage(), e);
} finally {
log.info("Main Finally - Shutting down Weld");
CDI.current().destroy(main);
weld.shutdown();
}
}
}
Here is my Scheduler class
#Named
public class Scheduler {
private ThreadPoolExecutor executor;
private Map<Future<?>, Importer> beanMap = new HashMap<>();
private static final Duration sleepDuration = Duration.ofSeconds(45);
#PostConstructor
public void init() {
//setup executor
}
#PreDestroy
public void cleanUp() {
log.info("PreDestroy - Scheduler");
//clean up omitted
}
public void schedule(String[] filenames) throws InterruptedException {
for(String filename : filenames) {
Importer caller = CDI.current().select(Importer.class).get();
caller.setFilename(filename);
Future<?> future = executor.submit(caller);
beanMap.put(future, caller);
}
while(!isDone()) {
Thread.sleep(sleepDuration.toMillis());
log.info("Remaining threads: {}, getRemaining());
destroyCompleted();
}
}
//Returns true only when all threads are completed
private boolean isDone() { //code omitted }
//Returns number of not completed threads
private int getRemaining() { //code omitted }
//Clean up any completed Importers
private boolean destroyCompleted() { //code omitted }
}
Here is my Importer code
#Named
public class Importer implements Callable<String> {
#Setter //lombok annotation
private String filename;
public void cleanUp() {
log.info("PreDestroy - Importer");
//clean up omitted
}
public String call() {
List<String> content = //extract file content
for(int i = 0; i < content.size; i++) {
if (i+1 % 10000 == 0) {
log.info("Processed {} rows in file {}", i+1, filename);
}
//rest of loop omitted
}
}
}
The file test.txt has around 100,000 lines so I can tell that the crash happens mid file processing. So after my code runs for a few minutes I get the following at the end of my log
Processed 50000 rows in file test.txt
Processed 60000 rows in file text.txt
PreDestroy - Scheduler
PreDestroy - Importer
PreDestroy - Main
WELD-ENV-002001: Weld SE container 03128098-039d-46db-97e5-8538a52a38cc shut down
I am logging "org.jboss" at DEBUG level. I can see that the container was shutdown midway through processing a file, but I don't see why it occurred. Is there some type of listener that I can extend/implement to see why shut down command happened?
For future reference: You can define an observer method for the BeforeShutdown event which you can use to at least close all open files and such before shutdown. It won't necessarily give you a reason for why it's shutting down, but it's a start.
If you say the JVM process was terminated from outside, then there's probably an InterruptedException thrown somewhere which you can catch. Note that catching InterruptedException and not re-interrupting the thread is a bug most of the time. In your case, you should react to it with an orderly shutdown when calling Thread#sleep. If the thread gets interrupted, someone wants to cancel the execution. They probably have a good reason for doing so; don't think you know better than them!
At the very least you can periodically check the interrupt flag of your threads at some convenient point during execution to initiate an orderly shutdown of your task.
Related
I have an array of 32 cache objects and I am refreshing each of the cache periodically after every 20 seconds. Attached code snippet below.
I am observing that few caches stops refreshing after some refresh cycles and there is no exception printed in the logs. Note that application is not crashing, but some caches are not getting refreshed.
public class CacheManager
{
private static class CacheRefresher extends Thread
{
Cache cache;
public CacheRefresher(Cache cache)
{
this(cache);
}
#Override
public final void run()
{
try {
LOG.info("cache is getting refreshed for " + cache.type);
cache.refreshCache();
}
catch (Exception e) {
String subject = "Cache refresh failed in BMW";
LOG.log(Level.WARN, subject + ". Exception thrown:", e);
}
}
}
public void refreshCaches(List<cache> caches)
{
ThreadFactory ourThreadFactory =
new NamedThreadFactory("CacheHandler", true);
ScheduledExecutorService scheduleService =
Executors.newScheduledThreadPool(32, ourThreadFactory);
initialDelay = 60;
for (Cache cache : caches) {
service.scheduleWithFixedDelay(new CacheRefresher(cache), initialDelay, 20, TimeUnit.SECONDS);
initialDelay += 2;
cacheContainers.add(cache);
}
}
}
Is it possible that a thread is exceeding max heap limit and error is getting thrown but since I am not catching error I am not observing it?
But I am not sure if outofmemory error is thrown by one thread then will the entire application stop or only a thread will get impacted? Any comment?
In my Spring application, there is a scheduler for executing some task. Scheduled annotation is not used there because the schedule is quite complicated - it is dynamic and it used some data from the database. So simple endless cycle with thread sleeping is used. And sleeping interval is changed according to some rules. Maybe all this can be done with Scheduled annotation, but the question is not about that.
Below is simple example:
#Service
public class SomeService {
#PostConstruct
void init() {
new Thread(() -> {
while (true) {
System.out.println(new Date());
try {
Thread.sleep(1000);
} catch (Exception ex) {
System.out.println("end");
return;
}
}
}).start();
}
}
The code works fine but there is some trouble with killing that new thread. When I stop the application from Tomcat this new thread is continuing to run. So on Tomcat manage page I see that application is stopped, but in Tomcat log files I still see the output from the thread.
So what the problem? How I should change the code so the thread would be killed when the application is stopped?
Have you tried to implement a #PreDestroy method which will be invoked before WebApplicationContext is closed to change a boolean flag used in your loop? Though it seems strange that your objects are not discarded even when application is stopped...
class Scheduler {
private AtomicBoolean booleanFlag = new AtomicBoolean(true);
#PostConstruct
private void init() {
new Thread(() -> {
while (booleanFlag.get()) {
// do whatever you want
}
}).start();
}
#PreDestroy
private void destroy() {
booleanFlag.set(false);
}
}
I am trying to change Quartz Sequential execution to Parallel Execution.
It is working fine, Performance wise, it is seems good but Spawned (created) threads are not destroyed.
It is Still in Runnable State; why and How can I fix that?
Please Guide me.
Code is here :
#Override
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
logger.error("Result Processing executed");
List<Object[]> lstOfExams = examService.getExamEntriesForProcessingResults();
String timeZone = messageService.getMessage("org.default_timezone", null, Locale.getDefault());
if(lstOfExams!=null&&!lstOfExams.isEmpty()){
ThreadPoolTaskExecutor threadPoolExecuter = new ThreadPoolTaskExecutor();
threadPoolExecuter.setCorePoolSize(lstOfExams.size());
threadPoolExecuter.setMaxPoolSize(lstOfExams.size()+1);
threadPoolExecuter.setBeanName("ThreadPoolTaskExecutor");
threadPoolExecuter.setQueueCapacity(100);
threadPoolExecuter.setThreadNamePrefix("ThreadForUpdateExamResult");
threadPoolExecuter.initialize();
for(Object[] obj : lstOfExams){
if(StringUtils.isNotBlank((String)obj[2]) ){
timeZone = obj[2].toString();
}
try {
Userexams userexams=examService.findUserExamById(Long.valueOf(obj[0].toString()));
if(userexams.getExamresult()==null){
UpdateUserExamDataThread task=new UpdateUserExamDataThread(obj,timeZone);
threadPoolExecuter.submit(task);
}
// testEvaluator.generateTestResultAsPerEvaluator(Long.valueOf(obj[0].toString()), obj[4].toString(), obj[3]==null?null:obj[3].toString(),timeZone ,obj[5].toString() ,obj[1].toString());
// logger.error("Percentage Marks:::::"+result.getPercentageCatScore());
} catch (Exception e) {
Log.error("Exception at ResultProcessingJob extends QuartzJobBean executeInternal(JobExecutionContext context) throws JobExecutionException",e);
continue;
}
}
threadPoolExecuter.shutdown();
}
}
UpdateUserExamDataThread .class
#Component
//#Scope(value="prototype", proxyMode=ScopedProxyMode.TARGET_CLASS)
//public class UpdateUserExamDataThread extends ThreadLocal<String> //implements Runnable {
public class UpdateUserExamDataThread implements Runnable {
private Logger log = Logger.getLogger(UpdateUserExamDataThread.class);
#Autowired
ExamService examService;
#Autowired
TestEvaluator testEvaluator;
private Object[] obj;
private String timeZone;
public UpdateUserExamDataThread(Object[] obj,String timeZone) {
super();
this.obj = obj;
this.timeZone = timeZone;
}
#Override
public void run() {
String threadName=String.valueOf(obj[0]);
log.info("UpdateUserExamDataThread Start For:::::"+threadName);
testEvaluator.generateTestResultAsPerEvaluator(Long.valueOf(obj[0].toString()), obj[4].toString(), obj[3]==null?null:obj[3].toString(),timeZone ,obj[5].toString() ,obj[1].toString());
//update examResult
log.info("UpdateUserExamDataThread End For:::::"+threadName);
}
}
TestEvaluatorImpl.java
#Override
#Transactional
public Examresult generateTestResultAsPerEvaluator(Long userExamId, String evaluatorType, String codingLanguage,String timeZoneFollowed ,String inctenceId ,String userId) {
dbSchema = messageService.getMessage("database.default_schema", null, Locale.getDefault());
try {
//Some Methods
return examResult;
}catch(Exception e){
log.erorr(e);
}
}
I can provide Thread Dump file if needed.
it seems you create a thread pool in the same size of exams which is not quite optimal.
// Core pool size is = number of exams
threadPoolExecuter.setCorePoolSize(lstOfExams.size());
// Max pool size is just 1 + exam size.
threadPoolExecuter.setMaxPoolSize(lstOfExams.size()+1);
You have to consider that:
- If you create a thread pool and started it as many threads as defined in core size started immediately.
The max pool size is only than effective when you submit more than core pool threads can process right now AND when the queue size is full (in this case 100). So that means a new thread will be only then created when the number of submitted tasks exceeded 100+exam size.
In your case I would set the core pool size 5 or 10 (it actually depends on the how many core your target CPU have and/or how IO bound the submitted tasks are).
The max pool size can be double of that but it doesn't effective until the queue is full.
To let the size of live threads decrease after the submitted work done you have to set 2 parameters.
setKeepAliveSeconds(int keepAliveSeconds) : Which let the threads shut down automatically if they are not used along the defined seconds (by default 60 seconds, which is optimal) BUT this is normally only used to shut down threads of non-core pool threads.
To shut down threads of core part after keepAliveSeconds you have to set setAllowCoreThreadTimeOut(boolean allowCoreThreadTimeOut) as true. Which is normally false to keep core pool alive as long as the application is running.
I hope it helps.
I suspect that one of your threads waits indefinitely for an IO request answer. For example, you try to connect to a remote host where you did not set connection timeout and the host does not answer. In this case, you can shutdown all executing tasks forcefully by running shutdownNow method of the underlying ExecutorService then you can analyze InterruptedIOException thrown by the offending threads.
Replace
threadPoolExecuter.shutdown();
with below so you can examine errors.
ExecutorService executorService = threadPoolExecuter.getThreadPoolExecutor();
executorService.shutdownNow();
This will send interrupt signal to all running threads.
The threads do not wait on IO from some remote server, because the executed method on the threads would be in some jdbc driver classes, but they are currently all in UpdateUserExamDataThread.run(), line 37.
Now the question is: what is the code at UpdateUserExamDataThread.java line 37 ?
Unfortunately, the UpdateUserExamDataThread.java given at the moment is incomplete and/or not the version really executed: the package declaration is missing and it ends at line 29.
I suspect the issue is simply that you are calling run() instead of execute() when spawning the task thread using submit(). There is probably some expectation when using submit that threads kill themselves when the task is finished rather than terminating at the end of the run method.
Just Needed to increase the priority of threads and create number of threads as per number of cores in processor.
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
logger.error("Result Processing executed");
List<Object[]> lstOfExams = examService.getExamEntriesForProcessingResults();
String timeZone = messageService.getMessage("org.default_timezone", null, Locale.getDefault());
int cores = Runtime.getRuntime().availableProcessors();
if(lstOfExams!=null&&!lstOfExams.isEmpty()){
ThreadPoolTaskExecutor threadPoolExecuter = new ThreadPoolTaskExecutor();
threadPoolExecuter.setCorePoolSize(cores);
// threadPoolExecuter.setMaxPoolSize(Integer.MAX_VALUE);
threadPoolExecuter.setBeanName("ThreadPoolTaskExecutor");
// threadPoolExecuter.setQueueCapacity(Integer.MAX_VALUE);
threadPoolExecuter.setQueueCapacity(lstOfExams.size()+10);
threadPoolExecuter.setThreadNamePrefix("ThreadForUpdateExamResult");
threadPoolExecuter.setWaitForTasksToCompleteOnShutdown(true);
threadPoolExecuter.setThreadPriority(10);
threadPoolExecuter.initialize();
for(Object[] obj : lstOfExams){
if(StringUtils.isNotBlank((String)obj[2]) ){
timeZone = obj[2].toString();
}
try {
Userexams userexam=examService.findUserExamById(Long.valueOf(obj[0].toString()));
if(userexam.getExamresult()==null){
UpdateUserExamDataThread task=new UpdateUserExamDataThread(obj,timeZone,testEvaluator);
// threadPoolExecuter.submit(task);
threadPoolExecuter.execute(task);
}
// testEvaluator.generateTestResultAsPerEvaluator(Long.valueOf(obj[0].toString()), obj[4].toString(), obj[3]==null?null:obj[3].toString(),timeZone ,obj[5].toString() ,obj[1].toString());
// logger.error("Percentage Marks:::::"+result.getPercentageCatScore());
} catch (Exception e) {
logger.error("Exception at ResultProcessingJob extends QuartzJobBean executeInternal(JobExecutionContext context) throws JobExecutionException",e);
continue;
}
}
threadPoolExecuter.shutdown();
}
}
I tested a multi-thread program in JUnit and main function, source code as follows:
public class TestDaemon {
#Test
public void test() {
Thread thread = new Thread(() -> {
try {
Thread.sleep(1000);
System.out.println("hello");
} catch (InterruptedException e) {
e.printStackTrace();
}
});
// default false
thread.setDaemon(false);
thread.start();
}
public static void main(String[] args) {
Thread thread = new Thread(() -> {
try {
Thread.sleep(1000);
System.out.println("hello");
} catch (InterruptedException e) {
e.printStackTrace();
}
});
// default false
thread.setDaemon(false);
thread.start();
}
}
It didn't print hello string in the JUnit test example.
In the main function example, it could print hello in the console, but when I set the thread.setDaemon(true), it also can't print hello.
I know this is related to Daemon thred and User thread, but I don't know how to explain it.
A daemon thread is a thread that does not prevent the JVM from exiting when the program finishes but the thread is still running. An example for a daemon thread is the garbage collection.
When you run your code from main it creates both beans, thus two threads - daemon and non-daemon. As long as non-daemon thread is running, your application won't exit. So it works.
It's different when run from JUnit. As soon as JUnit test method completes (and it completes immediately after the Spring context is up), JUnit assumes your tests are done. Thus it kills all your threads and basically the whole JVM.
Remember your Waitor1 bean spawns a background thread which JUnit doesn't care about. As soon as you leave #Test method JUnit will just stop everything.
We can analyze the source code of JUnit, part of junit.textui.TestRunner as follows:
public static final int SUCCESS_EXIT = 0;
public static final int FAILURE_EXIT = 1;
public static final int EXCEPTION_EXIT = 2;
...
public static void main(String args[]) {
TestRunner aTestRunner = new TestRunner();
try {
TestResult r = aTestRunner.start(args);
if (!r.wasSuccessful()) {
System.exit(FAILURE_EXIT);
}
System.exit(SUCCESS_EXIT);
} catch (Exception e) {
System.err.println(e.getMessage());
System.exit(EXCEPTION_EXIT);
}
}
/**
* Returns whether the entire test was successful or not.
*/
public synchronized boolean wasSuccessful() {
return failureCount() == 0 && errorCount() == 0;
}
/**
* Gets the number of detected failures.
*/
public synchronized int failureCount() {
return fFailures.size();
}
/**
* Gets the number of detected errors.
*/
public synchronized int errorCount() {
return fErrors.size();
}
In this source code, we can conclude that the TestRunner excutes the Unit Test method, no need to wait it finish their tasks, then calls System.exit() method, so that terminates the program. So, it couldn't print hello in the console.
In the main function, because the new thread is not daemon thread, the main program will wait it finishing their tasks, then teminates the program. So,hellostring could be seen in the console.
I'm trying to use the new Concurrency API to inject a ManagedThreadFactory and use it per the Oracle tutorial.
Here is an example of what I'm talking about:
#Singleton
#Startup
public class Demo {
#Resource(name="concurrent/__DefaultManagedThreadFactory") ManagedThreadFactory threadFactory;
#PostConstruct
public void startup() {
threadFactory.newThread(
new Runnable() {
#Override
public void run() {
System.out.println("Do something.");
}
}
).start();
}
}
I'm developing in Eclipse using the Glassfish plugin. When I republish after making a change, I always get this line in the server log. It appears once for every call to start() we make:
SEVERE: java.lang.IllegalStateException: Module (my application) is disabled
It's not actually throwing an IllegalStateException, just reporting that one has been thrown (and caught) inside Glassfish. The application deploys normally, but none of the threads start. If I subsequently republish and second time, the "error" goes away and the threads start as expected.
When I try to deploy the application to a "real" Glassfish setup (without Eclipse), it always reports successful deployment, and the logs do not contain the "error". But it still does not start the threads (even with repeated deployments).
Am I using the Concurrency API correctly? Could it be a configuration problem? For the record, I get the same behavior if I use a ManagedExcecutorService instead.
For the record, this question was asked a few months ago here: Can I start a ManagedThread in a Singleton Enterprise Java Bean?, but it was not really answered and I don't have the reputation yet to do anything but ask it again. Sorry!
UPDATE: This answer by Per-Axel Felth works. Thank you! I did some refactoring of that solution to attempt to isolate the workaround code from my original application logic:
#Singleton
#Startup
public class Demo {
#Resource(name="java:comp/DefaultManagedThreadFactory") ManagedThreadFactory threadFactory;
#EJB private ConcurrencyInitializer concurrencyInitializer;
#EJB private Demo self;
#PostConstruct
public void startup() {
self.startThread();
}
#Asynchronous
public void startThread() {
//This line applies the workaround
concurrencyInitializer.init();
//Everything beyond this point is my original application logic
threadFactory.newThread(
new Runnable() {
#Override
public void run() {
System.out.println("Do something.");
}
}
).start();
}
}
/**
* A utility class used to get around a bug in Glassfish that allows
* Concurrency resources (ManagedThreadFactory, ManagedExecutorService, etc)
* to be injected before they are ready to be used.
*
* Derived from solution by Per-Axel Felth in: https://stackoverflow.com/questions/23900826/glassfish-4-using-concurrency-api-to-create-managed-threads
*/
#Singleton
public class ConcurrencyInitializer {
/**
* The number of milliseconds to wait before try to
*/
public static final long RETRY_DELAY = 500L;
/**
* The maximum number of concurrency attempts to make before failing
*/
public static final int MAX_RETRIES = 20;
/**
* Repeatedly attempts to submit a Runnable task to an injected ManagedExecutorService
* to trigger the readying of the Concurrency resources.
*
* #return true if successful (Concurrency resources are now ready for use),
* false if timed out instead
*/
public boolean init() {
final AtomicBoolean done = new AtomicBoolean(false);
int i = 0;
try {
while (!done.get() && i++ < MAX_RETRIES) {
executorService.submit(new Runnable() {
#Override
public void run() {
done.set(true);
}
});
Thread.sleep(RETRY_DELAY);
}
} catch(InterruptedException e) {
//Do nothing.
}
return done.get();
}
}
It's related to a Glassfish bug. I ran into the same bug myself some time ago and built a workaround. Thing is, the thread factory is injected alright, but if you use it "too early" you'll end up with an IllegalStateException.
My workaround code is listed below. It uses an injected executor service to detect when app is loaded and concurrency utils are available and then executes the actual startup logic in method init.
#Singleton
#Startup
public class Demo {
#Resource(name = "concurrent/__DefaultManagedThreadFactory")
ManagedThreadFactory threadFactory;
#Resource
ManagedExecutorService executorService;
#EJB
Demo me;
#PostConstruct
public void startup() {
me.waitAndInitialize();
}
#Asynchronous
public Future<?> waitAndInitialize() {
try {
final AtomicInteger done = new AtomicInteger(0);
int i = 0;
while (done.intValue() == 0 && i < 20) {
System.out.println("Is executor service up?");
i++;
executorService.submit(
new Runnable() {
#Override
public void run() {
int incrementAndGet = done.incrementAndGet();
System.out.println("Run by executorservice");
}
});
Thread.sleep(500);
}
if (done.intValue() == 0) {
Logger.getAnonymousLogger().severe("Waited a long time for the ExecutorService do become ready, but it never did. Will not initialize!");
} else {
init();
}
} catch (Exception e) {
Logger.getAnonymousLogger().log(Level.SEVERE, "Exception in waitAndInitialize: " + e.getMessage(), e);
}
return new AsyncResult<>(null);
}
private void init() {
threadFactory.newThread(
new Runnable() {
#Override
public void run() {
System.out.println("Do something.");
}
}
).start();
}
}
I suspect that your ManagedThreadFactory is not being injected correctly so the "Demo" component is not started.
The Java EE 7 spec mandates that a managed thread factory be made available in JNDI with the name "java:comp/DefaultManagedThreadFactory", therefore try changing #Resource to
#Resource(name="java:comp/DefaultManagedThreadFactory")
I'm not familiar with Glassfish (I'm a WildFly kind of guy) but you may not see this reference in any JNDI tree display. It may be linked internally to "concurrent/__DefaultManagedThreadFactory" (which is not a resource name btw).
Failing that you can also try
#Resource(lookup="concurrent/__DefaultManagedThreadFactory")