I'm trying to figure out a way to implement an asynchronous retry mechanism using Hazelcast IExecutorService without recursive calls:
The recursive solution looks like that:
Callable task = ...
private sendToExecutor(){
Future future = submitToExecutorService(task);
((ICompletableFuture<ActionReply>) future).andThen(callback);
}
The callback is an ExecutionCallback:
#Override
public void onResponse(ActionReply response) {
// normal stuff
}
#Override
public void onFailure(Throwable t) {
// re-send if possible
if(numRetries < max_retries){
sendToExecutor();
}
}
I'm struggling to find a nice solution that does not involve the recursion. Any help will be appreciated. Thanks!
Create a wrapper class that implements Future and implement the get method which should catch RetryableHazelcastException. Note that you need have a limit on number of retries. If it crosses that limit means, there some major problem with your cluster.
public class RetryableFuture implements Future {
//Implement other methods.
#Override
public Object get() throws InterruptedException, ExecutionException {
try{
//get operation on future object
}catch(ExecutionException e){
if(e.getCause() instanceof RetryableHazelcastException){
//Log some warnings and submit the task back to Executors
}
}catch(Exception e){
//Not all exceptions are retryable
}finally {
//Close any kind of resources.
}
}
}
Related
I want to check the status of sending records to Kafka. I need to make sure that all records were successfully stored in Kafka. I was thinking about using the callback mechanism, e.g. creating a callback class like
public class MyCallback implememts Callback {
private AtomicReference<Exception> exceptionRef;
public MyCallback(){
exceptionRef=new AtomicReference<>(null);
}
#Override
public void onCompletion(final RecordMetadata metadata,final Exception exception){
if (exception!=null){
exceptionRef.set(exception);
}
}
public void check()
throws Exception
{
Exception exception=exceptionRef.get();
if (exception!=null) throw exception;
}
}
and then have a main program like
try{
Producer<Object,Object> producer=new KafkaProducer<>(props);
MyCallback callback=new MyCallback();
for (ProducerRecord<Object,Object> rec:myRecords){
producer.send(rec,callback);
}
producer.flush();
callback.check();
}
catch(Exception e){
handle(e);
}
My question: Can I be sure that the callback has been called for all sent records when flush() returns?
I should add that the setting acks=all is used.
I found that the answer is "yes". According to the javadoc:
The post-condition of flush() is that any previously sent record will have completed (e.g. Future.isDone() == true).
The question is then "Has the callback been called before the future completes?"
To answer that, you need to go into the internals code, i.e. the package
org.apache.kafka.clients.producer.internals. The future itself is implemented in the class FutureRecordMetadata, and the method is:
#Override
public boolean isDone() {
if (nextRecordMetadata != null)
return nextRecordMetadata.isDone();
return this.result.completed();
}
So we learn that the futures are linked, and the actual completion is delegated to the result, which is a ProduceFuture.
If you look in the class ProducerBatch, you will see that a ProduceFuture is on the batch level, in other words, a future on the record level is done, when the future of its batch is completed. Now looking at what sets batch level future to completed, we find that this method in ProducerBatch does that:
private void completeFutureAndFireCallbacks(long baseOffset, long logAppendTime, RuntimeException exception) {
// Set the future before invoking the callbacks as we rely on its state for the `onCompletion` call
produceFuture.set(baseOffset, logAppendTime, exception);
// execute callbacks
for (Thunk thunk : thunks) {
try {
if (exception == null) {
RecordMetadata metadata = thunk.future.value();
if (thunk.callback != null)
thunk.callback.onCompletion(metadata, null);
} else {
if (thunk.callback != null)
thunk.callback.onCompletion(null, exception);
}
} catch (Exception e) {
log.error("Error executing user-provided callback on message for topic-partition '{}'", topicPartition, e);
}
}
produceFuture.done();
}
The thunks are the record level futures, so it is clear that the callbacks are called prior to setting the future to done.
The issue is that I have a method starting a new thread for a time-consuming work. I want to test the callback result, but the child thread may still running, so as a result, what I get is not the right stub.
I think the code may explain itself:
public class JustAClass {
//it is a callback for async
public interface JustACallBack {
void callFunc(JustAResult result);
}
//this is the result interface
public interface JustAResult {
}
//this is a real class for the interface
public class JustAResultReal implements JustAResult{
public JustAResultReal(String content) {this.content = content;}
public String content;
}
//here is the key function
public void threadFunc(final JustACallBack callBack) {
BCCache.executorService.execute(new Runnable() {
#Override
public void run() {
//just to simulate a time-consuming task
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
//now we callback
callBack.callFunc(new JustAResultReal("can you reach me"));
}
});
}
}
and the test function could be(I am using mockito):
#Test
public void testThreadFunc() throws Exception {
JustAClass justAClass = new JustAClass();
JustAClass.JustACallBack callBack = Mockito.mock(JustAClass.JustACallBack.class);
justAClass.threadFunc(callBack);
//add this line, we can get the expected result
Thread.sleep(1200);
Mockito.verify(callBack).callFunc(captor.capture());
System.out.println(((JustAClass.JustAResultReal)captor.getValue()).content);
}
I know we can add a sleep to wait and expect that the child thread would exit within the period, but could there be a better way? Actually how could I know how long the child thread would take? Setting a very long time can be an approach but just seems not very nice.
The general approach in #stalet's answer is close, but doesn't quite work since any assertion failures from a separate thread are not noticed by the main thread. Therefore your test will always pass, even when it shouldn't. Instead, try using ConcurrentUnit (which I authored):
#Test
public void testInvoke() throws Throwable {
Waiter waiter = new Waiter();
JustAClass justAClass = new JustAClass();
JustAClass.JustACallBack callBack = new JustAClass.JustACallBack() {
#Override
public void callFunc(final JustAClass.JustAResult result) {
waiter.assertNotNull(result);
waiter.assertTrue(result instanceof JustAClass.JustAResultReal);
waiter.resume();
}
};
justAClass.threadFunc(callBack);
waiter.await(1200, TimeUnit.SECONDS);
}
The key here is ConcurrentUnit's Waiter will properly report any assertions failures to the main test thread and the test will pass or fail as it should.
I aggree with #Gimbys comment about this is no longer a unit-test when you start testing the the threading aspect.
Nevertheless it is interesting as a way to integration-test a asynchronous invokation.
To avvoid sleep i tend to use the class CountDownLatch to wait for invokations.
In order to count down you need an actuall implementation of the callback interface - so in my example I have made a mock implementation of this.
Since there is no actual methods to fetch the data - i am just testing that it is in fact a instance of the JustAReal interface.
#Test
public void testInvoke() throws Exception {
final CountDownLatch countDownLatch = new CountDownLatch(1); //1 is how many invokes we are waiting for
JustAClass justAClass = new JustAClass();
JustAClass.JustACallBack callBack = new JustAClass.JustACallBack() {
#Override
public void callFunc(final JustAClass.JustAResult result) {
assertNotNull("Result should not be null", result);
assertTrue("Result should be instance of JustAResultReal", result instanceof JustAClass.JustAResultReal);
countDownLatch.countDown();
}
};
justAClass.threadFunc(callBack);
if(!countDownLatch.await(1200, TimeUnit.MILLISECONDS)){
fail("Timed out, see log for errors");
}
}
In the disruptor (version 3.3.2), each event is a Runnable (since EventProcessor extends runnable).
I am writing an application that whenever an EventHandler throws an exception, the class who calls disruptor.start() needs to catch the exception and then react.
Now, if an EventProcessor would have been a Callable, that would have been easy.
Is there another way in Disruptor to propagate the exception?
I solved the problem by passing an implementation interface to the EventHandler,
as suggested by the Doug Lea book. The exception is set in a LinkedList, and at the end of the method call I retrieve the last element in the list. Sample code:
final LinkedList<Throwable> listExceptions = new LinkedList<Throwable>();
MyClassWithDisruptor at = MyClassWithDisruptor.getInstance();
at.send(message, transport, conf, new AuditExceptionHandler() {
#Override
public void handleException(final Throwable e) {
e.printStackTrace();
}
#Override
public void setException(final Exception e) throws AuditTrailException {
listExceptions.add(e);
}
});
The disruptor provides an ExceptionHandler which intended for dealing with this sort of problem.
I have a ScheduledExecutorService, which invokes a Runnable periodically via scheduleWithFixedDelay() (could have used scheduleAtFixedRate() instead).
Am now considering what to do if an error occurs. If it's something that can't easily be recovered from(*) I'd like the option of stopping all further invocations but not sure of the best way of doing this.
Apparently checked exceptions can't be thrown from a Runnable so would appreciate any guidance on how to choose from the following:
scheduledFuture.cancel(false);
...or...
scheduledFuture.cancel(true);
...or...
scheduledExecutorService.shutdown();
...or...
scheduledExecutorService.shutdownNow();
...or...
Throw a custom RuntimeException myself?
...or...
Something else?
(*) Would like to know the general case but in case anyone's interested, the checked exception I'm currently looking at is a ParserConfigurationException thrown from DocumentBuilderFactory.newDocumentBuilder(). If this is thrown, it indicates a serious problem so I'd basically like the scheduling to completely stop rather than potentially repeating the error every time.
You can use a Callable along with the Future perhaps. This will let you throw a checked exception from within an asynchronous task, yet still catch and handle as needed per task.
If you use that approach, then allowing the task itself to decide how to handle the exception probably makes the most sense. See this answer:
https://stackoverflow.com/a/4910682/123378 .
However, if you want to handle the exception outside of the task itself, then I think you will need another thread for each task. Here's one possible option:
ScheduledExecutorService scheduleExecutor;
scheduleExecutor = = Executors.newScheduledThreadPool(10); // or whatever
ExecutorService workerExecutor;
workerExecutor = Executors.newSingleThreadExecutor(); // or whatever
public void schedule(final long fixedDelay) {
scheduleExecutor.scheduleWithFixedDelay(new Runnable() {
#Override
public void run() {
Future<Void> future = workerExecutor.submit(new Callable<Void>() {
#Override
public Void call() throws Exception {
// Do work here. Throw appropiate exception as needed.
return null;
}
});
// Now you can catch and handle the exception in whatever
// way you need to. You can cancel just this task (which is likely
// redundant by this point), or you can choose to shutdown
// all other scheduled tasks (which I doubt is what you want).
try {
future.get();
} catch (Exception e) {
future.cancel(true);
}
}
}, 0, fixedDelay, TimeUnit.MILLISECONDS);
}
Based on a few of the helpful comments above here's the gist of my current code - a few q's remain within so would welcome any further comments:
public class ScheduledTask implements Runnable {
// Configurable values
private static final int CORE_THREAD_POOL_SIZE = 1;
private static final int INITIAL_DELAY_MS = 0;
private static final int INTERVAL_MS = 1000;
private final ScheduledExecutorService scheduledExecutorService =
Executors.newScheduledThreadPool(ScheduledTask.CORE_THREAD_POOL_SIZE);
private ScheduledFuture<?> scheduledFuture;
public void run() {
try {
try {
// Do stuff
} catch RecoverableCheckedException rce { // E.g. SAXException
// Log and handle appropriately
}
} catch UnrecoverableCheckedException uce { // E.g. ParserConfigurationException
// Not 100% happy with this. It means the caller would need to call
// getCause() to get the real Exception in this case. But other
// RuntimeExceptions wouldn't be wrapped. Could consider catching
// and wrapping all RuntimeExceptions but I like that even less!
throw new RuntimeException(uce);
}
}
public boolean isScheduling() {
return (this.scheduledFuture != null)
&& (!this.scheduledFuture.isDone());
}
// May not be needed but provided in case this class is shared.
public boolean isShutdown() {
return scheduledExecutorService.isShutdown();
}
public void start() {
// If the Executor Service has already been shutdown, would expect
// a RejectedExecutionException to be thrown here(?) Not sure what
// would happen if this method were called when isScheduling() is
// true?
this.scheduledFuture =
this.scheduledExecutorService.scheduleWithFixedDelay(
this,
ScheduledTask.INITIAL_DELAY_MS,
ScheduledTask.INTERVAL_MS,
TimeUnit.MILLISECONDS);
}
// To be called once at the very end - e.g. on program termination.
public void shutdown() {
this.scheduledExecutorService.shutdown();
}
}
What I want is a standard JDK class that look like this:
interface ExplodingRunnable {
void run() throws Exception;
}
Callable is no good, because its call() method is required to return a value, but I need a void.
Runnable is no good, because its run() method doesn't declare throws Exception.
I sort of need a combination of the two. Any ideas?
EDIT: I should have mentioned that I tried Callable<Void> but it requires you to define a method:
public Void call() {
// run your code;
return null; // ugly!
}
I'm looking for something a bit nicer.
Why do I want this?
I'm implementing a standard why of catching "will never happen" Exceptions (they will never happen, but various APIs define their methods throwing Exceptions) and throwing any Exceptions that might occur by wrapping them in an (unchecked) RuntimeException, so the caller can simply pass a "ExplodingRunnable" in and not have to code loads of perfunctory try/catch blocks that will never be exercised.
FINAL EDIT It looks like what I was looking for doesn't exist. The accepted answer is the closest to "correct", but it looks like there is no solution to answer the question as asked.
Could you just use Callable<Void>?
An interface with only one method, which returns void and throws Exception.
Among all java and javax classes, only one fits that description:
package java.lang;
public interface AutoCloseable
{
void close() throws Exception;
}
Well... the word "close" has many meanings...
You want to surround a bunch of statements with some extra handling, there is no sin to define your own interface here. You may find that your API requires users to learn 4 new phrases
Util.muckException( new ExplodingRunnable() { public void run() throws Exception
^1 ^2 ^3 ^4
You can actually cut down two, and user code would look like this
new MuckException(){ public void run() throws Exception
{
statement_1;
...
statement_n;
}};
public abstract class MuckException
{
public abstract run() throws Exception;
public MuckException()
{
try{ run(); }
catch(Exception e){ throw new Error(e); }
}
}
Just use Callable, ignore the return value and document things as ignoring the returned value and recommend returning null. Just because you can return something does not mean you have to.
I would just use Callable<Void> and learn to love it. ;)
You can have the checked exception not declared with the following.
Runnable runs = new Runnable() {
public void run() {
try {
// do something
} catch(Exception e) {
// rethrows anything without the compiler knowing.
// the method is deprecated but can be used on the current thread.
Thread.currentThread().stop(e);
}
}
});
Future future = executorService.submit(run);
try {
future.get();
} catch (ExecutionException ee) {
Throwable e = ee.getCause(); // can be the checked exception above.
}
and not have to code loads of perfunctory try/catch blocks that will never be exercised.
I had the same issue and fixed it a little differently
// Exceptions class
public RuntimeException wrap(Exception e) {
return e instanceof RuntimeException ? ((RuntimeException)e) : new RuntimeException(e);
}
// user code
try {
foo.bar();
} catch (Exception e) {
throw Exceptions.wrap(e);
}