Develop Reference

How to retry an operation when it fails

I have singleton client with the below contract
public interface MQPublisher {
void publish(String message) throws ClientConnectionException, ClientErrorException;
void start() throws ClientException;
void stop();
}
The class which is using this publisher is as below :
public class MessagePublisher {
#Autowired
private MQPublisher publisher;
private AtomicBoolean isPublisherRunning;
public void startPublisher() {
if (!isPublisherRunning.get()) {
publisher.start();
isPublisherRunning.compareAndSet(false, true);
}
}
#Retry(RETRY_MSG_UPLOAD)
public void sendMessage(String msg) {
try {
startPublisher();
publisher.publish(msg); // when multiple requests fail with the same exception, what will happen??
} catch (Exception e) {
log.error("Exception while publishing message : {}", msg, e);
publisher.stop();
isPublisherRunning.compareAndSet(true, false);
throw e;
}
}
We are using resilience4j retry functionality to retry the sendMessage method. This works fine in case of a single request. Consider a case when multiple requests are processed parallely and all of them fails with an exception. In this case, these requests will be retried and there is a chance that one thread will start the publisher while the other will stop it and it will throw exceptions again. How to handle this scenario in a cleaner way?

It isn't clear why the whole publisher should be stopped in case of failure. Nevertheless, if there are real reasons for that, I would change the stop method to use an atomic timer that will restart on each message sending and stop the publisher only after at least 5 seconds (or the time needed for a message to be successfully sent) have passed from the message sending.
Something like that:
#Slf4j
public class MessagePublisher {
private static final int RETRY_MSG_UPLOAD = 10;
#Autowired
private MQPublisher publisher;
private AtomicBoolean isPublisherRunning;
private AtomicLong publishStart;
public void startPublisher() {
if (!isPublisherRunning.get()) {
publisher.start();
isPublisherRunning.compareAndSet(false, true);
}
}
#Retryable(maxAttempts = RETRY_MSG_UPLOAD)
public void sendMessage(String msg) throws InterruptedException {
try {
startPublisher();
publishStart.set(System.nanoTime());
publisher.publish(msg); // when multiple requests fail with the same exception, what will happen??
} catch (Exception e) {
log.error("Exception while publishing message : {}", msg, e);
while (System.nanoTime() < publishStart.get() + 5000000000L) {
Thread.sleep(1000);
}
publisher.stop();
isPublisherRunning.compareAndSet(true, false);
throw e;
}
}
}
I think it is important to mention (as you just did) that this is a terrible design, and that such calculations should be done by the publisher implementer and not by the caller.

Cannot receive another websocket message after interruption of thread with websocket session

I have problem with receiving new messages from my JS client after I use the first received message in new thread and cancel the thread. I use spring boot for the back end. The interesting thing is that the session is not closed but I just cannot receive any more messages after the first one when I interrupt my thread.
Here is my websocket config:
#Configuration
#EnableWebSocket
public class WebSocketConfiguration implements WebSocketConfigurer {
#Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(new WebSocketHandler(), "/socket1").setAllowedOrigins("*");
}
}
Here is my handler with the executor service:
public class WebSocketHandler extends AbstractWebSocketHandler {
Logger logger = LoggerFactory.getLogger(WebSocketHandler.class);
Test test = new Test();
#Override
protected void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
logger.info("START");
List<UrlWithPageNumber> listings = new ArrayList<>();
listings.add(new UrlWithPageNumber( "www.somesite.com", 1));
listings.add(new UrlWithPageNumber( "www.anothersite.com", 1));
listings.add(new UrlWithPageNumber( "www.thirdsite.com", 1));
checkItemsAsync(listings, session);
logger.info("DONE");
session.sendMessage(new TextMessage("DONE"));
}
public void checkItemsAsync(List<UrlWithPageNumber> listings, WebSocketSession session) {
ExecutorService executorService = Executors.newFixedThreadPool(1);
final CountDownLatch latch = new CountDownLatch(listings.size());
for (UrlWithPageNumber listing : listings) {
executorService.submit(() -> {
if(Test.stop) {
return;
}
ListingInfo listingInfo = test.itemPage(listing.getLink(), 1, 1);
logger.info(listingInfo.toString());
synchronized(session) {
try {
session.sendMessage(new TextMessage(listingInfo.toString()));
} catch (IOException e) {
e.printStackTrace();
}
}
latch.countDown();
});
}
try {
latch.await();
} catch (InterruptedException ie) {
throw new RuntimeException(ie);
}
}
}
Now the problem is the following: I call my handleTextMessage method from JS client and the execution is starting then I press another button in my page and it is changing Test.stop boolean flag to true and in that way i stop the remaining threads to be executed. After that if I call handleTextMessage from the js like the first time it is not called. I checked if I close the WebSocketSession and then try to call the backend and the result is similar but in my case the session is not closed for sure ! The question is how to use the WebSocketSession many times and why the session is become broken when I use it in my executor service and stop the thread? Should I do something with the session if I shutdown threads or this is just some spring boot bug ?

The problem is in the latch it wait more than expected because I skip some threads..

Vertx.io cluster and service discovery

I am playing with vertx.io, it looks great. Now I build up a cluster of three verticles (three simple java main fat jars). One verticle expose a web interface (a poorly rest api), the other two simply awares the web verticle that they are up or down by the vertx.io's service discovery mechanism.
Here is my (relevant part) simple "non web" verticle:
public class FileReader extends AbstractVerticle {
private ServiceDiscovery discovery;
private Logger log = LogManager.getLogger(getClass());
private Record record;
#Override
public void start(Future<Void> startFuture) throws Exception {
record = EventBusService.createRecord(getServiceName(), getServiceAddress(), getClass());
setUpRecord(record);
discovery = ServiceDiscovery.create(vertx);
discovery.publish(record, h -> {
if (h.succeeded()) {
log.info("Record published.");
} else {
log.info("Record not published.", h.cause());
}
});
startFuture.complete();
}
...
#Override
public void stop(Future<Void> stopFuture) throws Exception {
log.info("Stopping verticle.");
discovery.unpublish(record.getRegistration(), h -> {
if (h.succeeded()) {
log.info("Service unpublished.");
stopFuture.complete();
} else {
log.error(h.cause());
stopFuture.fail(h.cause());
}
});
}
}
And here is how I deploy one of the two "non web" verticles:
public class FileReaderApp {
private static Logger log = LogManager.getLogger(FileReaderApp.class);
private static String id;
public static void main(String[] args) {
ClusterManager cMgr = new HazelcastClusterManager();
VertxOptions vOpt = new VertxOptions(new JsonObject());
vOpt.setClusterManager(cMgr);
Vertx.clusteredVertx(vOpt, ch -> {
if (ch.succeeded()) {
log.info("Deploying file reader.");
Vertx vertx = ch.result();
vertx.deployVerticle(new FileReader(), h -> {
if (h.succeeded()) {
id = h.result();
} else {
log.error(h.cause());
}
});
} else {
log.error(ch.cause());
}
});
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
log.info("Undeploying " + id);
Vertx.vertx().undeploy(id, h -> {
if (h.succeeded()) {
log.info("undeployed.");
} else {
log.error(h.cause());
}
});
}
});
}
}
When "non-web" verticles start, the "web" verticle is correctly notified. But when "non-web" verticles shutdown, I hit a keyboard Ctrl-C, I got this error and "web" verticle still think everyone is up:
2017-12-01 09:08:27 INFO FileReader:31 - Undeploying 82a8f5c2-e6a2-4fc3-84ff-4bb095b5dc43
Exception in thread "Thread-3" java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66)
at java.lang.Runtime.addShutdownHook(Runtime.java:211)
at io.vertx.core.impl.FileResolver.setupCacheDir(FileResolver.java:310)
at io.vertx.core.impl.FileResolver.<init>(FileResolver.java:92)
at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:185)
at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:144)
at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:140)
at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:34)
at io.vertx.core.Vertx.vertx(Vertx.java:82)
at edu.foo.app.FileReaderApp$1.run(FileReaderApp.java:32)
I don't fully get what's going on. Application shutdown while it was undeploying verticle? How to solve this? What is the vertx.io approach?

There are two problems
You should undeploy the verticle using the clustered Vert.x instance, not just any instance
undeploy is a non blocking operation so the shutdown hook thread must wait for completion.
Here's a modified version:
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
log.info("Undeploying " + id);
CountDownLatch latch = new CountDownLatch(1);
theClusteredVertxInstance.undeploy(id, h -> {
if (h.succeeded()) {
log.info("undeployed.");
} else {
log.error(h.cause());
}
latch.countDown();
});
try {
latch.await(5, TimeUnit.SECONDS);
} catch(Exception ignored) {
}
}
});

Subscriber.stopAsync() results in RejectedExecutionException

My code is basically following the official tutorials and the main purpose is to collect all messages from one subscription (Constants.UNFINISHEDSUBID) and republish them on another. But currently I'm facing a problem, that i can't solve. In my implementation calling subscriber.stopAsync() results in the following exception:
Mai 04, 2017 4:59:25 PM com.google.common.util.concurrent.AbstractFuture executeListener
SCHWERWIEGEND: RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6#6e13e898 with executor java.util.concurrent.Executors$DelegatedScheduledExecutorService#2f3c6ac4
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#60d40af2 rejected from java.util.concurrent.ScheduledThreadPoolExecutor#d55b6e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 320]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622)
at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:613)
at io.grpc.stub.ClientCalls$GrpcFuture.set(ClientCalls.java:458)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:437)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I also noticed that kind of randomly, sometimes all messages and sometimes just a few or not a single one get collected. Is calling subscriber.stopAsync() not the correct way?
My current implementation:
protected void pullUnfinished() throws Exception {
List<PubsubMessage> jobsToRepublish = new ArrayList<>();
SubscriptionName subscription =
SubscriptionName.create(Constants.PROJECTID, Constants.UNFINISHEDSUBID);
MessageReceiver receiver = new MessageReceiver() {
#Override
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
synchronized(jobsToRepublish){
jobsToRepublish.add(message);
}
String unfinishedJob = message.getData().toStringUtf8();
LOG.info("got message: {}", unfinishedJob);
consumer.ack();
}
};
Subscriber subscriber = null;
try {
ChannelProvider channelProvider = new PlainTextChannelProvider();
subscriber = Subscriber.defaultBuilder(subscription, receiver)
.setChannelProvider(channelProvider)
.build();
subscriber.addListener(new Subscriber.Listener() {
#Override
public void failed(Subscriber.State from, Throwable failure) {
System.err.println(failure);
}
}, MoreExecutors.directExecutor());
subscriber.startAsync().awaitRunning();
Thread.sleep(60000);
} finally {
if (subscriber != null) {
subscriber.stopAsync(); //Causes the exception
}
}
publishJobs(jobsToRepublish);
}
public class PlainTextChannelProvider implements ChannelProvider {
#Override
public boolean shouldAutoClose() {
// TODO Auto-generated method stub
return false;
}
#Override
public boolean needsExecutor() {
// TODO Auto-generated method stub
return false;
}
#Override
public ManagedChannel getChannel() throws IOException {
return NettyChannelBuilder.forAddress("localhost", 8085)
.negotiationType(NegotiationType.PLAINTEXT)
.build();
}
#Override
public ManagedChannel getChannel(Executor executor) throws IOException {
return getChannel();
}
}

I had the exact same issue when running similar code from a JUnit test, and found this related answer on multithreading in general, suggesting that a ThreadPool is closed while Listeners are still referring to it. I also looked into the code of Subscriber.java on GitHub, and found an example for receiving a number of messages in the JavaDoc on startAsync(), suggesting to wait for termination of stopAsync().
Try changing
subscriber.stopAsync();
to
subscriber.stopAsync().awaitTerminated();
Worked for me.

How to make JUnit4 "Wait" for asynchronous job to finish before running tests

I am trying to write a test for my android app that communicates with a cloud service.
Theoretically the flow for the test is supposed to be this:
Send request to the server in a worker thread
Wait for the response from the server
Check the response returned by the server
I am trying to use Espresso's IdlingResource class to accomplish that but it is not working as expected. Here's what I have so far
My Test:
#RunWith(AndroidJUnit4.class)
public class CloudManagerTest {
FirebaseOperationIdlingResource mIdlingResource;
#Before
public void setup() {
mIdlingResource = new FirebaseOperationIdlingResource();
Espresso.registerIdlingResources(mIdlingResource);
}
#Test
public void testAsyncOperation() {
Cloud.CLOUD_MANAGER.getDatabase().getCategories(new OperationResult<List<Category>>() {
#Override
public void onResult(boolean success, List<Category> result) {
mIdlingResource.onOperationEnded();
assertTrue(success);
assertNotNull(result);
}
});
mIdlingResource.onOperationStarted();
}
}
The FirebaseOperationIdlingResource
public class FirebaseOperationIdlingResource implements IdlingResource {
private boolean idleNow = true;
private ResourceCallback callback;
#Override
public String getName() {
return String.valueOf(System.currentTimeMillis());
}
public void onOperationStarted() {
idleNow = false;
}
public void onOperationEnded() {
idleNow = true;
if (callback != null) {
callback.onTransitionToIdle();
}
}
#Override
public boolean isIdleNow() {
synchronized (this) {
return idleNow;
}
}
#Override
public void registerIdleTransitionCallback(ResourceCallback callback) {
this.callback = callback;
}}
When used with Espresso's view matchers the test is executed properly, the activity waits and then check the result.
However plain JUNIT4 assert methods are ignored and JUnit is not waiting for my cloud operation to complete.
Is is possible that IdlingResource only work with Espresso methods ? Or am I doing something wrong ?

I use Awaitility for something like that.
It has a very good guide, here is the basic idea:
Wherever you need to wait:
await().until(newUserIsAdded());
elsewhere:
private Callable<Boolean> newUserIsAdded() {
return new Callable<Boolean>() {
public Boolean call() throws Exception {
return userRepository.size() == 1; // The condition that must be fulfilled
}
};
}
I think this example is pretty similar to what you're doing, so save the result of your asynchronous operation to a field, and check it in the call() method.

Junit will not wait for async tasks to complete. You can use CountDownLatch to block the thread, until you receive response from server or timeout.
Countdown latch is a simple yet elegant solution and does NOT need an external library. It also helps you focus on the actual logic to be tested rather than over-engineering the async wait or waiting for a response
void testBackgroundJob() {
Latch latch = new CountDownLatch(1);
//Do your async job
Service.doSomething(new Callback() {
#Override
public void onResponse(){
ACTUAL_RESULT = SUCCESS;
latch.countDown(); // notify the count down latch
// assertEquals(..
}
});
//Wait for api response async
try {
latch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
assertEquals(expectedResult, ACTUAL_RESULT);
}