Develop Reference

Kinesis 2.2.11 java unable to create consumer

I need to migrate to Kinesis library to version 2.2.11 so I followed the tutorial: https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html
I need to run multiple instances of my consumer app, so every one of them needs to have an unique application name in order to have a separate lease table in DynamoDb.
When initializing the consumer Kinesis runs DynamoDBLeaseRefresher.createLeaseTableIfNotExists which checks if a new table needs to be created for this application name and creates one if it cannot be found.
So 2 operations are performed:
DescribeTable - it returns the table info or throws a ResourceNotFoundExecption,
if needed - CreateTable.
The problem for me is with the DescribeTable method. When I am looking for an existing table it returns it with no problem. But when I am looking for a non-existent table it throws the ResourceNotFoundExecption -> so far so good. Unfortunately it then gets wrapped and is now:
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: software.amazon.awssdk.awscore.exception.AwsServiceException$Builder.extendedRequestId(Ljava/lang/String;)Lsoftware/amazon/awssdk/awscore/exception/AwsServiceException$Builder;
and the app expecting ResourceNotFoundException gets something different instead and crashes.
The wrapped exception message is a bit misleading: "Unable to execute HTTP request" since the request was performed and returned the proper message: "Resource not found".
Funny thing is that it sometimes works, the exception does not get wrapped, the CreateTable operation is performed and the consumer starts properly.
I have made a workaround for it for now where I just create the table before the initialization of the LeaseCoordinator, so it always gets the existing table.
here is my code:
public KinesisStreamReaderService(String streamName, String applicationName, String regionName) {
KinesisAsyncClient kinesisClient = KinesisAsyncClient.builder()
.credentialsProvider(EnvironmentVariableCredentialsProvider.create())
.region(Region.of(connectionProperties.getRegion()))
.httpClientBuilder(createHttpClientBuilder())
.build();
DynamoDbAsyncClient dynamoClient = DynamoDbAsyncClient.builder().region(Region.of(regionName)).build();
CloudWatchAsyncClient cloudWatchClient = CloudWatchAsyncClient.builder().region(Region.of(regionName)).build();
// if(!dynamoDbTableExists(dynamoClient, applicationName)) {
// createDynamoDbTable(dynamoClient, applicationName);
// }
ConfigsBuilder configsBuilder = new ConfigsBuilder(streamName, applicationName, kinesisClient,
dynamoClient, cloudWatchClient, workerId(), KinesisReaderProcessor::new);
configsBuilder.retrievalConfig().initialPositionInStreamExtended(
InitialPositionInStreamExtended.newInitialPosition(
InitialPositionInStream.LATEST));
scheduler = new Scheduler(
configsBuilder.checkpointConfig(),
configsBuilder.coordinatorConfig(),
configsBuilder.leaseManagementConfig(),
configsBuilder.lifecycleConfig(),
configsBuilder.metricsConfig(),
configsBuilder.processorConfig(),
configsBuilder.retrievalConfig().retrievalSpecificConfig(new PollingConfig(streamName, kinesisClient))
);
}
private void createDynamoDbTable(DynamoDbAsyncClient dynamoClient, String applicationName) {
log.info("Creating new lease table: {}", applicationName);
CompletableFuture<CreateTableResponse> createTableFuture = dynamoClient
.createTable(CreateTableRequest.builder()
.provisionedThroughput(ProvisionedThroughput.builder().readCapacityUnits(10L).writeCapacityUnits(10L).build())
.tableName(applicationName)
.keySchema(KeySchemaElement.builder().attributeName("leaseKey").keyType(KeyType.HASH).build())
.attributeDefinitions(AttributeDefinition.builder().attributeName("leaseKey").attributeType(
ScalarAttributeType.S).build())
.build());
try {
CreateTableResponse createTableResponse = createTableFuture.get();
log.debug("Created new lease table: {}", createTableResponse.tableDescription().tableName());
} catch (InterruptedException | ExecutionException e) {
throw new DataStreamException(e.getMessage(), e);
}
}
private boolean dynamoDbTableExists(DynamoDbAsyncClient dynamoClient, String tableName) {
CompletableFuture<DescribeTableResponse> describeTableResponseCompletableFutureNew = dynamoClient
.describeTable(DescribeTableRequest.builder()
.tableName(tableName).build());
try {
DescribeTableResponse describeTableResponseNew = describeTableResponseCompletableFutureNew
.get();
return nonNull(describeTableResponseNew);
} catch (InterruptedException | ExecutionException e) {
log.info(e.getMessage(), e);
}
return false;
}
private static String workerId() {
String workerId;
try {
workerId = format("%s_%s", getLocalHost().getCanonicalHostName(), randomUUID().toString());
} catch (UnknownHostException e) {
workerId = randomUUID().toString();
}
return workerId;
}
#Override
public void read(Consumer<String> consumer) {
this.consumer = consumer;
scheduler.run();
}
private class KinesisReaderProcessor implements ShardRecordProcessor {
private String shardId;
#Override
public void initialize(InitializationInput initializationInput) {
this.shardId = initializationInput.shardId();
log.info("Initializing record processor for shard: {}", shardId);
}
#Override
public void processRecords(ProcessRecordsInput processRecordsInput) {
log.debug("Checking shard {} for new records", shardId);
List<KinesisClientRecord> records = processRecordsInput.records();
if (!records.isEmpty()) {
log.debug("Processing {} records from kinesis stream shard {}", records.size(), shardId);
records.forEach(record -> {
String json = UTF_8.decode(record.data()).toString();
log.info(json);
consumer.accept(json);
});
}
}
#Override
public void leaseLost(LeaseLostInput leaseLostInput) {
log.info("Record processor has lost lease, terminating");
}
#Override
public void shardEnded(ShardEndedInput shardEndedInput) {
try {
shardEndedInput.checkpointer().checkpoint();
} catch (ShutdownException | InvalidStateException e) {
log.error(e.getMessage(), e);
}
}
#Override
public void shutdownRequested(ShutdownRequestedInput shutdownRequestedInput) {
try {
shutdownRequestedInput.checkpointer().checkpoint();
} catch (ShutdownException | InvalidStateException e) {
log.error(e.getMessage(), e);
}
}
}
}
Am I missing some configuration for the scheduler or something? Why is it sometimes working?
Thanks
Edit:
The problem is this block of code in DynamoDBLeaseRefresher.tableStatus() is invoked to check if the table exists:
DescribeTableResponse result;
try {
try {
result =
(DescribeTableResponse)FutureUtils.resolveOrCancelFuture(this.dynamoDBClient.describeTable(request), this.dynamoDbRequestTimeout);
} catch (ExecutionException var5) {
throw exceptionManager.apply(var5.getCause());
} catch (InterruptedException var6) {
throw new DependencyException(var6);
}
} catch (ResourceNotFoundException var7) {
log.debug("Got ResourceNotFoundException for table {} in leaseTableExists, returning false.", this.table);
return null;
}
and in my case it should get ResourceNotFoundException if the table is not found, but as I said the expection gets wrapped to CompletionException before it reaches the appropriate catch block and is caught in the code here:
catch (ExecutionException var5) {
throw exceptionManager.apply(var5.getCause());
This is happening 20 times in the loop while trying to Initialize the LeaseCoordinator and then just stops trying to initialize the connection. (As mentioned above it works occasionally, but that makes it even stranger to me)
With my workaround it only needs 1 try to get initialized

You don't need to create a lease table manually - DynamoDBLeaseCoordinator will create one if not exists on initialization and wait until it exists:
#Override
public void initialize() throws ProvisionedThroughputException, DependencyException, IllegalStateException {
final boolean newTableCreated =
leaseRefresher.createLeaseTableIfNotExists(initialLeaseTableReadCapacity, initialLeaseTableWriteCapacity);
if (newTableCreated) {
log.info("Created new lease table for coordinator with initial read capacity of {} and write capacity of {}.",
initialLeaseTableReadCapacity, initialLeaseTableWriteCapacity);
}
// Need to wait for table in active state.
final long secondsBetweenPolls = 10L;
final long timeoutSeconds = 600L;
final boolean isTableActive = leaseRefresher.waitUntilLeaseTableExists(secondsBetweenPolls, timeoutSeconds);
if (!isTableActive) {
throw new DependencyException(new IllegalStateException("Creating table timeout"));
}
}
The issue in your case, I think, is that it's eventually created and you probably should periodically check until table appears - like DynamoDBLeaseCoordinator#initialize() does.

How to pass variable parameters to Observer?

I have two Observers that are merged with a flatMap. The first observer returns a value that is used when the second is called.
Observable<Integer> mergedObservers = firstAPI.getFirstInfo(userLat, userLong)
.flatMap(resultFirstObservable -> {
try {
return secondApi.getSecondInfo(resultFirstObservable.body().string(), "3")
.onErrorResumeNext(e -> {
e.printStackTrace();
return secondApi.getSecondInfo("defaultValue", "3");
});
} catch (IOException e) {
e.printStackTrace();
secondApi.getSecondInfo("defaultValue", "3")
.onErrorResumeNext(e -> {
e.printStackTrace();
return secondApi.getSecondInfo("defaultValue", "3");
});
});
}
}, (resultFirstObservable, resultSecondObservable) -> {
try {
return transformToWhatINeed(resultSecondObservable.body().string());
} catch (IOException ex) {
ex.printStackTrace();
return transformToWhatINeed([]);
}
});
userLat and userLong are declared outside my method and are changed during the time the activity is open, but my Subscription takes into account only the first value of these. I would have expected that each time there's a new call, they will take the newest values.
What am I doing wrong ?

If I understand you problem correctly using Observable.defer should solve problem
Observable<Integer> mergedObservers = Observable.defer {
firstAPI.getFirstInfo(userLat, userLong)
}.flatMap ...

Your method should be like:
Observable.combineLatest(userLatObservable, userLongObervable, yourmergerfunction).flatmap( lat, long –> firstApi.get(lat, long))... Etc..
I think you problem is that how do you get the value of userLat amd userLong... Those values should first be converted to Observables to join them in the chain.

How to not catch a particular line exception in try catch box in JAVA?

Here is my code:
whatever exception it throws I don't want to catch it outside, I want to continue my loop again by handling it separately. I don't want to use another try catch inside this try catch. Can someone guide me on this?

I don't want to use another try catch inside this try catch.
Yes you do.
MarketplaceBO marketplaceBOObject = new MarketplaceBO(entity.getMarketplaceID());
try {
marketplaceBOObject.loadFromSable();
} catch (WhateverException e) {
// Do something here, or, if you prefer, add the exception to a list and process later
doSomething() ;
// Continue your loop above
continue ;
}
if (marketplaceBOObject.isActive()) {
If you REALLY don't want to do this, your loadFromSable() method could return some object that provides information about success/failure of the call. But I wouldn't recommend that.

do this way -- this way your rest of the code will run no matter there is an exception or not
for (MerchantMarketplaceBO entity : merchantMarketplaceBOList) {
MarketplaceBO marketplaceBOObject = new MarketplaceBO(entity.getMarketplaceID());
try{
marketplaceBOObject.loadFromSable();
if (marketplaceBOObject.isActive()) {
resultVector.add(marketplaceBOObject.getCodigoMarketplace());
}
}
catch{
if (marketplaceBOObject.isActive()) {
resultVector.add(marketplaceBOObject.getCodigoMarketplace());
}
}
}

Another "trick" to deal with that is to move the body to the loop into a separate method having the "additional" try/catch block:
private MarketplaceBO loadFromSable(MerchantMarketplaceBO entity){
MarketplaceBO marketplaceBOObject = new MarketplaceBO(entity.getMarketplaceID());
try {
marketplaceBOObject.loadFromSable();
} catch (WhateverException e) {
// do something to make marketplaceBOObject a valid object
// or at least log the exception
}
return marketplaceBOObject;
}
But since we want to stick to the Same Layer of Abstraction principle we also need to move other part of that method to new smaller methods:
public void serveFromSableV2() {
String merchantCustomerID = ObfuscatedId.construct(request.getMerchantCustomerID()).getPublicEntityId();
try {
List<MerchantMarketplaceBO> merchantMarketplaceBOList =
getAllMerchantMarketplacesBOsByMerchant();
Vector<Marketplace> resultVector = new Vector<>();
for (MerchantMarketplaceBO entity : merchantMarketplaceBOList) {
MarketplaceBO marketplaceBOObject = loadFromSable(entity);
addToActiveMarketplacesList(marketplaceBOObject,resultVector);
}
verifyHavingActiveMarketPlaces(resultVector);
setResponseWithWrapped(resultVector);
} catch (EntityNotFoundException | SignatureMismatchException | InvalidIDException e) {
throw new InvalidIDException("merch=" + merchantCustomerID + "[" + request.getMerchantCustomerID() + "]"); //C++ stack throws InvalidIDException if marketplace is not found in datastore
}
}

You could refactor the load into a separate method that catches and returns the exception instead of throwing it:
private Optional<Exception> tryLoadFromSable(MarketplaceBO marketplaceBOObject) {
try {
marketplaceBOObject.loadFromSable();
return Optional.empty();
}
catch(Exception e) {
return Optional.of(e);
}
}
Then inside your loop:
//inside for loop...
MarketplaceBO marketplaceBOObject = new MarketplaceBO(entity.getMarketplaceID());
Optional<Exception> loadException = tryLoadFromSable(marketplaceBOObject);
if(loadException.isPresent()) {
//Do something here, log it, save it in a list for later processing, etc.
}

Java 8 generic Retry if specific exception

I'm finding myself writing alot of retry loops that look like
int triesRemaining = 3;
while (triesRemaining > 0) {
try {
<MY FUNCTION CALL>
LOGGER.info("success");
break;
} catch (Exception e) {
if (e.getCause() instanceof SocketTimeoutException) {
triesRemaining--;
LOGGER.info(e.getMessage() + " trying again. Tries Remaining: " + triesRemaining);
} else {
LOGGER.error(e.getMessage(), e);
return;
}
}
}
if (triesRemaining == 0) {
LOGGER.error("Failed over too many times");
}
I want to write a generic function that accepts a Lambda and only retries on a specific error (in the above case thats SocketTimeoutException). I've seen some functions that accept a Runnable which is fine, but they don't seem to allow limiting to specific exceptions.
Any advice?

Well it's already done. It also accepts list of exceptions on which you want to retry. It also provides linear/exponential retry strategies.
Have a look https://github.com/rholder/guava-retrying
A simple example from it's readme, you can compose and use a retryer like:-
Callable<Boolean> callable = new Callable<Boolean>() {
public Boolean call() throws Exception {
return true; // do something useful here
}};
Retryer<Boolean> retryer = RetryerBuilder.<Boolean>newBuilder()
.retryIfResult(Predicates.<Boolean>isNull())
.retryIfExceptionOfType(IOException.class)
.retryIfRuntimeException()
.withStopStrategy(StopStrategies.stopAfterAttempt(3))
.build();
try {
retryer.call(callable);
} catch (RetryException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}

Have a look to org.springframework.retry
There is an annotation #Retryable which corresponding to your need. You can specify the type of exception to retry and configure the number of attempt, etc...

Check out Failsafe:
RetryPolicy retryPolicy = new RetryPolicy()
.retryOn(SocketTimeoutException.class)
.withMaxRetries(3);
Failsafe.with(retryPolicy)
.onRetry((c, f, ctx) -> log.warn("Failure #{}. Retrying.", ctx.getExecutions()))
.onFailure(e -> LOGGER.error(e.getMessage(), e))
.run(() -> myFunctionCall());

What's the problem of just making this function to accept a Runnable argument and then run it in <MY FUNCTION CALL>?
public static void retry(Runnable r) {
// ...
while (triesRemaining > 0) {
try {
r.run();
LOGGER.info("success");
break;
}
// ...
}
then call it (if you prefer - with a lambda):
retry(() -> {
connectToServer();
// todo what-ever-you-want
});

I believe you're looking for pure Java based solution. Based on assumption, I would say Java 8 uses functional interface, an interface with single abstract method. I would create a new RetryCommand class that has a run method which takes in a function.

Java exception handling in non sequential tasks (pattern/good practice)

There are some task that should't be done in parallel, (for example opening a file, reading, writing, and closing, there is an order on that...)
But... Some task are more like a shoping list, I mean they could have a desirable order but it's not a must..example in communication or loading independient drivers etc..
For that kind of tasks,
I would like to know a java best practice or pattern for manage exceptions..
The java simple way is:
getUFO {
try {
loadSoundDriver();
loadUsbDriver();
loadAlienDetectorDriver();
loadKeyboardDriver();
} catch (loadSoundDriverFailed) {
doSomethingA;
} catch (loadUsbDriverFailed) {
doSomethingB;
} catch (loadAlienDetectorDriverFailed) {
doSomethingC;
} catch (loadKeyboardDriverFailed) {
doSomethingD;
}
}
But what about having an exception in one of the actions but wanting to
try with the next ones??
I've thought this approach, but don't seem to be a good use for exceptions
I don't know if it works, doesn't matter, it's really awful!!
getUFO {
Exception ex=null;
try {
try{ loadSoundDriver();
}catch (Exception e) { ex=e; }
try{ loadUsbDriver();
}catch (Exception e) { ex=e; }
try{ loadAlienDetectorDriver();
}catch (Exception e) { ex=e; }
try{ loadKeyboardDriver()
}catch (Exception e) { ex=e; }
if(ex!=null)
{ throw ex;
}
} catch (loadSoundDriverFailed) {
doSomethingA;
} catch (loadUsbDriverFailed) {
doSomethingB;
} catch (loadAlienDetectorDriverFailed) {
doSomethingC;
} catch (loadKeyboardDriverFailed) {
doSomethingD;
}
}
seems not complicated to find a better practice for doing that.. I still didn't
thanks for any advice

Consider the execute around idiom.
Another option (which isn't really all that different, it just decouples them more) is to do each task in a separate thread.
Edit:
Here is the kind of thing I have in mind:
public interface LoadableDriver {
public String getName();
public void loadDriver() throws DriverException;
public void onError(Throwable e);
}
public class DriverLoader {
private Map<String, Exception> errors = new HashMap<String, Exception>();
public void load(LoadableDriver driver) {
try {
driver.loadDriver();
} catch (DriverException e) {
errors.put(driver.getName(), e);
driver.onError(e);
}
}
public Map<String, Exception> getErrors() { return errors; }
}
public class Main {
public void loadDrivers() {
DriverLoader loader = new DriverLoader();
loader.loadDriver(new LoadableDriver(){
public String getName() { return "SoundDriver"; }
public void loadDriver() { loadSoundDriver(); }
public void onError(Throwable e) { doSomethingA(); }
});
//etc. Or in the alternative make a real class that implements the interface for each driver.
Map<String, Exception> errors = loader.getErrors();
//react to any specific drivers that were not loaded and try again.
}
}
Edit: This is what a clean Java version would ultimately look like if you implemented the drivers as classes (which is what the Java OO paradigm would expect here IMHO). The Main.loadDrivers() method would change like this:
public void loadDrivers(LoadableDriver... drivers) {
DriverLoader loader = ...
for(LoadableDriver driver : drivers) {
loader.load(driver);
}
//retry code if you want.
Set<LoadableDriver> failures = loader.getErrors();
if(failures.size() > 0 && tries++ > MAX_TRIES) {
//log retrying and then:
loadDrivers(drivers.toArray(new LoadableDriver[0]));
}
}
Of course I no longer use a map because the objects would be self-sufficient (you could get rid of the getName() method as well, but probably should override toString()), so the errors are just returned in a set to retry. You could make the retry code even simpler if each driver was responsible for knowing how often it should it retry.
Java won't look as nice as a well done C++ template, but that is the Java language design choice - prefer simplicity over complex language features that can make code hard to maintain over time if not done properly.

Try this:
protected void loadDrivers() {
loadSoundDriver();
loadUsbDriver();
loadAlienDetectorDriver();
loadKeyboardDriver();
}
Then:
protected void loadSoundDriver() {
try {
// original code ...
}
catch( Exception e ) {
soundDriverFailed( e );
}
}
protected void soundDriverFailed( Exception e ) {
log( e );
}
This gives subclasses a chance to change the behaviour. For example, a subclass could implement loading each driver in a separate thread. The main class need not care about how the drivers are loaded, nor should any users of the main class.

IMO, for your case, if the exception is "ignorable" it's best if the "loadSoundDriver" method catches the exception and simply returns an error.
Then in the function that loads stuff, you can record all the errors and at the end of the sequence, decide what to do with them.
[edit]
Something like this:
// init
MyError soundErr = loadSoundDriver();
MyError otherErr = loadOtherDriver();
if(soundErr!=null || otherErr !=null){
// handle the error(s)
}

Just surround every single load operation with its own try / catch block.
try {
loadSoundDriver();
} catch (loadSoundDriverFailed) {
doSomethingA;
}
try {
loadUsbDriver();
} catch (loadUsbDriverFailed) {
doSomethingB;
}
// ...
So you can handle every exception by itself and continue processing the oder operations.