Develop Reference

How to error handle for 20 reactive async API calls in java?

I'm writing a service that calls 20 external vendor APIs, aggregates that data and writes it to a blob storage. This is how I am calling each api, after I am using Mono.zip() and writing that result into a blob storage. However I feel that the way I am writing the code is really redundant, specifically the error handling using .onErrorResume() and .doOnSuccess() Is there a way I can maek this code cleaner by using generics or utilizing inheritance some way? I just dont want hundreds of lines of code that are basically doing the same thing...
Mono<MailboxProvidersDTO> mailboxProvidersDTOMono = partnerAsyncService.asyncCallPartnerApi(getMailboxProvidersUrl, MailboxProvidersDTO.class)
.retryWhen(getRetrySpec())
//need to have error handling for all 20 api calls
.doOnSuccess(res -> {
log.info(Logger.EVENT_SUCCESS, "Mailbox Providers report successfully retrieved.");
res.setStatus("Success");
})
.onErrorResume(BusinessException.class, ex -> {
log.error(Logger.EVENT_FAILURE, ex.getMessage());
MailboxProvidersDTO audienceExplorerDTO = new MailboxProvidersDTO();
audienceExplorerDTO.setStatus("Failed");
return Mono.just(audienceExplorerDTO);
});
Mono<TimeZonesDTO> timeZonesDTOMono = partnerAsyncService.asyncCallPartnerApi(getTimeZonesUrl, TimeZonesDTO.class);
Mono<RegionsDTO> regionsDTOMono = partnerAsyncService.asyncCallPartnerApi(getRegionsUrl, RegionsDTO.class);
Mono<AudienceExplorerDataSourcesDTO> audienceExplorerDataSourcesDTOMono = partnerAsyncService.asyncCallPartnerApi(getAudienceExplorerDataSourcesUrl, AudienceExplorerDataSourcesDTO.class);
...

You can effectively use generics to refactor your code. You can couple functional interfaces and Generics to create what you need:
On your example, you need both to "setStatus" and create new instances of different classes. You could then create a utility function to add onSuccess/onFailure behaviours over your initial data fetching Mono:
public <T> Mono<T> withRecovery(Mono<T> fetchData, BiConsumer<T, String> setStatus, Supplier<T> createFallbackDto) {
return fetchData
.doOnSuccess(result -> {
log.info...
setStatus.accept(result, "Success");
})
.doOnError(BusinessException.class, err -> {
log.error...
T fallback = createFallbackDto.get();
setStatus.accept(fallback, "Error");
return Mono.just(fallback);
});
}
Then, you can use this method like that:
Mono<MailProvidersDto> mails = withRecovery(
partnersAsyncService.asyncCallPartnerApi(getMailboxProvidersUrl, MailboxProvidersDTO.class),
MailProvidersDto::setStatus,
MailProvidersDto::new
);
Mono<TimeZoneDto> timezone = withRecoery(
partnersAsyncService.asyncCallPartnerApi(getMailboxProvidersUrl, TimeZoneDto.class),
TimeZoneDto::setStatus,
TimeZoneDto::new
);
... // Repeat for each api
Notes:
If the setStatus method is available through a common interface that all DTO implement, you can get rid of the biconsumer, and directly call result.setStatus(String), by specializing T generic to T extends StatusInterface.
With it, you could also factorize initial fetching and retry calls, by passing related parameters (url, class, retry spec) as method input.

Handling additional data in Apache ServiceComb compensation methods

I'm currently looking at the implementations of saga pattern for distributed transactions and I found that Apache ServiceComp pack might be something that works for me.
However, I have found a problem that the limitation of compensating methods to have the same declaration as the methods they compensate may be a bottleneck.
From Apache's example:
#Compensable(compensationMethod = "cancel")
void order(CarBooking booking) {
booking.confirm();
bookings.put(booking.getId(), booking);
}
void cancel(CarBooking booking) {
Integer id = booking.getId();
if (bookings.containsKey(id)) {
bookings.get(id).cancel();
}
}
You can see that we have the same declaration for both methods.
But, what if I need additional information to compensate my transaction? For instance, I have a call to external system to update some flag to "true". When I need to compensate it, how do I make "cancel" method know what the original value of this flag was?
The things get more tricky when we update the whole object. How do I send the whole object before modification to the cancel transaction?
These limitation doesn't look quite promising. Do you know if there are approaches to fight with this limitation?

You can save localTxId and flag an in your application and use localTxId in the compensation method to get the flag
Map extmap = new HashMap();
#Autowired
OmegaContext omegaContext;
#Compensable(compensationMethod = "cancel")
void order(CarBooking booking) {
booking.confirm();
bookings.put(booking.getId(), booking);
//save flag
extmap.put(omegaContext.localTxId(),'Your flag')
}
void cancel(CarBooking booking) {
//get flag
extmap.get(omegaContext.localTxId());
Integer id = booking.getId();
if (bookings.containsKey(id)) {
bookings.get(id).cancel();
}
}

How Do You Check For DELETE/UPDATE Without A WHERE Clause

I currently have a listener that we use to do a few different monitoring-type activities (like log a warning if a query takes more than 5 seconds), but it also watches for and kills "silly bugs" -- especially UPDATE and DELETE queries that are missing a WHERE clause.
In the past we did the following (note that we are using com.foundationdb.sql):
/**
* Hook into the query execution lifecycle before rendering queries. We are checking for silly mistakes,
* pure SQL, etc.
*/
#Override
public void renderStart(final #NotNull ExecuteContext ctx) {
if (ctx.type() != ExecuteType.WRITE)
return;
String queryString = ctx.sql();
try (final Query query = ctx.query()) {
// Is our Query object empty? If not, let's run through it
if (!ValidationUtils.isEmpty(query)) {
queryString = query.getSQL(ParamType.INLINED);
final SQLParser parser = new SQLParser();
try {
final StatementNode tokens = parser.parseStatement(query.getSQL());
final Method method = tokens.getClass().getDeclaredMethod("getStatementType");
method.setAccessible(true);
switch (((Integer) method.invoke(tokens)).intValue()) {
case StatementType.UPDATE:
SelectNode snode = ConversionUtils.as(SelectNode.class,
((DMLStatementNode) tokens).getResultSetNode());
// check if we are a mass delete/update (which we don't allow)
if ((Objects.isNull(snode)) || (Objects.isNull(snode.getWhereClause())))
throw new RuntimeException("A mass update has been detected (and prevented): "
+ DatabaseManager.getBuilder().renderInlined(ctx.query()));
break;
case StatementType.DELETE:
snode = ConversionUtils.as(SelectNode.class,
((DMLStatementNode) tokens).getResultSetNode());
// check if we are a mass delete/update (which we don't allow)
if ((Objects.isNull(snode)) || (Objects.isNull(snode.getWhereClause())))
throw new RuntimeException("A mass delete has been detected (and prevented): "
+ DatabaseManager.getBuilder().renderInlined(ctx.query()));
break;
default:
if (__logger.isDebugEnabled()) {
__logger
.debug("Skipping query because we don't need to do anything with it :-): {}", queryString);
}
}
} catch (#NotNull StandardException | IllegalAccessException
| IllegalArgumentException | InvocationTargetException | NoSuchMethodException
| SecurityException e) {
// logger.error(e.getMessage(), e);
}
}
// If the query object is empty AND the SQL string is empty, there's something wrong
else if (ValidationUtils.isEmpty(queryString)) {
__logger.error(
"The ctx.sql and ctx.query.getSQL were empty");
} else
throw new RuntimeException(
"Someone is trying to send pure SQL queries... we don't allow that anymore (use jOOQ): "
+ queryString);
}
}
I really don't want to use yet another tool -- especially since most SQL parsers can't handle UPSERTs or the wide variety of queries that jOOQ can, so a lot just get cut out -- and would love to use jOOQ's constructs, but I'm having trouble. Ideally I could just check the query class and if it's an Update or Delete (or a subclass), I would just scream if it isn't an instance of UpdateConditionStep or DeleteConditionStep, but that doesn't work because the queries are coming back as UpdateQueryImpl... and without crazy reflection, I can't see if there is a condition in use.
So... right now I'm doing:
/**
* Hook into the query execution lifecycle before rendering queries. We are checking for silly mistakes, pure SQL,
* etc.
*/
#Override
public void renderStart(final #NotNull ExecuteContext ctx) {
if (ctx.type() != ExecuteType.WRITE)
return;
try (final Query query = ctx.query()) {
// Is our Query object empty? If not, let's run through it
if (!ValidationUtils.isEmpty(query)) {
// Get rid of nulls
query.getParams().entrySet().stream().filter(entry -> Objects.nonNull(entry.getValue()))
.filter(entry -> CharSequence.class.isAssignableFrom(entry.getValue().getDataType().getType()))
.filter(entry -> NULL_CHARACTER.matcher((CharSequence) entry.getValue().getValue()).find())
.forEach(entry -> query.bind(entry.getKey(),
NULL_CHARACTER.matcher((CharSequence) entry.getValue().getValue()).replaceAll("")));
if (Update.class.isInstance(query)) {
if (!UpdateConditionStep.class.isInstance(query)) {
if (!WHERE_CLAUSE.matcher(query.getSQL(ParamType.INDEXED)).find()) {
final String queryString = query.getSQL(ParamType.INLINED);
throw new RuntimeException(
"Someone is trying to run an UPDATE query without a WHERE clause: " + queryString);
}
}
} else if (Delete.class.isInstance(query)) {
if (!DeleteConditionStep.class.isInstance(query)) {
if (!WHERE_CLAUSE.matcher(query.getSQL(ParamType.INDEXED)).find()) {
final String queryString = query.getSQL(ParamType.INLINED);
throw new RuntimeException(
"Someone is trying to run a DELETE query without a WHERE clause: " + queryString);
}
}
}
} else
throw new RuntimeException(
"Someone is trying to send pure SQL queries... we don't allow that anymore (use jOOQ): "
+ ctx.sql());
}
}
This let's me get rid of the third party SQL parser, but now I'm using a regular expression on the non-inlined query looking for \\s[wW][hH][eE][rR][eE]\\s, which isn't ideal, either.
Is there a way to use jOOQ to tell me if an UPDATE, DELETE, has a WHERE clause?
Similarly, is there a way that let's me see what table the query is acting against (so that I can limit the tables someone can perform mutable actions against -- obviously that one wouldn't check if it's UPDATE or DELETE, instead using the ExecuteType)?

That's an interesting idea and approach. One problem I can see with it is performance. Rendering the SQL string a second time and then parsing it again sounds like a bit of overhead. Perhaps, this ExecuteListener should be active in development and integration test environments only, not in production.
Regarding your questions
Is there a way to use jOOQ to tell me if an UPDATE, DELETE, has a WHERE clause?
Since you seem to be open to use reflection to access a third party library's internals, well of course, you could check if the ctx.query() is of type org.jooq.impl.UpdateQueryImpl or org.jooq.impl.DeleteQueryImpl. In version 3.10.1, both of them have a private condition member, which you could check.
This will obviously break any time the internals are changed, but it might be a pragmatic solution for now.
Similarly, is there a way that let's me see what table the query is acting against
A more general and more robust approach would be to implement a VisitListener, which is jOOQ's callback that is called during expression tree traversal. You can hook into the generation of the SQL string and the collection of bind variables, and throw your errors as soon as you encounter:
An UPDATE or DELETE statement
... without a WHERE clause
... updating a table from a specific set of tables
You "just" have to implement a stack machine that remembers all of the above things prior to throwing the exception. An example of how VisitListener can be implemented is given here:
https://blog.jooq.org/2015/06/17/implementing-client-side-row-level-security-with-jooq
New feature in the future
This kind of feature has been discussed a couple of times on the mailing list as well. It's a low hanging fruit to support by jOOQ natively. I've created a feature request for jOOQ 3.11, for this:
https://github.com/jOOQ/jOOQ/issues/6771

Using monads in Springboot application to catch exceptions

So our project back-end is a Java 8 Springboot application, springboot allows you to do some stuff really easily. ex, request validation:
class ProjectRequestDto {
#NotNull(message = "{NotNull.DotProjectRequest.id}")
#NotEmpty(message = "{NotEmpty.DotProjectRequest.id}")
private String id;
}
When this constraint is not meet, spring (springboot?) actually throws a validation exception, as such, we catch it somewhere in the application and construct a 404 (Bad Request) response for our application.
Now, given this fact, we kinda followed the same philosophy throughout our application, that is, on a deeper layer of the application we might have something like:
class ProjectService throws NotFoundException {
DbProject getProject(String id) {
DbProject p = ... // some hibernate code
if(p == null) {
Throw new NotFoundException();
}
return p;
}
}
And again we catch this exception on a higher level, and construct another 404 for the client.
Now, this is causing a few problems:
The most important one: Our error tracing stops being useful, we cannot differentiate (easily) when the exception is important, because they happen ALL the time, so if the service suddenly starts throwing errors we would not notice until it is too late.
Big amount of useless logging, on login requests for example, user might mistyped his password, and we log this and as a minor point: our analytics cannot help us determine what we are actually doing wrong, we see a lot of 4xx's but that is what we expect.
Exceptions are costly, gathering the stack trace is a resource intensive task, minor point at this moment, as the service scales up with would become more of a problem.
I think the solution is quite clear, we need to make an architectural change to not make exceptions part of our normal data flow, however this is a big change and we are short on time, so we plan to migrate over time, yet the problem remains for the short term.
Now, to my actual question: when I asked one of our architects, he suggested the use of monads (as a temporal solution ofc), so we don't modify our architecture, but tackle the most contaminating endpoints (ex. wrong login) in the short term, however I'm struggling with the monad paradigm overall and even more in java, I really have no idea on how to apply it to our project, could you help me with this? some code snippets would be really good.
TL:DR: If you take a generic spring boot application that throws errors as a part of its data flow, how can you apply the monad pattern to avoid login unnecessary amount of data and temporarily fix this Error as part of data flow architecture.

The standard monadic approach to exception handling is essentially to wrap your result in a type that is either a successful result or an error. It's similar to the Optional type, though here you have an error value instead of an empty value.
In Java the simplest possible implementation is something like the following:
public interface Try<T> {
<U> Try<U> flatMap(Function<T, Try<U>> f);
class Success<T> implements Try<T> {
public final T value;
public Success(T value) {
this.value = value;
}
#Override
public <U> Try<U> flatMap(Function<T, Try<U>> f) {
return f.apply(value);
}
}
class Fail<T> implements Try<T> {
// Alternatively use Exception or Throwable instead of String.
public final String error;
public Fail(String error) {
this.error = error;
}
#Override
public <U> Try<U> flatMap(Function<T, Try<U>> f) {
return (Try<U>)this;
}
}
}
(with obvious implementations for equals, hashCode, toString)
Where you previously had operations that would either return a result of type T or throw an exception, they would return a result of Try<T> (which would either be a Success<T> or a Fail<T>), and would not throw, e.g.:
class Test {
public static void main(String[] args) {
Try<String> r = ratio(2.0, 3.0).flatMap(Test::asString);
}
static Try<Double> ratio(double a, double b) {
if (b == 0) {
return new Try.Fail<Double>("Divide by zero");
} else {
return new Try.Success<Double>(a / b);
}
}
static Try<String> asString(double d) {
if (Double.isNaN(d)) {
return new Try.Fail<String>("NaN");
} else {
return new Try.Success<String>(Double.toString(d));
}
}
}
I.e. instead of throwing an exception you return a Fail<T> value which wraps the error. You can then compose operations which might fail using the flatMap method. It should be clear that once an error occurs it will short-circuit any subsequent operations - in the above example if ratio returns a Fail then asString doesn't get called and the error propagates directly through to the final result r.
Taking your example, under this approach it would look like this:
class ProjectService throws NotFoundException {
Try<DbProject> getProject(String id) {
DbProject p = ... // some hibernate code
if(p == null) {
return new Try.Fail<DbProject>("Failed to create DbProject");
}
return new Try.Succeed<DbProject>(p);
}
}
The advantage over raw exceptions is it's a bit more composable and allows, for example, for you to map (e.g. Stream.map) a fail-able function over a collection of values and end up with a collection of Fails and Successes. If you were using exceptions then the first exception would fail the entire operation and you would lose all results.
One downside is that you have to use Try return types all the way down your call stack (somewhat like checked exceptions). Another is that since Java doesn't have built-in monad support (al la Haskell & Scala) then the flatMap'ing can get slightly verbose. For example something like:
try {
A a = f(x);
B b = g(a);
C c = h(b);
} catch (...
where f, g, h might throw, becomes instead:
Try<C> c = f(x).flatMap(a -> g(a))
.flatMap(b -> h(b));
You can generalise the above implementation by making the error type an generic parameter E (instead of String), so it then becomes Try<T, E>. whether this is useful depends on your requirements - I've never needed it.
I have a more fully-implemented version here, alternatively the Javaslang and FunctionalJava libraries offer their own variants.

Mongo result docs not store as object

I'm trying to get the results after executing a query and store it in a variable called "model".
db.collection.findOne({object},function(err,docs){
model["output"]= docs;
})
The above code stores model["output"] as "undefined". How do I get hold of this value?
Sorry there was not enough code.
So there are two files.
FILE1
dbStmtModel.insertRecords(collectionName, record).then(
function (results) {
console.log("results",results);
result = results;
}, function (err){
return err;
});
model[statement.output] = result;
FILE2
function insertRecords(operand1,operand2){
var deferred = q.defer();
db.collection(operand1).update(operand2,operand2,{upsert:true},function (err,docs) {
if(err) {
deferred.reject(err);
}
else {
deferred.resolve(docs);
}
});
return deferred.promise
}
So tried using promises, tried using async. Still do not seem to get the model store the output of the result of the query. Also, there are no errors, since the callback returns correctly and prints the results. I'm not sure I'm using promises correctly though, although the console statement seems to print the results correctly. Also I'm using mongojs and not mongoose over mongodb(since the schema here is dynamic), don't know if thats going to change anything.

First of all, check for the err param you're ignoring
There is no enough code to be sure, but I'm guessing you defined model before in the same scope you're calling findOne and just below that you try to use model['output'].
This won't work since findOne (as almost every mongodb driver method) is asynchronous, so it's unlikely for its callback to be called before you try to use model['output'].
There's no quick solution for your problem, you need to understand asynchronism. Checkout this answer.

db.collection.findOne({object},function(err,docs){
model["output"]= docs;
})
First - ill assume that your "model" object is defined. (It would throw that model is undefined, not model output)
Second, You're not checking for error, if there was error docs end up empty.
If still you've got Undefinded. There could be also error with model object. For instance - check if its even an object.
db.collection.findOne({object}, (err, docs) => {
if(err) {
return console.log(err)
}
model.output = docs;
}
Also! I'm just guessing but maybe you're trying to use it out of .findOne scope? What I mean - it is asynchronous call. So if you do something like this
db.collection.findOne({object}, (err, docs) => {
if(err) {
return console.log(err)
}
model.output = docs;
}
console.log(model.output);
then your model.output is undefined cause you call it before database returns data - it does not wait. You'll have to use callback (or promise) then.
callDB (object, cb) => {
db.collection.findOne(object, (err, docs) => {
if(err) {
return (err)
}
return (null, docs);
}
}
then you could call it
callDB({object}, (err, result) => {
model.result = result;
});
But be advised that your new call for function is still asynchronous. So still your model.result will work only inside of its scope.
// I've seen you've updated your question, but I'll leave it here.

First be sure that you are getting your results make a
console.log("RESULT",docs) if you getting your results then try below methods
As mongo query return doc which is a model schema that can not be modify.
Try this with lean it return document JSON object
var model={};
db.collection.findOne({object},function(err,docs){
model.output= docs;
}).lean(true);
OR
var result={}
db.collection.findOne({object},function(err,docs){
result.model.output = docs;
})