Develop Reference

Rxjava2 + Retrofit2 + Android. Best way to do hundreds of network calls

I have an app. I have a big button that allows the user to sync all their data at once to the cloud. A re-sync feature that allows them to send all their data again. (300+ entries)
I am using RXjava2 and retrofit2. I have my unit test working with a single call. However I need to make N network calls.
What I want to avoid is having the observable call the next item in a queue. I am at the point where I need to implement my runnable. I have seen a bit about Maps but I have not seen anyone use it as a queue. Also I want to avoid having one item fail and it report back as ALL items fail, like the Zip feature would do. Should I just do the nasty manager class that keeps track of a queue? Or is there a cleaner way to send several hundred items?
NOTE: SOLUTION CANNOT DEPEND ON JAVA8 / LAMBDAS. That has proved to be way more work than is justified.
Note all items are the same object.
#Test
public void test_Upload() {
TestSubscriber<Record> testSubscriber = new TestSubscriber<>();
ClientSecureDataToolKit clientSecureDataToolKit = ClientSecureDataToolKit.getClientSecureDataKit();
clientSecureDataToolKit.putUserDataToSDK(mPayloadSecureDataToolKit).subscribe(testSubscriber);
testSubscriber.awaitTerminalEvent();
testSubscriber.assertNoErrors();
testSubscriber.assertValueCount(1);
testSubscriber.assertCompleted();
}
My helper to gather and send all my items
public class SecureDataToolKitHelper {
private final static String TAG = "SecureDataToolKitHelper";
private final static SimpleDateFormat timeStampSimpleDateFormat =
new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
public static void uploadAll(Context context, RuntimeExceptionDao<EventModel, UUID> eventDao) {
List<EventModel> eventModels = eventDao.queryForAll();
QueryBuilder<EventModel, UUID> eventsQuery = eventDao.queryBuilder();
String[] columns = {...};
eventsQuery.selectColumns(columns);
try {
List<EventModel> models;
models = eventsQuery.orderBy("timeStamp", false).query();
if (models == null || models.size() == 0) {
return;
}
ArrayList<PayloadSecureDataToolKit> toSendList = new ArrayList<>();
for (EventModel eventModel : models) {
try {
PayloadSecureDataToolKit payloadSecureDataToolKit = new PayloadSecureDataToolKit();
if (eventModel != null) {
// map my items ... not shown
toSendList.add(payloadSecureDataToolKit);
}
} catch (Exception e) {
Log.e(TAG, "Error adding payload! " + e + " ..... Skipping entry");
}
}
doAllNetworkCalls(toSendList);
} catch (SQLException e) {
e.printStackTrace();
}
}
my Retrofit stuff
public class ClientSecureDataToolKit {
private static ClientSecureDataToolKit mClientSecureDataToolKit;
private static Retrofit mRetrofit;
private ClientSecureDataToolKit(){
mRetrofit = new Retrofit.Builder()
.baseUrl(Utilities.getSecureDataToolkitURL())
.addCallAdapterFactory(RxJavaCallAdapterFactory.create())
.addConverterFactory(GsonConverterFactory.create())
.build();
}
public static ClientSecureDataToolKit getClientSecureDataKit(){
if(mClientSecureDataToolKit == null){
mClientSecureDataToolKit = new ClientSecureDataToolKit();
}
return mClientSecureDataToolKit;
}
public Observable<Record> putUserDataToSDK(PayloadSecureDataToolKit payloadSecureDataToolKit){
InterfaceSecureDataToolKit interfaceSecureDataToolKit = mRetrofit.create(InterfaceSecureDataToolKit.class);
Observable<Record> observable = interfaceSecureDataToolKit.putRecord(NetworkUtils.SECURE_DATA_TOOL_KIT_AUTH, payloadSecureDataToolKit);
return observable;
}
}
public interface InterfaceSecureDataToolKit {
#Headers({
"Content-Type: application/json"
})
#POST("/api/create")
Observable<Record> putRecord(#Query("api_token") String api_token, #Body PayloadSecureDataToolKit payloadSecureDataToolKit);
}
Update. I have been trying to apply this answer to not much luck. I am running out of steam for tonight. I am trying to implement this as a unit test, like I did for the original call for one item.. It looks like something is not right with use of lambda maybe..
public class RxJavaBatchTest {
Context context;
final static List<EventModel> models = new ArrayList<>();
#Before
public void before() throws Exception {
context = new MockContext();
EventModel eventModel = new EventModel();
//manually set all my eventmodel data here.. not shown
eventModel.setSampleId("SAMPLE0");
models.add(eventModel);
eventModel.setSampleId("SAMPLE1");
models.add(eventModel);
eventModel.setSampleId("SAMPLE3");
models.add(eventModel);
}
#Test
public void testSetupData() {
Assert.assertEquals(3, models.size());
}
#Test
public void testBatchSDK_Upload() {
Callable<List<EventModel> > callable = new Callable<List<EventModel> >() {
#Override
public List<EventModel> call() throws Exception {
return models;
}
};
Observable.fromCallable(callable)
.flatMapIterable(models -> models)
.flatMap(eventModel -> {
PayloadSecureDataToolKit payloadSecureDataToolKit = new PayloadSecureDataToolKit(eventModel);
return doNetworkCall(payloadSecureDataToolKit) // I assume this is just my normal network call.. I am getting incompatibility errors when I apply a testsubscriber...
.subscribeOn(Schedulers.io());
}, true, 1);
}
private Observable<Record> doNetworkCall(PayloadSecureDataToolKit payloadSecureDataToolKit) {
ClientSecureDataToolKit clientSecureDataToolKit = ClientSecureDataToolKit.getClientSecureDataKit();
Observable observable = clientSecureDataToolKit.putUserDataToSDK(payloadSecureDataToolKit);//.subscribe((Observer<? super Record>) testSubscriber);
return observable;
}
Result is..
An exception has occurred in the compiler (1.8.0_112-release). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
com.sun.tools.javac.code.Symbol$CompletionFailure: class file for java.lang.invoke.MethodType not found
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:compile<MyBuildFlavorhere>UnitTestJavaWithJavac'.
> Compilation failed; see the compiler error output for details.
Edit. No longer trying Lambdas. Even after setting up the path on my mac, javahome to point to 1.8, etc. I could not get it to work. If this was a newer project I would push harder. However as this is an inherited android application written by web developers trying android, it is just not a great option. Nor is it worth the time sink to get it working. Already into the days of this assignment instead of the half day it should have taken.
I could not find a good non lambda flatmap example. I tried it myself and it was getting messy.

If I understand you correctly, you want to make your calls in parallel?
So rx-y way of doing this would be something like:
Observable.fromCallable(() -> eventsQuery.orderBy("timeStamp", false).query())
.flatMapIterable(models -> models)
.flatMap(model -> {
// map your model
//avoid throwing exceptions in a chain, just return Observable.error(e) if you really need to
//try to wrap your methods that throw exceptions in an Observable via Observable.fromCallable()
return doNetworkCall(someParameter)
.subscribeOn(Schedulers.io());
}, true /*because you don't want to terminate a stream if error occurs*/, maxConcurrent /* specify number of concurrent calls, typically available processors + 1 */)
.subscribe(result -> {/* handle result */}, error -> {/* handle error */});
In your ClientSecureDataToolKit move this part into constructor
InterfaceSecureDataToolKit interfaceSecureDataToolKit = mRetrofit.create(InterfaceSecureDataToolKit.class);

Why doesn't EJBException use Throwable.cause?

Quite for a long time I'm wondering why doesn't EJBException use standard Throwable.cause field to reach an exception it wraps?
It complicates getting the original root cause to something like that
private String getRootCauseErrorMessage(final Exception ex) {
Throwable currentException = ex;
Throwable nextException = null;
do {
if (nextException != null) {
currentException = nextException;
}
/* For some reason EJBException stores cause in a separate field rather the all generic Throwables */
if (currentException instanceof EJBException) {
nextException = ((EJBException) currentException).getCausedByException();
} else {
nextException = currentException.getCause();
}
} while (nextException != null);
return currentException.getMessage();
}
ps: I'm on Java6 and EJB3

Throwable.getCause was not added until Java 1.4. Some implementations of EJBException do retrofit the getCausedByException method to use the getCause method (similar to how the RemoteException.getCause method was retrofitted), but it sounds like your application server does not do this.

auto log java exceptions

I have written a huge Web-Application and 'forgot' to include logging (I only print the errors with the standard e.printStackTrace() method).
My question is, if there is any method to auto-log (getLogger.LOG(SEVERE,"...")) any thrown exception?
maybe with a custom exception-factory like in exceptionFactory JSF?
I want to log every thrown exception with my logger, e.g. before the program enters the catch-block, the exception has to be logged already:
try{
...
} catch(Exception1 e){
//Exception must have been already logged here (without adding getLogger().LOG(...) every time)
System.out.println(e.printStackTrace());
} catch(Exception2 e){
//Exception must have been already logged here (without adding getLogger().LOG(...) every time)
System.out.println(e.printStackTrace());
}

Take a look at aspect oriented programming which can insert logging code at runtime for your favorite logging framework. The JDK includes the java.lang.instrument package which can insert bytecodes during classloading to perform your logging.
Otherwise, you can install a servlet Filter as the top most filter in the call chain which will catch most of your exceptions.
public class LogFilter implements javax.servlet.Filter {
private static final String CLASS_NAME = LogFilter.class.getName();
private static final Logger logger = Logger.getLogger(CLASS_NAME);
#Override
public void init(FilterConfig filterConfig) throws ServletException {
}
#Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
logger.entering(CLASS_NAME, "doFilter", new Object[]{request, response});
try {
chain.doFilter(request, response);
} catch (IOException | ServletException | RuntimeException | Error ioe) {
logger.log(Level.SEVERE, "", ioe);
throw ioe; //Keep forwarding.
} catch (Throwable t) {
logger.log(Level.SEVERE, "", t);
throw new ServletException(t);
}
logger.exiting(CLASS_NAME, "doFilter");
}
#Override
public void destroy() {
}
}

You can set uncaught exception handler for main thread and every other you create using Thread.setUncaughtExceptionHandler() method and do all the required logging there.

I am now also in front of new larger project and interested in elimination of not necessary code to be produced. First I wanted to log every entry and exit from method including input and output data. In my case of event driven architecture I am pushing these data to elastic and analyse continuously method processing timeouts, that is lot of code lines. So I handled this with AspectJ. Very nice example of this is here:
http://www.baeldung.com/spring-performance-logging
Same applies for auto Error logging, here is dummy example which I will extend to work with slf4j, but these are details:
public aspect ExceptionLoggingAspect {
private Log log = LogFactory.getLog(this.getClass());
private Map loggedThrowables = new WeakHashMap();
public pointcut scope(): within(nl.boplicity..*);
after() throwing(Throwable t): scope() {
logThrowable(t, thisJoinPointStaticPart,
thisEnclosingJoinPointStaticPart);
}
before (Throwable t): handler(Exception+) && args(t) && scope() {
logThrowable(t, thisJoinPointStaticPart,
thisEnclosingJoinPointStaticPart);
}
protected synchronized void logThrowable(Throwable t, StaticPart location,
StaticPart enclosing) {
if (!loggedThrowables.containsKey(t)) {
loggedThrowables.put(t, null);
Signature signature = location.getSignature();
String source = signature.getDeclaringTypeName() + ":" +
(enclosing.getSourceLocation().getLine());
log.error("(a) " + source + " - " + t.toString(), t);
}
}
}
I would be happy to hear what else is good example of boiler plate code reduction. I of course use Loombok which does superior task...
NOTE: do not reinvent wheel, so look here as other people collected usefull AOP to be reused in your project out of the box :-)) open source is great community: https://github.com/jcabi/jcabi-aspects

How do I get the server-side exception message with GWT?

I tried
public void onFailure(Throwable caught) {
Throwable cause = caught.getCause();
String causeStr = (cause==null) ? "" : ", "+cause.getMessage();
errorLabel.setText(SERVER_ERROR + ": " + caught.getMessage() + causeStr);
But cause is always null and caught.getMessage() always equals the very generic 500 The call failed on the server; see server log for details. I want to throw IllegalArgumentExceptions from the server and be able to show it on the client:
throw new IllegalArgumentException("Email address is invalid.");

Your Exception needs to be Serializable to travel through the cables.
In addition, the best practice says: You should have two exception kinds:
SystemException: that is a fatal exception, the user can't recover (this should not be serializable since you will give the user a feedback of the type "an error occured on the server, please contact the administrator"
BusinessException: which is due to a misuse of your ammplication by the user (ex: UnauthorizedException, BadMailException or more generally InvalidvalueException)
This way you will write System exceptions in the logs and send back business exceptions to the user

You can use com.google.gwt.core.client.GWT.UncaughtExceptionHandler to catch the exception on the server, and then throw your own exception that
implements Serializable, and
is defined in a source folder that is acccessible to (and compiled for) the client.

You could also override the RequestFactoryServlet and pass it a custom exception handler::
public class CustomRequestFactoryServlet extends RequestFactoryServlet {
private static class ApplicationExceptionLogger implements ExceptionHandler {
private final Logger log = LoggerFactory.getLogger(ApplicationExceptionLogger.class);
#Override
public ServerFailure createServerFailure(Throwable throwable) {
log.error("Server Error", throwable);
return new ServerFailure(throwable.getMessage(), throwable.getClass().getName(), throwable.getStackTrace().toString(), true);
}
}
public CustomRequestFactoryServlet() {
super(new ApplicationExceptionLogger());
}
}
In web.xml ::
<servlet>
<servlet-name>requestFactoryServlet</servlet-name>
<servlet-class>com.myvdm.server.CustomRequestFactoryServlet</servlet-class>
</servlet>

I also found you can send back a Google UmbrellaException, but you have to instantiate it a little funny because it only takes Sets in the constructor:
Server
public String getUserId () throws Exception {
Set<Throwable> s = new HashSet<Throwable>(Arrays.asList(new IllegalArgumentException("Hidey hidey ho!")));
if (true) throw new com.google.gwt.event.shared.UmbrellaException(s);
Client
public void onFailure(Throwable caught) {
log.severe("fetchUserName(), Could not fetch username: " + caught.getMessage());
Console
Mon Oct 14 12:05:28 EDT 2013 com.example.client.Login
SEVERE: fetchUserName(), Could not fetch username: Exception caught: Hidey hidey ho!

I liked Zied's and Fred's answers the best as they are the simplest and most transparent. However, no need to use UncaughtExceptionHandler or create SystemExceptions, so it can be even simpler. Just capture exceptions as normal, re-wrap, and throw. No need to litter the server Interface with exceptions (just yours). Any serious errors like OutOfMemoryError will be handled by GWT as normal. Also simpler instantiation than my other answer. GWT already has pass/fail handlers with onSuccess/onFailure so no need to re-check for a failure within onSuccess with a special return value. However, the only way to reach onFailure is with an Exception, so even though a boolean might be sufficient, an Exception is required to indicate an error to the client handler.
MyException
package com.example.shared;
import java.io.Serializable;
public class MyException extends Exception implements Serializable {
private static final long serialVersionUID = 1104312904865934899L;
public MyException() {}
public MyException (String s) {
super(s);
}
}
Server
public void cancelSend() throws MyException {
throw new MyException("Because I said so");

Detecting when a handler couldn't be started when embedding Jetty

I'm embedding Jetty in a similar manner as described here. When the RequestLogHandler can't open the specified logfile, it throws an exception which is unfortunately caught by org.eclipse.jetty.server.Server and swallowed (but logged first, at least). This means that there's no obvious way for me to tell if the log handler was started correctly.
Is there a way that I'm missing to detect when a handler couldn't start?

This idea is based on the implementation of WebAppContext where you can use WebAppContext.getUnavailableException() to determine whether the context was initialized successfully.
Simply replace the default implementation of Server and Context with your own:
public static class MyContext extends Context {
private Exception _exception;
#Override
protected void doStart() throws Exception {
try {
super.doStart();
} catch (final Exception e) {
_exception = e;
}
}
#Override
protected void doStop() throws Exception {
try {
super.doStop();
} finally {
_exception = null;
}
}
public Exception getException() {
return _exception;
}
}
public static class MyServer extends Server implements InitializingBean {
public void afterPropertiesSet() throws Exception {
start();
for (final Handler h : getHandlers()) {
if (h instanceof MyContext) {
final MyContext c = (MyContext) h;
if (c.getException() != null) {
throw new RuntimeException("failed to init context " + c.getDisplayName(),
c.getException());
}
}
}
}
}
In your beans.xml, simply replace org.mortbay.jetty.Server (and remove init-method="start") and org.mortbay.jetty.servlet.Context with your own implementations.
This code is for Jetty 6 though (as is the example you linked to), as that's what I have around. I didn't test it though, but it's pretty much the same as we are successfully using in conjunction with WebAppContext. In order to extend this to RequestLogHandler, you could either do the same for just any handler you are using or create a decorator to wrap any handler. You may want to look at org.mortbay.jetty.handler.HandlerWrapper for this purpose.

How about modifying the jetty code? You could add some simple println statements in strategic places in the RequestLogHandler which would indicate to you whether or not the handler was started.