I am trying to figure out, how to re-run failed tests with usage of Espresso. I think it's a bit more complicated from common JUnit test case as you need to restore status in your app from before test start.
My approach is to create my own ActivityTestRule so I just copied whole code from this class and named it MyActivityTestRule.
In case of instrumentation tests rule will also need information of how we want to start our activity. I prefer to launch it myself rather than have environment to do it for me. So for example:
#Rule
public MyActivityTestRule<ActivityToStartWith> activityRule = new MyActivityTestRule<>(
ActivityToStartWith.class, true, false
);
So I also launch my activity in #Before annotation method:
#Before
public void setUp() throws Exception {
activityRule.launchActivity(new Intent());
}
And make clean up in #After annotation method:
#After
public void tearDown() throws Exception {
cleanUpDataBaseAfterTest();
returnToStartingActivity(activityRule);
}
Those methods - setUp(), tearDown() are essential to be called before/after each test run - to ensure the app state during the test start is correct.
Inside MyActivityTestRule I did few modifications so far. First thing is change of apply method from:
#Override
public Statement apply(final Statement base, Description description) {
return new ActivityStatement(super.apply(base, description));
}
It's a but unknown thing for me, as ActivityStatement placed in ActivityTestRule has super.apply method so it also wraps test statement in UiThreadStatement:
public class UiThreadStatement extends Statement {
private final Statement mBase;
private final boolean mRunOnUiThread;
public UiThreadStatement(Statement base, boolean runOnUiThread) {
mBase = base;
mRunOnUiThread = runOnUiThread;
}
#Override
public void evaluate() throws Throwable {
if (mRunOnUiThread) {
final AtomicReference<Throwable> exceptionRef = new AtomicReference<>();
getInstrumentation().runOnMainSync(new Runnable() {
public void run() {
try {
mBase.evaluate();
} catch (Throwable throwable) {
exceptionRef.set(throwable);
}
}
});
Throwable throwable = exceptionRef.get();
if (throwable != null) {
throw throwable;
}
} else {
mBase.evaluate();
}
}
}
No mather what I do with my tests I can never create case mRunOnUiThread boolean to be true. It will be true if within my test cases, tests with annotation #UiThreadTest will be present - or that's what I understood from code. It never happens though, I don't use anything like that so I decided to ignore this UiThreadStatement and change MyActivityTestRule to:
#Override
public Statement apply(final Statement base, Description description) {
return new ActivityStatement(base);
}
And my test cases run without any problem. Thanks to that all I have left - that wraps around mBase.evaluate() is:
private class ActivityStatement extends Statement {
private final Statement mBase;
public ActivityStatement(Statement base) {
mBase = base;
}
#Override
public void evaluate() throws Throwable {
try {
if (mLaunchActivity) {
mActivity = launchActivity(getActivityIntent());
}
mBase.evaluate();
} finally {
finishActivity();
afterActivityFinished();
}
}
}
In general launchActivity will be called only if I set in the 3rd parameter of ActivityTestRule constructor value true. But I launch tests by myself in setUp() so it never happens.
From what I understood mBase.evaluate() runs my code inside #Test annotation method. It also stops the test case during throwable being thrown. That means I can catch it and restart it - like proposed there:
How to Re-run failed JUnit tests immediately?
And okay I did something like that:
public class ActivityRetryStatement extends Statement {
private final Statement mBase;
private final int MAX_RUNS = 2;
public ActivityRetryStatement(Statement base) {
mBase = base;
}
#Override
public void evaluate() throws Throwable {
Throwable throwable = null;
for (int i = 0; i < MAX_RUNS; i++) {
try {
mBase.evaluate();
// if you reach this lane that means evaluate passed
// and you don't need to do the next run
break;
} catch (Throwable t) {
// save first throwable if occurred
if (throwable == null) {
throwable = t;
}
// my try that didn't work
launchActivity(testInitialIntent);
// I've returned test to starting screen but
// mBase.envaluate() didn't make it run again
// it would be cool now to:
// - invoke #After
// - finish current activity
// - invoke #Before again and start activity
// - mBase.evaluate() should restart #Test on activity started again by #Before
}
}
finishActivity();
afterActivityFinished();
// if 1st try fail but 2nd passes inform me still that there was error
if (throwable != null) {
throw throwable;
}
}
}
So those comments in catch block are parts I don't know how to do. I tried to perform launchActivity on intent I used in setUp() to run the test for the 1st time. But mBase.evaluate() didn't make it react (test case didn't go again) - nothing happened + it wouldn't really save me there I think. I lacked some initiations I do in #SetUp, it wasn't called again. I would really like to find a way how to properly restart whole test lifecycle #Before #Test #After over again. Maybe some call on Instrumentation or TestRunner from code.
Any thoughts about how it could be done?
You can use https://github.com/AdevintaSpain/Barista library for rerunning failed tests.
You can see more details about dealing with flaky tests here: https://github.com/AdevintaSpain/Barista#dealing-with-flaky-tests
The answer is super simple. Just make sure you upgrade your espresso tests to Junit 4, and then see this answer
Related
I'm looking for a way to cause a succeed through an custom exception without expecting it all the time in junit4. Is this possible with a testrule or something, without touching every single testcase?
I know these options exist but then the exception is expected and the test fails, if no exception is thrown. I want the test to continue even if no exception is thrown and just use the exception to end the test in some special cases through aspectj.
#Test(TestSuccessException.class)
public void testCase() {
...
}
public class TestClass{
#Rule
public ExpectedException thrown = ExpectedException.none();
#Test
public void someTest() {
thrown.expect(MyRuntimeException.class);
...
}
}
As far as the junit4 source code looks, there isn't a way to achieve this.
The only way I found is by customizing the runner itself.
So something like this:
public class CustomTestRunner extends Runner {
private Class testClass;
public CustomTestRunner(Class testClass) {
super();
this.testClass = testClass;
}
#Override
public Description getDescription() {
return Description.EMPTY;
}
#Override
public void run(RunNotifier notifier) {
// Load all methods with #Test annotation from the given class and fire the execution
try {
Object testObject = testClass.getConstructor().newInstance();
for (Method method : testClass.getMethods()) {
if (method.isAnnotationPresent(Test.class)) {
fire(notifier, testObject, method);
}
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private void fire(RunNotifier notifier, Object testObject, Method method) throws IllegalAccessException, InvocationTargetException {
notifier.fireTestStarted(Description
.createTestDescription(testClass, method.getName()));
try {
// Call the test method
method.invoke(testObject);
} catch (InvocationTargetException e) {
// method.invoke wraps the original exception with InvocationTargetException
// The original exception is accessible via getCause()
// Check if the type of the original exception is the custom "early exist" exception
// If it doesn't match, throw the exception again; otherwise, ignore and mark the test as successful
if (!(e.getCause() instanceof EndTestEarlyException)) {
throw e;
}
}
notifier.fireTestFinished(Description
.createTestDescription(testClass, method.getName()));
}
}
You can use this by annotating the Test class as follows:
#RunWith(CustomTestRunner.class)
class MyIntegrationTest {
...
}
Note: Runner is the most generic Runner possible.
You could also attempt overriding a more specific runner if you already use one.
Edit:
As you are working with legacy, I intentionally tried not to use newer language features, like generics (Class<?>).
The solution is based on this baeldung article.
Junit5
Last but not least:
This is probably not relevant in your particular case but might be interesting for future readers.
If you manage to upgrade to Junit5, you could handle this within an extension.
You could implement a custom extension like this:
public class IgnoreEndTestEarlyException implements TestExecutionExceptionHandler {
#Override
public void handleTestExecutionException(ExtensionContext context,
Throwable throwable) throws Throwable {
if (throwable instanceof EndTestEarlyException ) {
return;
}
throw throwable;
}
}
And use it like this:
#ExtendWith(IgnoreEndTestEarlyException.class)
public class MyIntegrationTest
I tend to create another annotation (something like #IntegrationTest), put the #ExtendsWith on there, and use the new annotation.
It would be cleaner and easier to add multiple extensions.
You can run Junit4 and Junit5 within the same module, but you must replace all annotations within your integration test suit.
It might not be worth the effort for multiple thousand tests.
For Junit4 I found a better solution for my usecase. Just override the runChild Method from BlockJUnit4ClassRunner and add a try catch for the EndTestEarlyException.
#Override
protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
Description description = describeChild(method);
if (isIgnored(method)) {
notifier.fireTestIgnored(description);
} else {
Statement statement = new Statement() {
#Override
public void evaluate() throws Throwable {
try {
methodBlock(method).evaluate();
} catch (EndTestEarlyException e) {
System.out.println("EndTestEarlyException - ignore");
}
}
};
runLeaf(statement, description, notifier);
}
}
According to the documentation, non stubbed methods return null.
I want to test a method that should return "null" under some circumstances, however the test fails with the exception indicating the method was not called.
This is the test init function:
#Before
public void setup() throws Exception {
MockitoAnnotations.initMocks(this);
mDataEntry = getFakeEntry();
when(mRepository.getEntry(1L)).thenReturn(mDataEntry);
mGetEntry = new GetEntryImpl(mRepository);
}
and this is the failed test:
#SuppressWarnings("unchecked")
#Test
public void testGetEntry_failure() throws Exception {
mGetEntry.execute(2L, mCallback);
verify(mRepository).getEntry(eq(2L));
verify(mCallback).onError(anyString(), Mockito.any(Exception.class));
}
the execute method calls the mocked object mRepository function getEntry(2L) which I was expecting to return null. However, this is what Mockito tells me when I run the test:
Wanted but not invoked:
mRepository.getEntry(2);
-> at com.xyz.interactor.GetEntryTest.testGetEntry_failure(GetEntryTest.java:54)
Actually, there were zero interactions with this mock.
I tried adding
when(mRepository.getEntry(2L)).thenReturn(null);
to the init function, but it makes no difference. If I return a valid object instead of null, then the test fails as expected, because the onError function is not called (so the mocked object's function for value 2L is being called when I specify a valid return value).
How can I have the mocked object return null for a set of values?
Edit:
here's the code for the function under test:
#Override
public void execute(final long id, final Callback<DataEntry> callback) {
AsyncTask.execute(new Runnable() {
#Override
public void run() {
DataEntry dataEntry = mDataEntryRepository.getEntry(id);
if (dataEntry != null) {
callback.onResult(dataEntry);
} else {
callback.onError("TODO", null);
}
}
});
}
and for reference, the success test works:
#SuppressWarnings("unchecked")
#Test
public void testGetEntry_success() throws Exception {
mGetEntry.execute(1L, mCallback);
verify(mRepository).getEntry(eq(1L));
verify(mCallback).onResult(eq(mDataEntry));
}
I don't think the problem is with Mockito default values / returning null.
I wrote a modified SSCCE, and the tests run fine for me. I don't have the android API, so I couldn't use the AsynchTask.execute(). From what I understand, this code would run in a separate thread, so you might not be guaranteed that the code was run before verify is called. If you take out the AsynchTask and implement execute as follows, does it still fail? Is it possible for an exception to be thrown in execute?
public void execute( final long id, final Callback<DataEntry> callback) {
DataEntry dataEntry = mDataEntryRepository.getEntry(id);
if (dataEntry != null) {
callback.onResult(dataEntry);
} else {
callback.onError("TODO", null);
}
}
I'll try to provide a hackneyed, useless example that reduces the problem nicely :-)
I have a GenericException, and a MoreSpecificException which extends GenericException.
I need to test that SomeService.doThis() throws a MoreSpecificException. JUnit lets me do this elegantly like so.
#Test(expected = MoreSpecificException.class)
public void testDoThis() throws GenericException {
new SomeService().doThis();
}
However, I also need to test that SomeService.doThat() throws a GenericException, so I tried this.
#Test(expected = GenericException.class)
public void testDoThat() throws GenericException {
new SomeService().doThat();
}
However, I found that if doThat() actually throws a MoreSpecificException then the second test still passes. I assume this is because MoreSpecificException is a GenericException and the annotation is implemented to respect that relationship.
While this is a sensible default behaviour, I don't want this. I want to test that doThat() throws a GenericException and only a GenericException. If it throws a MoreSpecificException or any other subclass of GenericException, I want the test to fail.
Reading the docs it doesn't seem I can do anything with the annotation to change this behaviour, so looks like I'll have to use another solution.
At the moment I'm resorting to the following ugly solution - EDIT made significantly less ugly by Nathan Hughes' answer :-)
#Test
public void testDoThat() {
try {
new SomeService().doThat();
Assert.fail();
} catch(GenericException ex) {
Assert.assertEquals(GenericException.class, ex.getClass());
}
}
Is there a more elegant way to achieve what I want within the JUnit framework?
BDD Style Solution
JUnit 4 + Catch Exception + AssertJ
The most elegant solution ;) Readable, without boilerplate code.
#Test
public void testDoThat() {
when(new SomeService()).doThat();
then(caughtException()).isExactlyInstanceOf(GenericException.class);
}
The code is identical for FEST Assertions 2 + Catch-Exceptions.
Source code
https://gist.github.com/mariuszs/7489706
Dependencies
org.assertj:assertj-core:1.4.0
com.googlecode.catch-exception:catch-exception:1.2.0
You can assert that the class of the Exception is what you expect:
#Test
public void testDoThat() {
try {
new SomeService().doThat();
Assert.fail();
} catch(GenericException ex) {
assertEquals(GenericException.class, ex.getClass());
}
}
Also got rid of the flag, instead having the test fail if no exception is thrown.
You can use the ExpectedException rule and a custom Hamcrest matcher that specifies which class can be thrown.
The following test will print that you expected an instance of RuntimeException, but got an IllegalArgumentException.
#Rule
public ExpectedException thrown = ExpectedException.none();
#Test
public void testThrows() {
thrown.expect(isClass(RuntimeException.class));
throw new IllegalArgumentException("FAKE");
}
public class ClassMatchMatcher extends BaseMatcher<Object> {
private final Class<?> expectedClass;
private ClassMatchMatcher(Class<?> expectedClass) {
this.expectedClass = expectedClass;
}
#Override
public boolean matches(Object item) {
return expectedClass.equals(item.getClass());
}
#Override
public void describeTo(Description description) {
description.appendText("an instance of ")
.appendText(expectedClass.getName());
}
}
public class ExtraMatchers {
public static Matcher<Object> isClass(Class<?> aClass) {
return new ClassMatchMatcher(aClass);
}
}
Edit: Added a static factory method to make the test code cleaner.
I have a JUnit test class that has two test methods:
#Test
public void test1() {
// Setup: Create foo1.m
// Exercise
// Tear Down: Delete foo1.m
}
#Test
public void test2() {
// Setup: Create foo2.m
// Exercise
// Tear Down: Delete foo2.m
}
For each method, I would like to make sure that, if the Exercise section fails for any reason, the Tear Down will still run. Note that the Setup and Tear Down code for both test methods are different, so I don't think I can use JUnit's #Before and #After annotations to do what I want.
I could put TRY-CATCH blocks into each test method:
#Test
public void test2() {
// Setup: Create foo2.m
try {
// Exercise
} finally {
// Tear Down: Delete foo2.m
}
}
but that seems ugly. Is there a way to make sure the test-method-specific tear down code in each test method is executed, without using a TRY-CATCH block?
If the setup and teardown are different, you are essentially cramming two different test fixtures into a single file. The sensible answer is to put them in separate files and use the normal annotations. If they have anything in common separate that out into a common abstract class.
Adding multiple setups in the same file can easily result in a situation where it's not clear which instance members are used in which tests, so that maintaining the tests becomes a lot harder than it needs to be.
Update:
I found a better solution, so I include here, the original answer can be found below. I think JUnit 4 rules can be used here:
class PrepareFile implements org.junit.rules.TestRule {
#Retention(RetentionPolicy.RUNTIME)
public #interface FileName {
String value() default "";
}
#Override
public Statement apply(final Statement statement, final Description description) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
String fileName = description.getAnnotation(FileName.class).value();
File file = new File(fileName);
try {
file.createNewFile();
statement.evaluate();
} finally {
file.delete();
}
}
};
}
}
Using it in the test:
#Rule
public PrepareFile prepareFile = new PrepareFile();
#Test
#PrepareFile.FileName("foo1.m")
public void test1() {
// Exercise
}
#Test
#PrepareFile.FileName("foo2.m")
public void test2() {
// Exercise
}
Here comes my original answer:
You may try to use the #BeforeClass and #AfterClass annotations.
#BeforeClass
public static void setUp() {
// Setup1: Create foo1.m
// Setup2: Create foo2.m
}
#AfterClass
public static void tearDown() {
// Tear Down1: Delete foo1.m
// Tear Down2: Delete foo2.m
}
#Test
public void test1() {
// Exercise
}
#Test
public void test2() {
// Exercise
}
This way you can setup and tear down all test cases once and the framework ensures that teadDown() is called in case of errors as well.
I have to write a Test case in JUnit for a Class lets call it C1 which internally calls Runtime.getRuntime.exit(somevalue).
The class C1 has a main method which accepts some arguments and the creates a CommandLine and then depending on the passed arguments does the specific tasks.
Now all tasks after executing call a Runtime.getRuntime.exit(somevalue). The somevalue defines whether the task was executed successfully (means somevalue is 0) or had errors (means somevalue is 1).
In the JUnit test case of this I have to get this somevalue and check whether it is the desired somevalue or not.
How do I get the somevalue in the JUnit test case.
You can override security manager to catch the exit code, if you use a mocking framework it would be more concise:
#Test
public void when_main_is_called_exit_code_should_be_1() throws Exception {
final int[] exitCode = new int[1];
System.setSecurityManager(new SecurityManager() {
#Override
public void checkExit(int status) {
exitCode[0] = status;
throw new RuntimeException();
}});
try { main(); } catch(Exception e) {}
assertEquals(exitCode[0], 1);
}
public static void main() {
System.exit(1);
}