My application has a flow that in the end of it the method System.exit(int) is being called.
I'm trying to test this flow by running a test using TestNG.
However, when running the test I'm getting this weird message although the test was completed:
Just for finding the root cause, I removed the System.exit(int) from the real flow and the test passed as expected, so the problem here is the System.exit(int) method.
In order to solve this issue I've tried to mock the problematic method but couldn't find the right way to do it. Here is what I did
I added java.lang.System.class under #PrepareForTest in the tests class.
added PowerMockito.mockStatic(java.lang.System.class) in the test
I've tried to mock the method in two ways:
a.
PowerMockito.replace(PowerMockito.method(System.class, "exit", int.class))
.with((proxy, method, args) -> null);
When running this way looks like the mock is not working, because I'm getting the same message at the end of the test which I also got when not applying any mocks on System.exit(int)
b.
PowerMockito.doNothing().when(System.class, "exit", Mockito.any());
In this way I'm getting this exception at the beginning of the test:
org.powermock.reflect.exceptions.MethodNotFoundException: No method found with name 'exit' with parameter types: [ <none> ] in class java.lang.System.
I already mocked some methods in this ways, not sure why with System.exit(int) it is not working.
Any ideas?
Thanks
Interesting question, I didn't know either, but apparently this is possible without the use of Powermock, through the use of SecurityManagers. Quoting from the original post:
Today I was writing test for one of our command line tools, and I had
this problem where the method dumping everything, which I really
needed to be called since that’s the outcome I was checking, also
called System.exit(). I had to find a way to test this anyway. I
thought about using PowerMock and mocking system but that would have
been complicated because I would have to find the exact class calling
the System.exit(). So here is another solution to avoid the
System.exit to exit (yes that’s possible I didn’t know about that
either).
The secrets lays in the SecurityManager mechanism of Java, this class
not only allows you to check permissions, but also to check exit
event. Therefore you can throw an exception if you want to stop the
exit
Below is a full sample I tested in IJ. Please note the sample is supposed to fail on purpose with:
java.lang.AssertionError:
Expected: is <10>
but: was <5>
Expected :is <10>
Actual :<5>
package com.example;
import org.junit.Test;
import java.security.Permission;
import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
public class SystemExitTest {
#Test
public void shouldExitWithSpecificCode() {
//save the initial security manager (probably null)
SecurityManager initialSecurityManger = System.getSecurityManager();
try {
// set security manager
System.setSecurityManager(new NoExitSecurityManager());
// execute code under test
new MyClass().exit(5);
// ensure this point is not reached by any chance
fail("Should've thrown ExitException");
} catch (ExitException e) {
// validate exit code
assertThat(e.status, is(10)); // <== this fails on purpose
} finally {
// restore initial security manager (otherwise a new ExitException will be thrown when the JVM will actually exit)
System.setSecurityManager(initialSecurityManger);
}
}
// class under test
public static class MyClass {
public void exit(int code) {
System.exit(code);
}
}
// exception to be thrown by security manager when System.exit is called
public static class ExitException extends SecurityException {
public final int status;
public ExitException(int status) {
this.status = status;
}
}
// custom security manager
public static class NoExitSecurityManager extends SecurityManager {
#Override
public void checkPermission(Permission perm) {
}
#Override
public void checkPermission(Permission perm, Object context) {
}
#Override
public void checkExit(int status) {
super.checkExit(status);
throw new ExitException(status);
}
}
}
Related
I have a TestRunner class:
public class TestRunner {
private String result = "";
public void runTests(List<String> testClassNames) {
for (String testClassName : testClassNames) {
}
}
public String getResult() {
return result;
}
}
Then I have two test classes and I should run only the tests with #MyTest from both classes. getResult() should essentialy return "exampleTest() - OK" or "exampleTest() - FAILED", depending on if the test passes. Test class looks like this:
public class ExampleTests1 {
#MyTest (expected = IllegalStateException.class)
public void test1() {
throw new IllegalStateException();
}
#MyTest (expected = IllegalStateException.class)
public void test2() {
throw new RuntimeException();
}
}
I tried to do it with JUnitCore and Result, but couldn't find a way to get the test method name to add to the result. Is there another way?
You'll have to use reflection in this case.
This is kind of how JUnit works internally:
Since this sounds as a homework/assigment for educational purposes (If you're developing a "real" application - then just use JUnit or TestNG and don't write anything like this by yourself :) ), I won't provide a full solution however this is what you should do:
For each class identified by a className you should get a java.lang.Class that describes this class name. You can use Class.forName(...)
Then you should get all the methods of that class (by reflection) and for each method run the following:
2.1 Check whether the method is marked with an annotation #MyTest. If it doesn't - don't handle it
2.2 Also check whether the method name starts with test (String has startsWith method)
2.3 If you found out that the test class contains test methods, then:
2.3.1 Create an instance of the Test Class (probably you can assume that it has no-ops constructor, then use newInstance())
2.3.2 Run the method (again by reflection). Check the result / surround the execution call with try/catch block to intercept errors.
2.3.3 Print the Result as specified in the assignment :)
I have written some unit tests for a static method. The static method takes only one argument. The argument's type is a final class. In terms of code:
public class Utility {
public static Optional<String> getName(Customer customer) {
// method's body.
}
}
public final class Customer {
// class definition
}
So for the Utility class I have created a test class UtilityTests in which I have written tests for this method, getName. The unit testing framework is TestNG and the mocking library that is used is Mockito. So a typical test has the following structure:
public class UtilityTests {
#Test
public void getNameTest() {
// Arrange
Customer customerMock = Mockito.mock(Customer.class);
Mockito.when(...).thenReturn(...);
// Act
Optional<String> name = Utility.getName(customerMock);
// Assert
Assert.assertTrue(...);
}
}
What is the problem ?
Whereas the tests run successfully locally, inside IntelliJ, they fail on Jenkins (when I push my code in the remote branch, a build is triggered and unit tests run at the end). The error message is sth like the following:
org.mockito.exceptions.base.MockitoException: Cannot mock/spy class
com.packagename.Customer Mockito
cannot mock/spy because :
- final class
What I tried ?
I searched a bit, in order to find a solution but I didn't make it. I note here that I am not allowed to change the fact that Customer is a final class. In addition to this, I would like if possible to not change it's design at all (e.g. creating an interface, that would hold the methods that I want to mock and state that the Customer class implements that interface, as correctly Jose pointed out in his comment). The thing that I tried is the second option mentioned at mockito-final. Despite the fact that this fixed the problem, it brake some other unit tests :(, that cannot be fixed in none apparent way.
Questions
So here are the two questions I have:
How that is possible in the first place ? Shouldn't the test fail both locally and in Jenkins ?
How this can be fixed based in the constraints I mentioned above ?
Thanks in advance for any help.
An alternative approach would be to use the 'method to class' pattern.
Move the methods out of the customer class into another class/classes, say CustomerSomething eg/CustomerFinances (or whatever it's responsibility is).
Add a constructor to Customer.
Now you don't need to mock Customer, just the CustomerSomething class! You may not need to mock that either if it has no external dependencies.
Here's a good blog on the topic: https://simpleprogrammer.com/back-to-basics-mock-eliminating-patterns/
How that is possible in the first place? Shouldn't the test fail both locally and in Jenkins ?
It's obviously a kind of env-specifics. The only question is - how to determine the cause of difference.
I'd suggest you to check org.mockito.internal.util.MockUtil#typeMockabilityOf method and compare, what mockMaker is actually used in both environments and why.
If mockMaker is the same - compare loaded classes IDE-Client vs Jenkins-Client - do they have any difference on the time of test execution.
How this can be fixed based in the constraints I mentioned above?
The following code is written in assumption of OpenJDK 12 and Mockito 2.28.2, but I believe you can adjust it to any actually used version.
public class UtilityTest {
#Rule
public InlineMocksRule inlineMocksRule = new InlineMocksRule();
#Rule
public MockitoRule mockitoRule = MockitoJUnit.rule();
#Test
public void testFinalClass() {
// Given
String testName = "Ainz Ooal Gown";
Client client = Mockito.mock(Client.class);
Mockito.when(client.getName()).thenReturn(testName);
// When
String name = Utility.getName(client).orElseThrow();
// Then
assertEquals(testName, name);
}
static final class Client {
final String getName() {
return "text";
}
}
static final class Utility {
static Optional<String> getName(Client client) {
return Optional.ofNullable(client).map(Client::getName);
}
}
}
With a separate rule for inline mocks:
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
import org.mockito.internal.configuration.plugins.Plugins;
import org.mockito.internal.util.MockUtil;
import java.lang.invoke.MethodHandles;
import java.lang.invoke.VarHandle;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
public class InlineMocksRule implements TestRule {
private static Field MOCK_MAKER_FIELD;
static {
try {
MethodHandles.Lookup lookup = MethodHandles.privateLookupIn(Field.class, MethodHandles.lookup());
VarHandle modifiers = lookup.findVarHandle(Field.class, "modifiers", int.class);
MOCK_MAKER_FIELD = MockUtil.class.getDeclaredField("mockMaker");
MOCK_MAKER_FIELD.setAccessible(true);
int mods = MOCK_MAKER_FIELD.getModifiers();
if (Modifier.isFinal(mods)) {
modifiers.set(MOCK_MAKER_FIELD, mods & ~Modifier.FINAL);
}
} catch (IllegalAccessException | NoSuchFieldException ex) {
throw new RuntimeException(ex);
}
}
#Override
public Statement apply(Statement base, Description description) {
return new Statement() {
#Override
public void evaluate() throws Throwable {
Object oldMaker = MOCK_MAKER_FIELD.get(null);
MOCK_MAKER_FIELD.set(null, Plugins.getPlugins().getInlineMockMaker());
try {
base.evaluate();
} finally {
MOCK_MAKER_FIELD.set(null, oldMaker);
}
}
};
}
}
Make sure you run the test with the same arguments. Check if your intellij run configurations match the jenkins. https://www.jetbrains.com/help/idea/creating-and-editing-run-debug-configurations.html. You can try to run test on local machine with the same arguments as on jenkins(from terminal), if it will fail that means the problem is in arguments
Here is the code that I am working with. In this test I want to verify that the log method is being called when an exception is caught.
public class SuperClass(){
public void log()
{
do some logging;
}
}
public class ClassUnderTest extends SuperClass(){
public String methodbeingtested(Object param)
{
try
{
String a = SomeObject.
methodthatthrowsexception(param);//static method, throws JAXB/NPE
}
catch(Exception exp)
{
log("log msg",exp);//inherited method
}
}
}
public class ClassUnderTestTest {
#Test
public testmethodbeingtested(){
ClassUnderTest cut = new ClassUnderTest()
ClassUnderTest cutspy = Mockito.spy(cut);
cutspy.methodbeingtested(param);
Mockito.verify(cutspy).log("log msg", new Exception()); // exp is needed to here.
}
}
After looking at several samples, the above was the closest I could get. This testcase forces an exception. But it fails to verify the log method call as Mockito.verify requires the exact exception (exp) that is caught, which the test case does not have access to.
Is there any other way to test this scenario?
Mockito's verify method can be used with argument matchers. If you want to verify that log was called, with any Exception at all as the second argument, you can just write
verify(cutspy).log(eq("log msg"), any(Exception.class));
I've assumed that you have the right static imports for verify, eq and any.
As an aside, this test does not need PowerMock. Your line PowerMock.expectLastCall().once(); is both redundant and confusing, and should probably be removed, along with the #PrepareForTest annotation.
Instead of spying on ClassUnderTest, you should mock the logging framework, inject it into the class and then verify that the log method gets called. Id' also mock the SomeObject class and have it throw exception.
As an aside, you should really evaluate if you need to verify your log statements. Perhaps you have a valid reason to do so but typically, asserting/verifying to this extent is not required and will only make your tests brittle.
How can I make the 3rd test to check for the existence of cause1 in the message of the exception? I also listed in the first two tests that have drawbacks. First is not checking for the message second needs a lot of boilerplate code.
public class CheckExceptionsWithMockitoTest {
#Test(expected = RuntimeException.class)
public void testExpectedException1() {
A a = new A();
a.doSomethingThatThrows();
}
#Test
public void testExpectedException2() {
A a = new A();
try {
a.doSomethingThatThrows();
fail("no exception thrown");
} catch (RuntimeException e) {
assertThat(e.getMessage(), org.hamcrest.Matchers.containsString("cause1"));
}
}
#Test
public void testExpectedException3() {
A a = new A();
A spyA = org.mockito.Mockito.spy(a);
// valid but doesnt work
// doThrow(new IllegalArgumentException()).when(spyA).doSomethingThatThrows();
// invalid but in the spirit of what i want
//chekThrow(RuntimeException.class,containsString("cause1")).when(spyA).doSomethingThatThrows();
}
}
I couldn't find in Mockito something that works but there is something that looks like could be possible (at the level of syntax) and capabilities.
Using catchexception I created the test like this
import static com.googlecode.catchexception.CatchException.*;
import static com.googlecode.catchexception.apis.CatchExceptionHamcrestMatchers.*;
import static org.hamcrest.Matchers.*;
import static org.junit.Assert.*;
import org.junit.*;
public class CheckExceptionsWithMockitoTest{
//...
#Test
public void testExpectedException3() {
A a = new A();
verifyException(a,IllegalArgumentException.class)
.doSomethingThatThrows();
//if more details to be analized are needed
assertThat(
(IllegalStateException) caughtException(),
allOf(
is(IllegalStateException.class),
hasMessageThat(
containsString("is not allowed to add counterparties")),
hasNoCause()));
//more asserts could come
assertNotNull(a);
}
}
Use catch-exception library, or I guess that the solution you are looking for is your second implementation.
#expected doesn't provide any way to assert on the thrown exception except for its class, so you can't avoit try/catching (not that much boiler plate code !)
Mockito doesn't provide something likes a verifyThrows method.
So you can trade try/catching for an additional library : using catch-exception, you'll be able to catch exception in a single line and have it ready for further assertion(s).
Sample source code
A a = new A();
when(a).doSomethingThatThrows();
then(caughtException())
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("is not allowed to add counterparties")
.hasNoCause();
Dependencies
'com.googlecode.catch-exception:catch-exception:1.2.0'
If A is your system under test, it doesn't make any sense to mock it, and it rarely makes sense to spy on it. Your implementation in testExpectedException2 is the right one; the boilerplate code is necessary because without a try block Java will not let any code run after the method is intercepted (as I described in this previous SO answer).
Though Mockito won't be any help, JUnit will. The #Test(expected=foo) parameter actually has a more-flexible alternative, the built-in ExpectedException JUnit rule:
public class CheckExceptionsWithMockitoTest {
#Rule public ExpectedException thrown = ExpectedException.none();
#Test
public void testExpectedException1() {
A a = new A();
thrown.expect(RuntimeException.class);
thrown.expectMessage(containsString("cause1"));
a.doSomethingThatThrows();
}
}
Mockito would come in handy in a separate test checking whether your method wraps an arbitrary exception while preserving its message, which would look roughly like this:
#Test
public void doSomethingShouldWrapExceptionWithPassedMessage() {
Dependency dependency = Mockito.mock(Dependency.class);
when(dependency.call()).thenThrow(new IllegalArgumentException("quux"));
A a = new A(dependency);
thrown.expect(RuntimeException.class);
thrown.expectMessage(containsString("quux"));
a.doSomethingThatThrows();
}
Be careful to avoid the temptation to make this a common pattern in your tests. If you are catching an exception thrown from your system under test, you're effectively ceding control back to the SUT's consumer. There should be little left to test in the method afterwards, except the properties of the exception and MAYBE the state of your system, both of which should be rare enough that try/catch boilerplate is forgivable.
If you have the opportunity to use scala, scalaTest's fun suite has concise way of testing exceptions using intercept (http://www.scalatest.org/getting_started_with_fun_suite).
It's as simple as
test(a list get method catches exceptions){
intercept[IndexOutBoundsException]{
spyListObject.get(-1)
}
}
You could potentially write your tests to your java project in scala if you are looking for easy to write / clear test. But this may present other challenges.
Updated answer for 06/19/2015 (if you're using java 8)
Using assertj-core-3.0.0 + Java 8 Lambdas
#Test
public void shouldThrowIllegalArgumentExceptionWhenPassingBadArg() {
assertThatThrownBy(() -> myService.sumTingWong("badArg"))
.isInstanceOf(IllegalArgumentException.class);
}
Reference: http://blog.codeleak.pl/2015/04/junit-testing-exceptions-with-java-8.html
Using catchexception I created the test like this
import static com.googlecode.catchexception.CatchException.*;
import static com.googlecode.catchexception.apis.CatchExceptionHamcrestMatchers.*;
import static org.hamcrest.Matchers.*;
import static org.junit.Assert.*;
import org.junit.*;
public class CheckExceptionsWithMockitoTest{
//...
#Test
public void testExpectedException3() {
A a = new A();
verifyException(a,IllegalArgumentException.class)
.doSomethingThatThrows();
//if more details to be analized are needed
assertThat(
(IllegalStateException) caughtException(),
allOf(
is(IllegalStateException.class),
hasMessageThat(
containsString("is not allowed to add counterparties")),
hasNoCause()));
//more asserts could come
assertNotNull(a);
}
}
If you have a look in Mockito.class on spy method it creates mock with spiedInstance:
public static <T> T spy(T object) {
return MOCKITO_CORE.mock((Class<T>) object.getClass(), withSettings()
.spiedInstance(object)
.defaultAnswer(CALLS_REAL_METHODS));
}
In MockSettings it is possible to register Invocation listeners: https://static.javadoc.io/org.mockito/mockito-core/3.0.0/org/mockito/listeners/InvocationListener.html
I created simple listener which stores all reported invocations:
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import org.mockito.listeners.InvocationListener;
import org.mockito.listeners.MethodInvocationReport;
public class StoringMethodInvocationListener implements InvocationListener {
private List<MethodInvocationReport> methodInvocationReports = new ArrayList<>();
#Override
public void reportInvocation(MethodInvocationReport methodInvocationReport) {
this.methodInvocationReports.add(methodInvocationReport);
}
public List<MethodInvocationReport> getMethodInvocationReports() {
return Collections.unmodifiableList(methodInvocationReports);
}
}
After the invocation you can go through reports and find the one needed and verify that stored throwable is the one expected.
Example:
StoringMethodInvocationListener listener = new StoringMethodInvocationListener();
Consumer mock2 = mock(Consumer.class, withSettings()
.spiedInstance(consumerInstance)
.defaultAnswer(CALLS_REAL_METHODS)
.invocationListeners(listener));
try {
mock2.listen(new ConsumerRecord<String, String>(RECEIVER_TOPIC, 0, 0, null, "{}"));
} catch (Exception e){
//nothing
}
Assert.notEmpty(listener.getMethodInvocationReports(), "MethodInvocationReports list must not be empty");
Assert.isInstanceOf(BindException.class, listener.getMethodInvocationReports().get(1).getThrowable());
OK, so the #Ignore annotation is good for marking that a test case shouldn't be run.
However, sometimes I want to ignore a test based on runtime information. An example might be if I have a concurrency test that needs to be run on a machine with a certain number of cores. If this test were run on a uniprocessor machine, I don't think it would be correct to just pass the test (since it hasn't been run), and it certainly wouldn't be right to fail the test and break the build.
So I want to be able to ignore tests at runtime, as this seems like the right outcome (since the test framework will allow the build to pass but record that the tests weren't run). I'm fairly sure that the annotation won't give me this flexibility, and suspect that I'll need to manually create the test suite for the class in question. However, the documentation doesn't mention anything about this and looking through the API it's also not clear how this would be done programmatically (i.e. how do I programatically create an instance of Test or similar that is equivalent to that created by the #Ignore annotation?).
If anyone has done something similar in the past, or has a bright idea of how else I could go about this, I'd be happy to hear about it.
The JUnit way is to do this at run-time is org.junit.Assume.
#Before
public void beforeMethod() {
org.junit.Assume.assumeTrue(someCondition());
// rest of setup.
}
You can do it in a #Before method or in the test itself, but not in an #After method. If you do it in the test itself, your #Before method will get run. You can also do it within #BeforeClass to prevent class initialization.
An assumption failure causes the test to be ignored.
Edit: To compare with the #RunIf annotation from junit-ext, their sample code would look like this:
#Test
public void calculateTotalSalary() {
assumeThat(Database.connect(), is(notNull()));
//test code below.
}
Not to mention that it is much easier to capture and use the connection from the Database.connect() method this way.
You should checkout Junit-ext project. They have RunIf annotation that performs conditional tests, like:
#Test
#RunIf(DatabaseIsConnected.class)
public void calculateTotalSalary() {
//your code there
}
class DatabaseIsConnected implements Checker {
public boolean satisify() {
return Database.connect() != null;
}
}
[Code sample taken from their tutorial]
In JUnit 4, another option for you may be to create an annotation to denote that the test needs to meet your custom criteria, then extend the default runner with your own and using reflection, base your decision on the custom criteria. It may look something like this:
public class CustomRunner extends BlockJUnit4ClassRunner {
public CTRunner(Class<?> klass) throws initializationError {
super(klass);
}
#Override
protected boolean isIgnored(FrameworkMethod child) {
if(shouldIgnore()) {
return true;
}
return super.isIgnored(child);
}
private boolean shouldIgnore(class) {
/* some custom criteria */
}
}
Additionally to the answer of #tkruse and #Yishai:
I do this way to conditionally skip test methods especially for Parameterized tests, if a test method should only run for some test data records.
public class MyTest {
// get current test method
#Rule public TestName testName = new TestName();
#Before
public void setUp() {
org.junit.Assume.assumeTrue(new Function<String, Boolean>() {
#Override
public Boolean apply(String testMethod) {
if (testMethod.startsWith("testMyMethod")) {
return <some condition>;
}
return true;
}
}.apply(testName.getMethodName()));
... continue setup ...
}
}
A quick note: Assume.assumeTrue(condition) ignores rest of the steps but passes the test.
To fail the test, use org.junit.Assert.fail() inside the conditional statement. Works same like Assume.assumeTrue() but fails the test.