I am struggling to know how to decide what objects my test object interacts with should be mocked.
class MyClass {
private Customer customer;
private Invoice invoice;
private PrintService ps;
private DBAccessService da;
private EmailService em;
..........
}
I can see that the last 3 are to be mocked because they deal with some external systems. What about Customer and Invoice? If not, why?
The general answer to the question "Should I mock all objects that my test object interacts with?" is a resounding no: that is not a rule you should internalize or follow. Mocks are one of many types of test double, and you'll have to use judgment about where to use mocks, where to use other fakes or test doubles, and where to use real collaborators. I do agree with your decision that the last three fields (services) are probably worth mocking, and if it were up to me, I would use real objects for the former two fields (Customer and Invoice).
Here, I would adhere to a guideline, don't mock data objects, which follows from a few observations:
Data objects are often extremely stateful, and mocking frameworks tend to stub state poorly. Mockito doesn't really have a great syntax for "getX returns 15 indefinitely, until you call setX(20), then getX returns 20". Thus, stubbing correctly is often hard.
For data objects that are just fields/getters/setters, there's not much value to verifying that couldn't be done by reading mutable state out of a real implementation. Who cares whether getY was called, as long as the value in the object was read? Who cares how many times setY was called, as long as the correct value ends up in the object? Thus, verifying is often unnecessary.
Data objects are usually written before the objects that consume them, so there's often a working implementation that already exists.
Data objects often have determistic behavior with few external interactions, so there's often little to be gained in improving test stability or reducing test flakiness.
As biziclop mentioned in the question comments,
always remember what the purpose of automated testing is: to find errors in the shortest possible time with the smallest possible effort. Or to turn it around: to build up confidence about the correctness of your code.
In this sense, mocking data objects has a high cost to the readability and correctness of your test, and is also unlikely to provide benefit regarding test correctness or stability. I'd avoid it.
Related
I have a class that I am trying to write a few unit tests for, specifically, a void method within the class. Im not entirely sure whether or not the fact that the method is void is of any consequence here, but this is the basic structure of the class.
public class MyClass {
private Map<String,Object> myMap;
public void process(Bucket bucket){
//parses bucket, updates myMap
}
public int getAnswer(String input){
//searches map data
}
}
The basic usage of the class is that the process method is called for each bucket to ingest it’s data, and the getAnswer method returns a summation of sorts about the data that has been processed from the buckets.
I’ve written a few unit tests for the getAnswer method, but since the process method doesn’t return anything, but rather just updates the internal state of my object, is there a way to properly unit test it? Since it’s a public method, I figured that it needed to be unit tested, but haven’t figured out how to do so, independent of the getAnswer method.
We would need more code to answer you in details, but in short the answer is yes, you can and should definitely test it.
process is going to make some side-effects somewhere, right? Either you inject Mockito mocks of the classes/interfaces these side-effects are effected upon, or if you have an in-memory version of them you inject that instead.
In the case of mocks, you'll want to carefully assert that the side-effect methods were called exactly with the right parameters. If you're using an in-memory simulation, you check the state of your in-memory object and check it conforms to what you expected.
For unit testing of process method, I think you can do following:
Verify input bucket can be able to parse based on your method comments.
Exception handling -- if applicable -- if bucket is unable to parse.
For map update / add, you can always check if initial size is changed & valid bucket made successful attempt to add to map. If bucket key already present, you can also verify if size of map remains same.
All above points are based on information provided in original question & it is independent of getAnswer method.
I'm in a java context and am using Mockito (but I'm not bound to it) for basic mocking needs.
I have code like this
public class AuditInfoSerializer {
[..]
public Map<String, Object> doStuff(Object a) {
doOtherStuff("hello", new TempClass(someField, <someParams>));
doOtherStuff("world", new TempClass(someField, <otherParams>));
return getResult();
}
}
and in a test I want to verify that there are two instances of TempClass created with the correct set of parameters when I call the doStuff method.
Is this possible somehow?
You don't want to verify temporary data on the object under test. You want to mock dependencies and assert the object under test behavior : that is with this input you have this output.
Mock verifying is a trade off for methods to mock that return nothing but only produces side effect.
So use it only as you don't have the choice.
In your unit test, what you want is asserting what the method to test returns that is getResult().
Do that with Assert.assertEquals(...) not with Mockito.verify(...).
For the most part I agree with #davidxxx's point about the mock verifying tradeoff. If you have a setup that allows you to make assertions about an outcome like a map that is created as a result, go for it!
From an API perspective doStuff is a simple straight-forward method: Throw something at it, get something back. The information you are interested in will be contained in the map (this would be your assertion).
There is a lot going on under the hood before doStuff returns something. Many people tend to want to break up encapsulation when testing stuff. They are constantly looking for ways to uncover what is going on behind the curtains. I believe that's totally natural. But of course, it's also an anti pattern. It doesn't matter what tool you (mis)use to break natural boundaries (mocking frameworks, custom reflection, "back doors" in your code base, etc). It is always wrong. As #Michael already pointed out, the call to doOtherStuff is indeed an implementation detail. Take the perspective of client code that makes a call to doStuff. Is it interested in how the map is created? I doubt it. This should also be your testing perspective.
One last thing about using verification in tests. I would like to mitigate the trade off statement. I really don't like the generalization here. Verification is not always the less attractive choice compared to real assertions:
// Valid test without any verifaction
#Test
void method_foo_returns_gibberish (#Mock SomeInput someInput) {
// Maybe this is just to prevent an NPE ...
when(someInput.readStuff()).thenReturn("bla");
assertEquals("gibberish", Foo.foo(someInput));
}
// Test made possible by verification
#Test
void method_foo_is_readonly (#Mock SomeInput someInput) {
Foo.foo(someInput);
verify(someInput.readStuff());
verifyNoMoreInteractions(mockedList);
}
This is just the most obvious example that I could think of. There is a fraction of BDD geniuses who strive to build their whole architecture around verification driven tests! Here is an excellent article by Martin Fowler
When talking about testing, most of the time, there is no black and white. Using mocks and verification means writing different tests.
As always, it's about picking the right tool.
Since recently, I am trying out unit testing to get acquainted with the practice, and to ultimately write better code. I have started on one of my big projects, with a rather large untested code base, but am only unit testing utility classes, that don't have many dependencies and are quite easy to test.
I have read quite a bit of introductory material to unit testing in general, and what often comes up is that unit tests should always test the smallest possible unit of behaviour. That is to say, that whether a test passes or fails should only depend on a very specific piece of code, such as a small method.
However, I am already finding problems to put this idea into practice. Consider for example :
#Test
public void testSomethingWithFoo() {
Foo foo = new Foo(5);
Bar result = foo.bar(); //Suppose this returns new Bar(42) for some reason.
Bar expected = new Bar(42);
Assert.assertEquals(expected, result); // Implicitly calls Bar.equals, which returns true if both Bars' constructor parameters are equal.
}
Clearly, this test depends on two elements : the foo.bar() method, but also the bar.equals(otherBar) method, which is implicitly called by the assertion.
Now, I could write an other test, which asserts that this bar.equals() method works correctly. However, say it fails. Now, my first test also should fail, but for a reason beyond its scope.
My point is that, for this particular example, nothing really is problematic; I could maybe check for equality without using equals at all. However, I feel like this sort of issue will become a real problem with more complex behaviours, where avoiding existing methods because they might not work would involve rewriting large amounts of code just for tests.
How to translate these requirements into code, without making unit tests interdependent ?
Is it even possible ? If not, should I stop bothering about this issue, and assume all methods not under the current test work ?
However, say it fails. Now, my first test also should fail, but for a
reason beyond its scope.
Test isolation is a really important thing, you are right but note also that this has a limitation.
You will necessary fall into some cases where in your unit tests, you will have to rely on other general methods of other objects such as equals/hashCode or still constructors.
This is unavoidable. Sometimes, the coupling is not an issue.
For example, using a constructor to create the object passed to the method under test or a mock value is natural and should really not be avoided !
But in other cases, the coupling with the API of another class is not desirable.
It is for example the case in your sample code as your test relies on an equals() method which the fields tested may change through the time and also be functionally different from fields to watch/assert in the tested method.
In addition, this doesn't make the fields asserted explicit.
In this kind of case, to assert fields values of a returned object by the method under test, you should assert field by field via getter methods.
To avoid writing this repetitive and error prone code, I use a matcher testing library such as AssertJ or Hamcrest (included in JUnit at a time) that provides fluent and flexible way to assert the content of an instance.
For example with AssertJ :
Foo foo = new Foo(5);
Bar actualBar = foo.bar();
Assertions.assertThat(actualBar)
.extracting(Bar::getId) // supposing 42 is stored in id field
.containsExactly(42);
With such a simple example, matcher testing library has no real value. You could directly do :
Foo foo = new Foo(5);
Bar actualBar = foo.bar();
Assert.assertEquals(42, actualBar.getId())
But for cases where you want to check multiple fields (very common case), it is very helpful :
Foo foo = new Foo(5);
Bar actualBar = foo.bar();
Assertions.assertThat(actualBar)
.extracting(Bar::getId, Bar::getName, Bar::getType)
.containsExactly(42, "foo", "foo-type);
I've found it helpful to divide my classes into two main types:
Types which hold data. They represent your data's structure, and contain little to no logic.
Types which hold logic, and hold no state.
The reason for this is that many classes at many layers will inevitably be tied to your data model, so you want the representation of your data model to be rock solid and unlikely to yield any surprises.
If Bar is a data structure, then relying on the behavior of its .equals() method isn't much different than relying on the behavior of int or String's .equals() behavior. You should stop worrying about it.
On the other hand, if Bar is a logic class, you probably shouldn't be relying on its behavior at all when you're trying to test Foo. In fact, Foo shouldn't be creating a Bar at all (unless Foo is a BarFactory, e.g.). Instead, it should be relying on interfaces which can be mocked.
When it comes to testing your Data types (especially in the exceptional cases where the data type really needs to have some logic, like a Uri class, you will of a necessity be testing multiple methods within that class (like Bar's constructor and equals method). But the tests should all be testing that class specifically. Make sure you've got plenty of tests to keep these rock-solid.
When testing a service class (one with logic), you will effectively be assuming that any data types you're dealing with have been tested sufficiently so you're really testing the behavior of the type you're worried about (Foo) and not the other types you happen to be interacting with (Bar).
I am currently debugging a rather complicated algorithm that fixes errors in a bit stream. A BitReader interface is quite simple, and the main reading method is like this:
/**
Reads bits from the stream.
#param length number of bits to read (<= 64)
#return read bits in the least significant bits
*/
long read(int length) throws IOException;
The objective is to test whether BitStreamFixer actually fixes the stream (in a way that is too hard to describe here). Basically I need to provide “broken” inputs to it and test whether its output is as correct as it can be (some inputs can't be fixed completely), like this:
BitStreamFixer fixer = new BitStreamFixer(input);
int word1 = fixer.readWord();
int word2 = fixer.readWord();
// possibly a loop here
assertEquals(VALID_WORD1, word1);
assertEquals(VALID_WORD2, word2);
// maybe a loop here too
Now, the BitStreamFixer class accepts an instance of BitReader. When unit testing the fixer, I obviously need one such instance. But where do I get one? I have two obvious options: either give it a real implementation of BitReader or mock it.
The former option is not really appealing because it would create a dependency on another object which has nothing to do with the class being tested. Moreover, it's not that easy because existing BitReader implementations read form input streams, so I'll need either a file or somehow prepared byte array, which is a tedious thing to do.
The latter option looks better and fits the usual unit testing approach. However, since I'm not even supposed to know what arguments the fixer will give to read, mocking it is not easy. I'll have to go with when(bitReader.read(anyInt())).thenAnswer(...) approach, implementing a custom answer that will create a lot of bit-fiddling logic to spoon-feed the object under test with proper bits in chunks of whatever size it asks for. Considering that bit streams I'm working with have rather complicated higher-level structure, it's not easy. And introducing logic in unit tests also doesn't smell good.
What do you think, is there any other option? Or maybe one of these can be improved in a way I fail to notice?
Write, test, and use a clear reusable test helper.
In a general sense, in unit testing, you're supposed to establish confidence in a system by watching it successfully interact with systems that you DO have confidence in. Of course you also want the system to be fast, deterministic, and easy to read/modify, but ultimately those come secondary to the assertion that your system work.
You've listed two options:
Use a mock BitReader, where you have enough confidence in predicting your system's interactions that you can set up the entire "when A then B" conversation. Mocking can be pretty easy when you have a small API surface of independent methods, like an RPC layer, but mocking can be very difficult when you have a stateful object with unpredictable method calls. Mocking is further useful to deterministically stub nondeterministic systems, like external servers or pseudorandom sources, or systems that don't exist yet; none of those is the case for you.
Because your read method can take a wide variety of parameters, each of which is valid and changes your system's state, then it's probably not a smart idea to use mocking here. Unless the order of calls that BitStreamFixer makes to BitReader is deterministic enough to make part of its contract, a mock BitReader will likely result in a brittle test: one that breaks when the implementation changes even if the system is perfectly functional. You'll want to avoid that.
Note that mocking should never yield "complicated logic", only complicated set-up. You're using mocks to avoid using real logic in your tests.
Use a real BitReader, which sounds like it will be painful and opaque to construct. This is probably the most realistic solution, though, especially if you've already finished writing and testing it.
You worry about "introducing new dependencies", but if your BitReader implementation exists and is fast, deterministic, and well-tested, then you shouldn't feel any worse about using it than using a real ArrayList or ByteArrayInputStream in your test. It sounds like the only real problem here is that creating the byte array would make it hard to maintain your test, which is a valid consideration.
In the comments, though, the real answer comes through: Build the BitWriter you're missing.
#Test public void shouldFixBrokenStream() {
BitReader bitReader = new StreamBitReader(BitWriter.create()
.pushBits(16, 0x8080)
.pushBits(12, 0x000) // invalid 12-bit sequence
.pushBits(16, 0x8080)
.asByteArrayInputStream());
BitStreamFixer fixer = new BitStreamFixer(bitReader);
assertEquals(0x80808080, fixer.read(32));
}
/** Of course, you could skip the BitReader yourself, and just make a new one. */
#Test public void shouldFixBrokenStream_bitReader() {
BitReader bitReader = new InMemoryBitReader();
bitReader.pushBits(16, 0x8080);
bitReader.pushBits(12, 0x000); // invalid 12-bit sequence
bitReader.pushBits(16, 0x8080);
BitStreamFixer fixer = new BitStreamFixer(bitReader);
assertEquals(0x80808080, fixer.read(32));
}
This is more readable than constructing an opaque bitstream offline and copy-pasting it into your test (particularly if well-commented), less brittle than mocks, and much more testable itself than an anonymous inner class (or Answer-based version of the same). It is also likely that you can use a system like that across multiple test cases, and possibly even multiple tests.
Say I have a unit test that wants to compare two complex for objects for equality. The objects contains many other deeply nested objects. All of the objects' classes have correctly defined equals() methods.
This isn't difficult:
#Test
public void objectEquality() {
Object o1 = ...
Object o2 = ...
assertEquals(o1, o2);
}
Trouble is, if the objects are not equal, all you get is a fail, with no indication of which part of the object graph didn't match. Debugging this can be painful and frustrating.
My current approach is to make sure everything implements toString(), and then compare for equality like this:
assertEquals(o1.toString(), o2.toString());
This makes it easier to track down test failures, since IDEs like Eclipse have a special visual comparator for displaying string differences in failed tests. Essentially, the object graphs are represented textually, so you can see where the difference is. As long as toString() is well written, it works great.
It's all a bit clumsy, though. Sometimes you want to design toString() for other purposes, like logging, maybe you only want to render some of the objects fields rather than all of them, or maybe toString() isn't defined at all, and so on.
I'm looking for ideas for a better way of comparing complex object graphs. Any thoughts?
The Atlassian Developer Blog had a few articles on this very same subject, and how the Hamcrest library can make debugging this kind of test failure very very simple:
How Hamcrest Can Save Your Soul (part 1)
Hamcrest saves your soul - Now with less suffering! (part 2)
Basically, for an assertion like this:
assertThat(lukesFirstLightsaber, is(equalTo(maceWindusLightsaber)));
Hamcrest will give you back the output like this (in which only the fields that are different are shown):
Expected: is {singleBladed is true, color is PURPLE, hilt is {...}}
but: is {color is GREEN}
What you could do is render each object to XML using XStream, and then use XMLUnit to perform a comparison on the XML. If they differ, then you'll get the contextual information (in the form of an XPath, IIRC) telling you where the objects differ.
e.g. from the XMLUnit doc:
Comparing test xml to control xml [different]
Expected element tag name 'uuid' but was 'localId' -
comparing <uuid...> at /msg[1]/uuid[1] to <localId...> at /msg[1]/localId[1]
Note the XPath indicating the location of the differing elements.
Probably not fast, but that may not be an issue for unit tests.
Because of the way I tend to design complex objects, I have a very easy solution here.
When designing a complex object for which I need to write an equals method (and therefore a hashCode method), I tend to write a string renderer, and use the String class equals and hashCode methods.
The renderer, of course, is not toString: it doesn't really have to be easy for humans to read, and includes all and only the values I need to compare, and by habit I put them in the order which controls the way I'd want them to sort; none of which is necessarily true of the toString method.
Naturally, I cache this rendered string (and the hashCode value as well). It's normally private, but leaving the cached string package-private would let you see it from your unit tests.
Incidentally, this isn't always what I end up with in delivered systems, of course - if performance testing shows that this method is too slow, I'm prepared to replace it, but that's a rare case. So far, it's only happened once, in a system in which mutable objects were being rapidly changed and frequently compared.
The reason I do this is that writing a good hashCode isn't trivial, and requires testing(*), while making use of the one in String avoids the testing.
(* Consider that step 3 in Josh Bloch's recipe for writing a good hashCode method is to test it to make sure that "equal" objects have equal hashCode values, and making sure that you've covered all possible variations are covered isn't trivial in itself. More subtle and even harder to test well is distribution)
The code for this problem exists at http://code.google.com/p/deep-equals/
Use DeepEquals.deepEquals(a, b) to compare two Java objects for semantic equality. This will compare the objects using any custom equals() methods they may have (if they have an equals() method implemented other than Object.equals()). If not, this method will then proceed to compare the objects field by field, recursively. As each field is encountered, it will attempt to use the derived equals() if it exists, otherwise it will continue to recurse further.
This method will work on a cyclic Object graph like this: A->B->C->A. It has cycle detection so ANY two objects can be compared, and it will never enter into an endless loop.
Use DeepEquals.hashCode(obj) to compute a hashCode() for any object. Like deepEquals(), it will attempt to call the hashCode() method if a custom hashCode() method (below Object.hashCode()) is implemented, otherwise it will compute the hashCode field by field, recursively (Deep). Also like deepEquals(), this method will handle Object graphs with cycles. For example, A->B->C->A. In this case, hashCode(A) == hashCode(B) == hashCode(C). DeepEquals.deepHashCode() has cycle detection and therefore will work on ANY object graph.
Unit tests should have well-defined, single thing they test. This means that in the end you should have well-defined, single thing that can be different about those two object. If there are too many things that can differ, I would suggest splitting this test into several smaller tests.
I followed the same track you are on. I also had additionnal troubles:
we can't modify classes (for equals or toString) that we don't own (JDK), arrays etc.
equality is sometimes different in various contexts
For example, tracking entities equality might rely on database ids when available ("same row" concept), rely the equality of some fields (the business key) (for unsaved objects). For Junit assertion, you might want all fields equality.
So I ended up creating objects that run through a graph, doing their job as they go.
There is typically a superclass Crawling object:
crawl through all properties of the objects ; stop at:
enums,
framework classes (if applicable),
at unloaded proxies or distant connections,
at objects already visited (to avoid looping)
at Many-To-One relationship, if they indicate a parent (usually not included in the equals semantic)
...
configurable so that it can stop at some point (stop completely, or stop crawling inside the current property):
when mustStopCurrent() or mustStopCompletely() methods return true,
when encountering some annotations on a getter or a class,
when the current (class, getter) belong to a list of exceptions
...
From that Crawling superclass, subclasses are made for many needs:
For creating a debug string (calling toString as needed, with special cases for Collections and arrays that don't have a nice toString ; handling a size limit, and much more).
For creating several Equalizers (as said before, for Entities using ids, for all fields, or solely based on equals ;). These equalizers often need special cases also (for example for classes outside your control).
Back to the question : These Equalizers could remember the path to the differing values, that would be very useful your JUnit case to understand the difference.
For creating Orderers. For example, saving entities need to be done is a specific order, and efficiency will dictate that saving the same classes together will give a huge boost.
For collecting a set of objects that can be found at various levels in the graph. Looping on the result of the Collector is then very easy.
As a complement, I must say that, except for entities where performance is a real concern, I did choose that technology to implements toString(), hashCode(), equals() and compareTo() on my entities.
For example, if a business key on one or more fields is defined in Hibernate via a #UniqueConstraint on the class, let's pretend that all my entities have a getIdent() property implemented in a common superclass.
My entities superclass has a default implementation of these 4 methods that relies on this knowledge, for example (nulls need to be taken care of):
toString() prints "myClass(key1=value1, key2=value2)"
hashCode() is "value1.hashCode() ^ value2.hashCode()"
equals() is "value1.equals(other.value1) && value2.equals(other.value2)"
compareTo() is combine the comparison of the class, value1 and value2.
For entities where performance is of concern, I simply override these methods to not use reflexion. I can test in regression JUnit tests that the two implementations behave identically.
We use a library called junitx to test the equals contract on all of our "common" objects:
http://www.extreme-java.de/junitx/
The only way I can think of to test the different parts of your equals() method is to break down the information into something more granular. If you are testing a deeply-nested tree of objects, what you are doing is not truly a unit test. You need to test the equals() contract on each individual object in the graph with a separate test case for that type of object. You can use stub objects with a simplistic equals() implementation for the class-typed fields on the object under test.
HTH
I would not use the toString() because as you say, it is usually more useful for creating a nice representation of the object for display or logging purposes.
It sounds to me that your "unit" test is not isolating the unit under test. If, for example, your object graph is A-->B-->C and you are testing A, your unit test for A should not care that the equals() method in C is working. Your unit test for C would make sure it works.
So I would test the following in the test for A's equals() method:
- compare two A objects that have identical B's, in both directions, e.g. a1.equals(a2) and a2.equals(a1).
- compare two A objects that have different B's, in both directions
By doing it this way, with a JUnit assert for each comparison, you will know where the failure is.
Obviously if your class has more children that are part of determining equality, you would need to test many more combinations. What I'm trying to get at though is that your unit test should not care about the behavior of anything beyond the classes it has direct contact with. In my example, that means, you would assume C.equals() works correctly.
One wrinkle may be if you are comparing collections. In that case I would use a utility for comparing collections, such as commons-collections CollectionUtils.isEqualCollection(). Of course, only for collections in your unit under test.
If you're willing to have your tests written in scala you could use matchete. It is a collection of matchers that can be used with JUnit and provide amongst other things the ability to compare objects graphs:
case class Person(name: String, age: Int, address: Address)
case class Address(street: String)
Person("john",12, Address("rue de la paix")) must_== Person("john",12,Address("rue du bourg"))
Will produce the following error message
org.junit.ComparisonFailure: Person(john,12,Address(street)) is not equal to Person(john,12,Address(different street))
Got : address.street = 'rue de la paix'
Expected : address.street = 'rue du bourg'
As you can see here I've been using case classes, which are recognized by matchete in order to dive into the object graph.
This is done through a type-class called Diffable. I'm not going to discuss type-classes here, so let's say that it is the corner stone for this mechanism, which compare 2 instances of a given type. Types that are not case-classes (so basically all types in Java) get a default Diffable that uses equals. This isn't very useful, unless you provide a Diffable for your particular type:
// your java object
public class Person {
public String name;
public Address address;
}
// you scala test code
implicit val personDiffable : Diffable[Person] = Diffable.forFields(_.name,_.address)
// there you go you can now compare two person exactly the way you did it
// with the case classes
So we've seen that matchete works well with a java code base. As a matter of fact I've been using matchete at my last job on a large Java project.
Disclaimer : i'm the matchete author :)