This question already has answers here:
What is the difference between integration and unit tests?
(21 answers)
Closed 7 years ago.
I know that Unit Testing is all about testing individual units of code and mocking the other dependencies like database connections, file I/O etc.
I have a few questions on unit tests
what is that "UNIT" in unit testing that we need to test? Is it a method of a class? or a use-case? Because use-case testing seems more of an integration test to me.
Mocking external dependencies. Does it mean to mock all references to any external class even if it is same package?
The 'Unit' in unit testing is not well defined.
However it is often used to refer to the process of testing an individual function or method. Unit tests like these are often combined into a group which tests a group of related functions or methods (e.g. a class).
As you suspect, the unit under test should be isolated from any other code during testing. This allows you to be sure that any error occurs within the code you wish to test.
For a much better and more detailed explanation, I recommend Martin Fowler's post on the issue of Unit Testing, which directly addresses this issue:
Unit testing is often talked about in software development, and is a term that I've been familiar with during my whole time writing programs. Like most software development terminology, however, it's very ill-defined, and I see confusion can often occur when people think that it's more tightly defined than it actually is.
what is that "UNIT" in unit testing that we need to test? Is it a method of a class? or a use-case?
A chip is a unit to a RAM. Ram is a Unit to Motherboard and motherboard can be a unit in a border context. So there is no specific answer for your question. But mostly we use methods as units because they are the smallest elements that we can test.
Sometimes even a class or a bigger component can be a unit. It depends on your context and how important and complex that particular "unit" is?
Mocking external dependencies. Does it mean to mock all references to any external class even if it is same package?
You mock if you do not have direct access to some component. (class...etc) It doesn't matter where it is. If you want to test something which depends on something that hasn't been implemented yet, you need to mock. What should mock is depend on what you want to test!
(ex. I want to implement a Tsunami alert system but the hardware (which send the wave signal) which should be in the sea is not implemented yet. So if I want to test the alert system I built I may be need to mock the behavior of that hardware device)
Independent unit testing refers that each test should not depend on the result of other test.
Mock external dependencies is replacing complex dependencies for a simple one: I.E replace a DataBase to a Mock that hardcode the result.
Hope it helps.
Related
This question already has answers here:
What's the difference between unit, functional, acceptance, and integration tests? [closed]
(8 answers)
Closed 2 years ago.
I work on my thesis and it's about softwaretesting. The programming language is Java.
I have a huge program with over 450.000 lines of code (without comments and blankspaces). There are also many JUnit-tests.
My question right now is: How can I get to know if a test is a unittest or an intergrationtest?
My ideas: Can I use the execution time of the tests? Or can I measure the performance of the CPU?
Do you have any tips? Do you have more experience in softwaretesting? I am not new to this, but this case is a bit new and huge for me...
Thank you in advance! :)
Alexis described the difference between unit and integration tests completely above, but I felt the need to chime in on the particularities of java and how to write and notice the difference between unit and integration tests.
Unit Tests
Generally for unit tests, a mocking framework is used to mock certain dependencies of a particular class and of a particular method. Since with unit tests, we only want to test a single unit of the code (class methods) we don't want to have to worry about the other classes that method communicates with, so we mock those external dependencies and often verify that the method call to those external dependencies was carried out, without actually running the external method.
Mocking is a powerful tool to write strong tests for your application and it encourages developers to write clean, well-designed and loosely-coupled code.
A common example you will see throughout a java project is the Mockito Framework. I advise reading more on the topic here.
Integration Tests
"Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems."
In java terms, this often refers to the interaction between classes. How class A calls class B and ensuring the link between them works correctly in regard to data transmission, error handling and control flow. Integration tests don't typically use a mocking framework but in-practice they can. Generally you want to fully test the component interactions without mocking out the dependency link. However, this depends on your needs in regard to application design. More information on this can be found here
Typically with integration tests, the database is tested regarding the CRUD (Create-Read-Update-Delete) operations to it. Unit tests have no connection to a database. This is because tests requiring a database connection will have side effects by their very nature. There is leakage from the concept of testing a "single unit" of your code. Therefore, normally the database is tested from integration tests and up.
Requiring this database connection will often cause your integration tests to run dramatically slower than your unit tests. This is due to needing to roll up the database for each set of transactions for a test class and often if you're using an IOC container (like Spring) then further overhead to time will exist. Typically, for integration tests we use a test database, rather than the real database, and is often in-memory.
In summary
If your project is designed well, the unit tests and integration tests should be segregated by package. It makes sense conceptually, and architecturally it is a better approach for developers to adapt to and feel comfortable with. It also completely draws that line between what is an integration test and what is a unit test.
Unit tests are very fast and should be the foundation of your application test suite.
Integration tests can be slow in most cases due to requiring a database connection but can cover more variants than a unit test can.
Unit tests typically use a mocking framework to mock external dependencies.
Integration tests typically use the real dependency objects for testing.
Unit tests should not have a database connection.
Integration tests typically do have a connection to a test database.
There are tons of articles/books about this and with a single query in Google you will find more, but in short:
Unit tests are the tests that test only the smallest unit (in Java the smallest unit is a method) in isolation.
Integration tests test the communication/collaboration between dependent units (in Java you might test that certain classes communicate properly).
There is no specific hint or annotation that declare your tests as Units Tests or Integration Tests, you have to dive on the test and figure it out.
You can measure the execution time of the tests really easily. There are a lot of tools for that but the Intellij (which is the most common IDE for Java) has a built-in feature for that. In the picture, you can see the execution time of the test suite printed by this IDE.
I’m currently learning about unit tests and integration testing and as I understood it, unit tests are used to test the logic of a specific class and integration tests are used to check the cooperation of multiple classes and libraries.
But is it only used to test multiple classes and if they work together as expected, or is it also valid to access databases in an integration test? If so, what if the connection can‘t be established because of a server sided error, wouldn’t the tests fail, although the code itself would work as expected? How do I know what‘s valid to use in this kind of tests?
Second thing I don‘t understand is how they are set up. Unit tests seem to me have a quite common form, like:
public class classTest {
#BeforeEach
public void setUp(){
}
#Test
public void testCase(){
}
}
But how are integration tests written? Is it commonly done the same way, just including more classes and external factors or is there another way that is used for that?
[... ] is it also valid to access databases in an integration test? [...] How do I know what‘s valid to use in this kind of tests?
The distinction between unit-tests and integration tests is not whether or not more than one component is involved: Even in unit-testing you can get along without mocking all your dependencies if these dependencies don't keep you from reaching your unit-testing goals (see https://stackoverflow.com/a/55583329/5747415).
What distinguishes unit-testing and integration testing is the goal of the test. As you wrote, in unit-testing your focus is on finding bugs in the logic of a function, method or class. In integration testing the goal is then, obviously, to detect bugs that could not be found during unit-testing but can be found in the integrated (sub-)system. Always keeping the test goals in mind helps to create better tests and to avoid unnecessary redundancy between integration tests and unit-tests.
One flavor of integration testing is interaction testing: Here, the goal is to find bugs in the interaction between two or more components. (Additional components can be mocked, or not - again this depends on whether the additional components keep you from reaching your testing goals.) Typical questions in the interactions of two components A and B could be, for example if B is a library: Is component A calling the right function of component B, is component B in a proper state to be accessed by A via that function (B might not be initialized yet), is A passing the arguments in the correct order, do the arguments contain the values in the expected form, does B give back the results in the expected way and in the expected format?
Another flavor of integration testing is subsystem testing, where you do not focus on the interactions between components, but look at the boundaries of the subsystem formed by the integrated components. And, again, the goal is to find bugs that could not be found by the previous tests (i.e. unit-tests and interaction tests). For example, are the components integrated in the correct versions, can the desired use-cases be exercised on the integrated subsystem etc.
While unit-tests form the bottom of the test pyramid, integration testing is a concept that applies on different levels of integration and can even focus on interfaces orthogonal to the software integration strategy (for example when doing interaction testing of a driver and its corresponding hardware device).
Second thing I don‘t understand is how they are set up. [...] how are integration tests written?
There is an extreme variation here. For many integration tests you can just use the same testing framework that is used for unit-tests: There is nothing unit-test specific in these frameworks. You will, certainly, in the test cases have to ensure that the setup actually combines the components of interest in their proper versions. And, whether or not additional dependencies are just used or mocked needs to be decided (see above).
Another typical scenario is to perform integration tests in the fully integrated system, using a system-test-like setup. This is often done out of convenience, just to avoid the trouble to create different special setups for the different integration tests: The fully integrated system just has them all combined. Certainly, this has also disadvantages, because it is often impossible or at least impractical to perform all integration tests as desired this way. And, when doing integration testing this way the boundaries between integration testing and system testing get fuzzy. Keeping focused in such a case means you really have to have a good understanding of the different test goals.
There are also mixed forms, but there are too many to describe them here. Just one example, there is a possibility to mock some shared libraries with the help of LD_PRELOAD (What is the LD_PRELOAD trick?).
It would be valid to access a database as part of an integration test, as integration tests are supposed to show whether a feature is working correctly.
If a feature were to not work because of a failed connection to a server side error, you would want the test to fail to inform you that this feature is not working. Integration tests are not there to inform you where the fault lies, just that a feature is not working.
See https://stackoverflow.com/a/7876055/10461045 as this helps to clarify the widely accepted difference.
Using a database (or an external connection to a service you are using) in an integration test is not only valid, but should be done. However, do not rely on integration tests heavily. Unit test every logic element you have and set up integration tests for certain flows.
Integration tests can be written in the same way, except (as you mentioned) they include more methods etc. In fact, the code snippet you've shown above is a common start write-up of an integration test.
You can read up more on tests here: https://softwareengineering.stackexchange.com/questions/301479/are-database-integration-tests-bad
Integration tests depends on external systems such as database or network connection. These components should be mock in unit testig.
But when we testing three different classes which behavior depends on each other is this an integretion or unit test? Should that classes be mocked? Lets assume that these classes have no acces to external systems and their behavior is strictly connected.
First of all, the terminology varies from company to company.
According to the book "How Google Tests Software" they call tests "Small Tests", "Medium Tests" and "Large Tests", if I remember correctly.
Other companies call it "Whitebox Tests", "Unit Tests", "Integration Tests", "End-to-End Tests". Even though the same name might be used in one company, it could mean something different in another company.
Secondly:
To mock or not to mock depends on what you want to test. If you want to test the interaction of those 3 classes then I suggest using all three as is, unless: If you want to isolate the behaviour of one class or desire a hard-to-achieve-behaviour/unrealistic behaviour/not-yet-implemented-behaviour or it is hard to use the other classes, I suggest mocking the other classes.
As I know.
Purposes of both testing are different from each other.
https://ocw.mit.edu/courses/aeronautics-and-astronautics/16-842-fundamentals-of-systems-engineering-fall-2015/lecture-notes/MIT16_842F15_Ses9_Ver.pdf
But unit testing may support integration testing; If used unit testing framework supports.
I'm currently consulting on an existing system, and I suspect the right next step is to add unit tests because of the types of exceptions that are occurring (null pointers, null lists, invalid return data). However, an employee who has a "personal investment" in the application insists on integration tests, even though the problems being reported are not related to specific use cases failing. In this case is it better to start with unit or integration tests?
Typically, it is very difficult to retrofit an untested codebase to have unit tests. There will be a high degree of coupling and getting unit tests to run will be a bigger time sink than the returns you'll get. I recommend the following:
Get at least one copy of Working Effectively With Legacy Code by Michael Feathers and go through it together with people on the team. It deals with this exact issue.
Enforce a rigorous unit testing (preferably TDD) policy on all new code that gets written. This will ensure new code doesn't become legacy code and getting new code to be tested will drive refactoring of the old code for testability.
If you have the time (which you probably won't), write a few key focused integration tests over critical paths of your system. This is a good sanity check that the refactoring you're doing in step #2 isn't breaking core functionality.
Integration tests have an important role to play, but central to the testing of your code is unit-tests.
In the beginning, you will probably be forced to do integration tests only. The reason is that your code base is very heavily coupled (just a wild guess since there are no unit tests). Tight coupling means that you cannot create an instance of an object for test without creating a lot of related objects first. This makes any tests integration-tests per definition. It is crucial that you write these integration tests, as should be used as base lines for your bug-finding/refactoring efforts.
Write tests that document the bug.
Fix the bug so all created unit-tests are green.
It is time to be a good boyscout (leave the campsite/code in better order that it was when you entered) : Write tests that documents the functionality of the class that contained the bug.
As a part of your boyscout efforts, you start to decouple the class from others. Dependency Injection is THE tool here. Think that no other classes should be constructed inside other classses -- they should be injected as interfaces instead.
Finally, when you have decoupled the class, you can decouple the tests as well. Now, when you are injecting interfacing instead of creating concrete instances inside the tested class, you can make stubs/mocks instead. Suddenly your tests have become unit-tests!!
You can create integration tests as well, where you inject concrete classes instead of stubs and mocks. Just remember to keep them far away from the unit-tests; preferably in another assembly. Unit-tests should be able to run all the time, and run very fast don't let them be slowed down by slow integration tests.
The answer to the question depends on the context in which it is being asked. If you are looking to bring an existing codebase, and you are considering rewriting or replacing large portions of the code then it will be more valuable to design a comprehensive set of integration tests around the components you wish to rewrite or replace. On the other hand, if you are taking responsibility for an existing system that needs to be support and maintained, you might want to first start with unit tests to make sure that your more focused changes do not introduce errors.
I'll put it another way. If someone sends you an old car, take a look at it. If you are going to replace all of the components right away, then don't bother testing the minute performance characteristics of the fuel injector. If, on the other hand, you are going to be maintaining the car, as is, go ahead and write targeted unit tests around the components you are going to be fixing.
General rule, code without unit tests are brittle, systems without integrations are brittle. If you are going to be focused on low-level code changes, write Unit Tests first. If you are going to be focused on system-level changes, write integration tests.
And, also, make sure to ignore everything you read on sites like this. No one here knows the specifics of your project.
Choosing between integration tests and unit tests is highly subjective. It depends on various metrics of the codebase, most notably cohesion and coupling of the classes.
The generic advice that I would provide is that if classes that are loosely coupled, then test setup is going to consume lesser time, and hence, it would be much easier to start writing unit tests (especially against the more critical classes in the codebase).
On the other hand, in the event of high coupling, you might be better off writing integration tests against the more critical code paths, starting especially with a class that is loosely coupled (and resident much higher up in the execution stack). At the same time, attempts must be made to refactor the classes involved to reduce coupling (while using the integration tests as a safety net).
I have been working on a comparatively large system on my own, and it's my first time working on a large system(dealing with 200+ channels of information simultaneously). I know how to use Junit to test every method, and how to test boundary conditions. But still, for system test, I need to test all the interfacing and probably so some stress test as well (maybe there are other things to do, but I don't know what they are). I am totally new to the world of testing, and please give me some suggestions or point me to some info on how a good code tester would do system testing.
PS: 2 specific questions I have are:
how to test private functions?
how to testing interfaces and avoid side effects?
Here are two web sites that might help:
The first is a list of open source Java tools. Many of the tools are addons to JUnit that allow either easier testing or testing at a higher integration level.
Depending on your system, sometimes JUnit will work for system tests, but the structure of the test can be different.
As for private methods, check this question (and the question it references).
You cannot test interfaces (as there is no behavior), but you can create an abstract base test classes for testing that implementations of an interface follow its contract.
EDIT: Also, if you don't already have unit tests, check out Working Effectivly with Legacy Code; it is a must for testing code that is not set up well for testing.
Mocking is a good way to be able to simulate system tests in unit testing; by replacing (mocking) the resources upon which the other component depends, you can perform unit testing in a "system-like" environment without needing to have the entire system constructed to do it.
As to your specific questions: generally, you shouldn't be using unit testing to test private functions; if they're private, they're private to the class. If you need to test something, test a public method which uses that private method to do something. Avoiding side effects that can be potentially problematic is best done using either a complete test environment (which can easily be wiped back to a "virgin" state) or using mocking, as described above. And testing interfaces is done by, well, testing the interface methods.
Firstly, if you already have a large system that doesn't have any unit tests, and you're planning on adding some, then allow me to offer some general advice.
From maintaining the system and working with it, you'll probably already know the areas of the system which tend to be buggiest, which tend to change often and which tend not to change very much. If you don't, you can always look through the source control logs (you are using source control, right?) to find out where most of the bug fixes and changes are concentrated. Focus your testing efforts on these classes and methods. There's a general rule called the 80/20 rule which is applicable to a whole range of things, this being one of them.
It says that, roughly on average, you should be able to cover 80 percent of the offending cases by doing just 20% of the work. That is, by writing tests for just 20% of the code, you can probably catch 80% of the bugs and regressions. That's because most of the fragile code, commonly changed code and worst offending code makes up just 20% of the codebase. In fact, it may be even less.
You should use junit to do this and you should use something like JMock or some other mocking library to ensure you're testing in isolation. For system testing/integration testing, that is, testing things while they're working together, I can recommend FitNesse. I've had good experience with it in the past. It allows you to write your test in a web browser using simple table-like layouts, where you can easily define your inputs and expected outputs. All you have to do is write a small backing class called a Fixture, which handles the creation of the components.
Private functions will be tested when the public functions that call them. Your testing of the public function only cares that the result returned is correct.
When dealing with API (to other packages or URLS or even to file/network/database) you should mock them. A good unit test should run in a few milliseconds not in seconds. Mocking is the only way to do that. It means that bugs between packages can be dealt with a lot easier than logical bugs at the functional level. For Java easymock is a very good mocking framework.
You may have a look on this list : Tools for regression testing / test automation of database centric java application? for a list of interesting tools.
As you seem to already use Junit extensively it means that you're already "test infected", that is a good point...
In my personal experience, the most difficult thing to manage is data. I mean, controlling very acutely the data agaisnt which the tests are runned.
The lists of tools given before are useful. From personal experience these are the tools I find useful:
Mocking - Mockito is an excellent implementation and has clever techniques to ensure you only have to mock the methods you really care about.
Database testing - DBunit is indespensible for setting up test data and verifying database interactions.
Stress testing - Jmeter - once you see passed the slightly clunky gui this is a very robust tool for setting up scenarios and running stress tests.
As for general approach start by trying to get tests running for the usual "happy paths" through your application these can form a basis for regression testing and performance testing. Once this is complete you can start looking at edge cases and error scenarios.
Although this level of testing should be secondary to good unit testing.
Good luck!