We have built a REST API that exposes bunch of business services - a business service may invoke other platform/utility services to perform database read and writes, to perform service authorization etc.
We have deployed these services as WAR files in Tomcat.
We want to test this whole setup using a integration test suite which we would like to also treat as regression test suite.
What would be a good approach to perform integration testing on this and any tools that can speed up the development of suite? Here are few requirements we think we need to address:
Ability to define integration test cases which exercise business scenarios.
Set up the DB with test data before suite is run.
Invoke the REST API that is running on a remote server (Tomcat)
Validate the DB post test execution for verifying expected output
Have code coverage report of REST API so that we know how confident we should be in the scenarios covered by the suite.
At my work we have recently put together a couple of test suites to test some RESTful APIs we built. Like your services, ours can invoke other RESTful APIs they depend on. We split it into two suites.
Suite 1 - Testing each service in isolation
Mocks any peer services the API depends on using restito. Other alternatives include rest-driver, wiremock, pre-canned and betamax.
The tests, the service we are testing and the mocks all run in a single JVM
Launches the service we are testing in Jetty
I would definitely recommend doing this. It has worked really well for us. The main advantages are:
Peer services are mocked, so you needn't perform any complicated data setup. Before each test you simply use restito to define how you want peer services to behave, just like you would with classes in unit tests with Mockito.
The suite is super fast as mocked services serve pre-canned in-memory responses. So we can get good coverage without the suite taking an age to run.
The suite is reliable and repeatable as its isolated in it's own JVM, so no need to worry about other suites/people mucking about with an shared environment at the same time the suite is running and causing tests to fail.
Suite 2 - Full End to End
Suite runs against a full environment deployed across several machines
API deployed on Tomcat in environment
Peer services are real 'as live' full deployments
This suite requires us to do data set up in peer services which means tests generally take more time to write. As much as possible we use REST clients to do data set up in peer services.
Tests in this suite usually take longer to write, so we put most of our coverage in Suite 1. That being said there is still clear value in this suite as our mocks in Suite 1 may not be behaving quite like the real services.
With regards to your points, here is what we do:
Ability to define integration test cases which exercise business scenarios.
We use cucumber-jvm to define business scenarios for both of the above suites. These scenarios are English plain text files that business users can understand and also drive the tests.
Set up the DB with test data before suite is run.
We don't do this for our integration suites, but in the past I have used unitils with dbunit for unit tests and it worked pretty well.
Invoke the REST API that is running on a remote server (Tomcat)
We use rest-assured, which is a great HTTP client geared specifically for testing REST APIs.
Validate the DB post test execution for verifying expected output
I can't provide any recommendations here as we don't use any libraries to help make this easier, we just do it manually. Let me know if you find anything.
Have code coverage report of REST API so that we know how confident we should be in the scenarios covered by the suite.
We do not measure code coverage for our integration tests, only for our unit tests, so again I can't provide any recommendations here.
Keep your eyes peeled on our techblog as there may be more details on their in the future.
You may also check out the tool named Arquillian, it's a bit difficult to set up at first, but provides the complete runtime for integration tests (i.e. starts its own container instance and deploys your application along with the tests) and provides extensions that solve your problems (invoking REST endpoints, feeding the databases, comparing results after the tests).
Jacoco extension generates the coverage reports than can be later displayed i.e. by the Sonar tool.
I've used it for a relatively small-scale JEE6 project and, once I had managed to set it up, I was quite happy with how it works.
Related
I need to write some integration tests for my Spring application using Flowable. My tests must include the application BPMN workflow logic.
My question is - should I start and deploy normal Flowable engine during my tests as I do in the application? In official documentation I see some Flowable classes prepared for unit testing but nothing for integration.
Won't starting real Flowable engine cause performance issues during running IT? I'm afraid that they will take long time if I will need to run this with every test separately. How do you deal with this in your applications?
If you ask me, then you should definitely start and deploy a normal Flowable engine during your tests. The link you pasted from the documentation is the exact way how you can do the test. Keep in mind that you can use your own configuration, you don't need a special Spring configuration for the testing.
Starting the real Flowable engines won't cause any performance issues during your testing. All tests in the Flowable repository are actually tests that create and destroy an engine within a single test and that is quite fast. In your case it would be even faster as you won't be starting the engine for each test (the Spring application context is cached between tests). I also have to note that even if you start the engine for each test the time would be negligible as booting the engine is quite fast.
Keep in mind that other components from your Spring application might slow down the start of the tests.
As a reference in the flowable-spring module there are 76 tests in 28 test classes, where each test class has it's own Spring configuration, which means there is no Spring context reuse between tests. All those tests take 55s on my local machine. For those tests you need to keep into consideration that some tests are testing some complex scenarios where the async executors are running and are taking more time than usually. You most probably won't have such tests. With those specific tests disabled (3 from 3 test classes) the test time goes down to 28s.
NB: If you are not using #Deployment or you are relying on the auto deploy feature from Flowable then make sure that you are deleting the instances that you are creating in your tests. This would make sure that data from one test does not affect data from another test.
In general, I write integration test from my service/ remoting layer to the database so that I can check the server side layers are integrated and tested, I would like to keep the rollback as false if not we will miss out the database constraint level validation. It is a personal preference.
We can follow different approaches
- Create data for each test case and delete it once executed
- Run with a certain amount of existing common data such as (User)
There may be entities depends on other several entities and to be able to test such flows it requires a lot of effort to create every entity for each test case or class and maybe for a business flow if we make a decision we create a certain amount of data and execute a business flow with a certain number of test and clear the data. These things can consume a lot of time to run such test cases.
Is there an effective approach or best practice that is followed in the industry to write integration test in the continues integration environments. I normally use TestNG as it provides spring support. Is there any Java-based frameworks.
I think it really depends on a project and there is no silver bullet solution here.
There are indeed many approaches as you state, I'll mention a few:
Take advantage of Spring's #Transactional annotation on the test. In this case, spring will execute rollback after each test. so that the data changed by the test won't really be saved in the database even if the test passes.
Do not use #Transactional but organize tests so that they won't interfere (each test will use its own set of data that can co-exist with other tests data). If the test fails and doesn't "clean-up" its stuff, then other tests should still run. In addition, if the tests are being run in parallel, they still should not interfere.
Use new schema for each test (obviously expensive, but still can be a viable option to some projects).
Now, the real question is what do you test.
If you test a java code, like that your SQLs are created correctly, then probably the first way is a way to go.
Of course, it also depends on what commands are being executed during the tests, not in all databases all the commands can be in a transaction (for example in Postgres you can use DDL inside a transaction, in Oracle you can't, and so forth).
Another concern to think about during the continuous testing is the performance of tests.
Integration tests are slow and if you have a monolith application that runs hundreds of them, then the build will be really slow. Managing build that runs hours is a big pain.
I would like to mention here 2 ideas that can help here:
Moving to microservices helps a lot in this case (each microservice runs only a bunch of its tests and hence the build of each microservice on its own is much faster by nature)
Another interesting option to consider is running the tests against a docker container of the database that starts right in the test case (it also can be cached so that not every test will raise a docker container). A big benefit of such an approach is that everything runs locally (on the build server), so no interaction with the remote database (performance) + the clean-up of resources is done automatically, even if some tests fail. The Docker container dies and all the data put by the tets gets cleaned up automatically. Take a look at Testcontainers project maybe you'll find it helpful
I'm building a basic HTTP API and some actions like POST /users create a new user record in the database.
I understand that I could mock these calls, but at some level I'm wondering if it's easier to let my Junit tests run against a real (test) database? Is this a bad practice? Should only integration tests run against a real DB?
I'm using flyway to maintain my test schema and maven for my build, so I can have it recreate the test DB with the proper schema on each build. But I'm also worried that I'd need some additional overhead to maintain/clean the state of the database between each test, and I'm not sure if there's a good way to do that.
Unit tests are used to test single unit of code. This means that you write a unit test by writing something that tests a method only. If there are external dependencies then you mock them instead of actually calling and using those dependencies.
So, if you write code and it interacts with the real database, then it is not a unit test. Say, for some reason your call to db fails then unit test will also fail. Success or failure of your unit test should not be dependent on the external dependencies like db in your case. You have to assume that db call is successful and then hard code the data using some mocking framework(Mockito) and then test your method using that data.
As often, it depends.
On big projects with lots of JUnit tests, the overhead for the performance can be a point. Also the work time needed for the setup of the test data within the database as well as the needed concept for your tests not interfering with the test data of other tests while parallel execution of JUnit tests is a very big argument for only testing against a database if needed and otherwise mock it away.
On small projects this problems may be easier to handle so that you can always use a database but I personally wouldn't do that even on small projects.
As several other answers suggest you should create unit tests for testing small pieces of code with mocking all external dependencies.
However sometimes ( a lot of times) it should worth to test whole features. Especially when you use some kind of framework like Spring. Or you use a lot of annotations. When your classes or methods have annotations on them the effects of those annotations usually cannot be tested via unit-tests. You need the whole framework running during the test to make sure it works as expected.
In our current project we have almost as much integration tests as unit tests. We use the H2 in-memory DB for these tests, this way we can avoid failures because of connectivity problems, and Spring's test package could collect and run multiple integration tests into the same test-context, so it has to build the context only once for multiple tests and this way running these tests are not too expensive.
Also you can create separate test context for different part of the project (with different settings and DB content), so this way the tests running under different context won't interfere with each-other.
Do not afraid of using a lot of integration tests. You need some anyway, and if you already have a test-context it's not a big deal adding some more tests into the same context.
Also there are a lot of cases which would take a LOT of effort to cover with unit-tests (or cannot be covered fully at all) but can be covered simply by an integration tests.
A personal experience:
Our numerous integration tests were extremely useful when we switched from Spring Boot to Spring Boot 2.
Back to the original question:
Unit tests should not connect to real DB, but feel free to use more integration tests. (with in-memory DB)
Modern development practices recommend that every developer runs the full suite of unit tests often. Unit tests should be reliable (should not fail if the code is OK) Using an external database can interfere with those desiradata.
If the database is shared, simultaneous runs of the testsuite by different developers could interfere with each other.
Setting up and tearing down the database for each test is typically expensive, and thus can make the tests too slow for frequent execution.
However, using a real database for integration tests is OK. If you use an in-memory database instead of a fully real database, even set up and tear down of the database for each integration test can be acceptably fast.
A popular choice is the use of an in-memory database to run tests. This makes it easy to test, for example, repository methods and business logic involving database calls.
When opting for a "real" database, make sure that every developer has his/her own test database to avoid conflicts. The advantage of using a real database is that this prevents possible issues that could arise because of slight differences in behavior between in-memory and real database. However, test execution performance can be an issue when running a large test suite against a real database.
Some databases can be embedded in a way that the database doesn't even need to be installed for test execution. For example, there is an SO thread about firing up an embedded Postgres in Spring Boot tests.
I found myself last week having to start thinking about how to refactor an old application that only contains unit tests. My first idea was to add some component test scenarios with Cucumber to get familiarised with the business logic and to ensure I don't break anything with my changes. But at that point I had a conversation with one of the architects in the company I work for that made me wonder whether it was worth it and what was actually the code I had to actually test.
This application has many different types of endpoints: rest endpoints to be called from and to call to, Oracle stored procedures and JMS topics and queues. It's deployed in a war file to a Tomcat server and the connection factory to the broker and the datasource to the database are configured in the server and fetched using JNDI.
My first idea was to load the whole application inside an embedded Jetty, pointing to the real web.xml so everything is loaded as it would be loaded from a production environment but then mocking the connection factory and the datasource. By doing that, all the connectivity logic to the infrastructure where the application is deployed would be tested. Thinking about the hexagonal architecture, this seems like too much effort having in mind that those are only ports which logic should only be about transforming received data into application data. Shouldn't this just be unit tested?
My next idea was to just mock the stored procedures and load the Spring XMLs in my test without any web server, which makes it easier to mock classes. For this I would be using libraries like Spring MockMvc for the rest endpoints and Mockrunner for JMS. But again, this approach would still test some adapters and complicate the test as the result of the tests would be XML and JSON payloads. The transformations done in this application are quite heavy where the same message type could contain different versions of a class (each message could contain many complex object that implement several interfaces).
So right now I was thinking that maybe the best approach would be to just create my tests from the entry point to the application, the services called from the adapters, and verify that the services responsible to send messages to the broker or to call other REST endpoints are actually invoked. Then just ensure there are proper unit tests for the endpoints and verify everything works once deployed by just providing some smoke tests that are executed in a real environment. This would test the connectivity logic and the business logic would be tested in isolation, without caring if a new adapter is added or one is removed.
Is this approach correct? Would I be leaving something without testing this way? Or is it still too much and I should just trust the unit tests?
Thanks.
Your application and environment sound quite complicated. I would definitely want integration tests. I'd test the app outside-in as follows:
Write a smoke-test suite that runs against the application in the actual production environment. Cucumber would be a good tool to use. That suite should only do things that are safe in production, and should be as small as possible while giving you confidence that the application is correctly installed and configured and that its integrations with other systems are working.
Write an acceptance test suite that runs against the entire application in a test environment. Cucumber would be a good choice here too.
I would expect the acceptance-test environment to include a Tomcat server with test versions of all services that exist in your production Tomcat and a database with a schema, stored procedure, etc. identical to production (but not production data). Handle external dependencies that you don't own by stubbing and mocking, by using a record/replay library such as Betamax and/or by implementing test versions of them yourself. Acceptance tests should be free to do anything to data, and they shouldn't have to worry about availability of services that you don't own.
Write enough acceptance tests to both describe the app's major use cases and to test all of the important interactions between the parts of the application (both subsystems and classes). That is, use your acceptance tests as integration tests. I find that there is very little conflict between the goals of acceptance and integration tests. Don't write any more acceptance tests than you need for specification and integration coverage, however, as they're relatively slow.
Unit-test each class that does anything interesting whatsoever, leaving out only classes that are fully tested by your acceptance tests. Since you're already integration-testing, your unit tests can be true unit tests which stubb or mock their dependencies. (Although there's nothing wrong with letting a unit-tested class use real dependencies that are simple enough to not cause issues in the unit tests).
Measure code coverage to ensure that the combination of acceptance and unit tests tests all your code.
I have an API that I would like to test in an automated fashion. I'm doing it in java at the moment but I think that the problem is language agnostic
A little bit of context:
The main idea is to integrate with a payments system in a manner similar to this demo.
The main flow is something like:
You checkout your cart and click a pay button
Your webapp will initiate a transaction with the payment api and you'd get a reference number. This reference number would now be used as a query parameter for that payment and you'd get redirected to that website.
After the customer makes the payment, you'd get redirected back to the webapp where you can retrieve the transaction and display the result
My main problem is how do I approach automated integration testing for this type of scenario? The main ideas that I have:
Use stubbing or mocking for the transaction reference and response. But this is more in line with unit testing than integration testing. I don't mind doing this, but I would want to explore the integration testing first.
Perhaps I should try some sort of automated form filling. So I would do a curl type request on the redirect url and a curl type post request after inspecting what the redirect website does.
Use some sort of web testing tool like selenium or something like that.
I think the answer depends on the goals and scope of your integration test, and also the availability of a suitable platform to use for integration testing. Here are a couple of thoughts that may aid your decision, focussing first on establishing the goals of your tests before making any suggestions on what the appropriate testing tools would be.
(I make the assumption that you don't want to actually use the production version of the payments service when running your tests)
Testing Integration with a 'Real' Payment Service: The strongest form of integration test would be one where you actually invoke a real payments service as part of your test, as this would test the code in your application, the code in the payments service, and the communication between the two. However, this requires you to have an always running test version of the payment service available, and the lower the availability of this, the more fragile your tests become.
If the payment service is owned by your team/department/company, this might not be so bad because you have the necessary control to make sure it is always available for testing. However, if it is a vendor system, assuming they control the test version of the service, then you are opening yourself up to fragility issues if they don't effectively maintain that test service to provide a high level of availability (which, in my experience, they generally don't, issues frequently occur like the service doesn't get upgraded often enough, or their support teams don't notice if the service has gone down).
One scenario you may come across is that the vendor may provide a test service that is always running a pre-release version of their software, so that their clients can run tests with new versions of their software before they are released into production and flag any integration issues. If they do, this should definitely influence your decision.
Testing Integration with a 'Fake' Payment Service: An alternative is to build a fake version of the service, running in a similar environment to the real service and with the same API, that can be used for integration tests. This can be as simple or as complex as you like, ranging from a very thin service that simply returns a single canned response to each request, to something that can return a range of different responses (success, fail, service not found etc...), depending on what your test goals are.
The upside of this is less fragility - it is likely to have much higher availability because it is under your control, and it will also be much simpler to guarantee the responses from the service you are looking for from your tests. Also, it makes it much simpler to build more intelligent test cases (i.e. a test for how your code responds if the service is unavailable, or if it reports it is under heavy load and cannot process your transaction yet). The downsides are that it is likely to be more work to set up, and you are not exercising the code in the real payments service when you run your unit tests.
What is the Best Approach to Take to Test the Service?: This is highly context specific, however here is what I would consider an ideal approach to testing, based on striking a balance between effectively testing integration with the service, and minimizing the risk of impacting developer productivity through fragile tests.
If the vendor provides a test version of their service, write a small test suite that verifies you are getting the expected responses from their service, and run this once per day. This will allow the benefits of verifying your own assumptions about the behavior of their service, without introducing fragility to the rest of your tests. The appropriate tool would be the simplest tool for the job (even a shell script which emails you any issues would be absolutely fine here), as long as it is maintainable, at the end of the day these wouldn't be tests developers would be expected to run regularly.
For my end-to-end test suite (i.e. one that deploys a test version of the system and tests it end to end), I would build a fake version of the payment service that can be deployed with the test system, and use that for testing. This allows you to test your own system completely end to end, and with a stable payment service endpoint. These end-to-end tests should include scenarios where the service cannot be found, reports a failure, things like that. Given this is end-to-end testing, a tool such as Selenium would be appropriate for tests driven through a web UI.
For the rest of the automated tests I would mock out calls to the payment service. One very simple approach is to encapsulate all of your calls to the payment service in a single component in your system, which can be easily replaced by a mock version during tests (e.g. using Dependency Injection).