We are considering to use Cucumber on our project for acceptance testing.
When we write a scenario in a Cucumber feature, we write a list of Given, When and Then statements.
As we use cucumber-jvm project, the Given, When and Then statement are related to Java methods in (JUnit) classes.
I want to know what is the best organization for the code related to Given / When / Then in the project structure. My main concern is the maintenance of the cucumber tests on a big project, where the number of scenario is quite important, and especially regarding the items that are shared between features.
I can see at least 2 main approaches:
Each feature is related to it's own JUnit class. So if I have a foo/bar/baz.feature cucumber file, I will find the releated foo.bar.Baz JUnit class with the adequate #Given, #When and #Then annotated methods.
Separate #Given, #When and #Then methods into "thematic" classes and packages. For example, if in my cucumber scenario I have a statement Given user "foo" is logged, then the #Given("^user \"([^\"]*)\" is logged$") annotated method will be located in the foo.user.User class method, but potentially, the #When method used later in the same cucumber scenario will be in a different Java class and package (let say foo.car.RentCar).
For me, the first approach seems good in the way that I can easily do the relation between my cucumber features and my Java code. But the drawback is that I can have a lot of redundancies or code duplication. Also, it may be hard to find a possible existing #Given method, to avoid to recreate it (the IDE can help, but here we are using Eclipse, and it does not seem to give a list of existing Given statement?).
The other approach seems better essentially when you have Given conditions shared among several cucumber feature, and thus I want to avoid code duplication. The drawback here is that it can be hard to make the link between the #Given Java method and the Given cucumber statement (maybe, again, the IDE can help?).
I'm quite new to cucumber, so maybe that my question is not a good question, and with time and experience, the structure will be self-evident, but I want to get good feedbacks on its usage...
Thanks.
I would suggest grouping your code according to the objects it refers to, similar to option #2 you presented in your question. The reasons being:
Structuring your code based on how and where it's being used is a big no-no. It's actually creating coupling between your feature files and your code.
Imagine such a thing in your product's code- the SendEmail() function wouldn't be in a class called NewEmailScreenCommands, would it? It would be in EmailActions or some such.
So the same applies here; structure your code according to what it does, and not who uses it.
The first approach would make it difficult to re-organize your feature files; You'd have to change your code files whenever you change your feature files.
Keeping code grouped by theme makes DRYing it much easier; you know exactly where all the code dealing with the user entity is, so it's easier for you to reuse it.
On our project we use that approach (i.e BlogPostStepDefinitions class), with further separating the code, if the class gets too large, to types of steps (i.e BlogPostGivenStepDefinitions).
We have also started using Cucumber-JVM for acceptance testing and have similar problems with organising code. We have opted to have 1 step definition class for each feature. At the moment this is fine as the features we are testing aren't very complex and quite separate, there is very little overlap in our features.
The second approach you mentioned would be better I think, but it is often challenging to tie together several different step definition classes for a single scenario. I think the best project structure will become clearer once you start adding more features and refactor as normal.
In the meantime here is an Eclipse plugin for cucumber,
https://github.com/matthewpietal/Eclipse-Plugin-for-Cucumber
it has syntax highlighting as well as a list of existing available steps when writing a feature.
On the current project I am taking part in, we asked ourselves the very same question.
After fiddling a bit with the possibilities, what we opted for was a mix of both the solutions you exposed.
Have steps regrouped in theme-centric common steps classes
app-start steps
security check steps
[place random feature concern here] steps
And classes of scenario (and in some case even feature) specific steps
This was to have at the same time the grouping of factorized code which is pretty easily identifiable on it's whatabouts, whereabouts and whatnot.
Yet it allows not to clutter those common classes with overly specific code.
The wiring between all these classes is handled by spring (with cucumber spring which does a great job once you get the hang of it).
Related
I am new to Cucumber and I am using cucumber-java. The cucumber feature files look great. However, the reuse of step definitions really gets in the way about organizing the step definition code. The most intuitive way for me to organize the step definitions is just to follow the organization of the feature files, namely, one class for one feature. However, this way is screwed if there are two identical steps in two feature files, because you will miss one step in one of the classes, which breaks the coherence of the entire structure. However, I cannot convince myself to use other ways right now. So, what is a sensible way to organize step definitions when using cucumber-java?
Thank you very much.
Start with the way you are already doing and once you recognize that a step is reused, move it a class ReUsableSteps or something. This way, if a step does not exist in its class then it should be in ReUsableSteps. Just a thought.
You can have a separate package as placeholder for classes that defines common steps. Add that package as glue.
The best way to use cucumber is to minimize the duplication of step definition. What we should focus on while writing step definition is to avoid any code specific to a class or functionality in steps method of StepDefinition class because that way it can not be generalized, it should have common code which can be used for different steps and code for propagating to separate class for which step is meant and specific logic for each step should reside in separate class. This way we can avoid duplication.
Example:-
Suppose we have component feature file which contains step "Component A has to receive responses", its implementation class is ComponentStepDefinition class, when we implement this in step definition we should not use any logic for iterating or calculating responses for component A there, rather pass the value to a class which performs A related task, in this way to you have to implement "Component B has to receive responses" then same step definition can be used and B's logic will be in another class.
Sort your step definitions by common functionality as related to your domain. They do not have to be linked to specific feature files.
For example, one of our features is to provide our partners with scores (how are they doing on our platform). We have several step definitions related to those scores in a ScoreSteps file. We might have other steps related to partners, those would be in a PartnerSteps file. This also makes it easier to use helper methods related to a certain domain object.
These files would be completely different for you, depending on your domain.
Also note that your IDE will know if a step has already been defined or not (at least, in IntelliJ steps that are undefined are marked in yellow).
Reusing steps is critical for maintenance reasons. That doesn't mean trying to shoehorn steps here and there, but finding a balance between reusability and understanding. As already said above, arranging them into a Common or Reusable package is a pretty good idea. This is something to be done as you go, because you don't always know whether a step is going to be reused or not. In this sense, frequent refactoring of step definitions will be quite normal. Actually it is an indicator of code aliveness, so don't hesitate to make any changes to get the test scenarios clear enough and the testing code as clean as possible. It is just the same well-known coding principles, applied to testing.
One thing that helped me with this task was a utility class (actually it was a set of classes) that allowed me to know which steps and steps definitions exist, the class in which the step definition is defined, the feature files and test scenarios that make use of them, etc. You can even implement advanced options such as searching for steps or steps definitions that contain such and such keywords, or getting to know the step definitions that are not used any longer, etc. Kind of a dictionary.
It can be achieved by either processing the java classes that belong into the 'glue' folder and gather all the regular expressions associated to the gherkin annotations, or by parsing the feature files with the help of a Gherkin parser. Although you may want to have both approaches implemented, as they are not mutually-exclusive; on the contrary, they complement each other.
This is something you may not need when having just a few test scenarios. But as this number grows bigger and bigger, you will find such a mechanism really valuable.
This can be done easily by using multiple test runner classes for multiple feature files.
You can create same steps for different features as long as you are specifying the glue code in your test runner. Glue will make sure the cucumber checks only in the path specified for the steps during execution.
Then you can maintain a suite xml configuration like TestNG.xml for running which all features I want.
#RunWith(Cucumber.class)
#CucumberOptions(monochrome = true, features = "src/test/java/com/abc/batch/xyz/payout/bbq/monthly",
plugin = { "pretty", "html:target/cucumber-html-reports" }, strict = true, glue = {"com.abc.batch.xyz.payout.bbq.monthly" })
#Rollback(false)
The way I organize my cucumber tests are like I have a java package for a cucumber feature where I will have the steps definitions class, feature file, test data files for those tests and Hooks class.
The advantage is that when I try to look for the step definitions in the step class, I know where exactly to look at and the chances of getting a step definition conflict is zero unless you have added the same definition in that package twice.
Is there a way to strip the JUnit #Test code from my Java class.
At the moment I embedded the test code in the same file as the source code to be tested
(Yes I know it's bad, but it is an incentive for me to keep maintaining my test code)
I'd like to strip the test methods from the code, build the binary and deploy.
Thanks
No, I don't know how you can do that, but I could tell you why you might not want to use this approach in the first place.
You might be able to strip out the tests, but what about all the imports that reference test libraries?
What about any private methods that the test might be calling? They won't be marked with #Test.
Why go through all this trouble in the first place? Every IDE has tools for automatically generating a test class and shortcuts for switching between them.
what if the test annotation was accidentally applied to one of your class methods?
What if one of your class methods accidentally calls a test method?
These are just some of the reasons I can think of... why no just avoid the problem in the first place?
There are no specific tools to do this task that I'm aware of.
You'll probably need to "take your licks" and make the changes the hard way with your favorite text editor / IDE. (And maybe this exercise will teach you to pay more heed to good practice / good style ... )
There are some tools that might help you do this (things that aim to encrypt your bytecode also tend to remove unused things).
However I would NOT advise doing that. It is good that you found a way to encourage yourself to keep the code tested (great infact!). However, as you have found out, the way you chose isn't all that good. There are many reasons to keep the test code separate from the code itself.
What I would do is (one class at a time):
1) bite the bullet and make a parallel set of classes for testing
2) move anything that has an #Test before it to the new classes
3) move over anything else that keeps the tests from compiling/running.
Then make use of a code coverage tool (I like Cobertura but there are others) to give you a visual of how much of your code is tested. Add to that the idea of writing the tests before you run the code and you should do well.
Basically this is a very formulated way of working. If you follow the regiment of coding and code coverage and then fix the places where the coverage is poor you should find it just as easy as if the test code is in the class. It is all habit - and good habits are better than bad :-)
I have been working on a comparatively large system on my own, and it's my first time working on a large system(dealing with 200+ channels of information simultaneously). I know how to use Junit to test every method, and how to test boundary conditions. But still, for system test, I need to test all the interfacing and probably so some stress test as well (maybe there are other things to do, but I don't know what they are). I am totally new to the world of testing, and please give me some suggestions or point me to some info on how a good code tester would do system testing.
PS: 2 specific questions I have are:
how to test private functions?
how to testing interfaces and avoid side effects?
Here are two web sites that might help:
The first is a list of open source Java tools. Many of the tools are addons to JUnit that allow either easier testing or testing at a higher integration level.
Depending on your system, sometimes JUnit will work for system tests, but the structure of the test can be different.
As for private methods, check this question (and the question it references).
You cannot test interfaces (as there is no behavior), but you can create an abstract base test classes for testing that implementations of an interface follow its contract.
EDIT: Also, if you don't already have unit tests, check out Working Effectivly with Legacy Code; it is a must for testing code that is not set up well for testing.
Mocking is a good way to be able to simulate system tests in unit testing; by replacing (mocking) the resources upon which the other component depends, you can perform unit testing in a "system-like" environment without needing to have the entire system constructed to do it.
As to your specific questions: generally, you shouldn't be using unit testing to test private functions; if they're private, they're private to the class. If you need to test something, test a public method which uses that private method to do something. Avoiding side effects that can be potentially problematic is best done using either a complete test environment (which can easily be wiped back to a "virgin" state) or using mocking, as described above. And testing interfaces is done by, well, testing the interface methods.
Firstly, if you already have a large system that doesn't have any unit tests, and you're planning on adding some, then allow me to offer some general advice.
From maintaining the system and working with it, you'll probably already know the areas of the system which tend to be buggiest, which tend to change often and which tend not to change very much. If you don't, you can always look through the source control logs (you are using source control, right?) to find out where most of the bug fixes and changes are concentrated. Focus your testing efforts on these classes and methods. There's a general rule called the 80/20 rule which is applicable to a whole range of things, this being one of them.
It says that, roughly on average, you should be able to cover 80 percent of the offending cases by doing just 20% of the work. That is, by writing tests for just 20% of the code, you can probably catch 80% of the bugs and regressions. That's because most of the fragile code, commonly changed code and worst offending code makes up just 20% of the codebase. In fact, it may be even less.
You should use junit to do this and you should use something like JMock or some other mocking library to ensure you're testing in isolation. For system testing/integration testing, that is, testing things while they're working together, I can recommend FitNesse. I've had good experience with it in the past. It allows you to write your test in a web browser using simple table-like layouts, where you can easily define your inputs and expected outputs. All you have to do is write a small backing class called a Fixture, which handles the creation of the components.
Private functions will be tested when the public functions that call them. Your testing of the public function only cares that the result returned is correct.
When dealing with API (to other packages or URLS or even to file/network/database) you should mock them. A good unit test should run in a few milliseconds not in seconds. Mocking is the only way to do that. It means that bugs between packages can be dealt with a lot easier than logical bugs at the functional level. For Java easymock is a very good mocking framework.
You may have a look on this list : Tools for regression testing / test automation of database centric java application? for a list of interesting tools.
As you seem to already use Junit extensively it means that you're already "test infected", that is a good point...
In my personal experience, the most difficult thing to manage is data. I mean, controlling very acutely the data agaisnt which the tests are runned.
The lists of tools given before are useful. From personal experience these are the tools I find useful:
Mocking - Mockito is an excellent implementation and has clever techniques to ensure you only have to mock the methods you really care about.
Database testing - DBunit is indespensible for setting up test data and verifying database interactions.
Stress testing - Jmeter - once you see passed the slightly clunky gui this is a very robust tool for setting up scenarios and running stress tests.
As for general approach start by trying to get tests running for the usual "happy paths" through your application these can form a basis for regression testing and performance testing. Once this is complete you can start looking at edge cases and error scenarios.
Although this level of testing should be secondary to good unit testing.
Good luck!
I've got 2 questions about organising Unit tests.
Do I have to put test to the same package as tested class, or can I organise tests in different packages?
For example if I have validity and other tests, is it correct to split them into different packages, even if they are for same class?
What about mock and stub classes? Shall I separate them from packages containing only tests, or put them together?
The way we do our JUnit test cases is to put them in the same package, but in a different root directory. Since we use Maven, we just use the standard locations making the structure similar to the following.
src/main/java/com/foo/Bar.java
src/test/java/com/foo/BarTest.java
Obviously there's more to the structure, but this lets us build the tests separately from the mainline code, but still access protected classes and the like. With respect to different types of tests, this is very subjective. When we started our testing effort (which unfortunately started after development), I tried to keep things pretty isolated. Unfortunately, it quickly became a nightmare when we got to the 500+ test case point. I've since tried to do more consolidation. This led to reduced amounts of code to maintain. As I said, though, it's very subjective.
As far as test-only code, we keep it in a separate com.foo.test package that resides only in the src/test/java tree.
I too tend to put my tests in the same package but under a different root directory. This allows me to test package-private classes or access packing-private classes while testing something else in the package. They are kept in a separate directory tree to allow excluding them from the deployed result (in particular to ensure that test code didn't accidentally get into production code). What matters most, however, is what works for your situation.
In terms of how many test classes per production class, the theory I've seen is that you write one test class per fixture, that is per setup structure. In many cases that is the same (or close enough) to one test class per production class, but I have sometimes written more test classes (in particular equality tests tend to be separated) for a give production class, and occasionally one test class of for a group of (related) production classes (say, for testing the Strategy pattern).
Mostly, I don't worry too much about the theory, but rework the tests as needed to keep duplication to an absolute minimum.
Keeping it the same package allows you to use package-private visibility for code that is intended to be accessed via the test only.
Regarding using separate root directories, that is a good practice. It also has an advantage for us, since we use IDEA, IDEA recognizes that production code cannot reference test code.
In terms of keeping them separate, there is a great power in having one, and only one, test class per production class at the unit level. Of course, some classes get created in production as part of refactoring that have no test classes at all, and that is fine, but when you want to know what test tests a certain class, having a convention that says ClassNameTest is the tests for ClassName is very helpful.
TestNG is much friendlier to this paradigm than JUnit, though.
Test classes should be rather in different packages, it's easier to separate them from the production code when you package it for release. I usually keep lots of test fluff in those packages, all sorts of mocks, configurations, scenarios.. But when you build - it doesn't get it. In some situations, it's a good idea to keep your testing stuff even in different projects. Depends.
I've been asked to work on changing a number of classes that are core to the system we work on. The classes in question each require 5 - 10 different related objects, which themselves need a similiar amount of objects.
Data is also pulled in from several data sources, and the project uses EJB2 so when testing, I'm running without a container to pull in the dependencies I need!
I'm beginning to get overwhelmed with this task. I have tried unit testing with JUnit and Easymock, but as soon as I mock or stub one thing, I find it needs lots more. Everything seems to be quite tightly coupled such that I'm reaching about 3 or 4 levels out with my stubs in order to prevent NullPointerExceptions.
Usually with this type of task, I would simply make changes and test as I went along. But the shortest build cycle is about 10 minutes, and I like to code with very short iterations between executions (probably because I'm not very confident with my ability to write flawless code).
Anyone know a good strategy / workflow to get out of this quagmire?
As you suggest, it sounds like your main problem is that the API you are working with is too tightly coupled. If you have the ability to modify the API, it can be very helpful to hide immediate dependencies behind interfaces so that you can cut off your dependency graph at the immediate dependency.
If this is not possible, an Auto-Mocking Container may be of help. This is basically a container that automatically figures out how to return a mock with good default behavior for nested abstractions. As I work on the .NET framework, I can't recommend any for Java.
If you would like to read up on unit testing patterns and best practices, I can only recommend xUnit Test Patterns.
For strategies for decoupling tightly coupled code I recommend Working Effectively with Legacy Code.
First thing I'd try to do is shorting the build cycle. Maybe add in the options to only build and test the components currently under development.
Next I'd look at decoupling some of the dependencies by introducing interfaces to sit between each component. I'd also want to move the coupling out in the open most likely using Dependency Injection. If I could notmove to DI I would have two ctors, on no-arg ctor that used the service locator (or what have thee) and one injectable ctor.
the project uses EJB2 so when testing, I'm running without a container to pull in the dependencies I need!
Is that without meant to be a with? I would look at moving as much into POJOs as you can so it can be tested without needing to know anything EJB-y.
If you project can compile with Java 1.5 you shoul look at JMock? Things can get stubbed pretty quickly with 2.* version of this framework.
1.* version will work with 1.3+ Java compiler but the mocking is much more verbose, so I would not recommend it.
As for the strategy, my advice to you is to embrace interfaces. Even if you have a single implementation of the given interface, always create an interface. They can be mocked very easily and will allow you much better decoupling when testing your code.