I'd like to ask what's the practical difference between Cucumber and JUnit. I haven't worked with Cucumber at all; found some documentation but I'd greatly appreciate some feedback from someone who has worked with both (interested in a high lvl overview).
To break it down - what i'm interested in (I'll be using Selenium and not Protractor) :
Are there any things that Cucumber can't do vs Junit.
What's easier to use (coding, how fast you can write the tests) ?
Both work with Page Objects?
Some things that i need to get done
Test css styling
Test page responsiveness
Standard operation on WebElements (clicking, getting data etc)
Asserts.
Anything in addition to this is more than welcomed. Greatly appreciate your answer on this, thank you!
JUnit and Cucumber are aiming at different goals. They are rather complement to each other than replace each other.
Are there any things that Cucumber can't do vs Junit.
There isn't anything you can do with JUnit that you can't do with Cucumber. And the other way around.
The difference is that while JUnit aims at tests, Cucumber aims at collaboration with non technical people. Non technical people will not understand what a unit test does. They will, however, be able to understand and validate an example written in Gherkin.
What's easier to use (coding, how fast you can write the tests) ?
There is more overhead when you use Cucumber. You will have to implement each step as a method and not just one test method as you would do if you used JUnit. The readability you gain from expressing examples using plain text is sometimes worth the extra work.
Both work with Page Objects?
Page Objects are an abstraction for the web page you are verifying. It is a class you write as developer/tester. The Page Objects can be used by both JUnit and Cucumber. In fact, there is no difference between the tools from that perspective.
The choice to use JUnit or Cucumber is a matter of granularity and audience.
A work flow that works well is to mix the tools. Define examples of how the application should work using BDD, (Cucumber, Gherkin). Implement these scenarios using Cucumber. Then, use JUnit to work out details that may be important but not necessary important for the business stakeholders at a high level. Think of corner cases that are important but are too much details for your stakeholders.
An image that describes this mix is available here: https://cucumber.io/images/home/bdd-cycle.png
I wrote blog post a while back where I talk about the right tool for the job: http://www.thinkcode.se/blog/2016/07/25/the-right-tool-for-the-job
The right tool may be Cucumber. It can also be JUnit. It all depends on your audience.
Simply spoken, those two work on completely different levels of abstraction.
JUnit is mainly an automation framework; giving you the ability to rapidly write down test cases using the Java programming language. It provides annotations that make to easily declare: "this method over here is a JUnit test". It was intended as framework for unit tests; but many people also use it to drive full scale "integration" or "function" tests.
Cucumber on the other hand works on a much higher level of abstraction. You start by writing "test descriptions" in pure text. Leading to probably the key difference: you don't need a to know Java to write a cucumber test (you just need a java programmer to provide the "glue code" that allows Cucumber to turn your text input into some executable piece of code).
In that sense, you are somehow asking us to compare apples and turnips here; as one would be using these two toolsets for a different set of "problem solution". But as lined out; you can also use JUnit to drive "bigger" tests; so the main differentiation between these two tools is the level of abstraction that you are dealing with.
EDIT: your comment is correct; as those tools are for different "settings", you shouldn't expect that a non-technical person alone will be able to use cucumber to write good tests covering everything. Cucumber is a nice way to enable non-technical participation for creating tests; but in the end, you are solving technical (java related) problems; thus you need Java programming expertise at some point. Either "within the same person"; or at least within different people in your team.
Cucumber seems to make something more user friendly but I don't think business analysts really care what it is. Ultimately developers have to write unit tests, integration tests , cucumber tests (so Cucumber makes no sense for developer who has already written unit tests & integration tests & Business analyst don't care because they have already provided what they want).
Related
The Developers are usually writing test cases with Junits
The Testers are usually writing test cases with Cucumber
I am confused, that how are these(Cucumber & Junit) different, if at the end, both are meant for validating logic of our code !
Is my supposition correct, if I say...
simple method testing is done by Junits
tough scenarios, we have Cucumber
Unit tests, as their name suggests, are designed to test small units of code. A unit test is written in code, this makes it less readable to the business, but also makes it very powerful and potentially very fast.
Cucumber is a tool used to describe behaviour. Each cuke will exercise a much larger chunk of code.
The two sorts of tests are very different. So different that most Cucumber enthusiasts don't want to use the word test (https://cucumber.io/blog/collaboration/the-worlds-most-misunderstood-collaboration-tool/)
Scenarios are not at all about validating the logic of your code, they have nothing to do with code. They are about describing the behaviour of your application and supporting the development of that behaviour. After the scenario and the behaviour has been implemented they are used to confirm that the behaviour is 'probably' still working as intended.
Unit tests are all about code, they are written in code and (ideally) connect directly to code. They definitely can be used to validate the logic of your code.
There are some other major differences
Runtime
Scenarios are slow
Unit tests are fast (unless they are badly written)
This is not a small difference. An optimally written unit test can be 1000 times faster than an optimally written scenario.
Expressiveness and Power
scenarios use natural language to express themselves. They use abstraction and naming to wield power. Well written cukes are very readable by the business
unit tests use code to express themselves and to wield power. They are much more powerful. However its much harder for them to express themselves. Unit tests cannot be read by their stakeholders. Even the output of beautifully written unit tests is very hard to read for a non-coder
Summary
You have two very different things, working in different ways, using different tools (natural language vs code). So its not surprising you have different tools.
Unfortunately we have lots of people using Cucumber as a test tool to write unit like tests, and we have lots of people using unit test tools to write slow tests that test large chunks of code. Both these things are possible, maybe even viable, but they are far from optimal.
Cucumber is a Behavior Driven Design (BDD) framework. Where you can check the behavior of a piece of code.
JUnit is a lower level "Unit test" tool that allows developers to test every possible part of the code.
You can prefer Cucumber vs Junit for more clarity.
So, i'll put an example, explaining the difference between domain of Cucumber vs Junit
Considering a Web application's example..
Junits will be testing code logic of methods involved at Controller, Service & DAO layer
Cucumber Tests for this application, will bear cases having various REST request, exposed by API of this application. [It will have test requests for Get, Post, Put, Delete with multiple data for each request, as per case]
How to write a unit test framework?
Can anyone suggest some good reading?
I wish to work on basic building blocks that we use as programmers, so I am thinking of working on developing a unit test framework for Java.
I don't intend to write a framework that will replace junit;
my intention is to gain some experience by doing a worthy project.
There are several books that describe how to build a unit test framework. One of those is Test-Driven Development: By Example (TDD) by Kent Beck. Another book you might look at is xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros.
Why do you want to build your own unit test framework?
Which ones have you tried and what did you find that was missing?
If (as your comments suggest) your objective is to learn about the factors that go into making a good unit test framework by doing it yourself, then chapters 18-24 (Part II: The xUnit Example) of the TDD book show how it can be done in Python. Adapting that to Java would probably teach you quite a lot about Python, unit testing frameworks and possibly Java too.
It will still be valuable to you to have some experience with some unit test framework so that you can compare what you produce with what others have produced. Who knows, you might have some fundamental insight that they've missed and you may improve things for everyone. (It isn't very likely, I'm sorry to say, but it is possible.)
Note that the TDD people are quite adamant that TDD does not work well with databases. That is a nuisance to me as my work is centred on DBMS development; it means I have to adapt the techniques usually espoused in the literature to accommodate the realities of 'testing whether the DBMS works does mean testing against a DBMS'. I believe that the primary reason for their concern is that setting up a database to a known state takes time, and therefore makes testing slower. I can understand that concern - it is a practical problem.
Basically, it consists of three parts:
preparing set of tests
running tests
making reports
Preparing set of tests means that your framework should collect all tests which you want to run. You can specify these tests (usually classes with test methods which satisfy some convention or marked with certain annotation or implement marker interface) in a separate file (java or xml), or you can find them dynamically (making a search over classpath).
If you choose the dynamic searching, then you'll probably have to use some libraries which can analyse java bytecode. Otherwise you'll have to load all the classes in your classpath, and this a) requires much time and b) will execute all static initializers of loaded classes and can cause unexpected tests results.
Running tests can vary significantly depending on features of your framework. The simplest way is just calling test methods inside a try/catch block, analysing and saving results (you have to analyze 2 situations - when the assertion exception was thrown and when it was not thrown).
Making reports is all about printing analyzed results in xml/html/wiki or whatever else format.
The Cook's Tour is written by Kent Beck (I believe; it's not attributed), and describes the thought process that went into writing JUnit. I would suggest reading it and considering how you might choose an alternate line of development.
I am new to TDD and want to know which kind oft test i need for which part of the software.
Currently my team creates a relative complex editor on the Netbeans Platform, where i have to integrate an external editor and also write own stuff.
So how can i do the tests best for the GUI, the own code, integration code?
Where do i create coded tests and where should i use test cases and testers?
We are considering to use scala specs or junit for the coded tests.
Thank you for your help!
As a general rule you should consider writing a test case for every function/method you write in your class.
In our TDD process we simply follow the rule that every Java class must have a corresponding JUnit class that contains #Test method for every method of that Java class.
Then we follow this with 'code coverage' that will tell us how much of the code we have actually written has been tested. For this I suggest looking at a tool called Cobertura (link here) which provide us an easy visual way to examine the percentage of our code that has been tested. it works by simply detecting how many line of code your JUnit class has tested for every Java class. This will provide you with a good idea about what functionality in your code has or has not been tested.
We usually aim to have about 80% of the code tested (easier said than done though)
consider writing test cases for your high priority functionality first to get you started.
We don't usually write JUnit for the GUI. not sure if there is a way but we leave the GUI to be tested by the testers when the application is going through the usual Testing phase.
hope this helps a little.
This is very subjective, but I would suggest cleanly separating your gui and writing solid unit tests against the interface between the gui and the business layer. Don't worry about automating the gui testing. This will naturally force you to create a clean separation of concerns between the layers.
GUI is for human interaction, therefore automated GUI test will result in epic failures. If you want to test GUI properly, you'll just need humans.
However, you can test how your app will react to different user actions. Especially, think to test how your app should cope with user errors (for instance, when letters are written in a numeric only input field).
If the distinction between unit tests, functional tests and integration tests is not clear for you, don't bother too much. Try to test as much thing as you think necessary (again don't forget worst case scenarios). When in doubt, remember that it's better to test too much, than to little.
What is the most commonly used approach used for testing in Java projects (iterative development) ?
My suggestion is that you should have a healthy mix of automated and manual testing.
AUTOMATED TESTING
Unit Testing
Use NUnit to test your classes, functions and interaction between them.
http://www.nunit.org/index.php
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.peterkrantz.com/2005/selenium-for-aspnet/
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
Personal experience would suggest the most popular approach is none at all.
I've worked with TDD (Test Driven Development) before, and my feelings towards it are mixed. Essentially, you write your tests before you write your code, and write your code to satisfy the requirements of the test. TDD forces you to have an extremely clear idea about your requirements prior to starting. An added benefit is that, once you are done development, assuming you followed closely to TDD procedures, you'd have a complete set of test suites to go with the code. The down side is that it takes an extremely long time, and at times, you'd just want to skip a couple of steps (e.g. maybe writing code before tests like a sane person would prefer to do).
More can be read here (wiki link)
Unit testing?
Contract-based programming, a la Eiffel?
Waterfall model?
Different shops do different things. If there were one method to rule them all, you wouldn't be asking this question.
On the premise of doing testing at all, I would say that testing with JUnit is the common approach to do testing in Java.
Although most tests are written with JUnit mostly tests tend to be more integration tests than unit tests. (meaning not testing one thing in isolation but some things together)
Additionally test are mostly not written in a test first approach but in parallel or after a specific feature has been implemented.
If you go to a team that makes more advanced use of testing you might probably find some CI Server (Cruise Control, Hudson) running the tests at least once a day during a nightly build.
In the order of most commonly used approach:
no tests at all
manual tests: running the app,
clicking or providing input, check
results
try to write some JUnits, forget
about them, slide to 2 and 1
Start with TDD, see that it's hard
then slide to 3, 2 and 1
on the theoretical side there are loads of ways to properly test the code.
If you are looking for something practical take a look at
Clean Code Talk. Take a look at the whole series, about 5 talks (can't post more than one link).
My Suggestion for testing of the java project is to keep it simple.
Steps :-
Manual Testing :-Achieve a stable product.
Automation Testing :- Maintain the quality of the product.
Report Generation and reporting :- Let people know the quality of the product.
Continuous Integration :-Make it a complete automated,continuous tool.
When developer will commit the Functionality then Start testing the it module by module.Try to compare the Actual Output with the expected output and against that Log the issues.
When developer resolved with the issues,Start with the Integration Testing and also start testing the resolved state issues and check whether any regression occur due to issue Fixing.
At last when product become the stable one,Then start for automating the the modules.
You can also follow automation step by step like:-
1.Automating the modules.
2.Report generation and send mail for product HealthCheck.
3.Continuous Integration and Automation testing on private server on local machine.
I'm looking into how best to automate integration tests (by which I mean complete use cases entirely within our application)
The questions
Correct Approach for Unit Testing Complex Interactions
What are the pros and cons of automated Unit Tests vs automated Integration tests?
cover the "why" and "what" aspects very well.
The question Automated integration testing a C++ app with a database implies that xUnit frameworks are a good way to create and execute integration tests. Are xUnit's really well suited to that task? Are there common gotcha's to be aware of? A good approach to follow?
Are there better approaches (short of possibly purchasing the HP / former Mercury tool suite)?
My specific environment for this project is Java / SpringSource / Hibernate but am also interested in suggestions for the .Net platform.
The question Automated integration testing a C++ app with a database implies that xUnit frameworks are a good way to create and execute integration tests. Are xUnit's really well suited to that task? Are there common gotcha's to be aware of? A good approach to follow?
JUnit and TestNG are initially unit testing frameworks but can be used for integration testing as well. So to me, the answer is yes, they are well suited for integration testing, e.g. testing the service --> domain > persistence --> database layers (I'll actually come back on integration testing later). One of the tricky things when doing integration tests that involve the database is the data. A tool such as DbUnit can help and is typically used to put the database in a known state before to run each test (and to perform asserts on the database content). Another approach is to run the tests in a transaction and to rollback the transaction at the end of the test. Spring allows to do that very easily, so does the unitils library. In any case, a best practice is to avoid interdependent tests as much as possible (as mentioned in the link you gave), they are just a nightmare.
Are there better approaches (short of possibly purchasing the HP / former Mercury tool suite)?
To my knowledge, such tools are more end-to-end testing tools i.e. functional testing tools. So if by integration tests (which for me mean testing several components together) you actually mean functional tests (this is my understanding of complete use cases), I'd suggest to look at things like:
Abbot, Marathon, Frankenstein, etc for Swing applications
iMacros, Selenium, Cucumber, etc for Web applications
SoapUI for Web Services
Pay a special attention to the one in bold (all are actually great tools but I'm sure the one in bold provide good automation capabilities). And because HTTP and SOAP are standards (this doesn't apply to the Swing UI testing tools of course), these tools are not really Java specific (even if the tests themselves are written in Java/Groovy for SoapUI). And BTW, Selenium supports many programming languages).
Multi-threading can be a problem, since JUnit won't pick up exceptions in other threads. There are some Java Puzzlers about that. You also need to invent your own ways of doing statistical testing and the assert methods can be a bit rough. I also think that the semantics of JUnit are a bit unclear (JUnit uses one separate instance per test method for instance). For these reasons I switched to TestNG, which in my opinion is a better designed framework. The fact that JUnit was designed using extreme programming shows sometimes.
As mentioned you can do it with xUnit frameworks, but if you will want to mix Java and .Net, or web applications and desktop applications or add some more complexity to overall picture, than you won't be able to do it with just one unit test framework. So you will need to have many test tools, many test environments, many test script developers (for example one foe Java unit tests, one for .Net tests)... and this will add up complexity, troubles, costs...
As for HP Quick Test Pro you mentioned it should cover most of your needs. I mean most, because there may be some areas where it is not suitable (no way to run scripts on applications via Citrix virtualization), but for the most cases it will do the job. It is suitable for java/.net/web and other things (there are plugins for specialized uses). QTP usually operates on GUI objects so you can prepare test cases for user use cases, and test can be performed in a manner that normal user would perform actions (just a bit faster you have to intentionally slow it down to user speed if needed).
You will probably you will need one tool, one test environment, one test scrip developer (VB). It is expensive, but if it is for company it should be better choice in the long run.
And if you ask from company perspective it will play well with HP Quality Center if you decide to use it for your whole Testing division/team. Unless you use IBM solutions, than they have their own tool suite as part of their software Delivery Platform including Rational Robot