I've been thinking about acceptance testing and would like to gain a clearer understanding of how it can be automated (and fitted into a Continuous Delivery pipeline, for example). I have a desktop application written in Java, a messenger application. If I want to automate acceptance tests, how do I go about it? Here is an example manual test: "User A is messaging user B when the server crashes. Switching to a failover server, verify that the chat can be resumed where it left off and all chat functionality still works as expected in the client"
Can JUnit be used for automated acceptance and system testing like this? I've also heard of Selenium but I gathered it was only for web applications?
I understand the concept of unit testing with JUnit, but I'm lost as to how higher level testing of a desktop application's functionality can be automated. Some tests may involve interaction with the GUI, like sending messages to other clients, but others will involve server-side stuff like server crashes and so on. I've been thinking that it's probably possible to do this using JUnit and the application's code, maybe with some scripting involved (Perl, Ruby etc.) to launch the tests?
There are several software solutions that come at a cost for this. Some serious ones are QTP (now UFT), TestComplete, Ranorex, SilkTest. They should all have trial software available. If the Java application was written in Swing/AWT, Jemmy may be worth checking out. As Tony Hopkinson mentioned, AutoIt may fit, and is very fast to use. Selenium is great, but is only for web testing as far as I know. Automating a UI is very important for automated testing and complements acceptance testing because of its interaction with the end-user. Unit tests overlap a lot, and may cover functional integrations depending on how you organize them and have enough of them, but they are two different approaches with two different purposes.
Also, if you were using Swing, found this thread which may help: http://www.coderanch.com/t/96035/Testing/Automated-Swing-Testing
Related
I'm fascinated towards robots. You know usually it creates our work easy and for web you don't need to go anywhere on links once our robots is initialized on web server.
Let's come to our main question:
What exactly is the use of Robot Framework .. I'm good with Java but not with Python. Can I use it with Java? For what purpose Robot Framework is used for?
I'm good in HTMLUnit. But it can be used to parse the HTML source code and we could also implement it in Java to make source code. But what about Robot Framework?
I googled for 2 hours but couldn't find the exact answers. Please let me know if you want to know any information from me.
Overview
It's main use is to enable writing acceptance tests with a high level of abstraction for software products.
The framework requires less technical skill than programming language-based frameworks, and so can be used by team members who have very little programming experience. For example, an agile team product owner could express acceptance tests using the framework, without having to know the details of how the product is implemented.
On the other hand, because of its keyword-based design, those with a high degree of technical skill can write keywords in their language of choice to test low level functions, and even do unit and integration testing if so desired.
Uses
Because of its flexibility, robot framework can be used to test desktop applications, web applications, mobile applications, and RESTful and SOAP-based services. Because it provides an abstraction layer on top of the physical implementation of the system under test, it is possible to write tests that work cross-platform (ie: the same test case could be used to test both an android and iOS app, or for testing a web app that runs on chrome, firefox and safari).
Although robot framework is often used to test browser-based applications, it has been used to test databases, mobile devices, calculators, and many other things.
Extensibility
One of the great strengths of the robot framework is that it is highly extensible. Many of the features mentioned above are implemented as libraries. For example, you can plug in a library to use selenium to drive a browser. You can plug in a database library to directly access databases. There are libraries to support testing desktop apps, services, and many more.
Example
To illustrate how abstract a robot test case can be, a BDD-style testcase might look -- literally -- like this:
Given I am on the application login screen
When I enter a valid username and password
And I press the "login button"
Then I should be logged in
And my personal dashboard should be displayed
Note: this isn't the only way to write a test. Robot is very flexible, allowing you to choose between BDD (Behavior Driven Development) style, a traditional procedural style, or in a data-driven style.
The framework provides the infrastructure for converting those statements to actionable items. Ultimately they get boiled down to a function call -- either from an external library, or one provided by the development team. Those functions typically are written in python, but can be written in Java, a .NET language, or other languages through an interface, depending on how you actually run the test.
Reporting
In addition to being able to run tests, robot framework provides reports and logs that can be used to visualize the state of the product. Logs provide details of every single keyword and function call, reports show the status of complete suites and test cases, and can provide summary information based on arbitrary tags.
We have a large number of web applications developed at a fast pace. More often than not, we are unable to test them sufficiently on different browsers, leading to issues at times. (At the moment, we rely on developers to test their apps in different browsers.)
I am aware that I can use Selenium remote to test functioning of each app in different browser but that does not test the layout (is the top nav really on the top, does this long text break the layout, etc).
Is there an equivalent UI testing tool that could test not only the functionality but also the layout, even if on a high level?
Ideally, I would like to integrate these tests with a CI tool so we could keep a check on UI regression.
I don't know if this is even possible so checking - How do you test your apps in different browsers on a CI tool ?
Take a look at MogoTest. When I last checked, it detected cross-browser layout differences. I'm sure you could integrate it into a CI workflow.
I'm pretty sure the CI package from Hudson allows you to specify which browser to use. Failing that, you can specify it in your Junit-exported Selenium Tests. Just keep an array of the browser identifiers and loop over them for each test - bear in mind that taking this road will make it harder to see which browser failed for which test.
What is the most commonly used approach used for testing in Java projects (iterative development) ?
My suggestion is that you should have a healthy mix of automated and manual testing.
AUTOMATED TESTING
Unit Testing
Use NUnit to test your classes, functions and interaction between them.
http://www.nunit.org/index.php
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.peterkrantz.com/2005/selenium-for-aspnet/
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
Personal experience would suggest the most popular approach is none at all.
I've worked with TDD (Test Driven Development) before, and my feelings towards it are mixed. Essentially, you write your tests before you write your code, and write your code to satisfy the requirements of the test. TDD forces you to have an extremely clear idea about your requirements prior to starting. An added benefit is that, once you are done development, assuming you followed closely to TDD procedures, you'd have a complete set of test suites to go with the code. The down side is that it takes an extremely long time, and at times, you'd just want to skip a couple of steps (e.g. maybe writing code before tests like a sane person would prefer to do).
More can be read here (wiki link)
Unit testing?
Contract-based programming, a la Eiffel?
Waterfall model?
Different shops do different things. If there were one method to rule them all, you wouldn't be asking this question.
On the premise of doing testing at all, I would say that testing with JUnit is the common approach to do testing in Java.
Although most tests are written with JUnit mostly tests tend to be more integration tests than unit tests. (meaning not testing one thing in isolation but some things together)
Additionally test are mostly not written in a test first approach but in parallel or after a specific feature has been implemented.
If you go to a team that makes more advanced use of testing you might probably find some CI Server (Cruise Control, Hudson) running the tests at least once a day during a nightly build.
In the order of most commonly used approach:
no tests at all
manual tests: running the app,
clicking or providing input, check
results
try to write some JUnits, forget
about them, slide to 2 and 1
Start with TDD, see that it's hard
then slide to 3, 2 and 1
on the theoretical side there are loads of ways to properly test the code.
If you are looking for something practical take a look at
Clean Code Talk. Take a look at the whole series, about 5 talks (can't post more than one link).
My Suggestion for testing of the java project is to keep it simple.
Steps :-
Manual Testing :-Achieve a stable product.
Automation Testing :- Maintain the quality of the product.
Report Generation and reporting :- Let people know the quality of the product.
Continuous Integration :-Make it a complete automated,continuous tool.
When developer will commit the Functionality then Start testing the it module by module.Try to compare the Actual Output with the expected output and against that Log the issues.
When developer resolved with the issues,Start with the Integration Testing and also start testing the resolved state issues and check whether any regression occur due to issue Fixing.
At last when product become the stable one,Then start for automating the the modules.
You can also follow automation step by step like:-
1.Automating the modules.
2.Report generation and send mail for product HealthCheck.
3.Continuous Integration and Automation testing on private server on local machine.
I would like to make integration tests and system tests for my applications but producing good integration and system tests have often needed so much effort that I have not bothered. The few times I tried, I wrote custom, application-specific test harnesses, which felt like re-inventing the wheel each time. I wonder if this is the wrong approach. Is there a "standard" approach to integration and full system testing?
EDIT: To clarify, it's automated tests, for desktop and web applications. Ideally a complete test suite that exercises the full functionality of the application.
If by "make integration tests and system tests" you mean automated tests, then the answer is no, there is no standard approach. What approach choose will depend on:
the characteristics of the application (e.g. does it have a GUI?, is it read only?, how many external dependencies does it have, etc)
what you are trying to test (maybe you only need GUI tests, or perhaps the opposite is true and you don't really care about the GUI but the internal logic is critical)
how quickly you want to see the results (e.g. the more you stub out the faster your tests become)
the skillsets on your team
Personally, I love any approach that integrates with JUnit. JUnit is a fantastic framework which is well supported and easily tied into a continuous intergration server. Here are a few possible approaches:
Selenium with JUnit - Fantastic tool for driving web applications
Concordion - for all application types. Integrates with JUnit and allows plain english specifications of tests. Your 'fixture'/test code will hook into key words in the specification and assert or perform actions on them.
FEST - for swing applications, again it integrates with JUnit (see a theme yet? ;) (more choices here)
The above examples provide a tremendous amount of out of the box help for testing. Of course, they still require effort to wire to your application and maintain, but the advantages are well worth it. In addition to the above, you may need to be thinking about how to stub or mock out areas of your application. Perhaps you want to do all of your testing "under the GUI" or "above the database". In the first scenario, you'll need your tests to start at the points in your code where the GUI would interact with it and in the latter you'll need to stub out the services which interact with your database.
Point being, there's lots of ways to do this. Best start with a very clear understanding of what you want to get out of your testing. Then learn what existing frameworks are out there to help you based on what you want to test, and finally, don't try to conquer the world in a night. Start small getting a few tests running. Get the green bar (always a joy to see!) Get a stable proven platform for testing and make sure you're happy with it. Then add more as you go along.
I'm looking into how best to automate integration tests (by which I mean complete use cases entirely within our application)
The questions
Correct Approach for Unit Testing Complex Interactions
What are the pros and cons of automated Unit Tests vs automated Integration tests?
cover the "why" and "what" aspects very well.
The question Automated integration testing a C++ app with a database implies that xUnit frameworks are a good way to create and execute integration tests. Are xUnit's really well suited to that task? Are there common gotcha's to be aware of? A good approach to follow?
Are there better approaches (short of possibly purchasing the HP / former Mercury tool suite)?
My specific environment for this project is Java / SpringSource / Hibernate but am also interested in suggestions for the .Net platform.
The question Automated integration testing a C++ app with a database implies that xUnit frameworks are a good way to create and execute integration tests. Are xUnit's really well suited to that task? Are there common gotcha's to be aware of? A good approach to follow?
JUnit and TestNG are initially unit testing frameworks but can be used for integration testing as well. So to me, the answer is yes, they are well suited for integration testing, e.g. testing the service --> domain > persistence --> database layers (I'll actually come back on integration testing later). One of the tricky things when doing integration tests that involve the database is the data. A tool such as DbUnit can help and is typically used to put the database in a known state before to run each test (and to perform asserts on the database content). Another approach is to run the tests in a transaction and to rollback the transaction at the end of the test. Spring allows to do that very easily, so does the unitils library. In any case, a best practice is to avoid interdependent tests as much as possible (as mentioned in the link you gave), they are just a nightmare.
Are there better approaches (short of possibly purchasing the HP / former Mercury tool suite)?
To my knowledge, such tools are more end-to-end testing tools i.e. functional testing tools. So if by integration tests (which for me mean testing several components together) you actually mean functional tests (this is my understanding of complete use cases), I'd suggest to look at things like:
Abbot, Marathon, Frankenstein, etc for Swing applications
iMacros, Selenium, Cucumber, etc for Web applications
SoapUI for Web Services
Pay a special attention to the one in bold (all are actually great tools but I'm sure the one in bold provide good automation capabilities). And because HTTP and SOAP are standards (this doesn't apply to the Swing UI testing tools of course), these tools are not really Java specific (even if the tests themselves are written in Java/Groovy for SoapUI). And BTW, Selenium supports many programming languages).
Multi-threading can be a problem, since JUnit won't pick up exceptions in other threads. There are some Java Puzzlers about that. You also need to invent your own ways of doing statistical testing and the assert methods can be a bit rough. I also think that the semantics of JUnit are a bit unclear (JUnit uses one separate instance per test method for instance). For these reasons I switched to TestNG, which in my opinion is a better designed framework. The fact that JUnit was designed using extreme programming shows sometimes.
As mentioned you can do it with xUnit frameworks, but if you will want to mix Java and .Net, or web applications and desktop applications or add some more complexity to overall picture, than you won't be able to do it with just one unit test framework. So you will need to have many test tools, many test environments, many test script developers (for example one foe Java unit tests, one for .Net tests)... and this will add up complexity, troubles, costs...
As for HP Quick Test Pro you mentioned it should cover most of your needs. I mean most, because there may be some areas where it is not suitable (no way to run scripts on applications via Citrix virtualization), but for the most cases it will do the job. It is suitable for java/.net/web and other things (there are plugins for specialized uses). QTP usually operates on GUI objects so you can prepare test cases for user use cases, and test can be performed in a manner that normal user would perform actions (just a bit faster you have to intentionally slow it down to user speed if needed).
You will probably you will need one tool, one test environment, one test scrip developer (VB). It is expensive, but if it is for company it should be better choice in the long run.
And if you ask from company perspective it will play well with HP Quality Center if you decide to use it for your whole Testing division/team. Unless you use IBM solutions, than they have their own tool suite as part of their software Delivery Platform including Rational Robot