We have a large number of web applications developed at a fast pace. More often than not, we are unable to test them sufficiently on different browsers, leading to issues at times. (At the moment, we rely on developers to test their apps in different browsers.)
I am aware that I can use Selenium remote to test functioning of each app in different browser but that does not test the layout (is the top nav really on the top, does this long text break the layout, etc).
Is there an equivalent UI testing tool that could test not only the functionality but also the layout, even if on a high level?
Ideally, I would like to integrate these tests with a CI tool so we could keep a check on UI regression.
I don't know if this is even possible so checking - How do you test your apps in different browsers on a CI tool ?
Take a look at MogoTest. When I last checked, it detected cross-browser layout differences. I'm sure you could integrate it into a CI workflow.
I'm pretty sure the CI package from Hudson allows you to specify which browser to use. Failing that, you can specify it in your Junit-exported Selenium Tests. Just keep an array of the browser identifiers and loop over them for each test - bear in mind that taking this road will make it harder to see which browser failed for which test.
Related
I've been thinking about acceptance testing and would like to gain a clearer understanding of how it can be automated (and fitted into a Continuous Delivery pipeline, for example). I have a desktop application written in Java, a messenger application. If I want to automate acceptance tests, how do I go about it? Here is an example manual test: "User A is messaging user B when the server crashes. Switching to a failover server, verify that the chat can be resumed where it left off and all chat functionality still works as expected in the client"
Can JUnit be used for automated acceptance and system testing like this? I've also heard of Selenium but I gathered it was only for web applications?
I understand the concept of unit testing with JUnit, but I'm lost as to how higher level testing of a desktop application's functionality can be automated. Some tests may involve interaction with the GUI, like sending messages to other clients, but others will involve server-side stuff like server crashes and so on. I've been thinking that it's probably possible to do this using JUnit and the application's code, maybe with some scripting involved (Perl, Ruby etc.) to launch the tests?
There are several software solutions that come at a cost for this. Some serious ones are QTP (now UFT), TestComplete, Ranorex, SilkTest. They should all have trial software available. If the Java application was written in Swing/AWT, Jemmy may be worth checking out. As Tony Hopkinson mentioned, AutoIt may fit, and is very fast to use. Selenium is great, but is only for web testing as far as I know. Automating a UI is very important for automated testing and complements acceptance testing because of its interaction with the end-user. Unit tests overlap a lot, and may cover functional integrations depending on how you organize them and have enough of them, but they are two different approaches with two different purposes.
Also, if you were using Swing, found this thread which may help: http://www.coderanch.com/t/96035/Testing/Automated-Swing-Testing
At present I have a framework which support firefox browser only. Now I am going to enhance my frame work so that can any give your ideas to me by suggesting your thoughts.
At present I am using Selenium Java, Hybrid framework, maven setup, Webdriver surefire reports.
So
1) If I want to run test cases on multiple browsers and multiple version of browsers parallel. what is the best approach?
Thanks
R
Well you need to setup the grid for parallel runs.
You need to make your browser launch code depend on a configurable parameter.
You can either take it as a parameter of the testng xml or make it as a system argument.
Depending on the value of this parameter, your listeners or your launch code (I am hoping that would be isolated since u mention a framework is in place) should be able to use this parameter to make a decision on what browser/node to launch.
For parallel runs, it would be better if you can use some framework. We use testng as the framework which supports parallel runs with good results.
I know you dont want to hear it, but best way actually is trial and error
Your site is going to behave differently in different browsers. So you will have to update your script for almost every browser and you will have to do it manually
Browsing speed is going to be different. Maybe invest in Implicit wait into your script
And yes, you are going to hate it.
My "best approach" is:
Implement the script for one most used browser and automate as much as possible
In other versions do manual shakedown of most crucial functions
Communicate with business to write somewhere that some obscure browsers (like IE 6) are no longer supported so you have more time to polish the script
I did a GWT project in the past (GWT version 1.4) and it was extremely painful.
Interface is build with code (which it is really bad), requires a lot of slow compiling and waiting, unit testing was awful. Not to mention that integrating with Hibernate was the most annoying thing.
But it looks to me that GWT is really hot among Java developers and I'm reconsidering it.
Have you tried GWT 2.x? is it better now? I'm particularly interested in the three previous points (slow compiling, UI building and unit testing).
Let's address your three main complaints one-by-one.
Slow compilation
This is really a lot better now in a number of ways.
Compilation has become faster.
The GWT compiler can compile several permutations in parallel.
The (god awful) "hosted mode" browser has been replaced with a "development mode" browser plugin so that you can test in your favourite mainstream browser without compilation.
UI building
Yes. UiBinder.
Write HTML "templates" that include elements that act as placeholders for widgets. Elements representing panels (widgets that can contain widgets) can contain elements representing other widgets.
Yes, there will still be some aspect of composing widgets in Java, but this is now greatly reduced.
Unit testing
How was it awful before? Your logic code can still be run through JUnit. Recently, there has been a much heavier push toward MVP design in GWT, so presumably much more of your code can be tested with plain old JUnit.
GWT also has a manner of unit testing where a non-interactive browser is run. In my experience this can usually be safely avoided when using plenty of JUnit tests both for client (presenter) and server code.
What is the most commonly used approach used for testing in Java projects (iterative development) ?
My suggestion is that you should have a healthy mix of automated and manual testing.
AUTOMATED TESTING
Unit Testing
Use NUnit to test your classes, functions and interaction between them.
http://www.nunit.org/index.php
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.peterkrantz.com/2005/selenium-for-aspnet/
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
Personal experience would suggest the most popular approach is none at all.
I've worked with TDD (Test Driven Development) before, and my feelings towards it are mixed. Essentially, you write your tests before you write your code, and write your code to satisfy the requirements of the test. TDD forces you to have an extremely clear idea about your requirements prior to starting. An added benefit is that, once you are done development, assuming you followed closely to TDD procedures, you'd have a complete set of test suites to go with the code. The down side is that it takes an extremely long time, and at times, you'd just want to skip a couple of steps (e.g. maybe writing code before tests like a sane person would prefer to do).
More can be read here (wiki link)
Unit testing?
Contract-based programming, a la Eiffel?
Waterfall model?
Different shops do different things. If there were one method to rule them all, you wouldn't be asking this question.
On the premise of doing testing at all, I would say that testing with JUnit is the common approach to do testing in Java.
Although most tests are written with JUnit mostly tests tend to be more integration tests than unit tests. (meaning not testing one thing in isolation but some things together)
Additionally test are mostly not written in a test first approach but in parallel or after a specific feature has been implemented.
If you go to a team that makes more advanced use of testing you might probably find some CI Server (Cruise Control, Hudson) running the tests at least once a day during a nightly build.
In the order of most commonly used approach:
no tests at all
manual tests: running the app,
clicking or providing input, check
results
try to write some JUnits, forget
about them, slide to 2 and 1
Start with TDD, see that it's hard
then slide to 3, 2 and 1
on the theoretical side there are loads of ways to properly test the code.
If you are looking for something practical take a look at
Clean Code Talk. Take a look at the whole series, about 5 talks (can't post more than one link).
My Suggestion for testing of the java project is to keep it simple.
Steps :-
Manual Testing :-Achieve a stable product.
Automation Testing :- Maintain the quality of the product.
Report Generation and reporting :- Let people know the quality of the product.
Continuous Integration :-Make it a complete automated,continuous tool.
When developer will commit the Functionality then Start testing the it module by module.Try to compare the Actual Output with the expected output and against that Log the issues.
When developer resolved with the issues,Start with the Integration Testing and also start testing the resolved state issues and check whether any regression occur due to issue Fixing.
At last when product become the stable one,Then start for automating the the modules.
You can also follow automation step by step like:-
1.Automating the modules.
2.Report generation and send mail for product HealthCheck.
3.Continuous Integration and Automation testing on private server on local machine.
I would like to make integration tests and system tests for my applications but producing good integration and system tests have often needed so much effort that I have not bothered. The few times I tried, I wrote custom, application-specific test harnesses, which felt like re-inventing the wheel each time. I wonder if this is the wrong approach. Is there a "standard" approach to integration and full system testing?
EDIT: To clarify, it's automated tests, for desktop and web applications. Ideally a complete test suite that exercises the full functionality of the application.
If by "make integration tests and system tests" you mean automated tests, then the answer is no, there is no standard approach. What approach choose will depend on:
the characteristics of the application (e.g. does it have a GUI?, is it read only?, how many external dependencies does it have, etc)
what you are trying to test (maybe you only need GUI tests, or perhaps the opposite is true and you don't really care about the GUI but the internal logic is critical)
how quickly you want to see the results (e.g. the more you stub out the faster your tests become)
the skillsets on your team
Personally, I love any approach that integrates with JUnit. JUnit is a fantastic framework which is well supported and easily tied into a continuous intergration server. Here are a few possible approaches:
Selenium with JUnit - Fantastic tool for driving web applications
Concordion - for all application types. Integrates with JUnit and allows plain english specifications of tests. Your 'fixture'/test code will hook into key words in the specification and assert or perform actions on them.
FEST - for swing applications, again it integrates with JUnit (see a theme yet? ;) (more choices here)
The above examples provide a tremendous amount of out of the box help for testing. Of course, they still require effort to wire to your application and maintain, but the advantages are well worth it. In addition to the above, you may need to be thinking about how to stub or mock out areas of your application. Perhaps you want to do all of your testing "under the GUI" or "above the database". In the first scenario, you'll need your tests to start at the points in your code where the GUI would interact with it and in the latter you'll need to stub out the services which interact with your database.
Point being, there's lots of ways to do this. Best start with a very clear understanding of what you want to get out of your testing. Then learn what existing frameworks are out there to help you based on what you want to test, and finally, don't try to conquer the world in a night. Start small getting a few tests running. Get the green bar (always a joy to see!) Get a stable proven platform for testing and make sure you're happy with it. Then add more as you go along.