I am developing a library that extends Selenium 2 with some custom commands. The library should be usable from Selenium's Java and Python bindings as well as in Selenium IDE. From my research, these three target bindings should cover at least 80% of all Selenium 2 scripts.
In order to implement my custom commands for Selenium IDE, I think I need to write a plugin for it in JavaScript.
My question is this: If I already have an implementation of my custom commands in JavaScript, is it safe to re-use this implementation for the Java- and Python bindings of my library?
I am thinking of an approach that injects the JavaScript implementation of my commands via WebDriver#executeScript. Here is a pseudocode implementation of what I am thinking of.
In Java:
public void fooJava() {
executeScript("Inject code.js");
executeScript("fooJavaScript();");
}
In code.js:
function fooJavaScript() {
// Implementation of command "foo" from Selenium IDE plugin.
}
So, to execute my custom command fooJava() in Java, my library's code.js would be injected into the browser via executeScript. This would contain a JavaScript implementation of foo, say fooJavaScript. In a next executeScript call, this fooJavaScript would then be called.
While this approach would prevent me from having to implement my custom commands three times (Java, Python, Selenium IDE), I have a few concerns:
When I inject my code.js, am I in danger of destroying global state of the web site?
To which extent can I rely on JavaScript? Will it work if an alert dialog is present? In practice, how many of the drivers used with Selenium do not support JavaScript? Eg. HtmlUnit?
Will this work in all major browsers (somewhat recent versions of IE, Chrome, Firefox, Safari)?
Your real-life experiences with this would be much appreciated.
Principle states you should not be using JS as your testing mechanism if you are just delivering a payload with WebDriver.
WebDriver = integration testsJS = if you want unit tests
I don't know your use cases exactly, but:
If you're trying to run integration tests, stick with WebDriver to best simulate user interaction. You also avoid cross-browser JS issues in the future by relying on the WebDriver hooks to interact with the page, as in, you are better off relying on the community to provide reliable basic DOM interaction APIs for each browser. If you can't trigger test conditions with browser interactions, you're getting into unit/code testing territory instead of integration testing.
If you are trying to run the JS for sake of essentially testing a single function or piece of code rather than an integrated interaction, you are trying to run a unit test. Unit tests are best done in JS with something like Jasmine (name any framework here).
Reasoning:
Integration tests should be written to be as implementation independent as possible. You should not need to know a function name to trigger an integration test, since someone might change the function name in the future or restructure the code.
Since you are filling a QE/tester role, you do not want to be responsible for breaking integration tests when code changes - if you use this and are responsible, then you will need to change a test every time there is a code restructure.
Sources: Experience as a QE in 10,000+ employee software co.
Related
What is the advantage of using selenium with Typescript over selenium with Java.
Which type of web applications can be automated with Selenium+Typescript and why these cant be done with selenium+java?
In term of automation -> Type scripts basically used by tools like protractor which actually used javascript itself but provide extension to write scripts in typescript which then converted to javascript and pass further for execution.
The main use of typescript is its oops based so its easy to write maintainable code.
Protractor is a nodejs plug-in which use selenium wedriverjs binding internally
Your second question why typescripts:
Basically Protractor is use on Angular-JS client side websites. still you can automate non-angular websites also using Protractor
Normally selenium do not have the capability to understand if the element is ready due to even after page load completed, JavaScript is still working behind for Anugular/JS based website so Protractor provide wrapper which having additional functionality like waitForAngular
Additionally Protractor provide more locators identify strategy which is not present in normal selenium with java or c#
Refer:
http://www.protractortest.org/#/locators
As Protractor is build on wedriverjs we also need to script for it on JavaScript or typescript. basically even if you code in typescript internally it will convert it into typescript. people prefer typescript as it follows full oops unlike like JavaScript
Source:
http://www.protractortest.org/#/
there are more js based framework is in the market like:
WebdriverIO
Nightwatch
Now java also comes up with many library like JSWaiter etc which claim they can handle js based client side websites with java + selenium as well
No major benefits in my opinion, that's my tl,dr;
I can think of a couple of options you might have though:
You might be able to integrate the test framework with the front-end application repository. This can be useful if you are using a Typescript based framework (like React for example) and you can potentially use that to import the application data (for example stub responses for APIs that the developers are using for their dev work, assuming they are available there... just an example) and use that for testing the front-end code in isolation without the API dependencies.
The other reason I can think of is that if you did the above you can potentially execute your test cases as part of the CI of the front-end code, for example you can run a subset of the test cases with pull requests to make sure all is green before merging in the main branch.
Really you can do all the above with an external repo written in Java (hence, no major benefits)
At present I have a framework which support firefox browser only. Now I am going to enhance my frame work so that can any give your ideas to me by suggesting your thoughts.
At present I am using Selenium Java, Hybrid framework, maven setup, Webdriver surefire reports.
So
1) If I want to run test cases on multiple browsers and multiple version of browsers parallel. what is the best approach?
Thanks
R
Well you need to setup the grid for parallel runs.
You need to make your browser launch code depend on a configurable parameter.
You can either take it as a parameter of the testng xml or make it as a system argument.
Depending on the value of this parameter, your listeners or your launch code (I am hoping that would be isolated since u mention a framework is in place) should be able to use this parameter to make a decision on what browser/node to launch.
For parallel runs, it would be better if you can use some framework. We use testng as the framework which supports parallel runs with good results.
I know you dont want to hear it, but best way actually is trial and error
Your site is going to behave differently in different browsers. So you will have to update your script for almost every browser and you will have to do it manually
Browsing speed is going to be different. Maybe invest in Implicit wait into your script
And yes, you are going to hate it.
My "best approach" is:
Implement the script for one most used browser and automate as much as possible
In other versions do manual shakedown of most crucial functions
Communicate with business to write somewhere that some obscure browsers (like IE 6) are no longer supported so you have more time to polish the script
I'm really confused by the Selenium web site. At first they go to great lengths to explain Selenium IDE, but it outputs HTML test cases. But when you go to the Java documentation, it uses WebDriver which is pure java and your HTML test cases are useless. I found an export to JUnit feature from Selenium IDE but it doesn't use the WebDriver api, it wraps it inside the Selenium api, which doesn't even work. I get errors that I can only have one session at a time. I had only one test, made sure using netstat that I didn't have any other software listening on the port and disconnected the selenium instance. It just wouldn't work. Additionally the testcase extended from a deprecated class.
You cannot get back your test case to Selenium IDE from Java code so you can at that point throw the Selenium IDE away.
I converted the test case to pure WebDriver and I got it to work. So what is the recommended workflow for working with selenium and JUnit? Should I forget about Selenium IDE and recording actions in browser and just write everything in Java? Or is it still possible to use Selenium IDE somehow?
Having recently completed a project that used Selenium 2.0, I could find that the Selenium IDE is good only for prototying tests.
There are several drawbacks with the IDE that prevent it from being used to run Selenium tests. I could recall the following:
Typically you would want to run tests in a Suite. While the IDE does have this feature, I found that the IDE lacks a more important feature of running test setup and tear down scripts. This is trivial to achieve in JUnit/TestNG, but quite a pain with the recorded scripts in HTML. In short, the recorded tests aren't maintainable until you use a unit-testing library to run the tests from Java.
Data within the tests cannot be shared across tests; you will need to duplicate data in each test that requires it. This is expected when the tests are stored in a presentation language like HTML.
The default format of the exported tests does not use the page-object design pattern (which works very well for organizing Selenium tests). I didn't attempt creating my own format template for this, but this only convinced me that the best tests involving WebDriver and JUnit/TestNG are written by hand.
The optimal way of using the Selenium IDE is to create recordings of failed tests (by functional testers), instead of directly exporting the tests into your test suite. You could use the IDE to record the preliminary test so that the important aspects of the test (the assert/verify calls) are captured, and then rewrite it in your suite.
I use the IDE to create a "Work Flow Script", convert it into java code. Then I write everything from scratch in Java but with the info from the converted IDE script. That will have all ID's and so on but also in what order you have planned to "click" around, even some parts of it might be copied right off. Anyway it does speed things up a bit, but if you are using the webdriver it will complicate things a bit more and I have not yet moved over to the latest version.
Cheers
Stefan
The point is you can use either one, depending on your goal. In your case, WebDriver sounds like the way to go.
Selenium IDE is useful if you want to generate the HTML test cases.
Selenium WebDriver is useful for writing unit tests in Java (or other languages).
For a clear indication of this from the source, see the SeleniumHQ home page. It has a section on "Which part of Selenium is appropriate for me?", which answers your question.
I have to admit that I fell in love with Selenium for its record-and-play feature as well as the testcase generation functionality for those recorded actions from the IDE. But I am still hesitated to advance to the implementation stage because of the incidental details (e.g, locating the events with DOM, xpath..etc) that are built into the testcase during the recording, which could make the testcase failure prone whenever there is a html change once it's imported to the RC. I fully understand that it's a part of testers' jobs to adjust the expected results from time to time as part of the regression test, but I also do not wish the time spent on this is larger than the time that takes to do the manual test.
As far as I know Selenium with Robot framework has the keywords form of testcases. My guess is it allows us to extract the incidental details into various keywords, which could make the testcases being adjusted easier and are more maintainable. (Please correct me if I am wrong)
It will be appreciated to hear suggestions on how an effective UI automation environment should be setup. Should I just use Selenium RC or Selenium with Robot framework? And why?
Thanks in advance
You are absolutely right that incidental and often changing details in the produced scripts is the biggest problem of record-and-playback automation. You can obviously remove the details from the scripts after recording, but in my opinion it's better to build reusable libraries and code scripts manually from the start.
A good alternative for coding scripts using "real" programming languages is using some higher level automation framework such as Robot Framework that you mentioned. As you speculated, Robot's reusable keywords and also variables make extracting details away from tests very easy. The test cases in SeleniumLibrary's demo illustrates this very well and the demo also shows how to use Selenium through Robot.
You also asked about Sikuli. I've never used it myself but it sure looks interesting. You might be interested on this great how-to that explains how to use it through Robot Framework.
Our company is using Fitnesse, not Robot, to control Selenium however, we have the same problem. We switched from making assumptions about the DOM to only accessing elements by ID. Since this is cumbersome in Fitnesse we are currently working to add a Selenium backend to our own Framework (which previously only had backends for Java and Smalltalk).
So, by requiring that elements with certain ID's are present in the DOM we will of course break our tests if someone removes the elements from the page; however, we found that this behavious is very useful as this enforces the contract the tests made with the implementation and it is a good thing we find missing elements as soon as someone broke the implementation.
In addition, it is good practice to keep UI automation skin-deep: Only test what is present on the page with Selenium and test the business-logic by calling the underlying functions directly.
Is there an easy way to invoke a GWT RPC service endpoint directly from Java code? I mean real Java code, not Java code compiled down into javascript.
I ask because we want to run performance benchmarks/stress tests against a GWT RPC interface. I would like to write the test harness in Java and run it in a JVM (as opposed to javascript running in a browser).
I figure there must be a way to do this because I assume GWT Hosted mode requires such functionality. However, I can't really find any code in the GWT runtime that demonstrates how to cleanly do this. I've looked at the com.google.gwt.user.client.rpc package but the stuff in there seems to use JSNI which obviously wouldn't be usable by pure Java.
GWT SyncProxy allows you to access GWT RPC services (e.g methods) from pure Java (not JSNI) code. Thus you can use it to test your RPC interface.
See http://www.gdevelop.com/w/blog/2010/01/10/testing-gwt-rpc-services/ for details.
Are you trying to benchmark the business logic of the service, or how well GWT-RPC itself performs? If you are mostly worried about how well your backend code performs, you could just instantiate the class that implements your service directly:
MyServiceImpl impl = new MyServiceImpl();
impl.doSomething();
If you want to test a greater slice of the stack, including the RPC calls, take a look here. There is a section called "running your test in web mode" that has the following line: 'By default, tests run in hosted mode are run as normal Java bytecode in a JVM'. So if you use the described setup, I think you get your tests to run in java by default. Also on that page is info about GWT's built in profiling tools.
You could use a conventional load testing tool like Grinder to replay post requests to your service. That isn't quite what you are asking but it may be a better way to perform load testing on your application. Grinder can simulate many simultaneous users and so on.