Just a question to anyone here who is using ExtentReport as a listener.
Is it possible to use ExtentReport to generate the HTML Report during the tests being executed when its being used as a listener?
For example, instead of the report being generated when the tests are finished, the report is generated after the first test, and so on? I want to use ExtentReport to monitor the progress of my tests aswell and showing the results?
I've learnt its possible when using as a logger, as you can do a flush after each tests. However is it possible while using Extent as a Listener?
Thanks in advanced.
Kind regards,
Colin.
Yes it is possible. Since extent report is open source. One could edit the function which formulate the structure of our final report. But since extent report uses different functionalities, just modifying an endtest would cause a cascading effect on everything.
Or another way is to save the data before flush is called and by calling flussh everytime. One could replace the existing file over and over.
My best bet is too leave it as it is and get other listener jars to do the work for you.
Related
I am making an framework that internally user JUnit and REST Assured. This framework will have the 4 #Test methods for CRUD operations. Whenever the user want to do any operation, he will call only that particular Test method. But at the end of the each operation(say GET or DELETE or any other), it should generate the report.
I tried using surefire-report plugin. As I have read, that will generate report only when we build the project(running all the Test methods).
Is there any mechanism that fulfills my requirement of generation report for individual run also?
Execution will be like : final output will be the jar with individual CRUD facility.API.execute(GET, end_point_name);API.execute(POST, end_point_name,data);Test method get and post is called respectively for the above calls. Report should be generated for both the test cases for normal run as java application.
There are 3 solutions to your problem :
Either you write your logger statement and do proper logging of the events. You can either store it in DEBUG, INFO etc mode for better understanding and more control.
ExtentReports is another way to go :
http://www.ontestautomation.com/creating-html-reports-for-your-selenium-tests-using-extentreports/ refer the above link where they have a provided a detailed way of using the same.
You can also create a separate testng.xml file. Like maintaining a separate suite file this will internally make sure with the help surefire to create a separate reports.
I successfully use Cucumber to process my Java-based tests.
Sometimes these tests encounter regression issues, and it takes time to fix found issues (depending on issue priority, it can be weeks or even months). So, I'm looking for a way to mark some cucumber tests as known issues. Don't want these tests fail the entire set of tests, just want to mark them, for example, as pending with yellow color in report instead.
I know that I can specify a #tag for failed tests and exclude them from execution list, but that's not what I want to do as I still need these tests to be run continuously. Once the issue fixed, appropriate test should be green again without any additional tag manipulation.
Some other frameworks provide such functionality (run the test but ignore its result in case of fails). Is it possible to do the same trick somehow using Cucumber?
The final solution I use now - to mark known issues with specific tag, exclude these tests from regular round and to run them separately. But that's not the best solution I believe.
Any ideas appreciated. Thanks in advance.
I would consider throwing a pending exception in the step that causes the known failure. This would allow the step to be executed and not be forgotten.
I would also consider rewriting the failing steps in such a way that when the failure occurs, it is caught and a pending exception is thrown instead of the actual failure. This would mean that when the issue is fixed and the reason for throwing the pending exception is gone, you have a passing suite.
Another thing I would work hard for is not to allow a problem to be old. Problems are like kids, when they grow up the they get harder and harder to fix. Fixing a problem while it is young, perhaps a few minutes, is usually easy. Fixing problems that are months old are harder.
You shouldn't.
My opinion is that if you have tests that fail then you should add a bug/task ticket for these scenarios and to add them in a build status page with the related tag.
Another thing you could do is to add the ticket number as a tag and removed after the ticked is fixed.
If you have scenarios that fail due a bug, then the report should show that, if the scenario is not fully implemented then is better not to run it at all.
One thing you could do is to add a specific tag/name for those scenarios and to try in a before scenario method to get the tags and check for added specific tag/name and throw a pending exception.
What i suggest is to keep those scenarios running if there is a bug and to document that in the status page.
I think the client would understand better if those scenarios are red because they are failing rather than some yellow color with "grey" status of what is happening.
If you need the status of the run to trigger some CI job then maybe is better to change the condition there.
As i see it, the thing you need to give it a thought should be: what is the difference between yellow or red, pending or fail for you or the client?, you would wand to keep a clear difference and to keep track of the real status.
You should address these issues in an email, discuss them with the project team and with the QA team, and after a final decision is taken you should get a feedback from the customer also.
All,
I am using JUnit/Selenium (Java). I have over 400 test cases (separate java class files) distributed in different packages. I need to generate a basic test run report which would tell me if the test failed or passed and how much time it took.
TestNG is not an option since i am using TestWatcher along to make calls to a bug tracking tool API.
Any suggestions?
Thanks.
You may to use the RunListener class that listens to the test events. Then you also may to prepare the custom listener and create the report file. Unfortunately you probably will need add such listener to each your package.
The following link provides more details.
I have a web tool which when queried returns generated Java classes based upon the arguments in the URL.
The classes we retrieve from the webserver change daily and we need to ensure that they still can process known inputs.
Note these classes do not test the webserver, they run locally and transform xml into a custom format. I am not testing the webserver.
These classes must then be placed in specific package structure compiled and run against a known set of input data and compared against known output data.
I would like to do this automatically each night to make sure that the generated classes are correct.
What is the best way to achieve this?
Specifically whats the best way to:
retrieve the code from a webserver and place it in a file
compile the code and then call it
I'm sure a mix of junit and ant will be able to achieve this but is there and standard solution / approach for this?
First up, to answer your question: No, I do not think that there is a standard approach for this. This sounds like quite an unusual situation ;-)
Given that, what I would do is to write your JUnit tests to all call a class GeneratedCode, and then, once you download the code, rename the class to GeneratedCode, compile, and run your unit tests.
You have the same goal as continuous integration ;-)
Maybe a bit overkill for this simple task, but this is the standard way to get something, compile something and test something regularly.
E.g. you could try hudson.
You should be creating a "mock" interface for your web service that (a) behaves the same way and (b) returns a known answer.
You should then do some other integration testing with the live web service where a person looks at the results and decides if they worked.
Can you only test the generated classes after they were published on the webservice ? You have no way to test during or just after the generation ?
One idea, if the generated code isn't to complex, is to load it via the GroovyClassLoader and to run your tests against it. See this page for examples.
I'm learning JUnit. Since my app includes graphical output, I want the ability to eyeball the output and manually pass or fail the test based on what I see. It should wait for me for a while, then fail if it times out.
Is there a way to do this within JUnit (or its extensions), or should I just throw up a dialog box and assertTrue on the output? It seems like it might be a common problem with an existing solution.
Edit: If I shouldn't be using JUnit for this, what should I be using? I want to manually verify the build every so often, and unit test automatically, and it'd be great if the two testing frameworks got along.
Manually accepting/rejecting a test defeats the purpose of using an automated test framework. JUnit is not made for this kind of stuff. Unless you find a way to create and inject a mockup of the object representing your output device, you should consider alternatives (don't really know any there sorry).
I once wrote automated tests for a video decoding component. I dumped the decoded data to a file using some other decoder as a reference, and then compared the output of my decoder to that using the PSNR of each pair of images. This is not 100% self contained (needs external files as resources), but was automated at least, and worked fine for me.
Although you could probably code that, that is not what JUnit is about. It is about automated tests, not guided manual tests. Generally that "does it look right" test is regarded as an integration test, as it is something that is very hard to automate correctly in a way that doesn't break for trivial changes all the time.
Take a look at Abbot to give you a more robust way to test your GUI.
Unit tests shouldn't require human intervention. If you need a user to take an action then I think you're doing it wrong.
If you need a human to verify things, then don't do this as part of your unit tests. Just make it a required step for your test department to carry out when QA'ing builds. (this still works of your QA department is just you.)
I recommend using your unit tests for the Models if using MVC, or any utility method (i.e. with Swing it's common to have color mapping methods). If you have a good set of unit tests on things like model behavior, if you have a UI bug it'll help narrow your search.
Visual based unit tests are very difficult, at a company I worked at they had tried these visual tests but slight differences in video cards could produce failed tests. In the end this is where a good Q/A team is required.
Take a look at FEST-Swing. It provides an easy way to automatically test your GUIs.
The other thing you'll want to do is separate your the code which does the bulk of the work from your gui code as much as possible. You can then write unit tests on this work code without having to deal with the user interface. You'll also find that you'll run these tests much more frequently, as they can run quickly.