I successfully use Cucumber to process my Java-based tests.
Sometimes these tests encounter regression issues, and it takes time to fix found issues (depending on issue priority, it can be weeks or even months). So, I'm looking for a way to mark some cucumber tests as known issues. Don't want these tests fail the entire set of tests, just want to mark them, for example, as pending with yellow color in report instead.
I know that I can specify a #tag for failed tests and exclude them from execution list, but that's not what I want to do as I still need these tests to be run continuously. Once the issue fixed, appropriate test should be green again without any additional tag manipulation.
Some other frameworks provide such functionality (run the test but ignore its result in case of fails). Is it possible to do the same trick somehow using Cucumber?
The final solution I use now - to mark known issues with specific tag, exclude these tests from regular round and to run them separately. But that's not the best solution I believe.
Any ideas appreciated. Thanks in advance.
I would consider throwing a pending exception in the step that causes the known failure. This would allow the step to be executed and not be forgotten.
I would also consider rewriting the failing steps in such a way that when the failure occurs, it is caught and a pending exception is thrown instead of the actual failure. This would mean that when the issue is fixed and the reason for throwing the pending exception is gone, you have a passing suite.
Another thing I would work hard for is not to allow a problem to be old. Problems are like kids, when they grow up the they get harder and harder to fix. Fixing a problem while it is young, perhaps a few minutes, is usually easy. Fixing problems that are months old are harder.
You shouldn't.
My opinion is that if you have tests that fail then you should add a bug/task ticket for these scenarios and to add them in a build status page with the related tag.
Another thing you could do is to add the ticket number as a tag and removed after the ticked is fixed.
If you have scenarios that fail due a bug, then the report should show that, if the scenario is not fully implemented then is better not to run it at all.
One thing you could do is to add a specific tag/name for those scenarios and to try in a before scenario method to get the tags and check for added specific tag/name and throw a pending exception.
What i suggest is to keep those scenarios running if there is a bug and to document that in the status page.
I think the client would understand better if those scenarios are red because they are failing rather than some yellow color with "grey" status of what is happening.
If you need the status of the run to trigger some CI job then maybe is better to change the condition there.
As i see it, the thing you need to give it a thought should be: what is the difference between yellow or red, pending or fail for you or the client?, you would wand to keep a clear difference and to keep track of the real status.
You should address these issues in an email, discuss them with the project team and with the QA team, and after a final decision is taken you should get a feedback from the customer also.
Related
When I use the git bisect command, I run only the failing tests at
each bisection point for Java programs. However, I see that many tutorials related to git bisect propose running "make; make test". Is there any reason why I should run all the tests at each step?
Thanks a lot in advance.
I would have to say that the conditions mentioned by #bcmcfc are necessary but not sufficient. For reference, his conditions are
all tests pass at the commit marked as good
some tests fail at the commit marked as bad
My problem is not knowing what has happened in between the good commit and the bad. For example, was there another bug discovered and fixed in the intervening commits? It's conceivable that that bug or its fix influenced this bug.
Another issue is the possible presence of "dirty" commits in the history. I don't know your usage patterns, but some people allow commits with failing tests to be present on feature branches. bisect can land on those commits, and if you only run the tests that you expect to fail you may not fully understand what's happening in that commit, and that may lead you astray in fixing the bug. It may even be that the bug was introduced and then fixed in that feature branch, then introduced again later on another feature branch in a slightly different way, which will really confuse your efforts to fix it.
This seems to me to be an example of the old adage, "In theory there's no difference between theory and practice, but in practice there is." I would run every test every time. And if they all pass where you expect, then you shouldn't feel like you've wasted your effort, you should glow with confidence knowing that you know what's going on.
If:
all tests pass at the commit marked as good
some tests fail at the commit marked as bad
Then yes, it's safe to only run the failing tests to speed up the bisection process. You can infer from the test results at the good and bad commits that the rest of the tests should pass.
You would probably re-run the full test suite after fixing the bug in question in any case, which covers you for the case where your bugfix introduces a regression.
My question is from the perspective of language design.
Why is assert treated differently i.e. it raises a error and not an exception, it is not enabled by default etc..
It does seem elegant(very subjective opinion), easy to read(again subjective) for doing validations & also there are tools(IDE) which can live-evaluate it and provide warnings based on assertions.
I'd say that the reason is that defaults for Java are meant for production code (the "release" version of software) - if users need to build your code they will use provided defaults and if you are developer and want to have better reporting you can always make some additional effort.
Usually you don't want to ship assertions with a release version. Why? You can always design your code to perform some not disturbing background error handling and throwing AssertionError in users face is not always the way to go.
Most of the time I see them used as additional code testing - when you run regression tests and code coverage is high no assertion error suggest that there are no (obvious to spot) errors in your code. If some happens, you can deduce from stack trace what went wrong and why. On the other hand clients shouldn't be bothered with seeing descriptive error information.
So how should you actually use them? In my experience you should design code to not use assertions to perform error handling. If you want exception to be thrown somewhere throw it explicitly yourself. Once code can handle itself, you can add assertions to check pre- and postconditions as well as invariants - so basically used them to check algorithm correctness instead of data correctness. It has value for developers rather than users. Once you have enough confidence in your solution, you can disable assertions, your program still works fine and your users don't have to run program with additional runtime overhead.
Asserts are tools for the developer.
The core reason it's not enabled by default is that assertions via assert are not meant to provide run-time validation/protection for production code.
assert is a tool for use in development process and during testing that should not impact performance during actual running in production environment.
Imagine a very heavy weight assertions that are critical to check when building out a new feature or a change against a entire range of allowable inputs, but once built and properly tested, don't need to run until code changes again.
It raises an error, because the severity of an assertion violation is high enough to do so.
An example for an Exception is something like ArrayIndexOutOfBounds. This might be reasonable in some cases and you might even expect (and handle) such a case.
But an assertion violation is (like out of memory e.g.) nothing you would ever expect or you would like to deal with. It's an error and there is no excuse.
Assertions are not enabled by default, because they should be always fullfilled. You enable them to test for that, but then you "know" (as far as you can know) that they are not violated. So you don't need to check the conditions (which might be performance intensive) every time in production code.
The good thing about assertions in Java is, that the code actually performing the checks is never executed, when assertions are not enabled.
For example:
if (!doComplexChecks()) throw new AssertionError("Damn!");
might take a lot of time and you want to verify this when unit testing or debugging, but in production you just don't want that.
But this code
assert doComplexChecks();
is only executed, when assertions are enabled, so it saves you a lot of time in production code.
As said in the documentation we can continue build if one of the tasks has failed. But I can't get the point of that feature... Why do we need to execute other task if one of the tasks has been failed? Is it safe at all? Couldn't you provide an example?
Yep it makes sense, for example generating classes from wsdl, in case of service not available.
Then you should provide some logic in your application for not working this service.
The second sentence in your linked doc says:
This allows the build to complete sooner, but hides other failures that would have occurred. In order to discover as many failures as possible in a single build execution, you can use the --continue option.
So instead of failing on the first error just go and fail on all. Imagine a webform that only tells you each error at a time after submitting and it takes you ages to fill it out versus a form that shows you all you current of your current errors at once.
Examples are obviously include developing the original gradle file and testing it with your build. Or running on an integration service, where you would rather have "all" the errors at once instead of hitting the build button all the day.
If a task fails, any subsequent tasks that were depending on it will not be executed, as it is not safe to do so.
So you will most likely not end up with the result you are expecting. But could! That is on you to decide, as it depends on your build and what you are doing. So, 'it safe? Heck no. But sometimes we all have to do unsafe things...
If you want to get rid of something failing, that is not vital to the actual build result (e.g. the jar file you are after) but is part of the build process (e.g. a codenarc task as part of the tests) and you would rather fix a critical bug with ugly code, you might be better off to just exclude that task (gradle jar -x codenarc) instead of using this feature. Is this safe? Heck no... you get the picture!
This nearly identical to this question with two differences.
I want to run unit tests right before commit and not all of the time (accepted answer).
The answers and comments to the question seem to indicate that the suggested plugins are no longer supported.
Basic problem I am trying to solve is simply forgetting to run unit tests for small quick changes and it seems it should be possible to automate.
On the question you linked to, I mentioned Infinitest. It doesn't fit your item #1, but it only reruns tests that are likely to have broken (presumably it does some kind of clever code analysis -- I don't know the details) so you might find it useful anyway. It is definitely still supported, and in fact it's now open source!
Basic problem I am trying to solve is
simply forgetting to run unit tests
for small quick changes and it seems
it should be possible to automate.
If your tests are fast, you can run them on each save. See this blog by Misko Hevery
Alternatively, you could use a commit hook to run the tests before accepting the update.
I'm currently building a CI build script for a legacy application. There are sporadic JUnit tests available and I will be integrating a JUnit execution of all tests into the CI build. However, I'm wondering what to do with the 100'ish failures I'm encountering in the non-maintained JUnit tests. Do I:
1) Comment them out as they appear to have reasonable, if unmaintained, business logic in them in the hopes that someone eventually uncomments them and fixes them
2) Delete them as its unlikely that anyone will fix them and the commented out code will only be ignored or be clutter for evermore
3) Track down those who have left this mess in my hands and whack them over the heads with the printouts of the code (which due to long-method smell will be sufficently suited to the task) while preaching the benefits of a well maintained and unit tested code base
If you use Junit 4 you can annotate that tests with #Ignore annotation.
If you use JUnit 3 you can just rename tests so they don't start with test.
Also, try to fix tests for functionality you are modifying in order to not make code mess larger.
Follow the no broken window principle and take some action towards a solution of the problem. If you can't fix the tests, at least:
Ignore them from the unit tests (there are different ways to do this).
Enter as many issue as necessary and assign people to fix the tests.
Then to prevent such situation from happening in the future, install a plug in similar to Hudson Game Plugin. People gets assigned points during continuous integration, e.g.
-10 break the build <-- the worse
-1 break a test
+1 fix a test
etc.
Really cool tool to create a sense of responsibility about unit tests within a team.
The failing JUnit tests indicate that either
The source code under test has been worked on without the tests being maintained. In this case option 3 is definitely worth considering, or
You have a genuine failure.
Either way you need to fix/review the tests/source. Since it sounds like your job is to create the CI system and not to fix the tests, in your position i would leave a time-bomb in the tests. You can get very fancy with annotated methods with JUnit 4 (something like #IgnoreUntil(date="2010/09/16")) and a custom runner, so or you can simply add an an if statement to the first line of each test:
if (isBeforeTimeBomb()) {
return;
}
Where isBeforeTimeBomb() can simply check the current date against a future date of your choosing. Then you follow the advice given by others here and notify your development team that the build is green now, but is likely to explode in X days unless the timebombed tests are fixed.
Comment them out so that they can be fixed later.
Generate test coverage reports (with Cobertura for example). The methods that were supposed to be covered by the tests that you commented out will then be indicated as not covered by tests.
If they compile but fail: leave them in. That will get you a good history of test improvements over time when using CI. If the tests do not compile but break the build, comment them out and poke the developers to fix them.
This obviously does not preclude using option 3 (hitting them over the head), you should do that anyway, regardless of what you do with the tests.
You should definitely disable them in some way for now. Whether that's done by commenting, deleting (assuming you can get them back from source control) or some other means is up to you. You do not want these failing tests to be an obstacle for people trying to submit new changes.
If there are few enough that you feel you can fix them yourself, great -- do it. If there are too many of them, then I'd be inclined to use a "crowdsourcing" approach. File a bug for each failing test. Try to assign these bugs to the actual owners/authors of the tests/tested code if possible, but if that's too hard to determine then randomly selecting is fine as long as you tell people to reassign the bugs that were mis-assigned to them. Then encourage people to fix these bugs either by giving them a deadline or by periodically notifying everyone of the progress and encouraging them to fix all of the bugs.
A CI system that is steady red is pretty worthless. The main benefit is to maintain a quality bar, and that's made much more difficult if there's no transition to mark a quality drop.
So the immediate effort should be to disable the failing tests, and create a tracking ticket/work item for each. Each of those is resolved however you do triage - if nobody cares about the test, get rid of it. If the failure represents a problem that needs to be addressed before ship, then leave the test disabled.
Once you are in this state, you can now rely on the CI system to tell you that urgent action is required - roll back the last change, or immediately put a team on fixing the problem, or whatever.
I don't know your position in the company, but if it's possible leave them in and file the problems as errors in your ticket system. Leave it up to the developers to either fix them or remove the tests.
If that doesn't work remove them (you have version control, right?) and close the ticket with a comment like 'removed failing junit tests which apparently won't be fixed' or something a bit more polite.
The point is, junit tests are application code and as such should work. That's what developers get paid for. If a test isn't appropriate anymore (something that doesn't exist anymore got tested) developers should signal that and remove the test.