I have a gradle project using Kotlin and JUnit5 if it helps.
Currently I have a bunch of parametrized tests to run using #MethodSource. A tests with all parameters runs for a quite long time, and during local development itn's enough to run lesser with just one value.
Is it possible to define a switch or a configuration to run all tests, but with limited numbers of parameters?
Solutions tested, which didn't work:
Edit source method(s) everytime I need to run locally -> this leads to mistakes like "I forgot to uncomment tests".
Limit tests to a single test with all parameters -> This won't work as I need all tests running on every, may be just with a different parameter.
Make an abstract superclass, extend two with just different annotations -> I have enough tests to skip this solution, I prefer another way if possible.
check System.properties & System.env -> Something didn't work as I'd like to do.
Related
Preface
I'm deliberatly talking about system tests. We do have a rather exhaustive suite of unit tests, some of which use mocking, and those aren't going anywhere. The system tests are supposed to complement the unit tests and as such mocking is not an option.
The Problem
I have a rather complex system that only communicates via REST and websocket events.
My team has a rather large collection of (historically grown) system tests based JUnit.
I'm currently migrating this codebase to JUnit5.
The tests usually consist of an #BeforeAll in which the system is started in a configuration specific to the test-class, which takes around a minute. Then there is a number of independent tests on this system.
The problem we routinely run into is that booting the system takes a considerable amount of time and may even fail. One could say that the booting itself can be considered a test-case. JUnit handles lifecycle methods kind of weirdly - the time they take isn't shown in the report; if they fail it messes with the count of tests; it's not descriptive; etc.
I'm currently looking for a workaround, but what my team has done over the last few years is kind of orthogonal to the core idea of JUnit (cause it's a unit testing framework).
Those problems would go away if I replaced the #BeforeAllwith a test-method (let's call it #Test public void boot(){...}) and introduce an order-dependency (which is pretty easy using JUnit 5) that enforces boot to run before any other test is run.
So far so good! This looks and works great. The actual problem starts when the tests aren't executed by the CI server but by developers who try to troubleshoot. When I try to start a single test boot is filtered from the test execution and the test fails.
Is there any solution to this in JUnit5? Or is there a completely different approach I should take?
I suspect there may be a solution in using #TestTemplate but I'm really not sure how to procede. Also afaik that would only allow me to generate new named tests that would be filtered as well. Do I have to write a custom test-engine? That doesn't seem compelling.
This more general testing problem then related to Junit5. In order to skip very long boot up you can mock some components if it is possible. Having the booting system as a test does not make sense because there are other tests depending on that. Better to use #beforeAll in this case as it was before. For testing boot up, you can make separate test class for that which will run completely independent from other tests.
Another option is to group this kind of test and separate from the plain unit test and run it only if needed (for example before deployment on CI server). This really depends on specific use case and should those test be part of regular build on your local machine.
The third option is to try to reduce boot time if it possible. This is option if you can't use mocks/stubs or exclude those tests from regular build.
I've got a bunch of Selenium tests in my project and I'd love running them with IDEA. I need to pass certain VM arguments (where my firefox binary is located etc.) and I don't want to create a run config for every Test class that I have.
There are also too many tests to just run all every time.
So, does anyone know if it's possible to create a "parent" run config which would be used for all tests in a certain path whether I run them together or just a single one?
Now I feel silly :P
Run Configurations has a Defaults tab where you can set default values for JUnit tasks
I am using Maven and TestNG.
How to distinguish at runtime when a particular method is being called by a TestNG/JUnit test-case or by the main java code
Several comments are alluding to this, however it's generally extremely poor practice to build in statements that work one way under test, and another way when the app is running standalone. This increases the probability that the app will pass tests, but fail in production.
Instead, you should look at why you're wanting to make this distinction. In general, it will be for the sake of some dependent object, or due to input of one variety or another. In these cases, it's better to engineer the class to accept dependent objects to be inserted into it via configuration and under test, the only thing that changes is the configuration. The class under test should not distinguish the dependant classes from one another. Instead, just work with their interfaces, so you can create mock classes for testing.
When accepting input, redirect the input source to take scripted input.
For databases, redirect to an in-memory DB which is configured for the test, etc.
You will find this approach will VASTLY improve the quality of the code you write, and decrease the probability of bugs sneaking past your unit tests.
At runtime, your code is never running unit tests. Unless you invoke them explicitly from your code, which you should never do.
Unit tests are only run manually, or during the test phase of the maven lifecycle.
I have a JUnit 4 test suite with BeforeClass and AfterClass methods that make a setup/teardown for the following test classes.
What I need is to run the test classes also by them selves, but for that I need a setup/teardown scenario (BeforeClass and AfterClass or something like that) for each test class. The thing is that when I run the suite I do not want to execute the setup/teardown before and after each test class, I only want to execute the setup/teardown from the test suite (once).
Is it possible ? Thanks in advance.
I don't know of any standard way to do this with JUnit. The reason for it, as you probably already know, is that your test cases should run independently of each other. This concerns the "normal" setup/teardown methods which run before and after each test method. Class setup and teardown is a bit different though - although I would still prefer running my tests independently and staying out of the trouble zone.
However, if you really are convinced of what you are doing, you could use a global flag to signal whether or not the class setup/teardown is to run, and to check for its state in the class setup/teardown methods. In your test suite, you could include a special class as the very first one, which does nothing more than execute the setup and set the global flag to indicate to the real test cases that their class setup/teardown methods must not be run. Similarly, a special last class in the suite can execute the teardown code. The caveat is that I am afraid JUnit does not guarantee the order of execution of test classes inside a suite, although most probably it does execute them in the specified order - but this is just an implementation detail. Try it out, it may work for you - but there is no guarantee it will always do what you expect.
If you have jUnit 4.7+ I recommend looking into the new feature called Rules (which are explained in this blog post). They might not be exactly what you want, but they are probably the best you get with jUnit.
Supposedly TestNG has better test grouping possibilities, but I haven't really looked into it myself yet.
No, there's no standard way to do this in JUnit, though you could hack something up as Péter Török suggested.
Note however that you are more or less abusing JUnit in doing this. The whole point of unit tests it that they are independent of each other. This is because dependencies between tests create a total maintenance nightmare (tests failing because the run in the wrong order).
So I'd advise you to strongly consider if it's not better to just always run the setup...
I want to run my unit tests automatically when I save my Eclipse project. The project is built automatically whenever I save a file, so I think this should be possible in some way.
How do I do it? Is the only option really to get an ant script and change the project build to use the ant script with targets build and compile?
Update I will try 2 different approaches now:
Running an additional builder for my project that executes the ant target test (I have an ant script anyway)
ct-eclipse, recommended by Thorbjørn
For sure it it unwise to run all tests, because we can have for example 20.000 tests whereas our change could affect only, let's say 50 of them, among which are tests for the class we have changed and tests for classes that collaborate with our class.
There is an unseful plugin called infinitetest http://improvingworks.com/products/infinitest/ which runs only some tests ( related to class we've changed ) just after we save changes. It also integrate quite nicely with editor ( using annotations ) and problem view - displaying not-passing tests like errors.
Right click on your project > Properties > Builders > New, and there add your ant ant builder.
But, in my opinion, it is unwise to run the unit tests on each save.
See if Eclipse has a plugin for Infinitest.
I'd also consider TestNG as an alternative to JUnit. It has a lot of features that might be helpful in partitioning your unit test classes into shorter and longer running groups.
I believe you are looking for http://ct-eclipse.tigris.org/
I've experimented with the concept earlier, and my personal conclusion was that in order for this to be useful you need a lot of tests which take time. Personally I save very frequently so this would happen frequently, and I didn't find it to be an advantage. It might be different for you.
Instead we bit the bullet and set up a "build server" which watches our CVS repository and builds projects as they change. If the compilation fails or the tests fail we are notified quickly so we can remedy it.
It is as always a matter of taste what works for you. This is what I've found.
I would recommend Inifinitest for the described situation. Infinitest is nowadays a GPL v3 licensed product. Eclipse update site: http://infinitest.github.com
Then you must use INFINITEST. INFINITEST helps you to do Continuous Testing.
Whenever you make a change, Infinitest runs tests for you.
It selects tests intelligently, and only runs the ones you need. It reports unit test failures like compiler errors, and provides additional information that helps you write better tests.