I have a maven project with multiple modules and submodules. Each of these modules and submodule runs several unit tests of their own. As we know, maven-surefire-plugin runs unit tests for each module separately in a new JVM.
Now, as it would often occur, I have some global configuration/setup I need to perform before running tests. This configuration/setup applies to almost all the unit tests across the modules (that means across multiple JVMs). By default, this setup now runs every time, that is, for each (sub)module. If this configuration is a little expensive, it adds a lot to the exceution time.
One such example may be setting up db (clean the db and set it up). If I have 100 modules, this setup would run 100 times.
Is their any good/elegant way of splitting such configuration so it only runs once for the whole project? Also, I do not want to send in any properties through the maven command with -D, and this is make this process completely command-independent.
Related
We have a multi-module Maven project that takes about 2 hours to build and we would like to speed that up by making use of concurrency.
We are aware of the -T option which (as explained i.e. here) allows using multiple threads within the same JVM for the build.
Sadly, there is a lot of legacy code (which uses a lot of global states) in the project which makes executing multiple test in parallel in a single JVM very hard. Removing all of these blockers from the project would be a lot of work which we would like to avoid.
The surefire and failsafe plugins have multiple options regarding parallel execution behavior, however, as I understand it, this would only parallelize the test executions. Also, spawning a separate JVM for each test (class) seems kind of overkill to me. That would probably just as soon cause the build to take even longer than it does now.
Ideally, we would like to do the parallelization on the Maven reactor level and have it build each module in its own (single threaded) JVM with up to x JVMs running in parallel.
So my question is: is there a way to make maven create a separate JVM for each module build?
Alternatively, can we parallelize the build while making sure that tests (over all modules) are executed sequentially?
I am not completely sure this works but I guess if you use Maven Toolchains, then each module will start its own forked JVM for the tests, not reusing already running ones.
I guess it is worth a try.
I have a project consists with multi modules - I use Maven. Each module has an integration tests. Within module tests shared the same container - currently I use TestContainers. But there is a problem with long launching container with mysql database. Sometimes it takes too long, and whole build project with tests lasts even several dozen minutes. There is possibility to launch one container, shared between submodule? After each execution by submodule, I will erase database and configure for new submodule.
I see there is something like org.junit.platform.launcher from JUnit, but I don't see so much documentation and don't know if it help me in some way.
I have a maven project with test execution by the maven-surefire-plugin. An odd phenomenon I've observed and been dealing with is that running locally
mvn clean install
which executes my tests, results in a successful build with 0 Failures and 0 Errors.
Now when I deploy this application to our remote repo that Jenkins attempts to build, I get all sorts of random EasyMock errors, typically of the sort:
java.lang.IllegalStateException: 3 matchers expected, 4 recorded. at org.easymock.internal.ExpectedInvocation.createMissingMatchers
This is a legacy application being inherited, and we are aware that many of these tests are flawed if not plainly using EasyMock incorrectly, but I'm in a state where with test execution I get a successful build locally but not in Jenkins.
I know that the order of execution of these tests is not guaranteed, but I am wondering how I can introspect what is different in the Jenkins build pipeline vs. local to help identify the issue?
Is there anything I can do to force execute the tests in the way they're done locally? At this point, I have simply excluded many troublesome test classes but it seems that no matter how many times I see a Jenkins failure, I either fix the problem or exclude the test class, I'm only to find it complain about some other test class it didn't mention before.
Any ideas how to approach a situation like this?
I have experimented quite a similar situation, and the cause of mine was obviously some concurrency problems with the tests implementations.
And, after reading your comment:
What I actually did that fixed it (like magic am I right?) is for the maven-surefire plugin, I set the property reuseForks=false, and forkCount=1C, which is just 1*(number of CPU's of machine).
... I get more convinced that you have concurrency problems with your tests. Concurrency is not easy to diagnose, specially when your experiment runs OK on one CPU. But race conditions might arise when you run it on another system (which usually is faster or slower).
I recommend you strongly to review your tests one by one and ensure that each one of them is logically isolated:
They should not rely upon an expected previous state (files, database, etc). Instead, they should prepare the proper setup before each execution.
If they modify concurrently a common resource which might interfere other test's execution (files, database, singletons, etc), every assert must be done synchronizing as much as needed, and taking in account that its initial state is unknown:
Wrong test:
MySingleton.getInstance().put(myObject);
assertEquals(1, MySingleton.getInstance().size());
Right test:
synchronized(MySingleton.getInstance())
{
MySingleton.getInstance().put(myObject);
assertTrue(MySingleton.getInstance().contains(myObject));
}
A good start point for the reviewing is checking one of the failing tests and track the execution backwards to find the root cause of the fail.
Setting explicitly the tests' order is not a good practice, and I wouldn't recommend it to you even if I knew it was possible, because it only would hide the actual cause of the problem. Think that, in a real production environment, the executions' order is not usually guranteed.
JUnit test run order is non-deterministic.
Are the versions of Java and Maven the same on the 2 machines? If yes, make sure you're using the most recent maven-surefire-plugin version. Also, make sure to use a Freestyle Jenkins job with a Maven build step instead of the Maven project type. Using the proper Jenkins build type can either fix build problems outright or give you a better error so you can diagnose the actual issue.
You can turn on Maven debug logging to see the order tests are being run in. Each test should set up (and perhaps tear down) its own test data to make sure the tests may run independently. Perhaps seeing the test order will give you some clues as to which classes depend on others inappropriately. And - if the app uses caching, ensure the cache is cleaned out between tests (or explicitly populated depending on what the test needs to do). Also consider running the tests one package at a time to isolate the culprits - multiple surefile plugin executions might be useful.
Also check the app for classpath problems. This answer has some suggestions for cleaning the classpath.
And another possibility: Switching to a later version of JUnit might help - unless the app is using Spring 2.5.6.x. If the app is using Spring 2.5.6.x and cannot upgrade, the highest possible version of JUnit 4.x that may be used is 4.4. Later versions of JUnit are not compatible with Spring Test 2.5.6 and may lead to hard-to-diagnose test errors.
We have a large Java 8 Spring Hibernate Maven project that is getting larger.
Problems :
Build time is 10-12 minutes at best; 3 minutes without tests
We already have a command-line switch to skip rarely modified modules, which is the symptom of the build process reaching practical limits
Eclipse is struggling to manage the project (although IntelliJ is ok for now)
Things are getting worse as the project grows, and as more scenarios from the test team get incorported as integration tests in the code base.
How we work now
The project is configured in about 20 Maven modules, like so:
Parent
|--- Tier1
|--- Tier2
|--- WebTier
|---- ModuleA
|---- ModuleB
|---- ModuleC
|---- ...
|---- Entities
|---- Shared
|---- Batch
|---- IntegrationTests
The application is built as a single WAR
Developers deploy a single tier (typically WebTier) as an artefact from Eclipse or IntelliJ to their local Tomcat
Although the project seems nicely split in modules, there are many undesired coupling points between them. Specially in Shared, where modules needing "cross-modules" access put their services
All integration tests are in a dedicated module (no idea why)
Ideas to make it better
Add a MessageBroker module to allow loose coupling where relevant. Maybe JMS, or simply a dumb in-memory component for synchronous communication
Get rid of the Shared module
Make sure modules have coarse-grained entry-points
Remove undesired coupling between siblings and prefer the message broker when possible
Might keep Entities. At least the core-business entities (Customer, CustomerFile, ...). But some entities obviously belong to a single module (a batch execution info would be in the Batch module)
That way, anyone making a change to ModuleA would most of the time only build and run tests in that module without fearing to break the application.
Questions
Does that seem like a good plan? By good, I mean future-proof, with good chances to improve things, and not requiring an excessive amount of work given the situation
Should we have 1 Eclipse/IJ project per tier, let the IDE build the artefact and deploy it to Tomcat, or should we have 1 project per module, and dependencies toward Nexus? Or maybe the latter option is overkill?
Any other suggestions?
Some metrics
Windows 7, Java 8, Maven 3.0.3, TestNG.
SSD or 7200rpm HDD (limited impact)
6Gb RAM
Heap 1Gb (maven)
CI with Jenkins
Thanks a bunch!
CI would be real answer but it looks that your project is not modular as it should be. You don't build entire project from scratch every time. You build jars, test them in different projects and then use as single items. Each project should be small enough and cover just one area. Do you think that Java builds let's say security jars when they work on io package? Divide and conquer - that's the whole idea of OOP and encapsulation.
This might not be as bad as you think. Involved projects with unit tests and static analysis take a while.
A company I worked for had >1hr build + unitTest + CodeCoverage integration times. That doesn't count the time needed to ship the artifact off to vSphere for automated clickthrough testing of the installer on 26 languages x 8 supported OSes.
I have a set of legacy unit tests, most of which are Spring AbstractTransactionalJUnit4SpringContextTests tests, but some manage transactions on their own. Unfortunately, this seems to have introduced side-effects causing completely unrelated tests to fail when modifying the test data set, i.e., the failing test works when running it on its own (with the same initial data set), but fails when being run as part of the complete set of tests.
The tests are typically run through Maven's surefire plugin during the regular Maven build.
What I am looking for is an automated way to permute the amount and order of the executed tests to figure out the culprit. A naive, but pretty expensive approach, would take the power set of all tests and run all possible combinations. A more optimized approach would use the existing test execution order (which is mostly random, but stable) and test all potential ordered sub-sets. I am aware that the runtime of this process may be lengthy.
Are there any tools / Maven plugins that can do this out of the box?
I don't know of a tool which does specifically what you want, but you could play about with the runOrder parameter in maven surefire. From that page:
Defines the order the tests will be run in. Supported values are
"alphabetical", "reversealphabetical", "random", "hourly"
(alphabetical on even hours, reverse alphabetical on odd hours),
"failedfirst", "balanced" and "filesystem".
Odd/Even for hourly is
determined at the time the of scanning the classpath, meaning it could
change during a multi-module build.
So you could do a simple alphabetical runOrder and take the first failure, and start from there. At least you have a predictable run order. Then you run one by one (using -Dincludes) each test before the failing one & the failing one, to detect which one is making the failing test fail.
Then repeat the entire process for all of the failing tests. You could run this in a loop overnight or something.
Can you simply amend the tests to use a clean database copy each time? DBUnit is an excellent tool for doing this.
http://www.dbunit.org/