Added integration test for enrichment process using Java. I've used Eclipse IDE for local testing, tried running integration test as gradle build and also did verification as integrationtest.
Also triggered build for the same in teamcity, it is getting successful and all the testcases are passed.
But the problem here is, we have set automatic trigger for build in teamcity everyday at 11 p.m.
On automatic trigger, the integration test is failed.
Showing ITTestCases: AppStartup_Tests: java.lang.AssertionError: expected [true] but found [false]
Related
I'm running my integration tests on wildfly server. 1st we deploy our artifact and then another test project will consume the API of deployed project and do complete integration testing. Is there a way to report the code coverage sonarqube ?
Presumably your IT suite produces reports. You simply need to wait to do the analysis until after the IT is done, then feed the resulting reports into the analysis.
I have configured maven surefire plugin with parameter:
<configuration>
<forkedProcessTimeoutInSeconds>60</forkedProcessTimeoutInSeconds>
</configuration>
So when test's working more then 60 seconds, the surefire plugin interrupts it.
Everything works perfectly on my local machine when I use mvn test or mvn install, but when I try to build project on Jenkins it just swallows exception, writes into log [ERROR] There was a timeout or other error in the fork and continues the build. As result I get a Finished: SUCCESS message.
Question: Have anyone got this problem? Does anyone know any solution?
One important difference between default options of a maven local build and a Jenkins maven job is that locally the maven.test.failure.ignore option of the Maven Surefire Plugin is set to false (reasonably) so that test failures will also fail the build.
From official documentation:
Set this to "true" to ignore a failure during testing. Its use is NOT RECOMMENDED, but quite convenient on occasion.
However, a Maven Jenkins job will always run setting the same option to true, as such making the Maven build successful even with test failing and turn the status of the Jenkins job to UNSTABLE (and not SUCCESSFUL or FAILED, which may be a point of debate indeed).
This behavior is also documented in an official Jenkins issue ticket
Following the Jenkins Terminology, when (surefire or failsafe) tests fail, the Jenkins build status must be UNSTABLE:
<< A build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable. >>
So, in a Maven Jenkins job, if a test fails:
Maven build is SUCCESSFUL
Jenkins build is UNSTABLE
Instead, in a freestyle Jenkins job executing Maven, if a test fails:
Maven build is FAILED
Jenkins build is FAILED
Possible solutions:
Change the build to a freestyle Jenkins job running maven (which may be too much work though) or
Add the -Dmaven.test.failure.ignore=false option to your build (however, you would not have UNSTABLE builds any longer).
We are running TestNG tests using Gradle on Jenkins.
Job configuration:
Build section -> Invoke Gradle Script -> Use Gradle Wrapper -> Tasks:
clean test -Dgroups=myTestNGTestGroupName
In Jenkins Console Output I can see the logs from execution of gradlew.bat with specific parameters (one of them is -Dgroups=myTestNGTestGroupName)).
We have quite a lot of Jenkins jobs and automation Selenium tests.
Since that on daily basis we are checking only failed jobs.
During refactoring tests TestNG group name may be changed or typo may occur.
If you changed the test group name in test repository and forgot to update Jenkins job:
0 tests are executed and job is still passing (build is successful).
How can I tell Jenkins to mark build as non-successful if no tests were executed?
TestNG generates testng-results.xml file every time after test run
(even if 0 tests were executed).
We can analyze this file. The simplest solution which I found is using
Text-finder Plugin
(which in my case was already added to Jenkins)
I added Jenkins Text Finder in Post-build Actions as follow:
How it looks in Jenkins Console Output logs:
BUILD SUCCESSFUL
Total time: 42.105 secs
Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
Checking <testng-results skipped="0" failed="0" total="0" passed="0">
c:\jenkins\workspace\my-job-name\build\reports\tests\testng-results.xml:
<testng-results skipped="0" failed="0" total="0" passed="0">
Build step 'Jenkins Text Finder' changed build result to UNSTABLE
...
Finished: UNSTABLE
For instance, if regression test is failed then overall report should be failed.
I've configured jenkins with these steps, it gives the code coverage by unit test given by developer if unit tests are executed.
My question is how can I get the code coverage by executing my regression tests which are configured in Post Step.
Pre Steps: Configuring development environment, like changing some values in configs.
Pre-Build Step: clean install -fae -Pcode-coverage -DskipTests using maven.
Post Step:
Execute Shell - /var/lib/jenkins/qascripting/Vdopia_Automation/exe/hudson_tc_execute.sh to trigger my regression test.
My development environment is Netbeans 7.4 and Maven 3.0.5 (bundled with Netbeans). I'm using JUnit 4.11 to build unit (class names ending with Test) and integration tests (class names ending with IT). I'm running unit tests and excluding integration tests with the "-DskipITs" Maven option. I have a custom action which runs just the integration tests with the failsafe plugin. Both execute successfully. However, I only see the results in the "Test Results" window when running the unit tests. How can I get the integration tests to show in the "Test Results" window? With the integration tests, I'm only seeing the output in the console.
The maven-failsafe-plugin only executes in integration-test and verify (and of course the help) goal while the maven-surefire-plugin runs during test goal.
NetBeans Test Results window only shows the tests that were executed using the 'test' goal.
My solution for this situation is to categorize my integration tests into
testintegration - just lightweight ITs, I run them with surefire (to see them in Test Results window)
testheavy - those that will require me to bootstrap something I run with the fail-safe plugin
I hope you have the option of doing something close to that.