We are running TestNG tests using Gradle on Jenkins.
Job configuration:
Build section -> Invoke Gradle Script -> Use Gradle Wrapper -> Tasks:
clean test -Dgroups=myTestNGTestGroupName
In Jenkins Console Output I can see the logs from execution of gradlew.bat with specific parameters (one of them is -Dgroups=myTestNGTestGroupName)).
We have quite a lot of Jenkins jobs and automation Selenium tests.
Since that on daily basis we are checking only failed jobs.
During refactoring tests TestNG group name may be changed or typo may occur.
If you changed the test group name in test repository and forgot to update Jenkins job:
0 tests are executed and job is still passing (build is successful).
How can I tell Jenkins to mark build as non-successful if no tests were executed?
TestNG generates testng-results.xml file every time after test run
(even if 0 tests were executed).
We can analyze this file. The simplest solution which I found is using
Text-finder Plugin
(which in my case was already added to Jenkins)
I added Jenkins Text Finder in Post-build Actions as follow:
How it looks in Jenkins Console Output logs:
BUILD SUCCESSFUL
Total time: 42.105 secs
Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
Checking <testng-results skipped="0" failed="0" total="0" passed="0">
c:\jenkins\workspace\my-job-name\build\reports\tests\testng-results.xml:
<testng-results skipped="0" failed="0" total="0" passed="0">
Build step 'Jenkins Text Finder' changed build result to UNSTABLE
...
Finished: UNSTABLE
Related
Added integration test for enrichment process using Java. I've used Eclipse IDE for local testing, tried running integration test as gradle build and also did verification as integrationtest.
Also triggered build for the same in teamcity, it is getting successful and all the testcases are passed.
But the problem here is, we have set automatic trigger for build in teamcity everyday at 11 p.m.
On automatic trigger, the integration test is failed.
Showing ITTestCases: AppStartup_Tests: java.lang.AssertionError: expected [true] but found [false]
I'm doing a POC for a Java project in an AzureDevops CI/CD pipeline. I created a Maven project that has Selenium tests (TestNG) that run against a demo website which is independent of my project. I want to run unit tests in the build pipeline and UI Selenium tests in the release pipeline.
The Visual Studio test task seems to be the building block that I need. I think you can differentiate between unit tests & UI tests using the 'Test files' field like **\unit*Test.dll, **\ui*Test. Unfortunately, this task is not available/compatible for Java projects.
I was able to run the Selenium tests with the Maven task and Surefire plugin during build but remember, I only want to run unit tests during build.
I actually was able to run the Selenium tests in the release pipeline via a workaround which was:
Copy the whole project to the artifacts directory of the release during build (copy files task).
Add a Maven task to the release pipeline
Trigger the Selenium tests in pom.xml
Normally, you would only copy artifacts to the artifact directory so I think doing that is a huge hack.
Another problem is that Maven will build the project during build and release which is wasteful. To dial back the waste, some savvy Maven configuration might help. I was thinking about skipping compilation and resolve dependencies during release, but I don't know where to find the Maven dependencies in the DevOps ecosystem.
Am I missing something or is AzureDevops maybe not supporting Java all that well?
I do follow a method for Maven selenium tests on Azure DevOps. What I do is, in the build pipeline I build my tests in such a way that it produces a jar with all the dependencies and test classes in it. I also use testng in my approach. Next I copy my build Artifact to Artifactory. This completes the build. Now during the release I download my Artifact from Artifactory and I check the environment where i want to run and I inject the right testng file by running java -jar myfile.jar testngIT.xml. This runs my tests faster and better.
You can try just adding a test task in your release pipeline just as in the build pipeline.
And add a copy task in the build pipeline to copy the test codes and files to the build artifacts and publish it to release pipeline.
Below steps is just for reference(in classic view). Hope it can be of some help.
1, Add copy file task in the build pipeline to copy the all test files and all the dependent setting files to the test folder in artifacts.
2, Publish artifacts to release pipeline
3, In the release pipeline, add the task to execute the tests just like the way you do in build pipeline
I have configured maven surefire plugin with parameter:
<configuration>
<forkedProcessTimeoutInSeconds>60</forkedProcessTimeoutInSeconds>
</configuration>
So when test's working more then 60 seconds, the surefire plugin interrupts it.
Everything works perfectly on my local machine when I use mvn test or mvn install, but when I try to build project on Jenkins it just swallows exception, writes into log [ERROR] There was a timeout or other error in the fork and continues the build. As result I get a Finished: SUCCESS message.
Question: Have anyone got this problem? Does anyone know any solution?
One important difference between default options of a maven local build and a Jenkins maven job is that locally the maven.test.failure.ignore option of the Maven Surefire Plugin is set to false (reasonably) so that test failures will also fail the build.
From official documentation:
Set this to "true" to ignore a failure during testing. Its use is NOT RECOMMENDED, but quite convenient on occasion.
However, a Maven Jenkins job will always run setting the same option to true, as such making the Maven build successful even with test failing and turn the status of the Jenkins job to UNSTABLE (and not SUCCESSFUL or FAILED, which may be a point of debate indeed).
This behavior is also documented in an official Jenkins issue ticket
Following the Jenkins Terminology, when (surefire or failsafe) tests fail, the Jenkins build status must be UNSTABLE:
<< A build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable. >>
So, in a Maven Jenkins job, if a test fails:
Maven build is SUCCESSFUL
Jenkins build is UNSTABLE
Instead, in a freestyle Jenkins job executing Maven, if a test fails:
Maven build is FAILED
Jenkins build is FAILED
Possible solutions:
Change the build to a freestyle Jenkins job running maven (which may be too much work though) or
Add the -Dmaven.test.failure.ignore=false option to your build (however, you would not have UNSTABLE builds any longer).
I have a Gradle custom Test task to run my Integration Tests. I would like to be able to run this without Gradle automatically going through all previous phases and just running the test. Is there a way to do this without -x for each build step?
I had been introduced to concept of CI lately and was trying to work on jenkins CI. I was stuck up in one thing . How to trigger executable testng files in jenkins CI. For ex locally in our machines we just run testng.xml to execute couple of test cases. In the same way how can we trigger this xml file to run in jenkins CI ?
In most cases with jenkins you wouldn't use an executable. Normally you'd run the wrapper for the tests (Junit/Nunit etc.) which Jenkins is fully capable of running on it's own.
You can use this article to run TestNG tests using Maven:
Running TestNG tests using maven
After configuration is completed just add Invoke top-level Maven targets step to the Build Steps in Jenkins (Maven plugin should be installed). The target should be test in this case.
If you will face with any errors during configuration, try to google them.
If you are not using any build tool like maven or ant, you can invoke it from command line as we'll and specify your suite file. Make sure to set the correct class paths http://testng.org/doc/documentation-main.html#running-testng
You can put this as a build step in Jenkins.
Add a compilation step prior to this step. I haven't ever tried it - have always used ant or maven, but that is where I would start exploring.