I am trying to build a gitlab CI/CD pipeline on my java maven project. I tried all the stages on my local machine and they are passing successfully but when I try the same things on gitlab repo it is throwing error. I tried almost every solution and gone through numerous posts to resolve the errors but to no use. Here is my yml content:
image: maven:3.5-jdk-8
variables:
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
include:
- template: Security/SAST.gitlab-ci.yml
cache:
paths:
- .m2/settings.xml
# Define stages
# Stages group various steps into one block,
# if any step fails, the entire stage fails
stages:
- validate
- compile
- SonarQube
- test
validate:
stage: validate
script:
- mvn validate
compile:
stage: compile
script:
- mvn $MAVEN_CLI_OPTS compile
sonarqube-check:
image: maven:3.6.3-jdk-11
stage: SonarQube
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"
script:
- mvn sonar:sonar \ -Dsonar.projectKey=XX.XX.XXX.XXX \ -Dsonar.host.url=http://XX.XX.XXX.XXX \ -Dsonar.login=XX.XX.XXX.XXX -X
allow_failure: true
spotbugs-sast:
variables:
COMPILE: "false"
SECURE_LOG_LEVEL: "debug"
artifacts:
reports:
sast: gl-sast-report.json
#spotbugs-sast:
# variables:
# SECURE_LOG_LEVEL: "debug"
#FAIL_NEVER: 1
test:
stage: test
script:
- mvn $MAVEN_CLI_OPTS test -B
The stage validate and compile are running fine but I am getting warnings in sonar check and spotbugs sast. The same is running fine on my local.
In the sonarqube check the error is:
Error resolving version for plugin ' -Dsonar.host.url=http://XX.XX.XXX.XXX' from the repositories [local (/builds/XX.XX.XXX.XXX/.m2/repository), central (https://repo.maven.apache.org/maven2)]: Plugin not found in any plugin repository -> [Help 1]
org.apache.maven.plugin.version.PluginVersionResolutionException: Error resolving version for plugin ' -Dsonar.host.url=http://XX.XX.XXX.XXX' from the repositories [local (/builds/XXXX/XXXXX/.m2/repository), central (https://repo.maven.apache.org/maven2)]: Plugin not found in any plugin repository
and for the spotbugs-sast warning the error is:
[FATA] [Find Security Bugs] ▶ lstat /root/.m2/repository: no such file or directory
I have spent numerous hours resolving this and waiting for the project to compile successfully with all the stages but every time I was disappointed. Please help. I appreciate your time.
Maven recognized the String -Dsonar.host.url=http://XX.XX.XXX.XXX as a plugin/goal that it needs to resolve.
This is caused by the \ in your script.
Instead of:
mvn sonar:sonar \ -Dsonar.projectKey=XX.XX.XXX.XXX \ -Dsonar.host.url=http://XX.XX.XXX.XXX \ -Dsonar.login=XX.XX.XXX.XXX -X
try:
mvn sonar:sonar -Dsonar.projectKey=XX.XX.XXX.XXX -Dsonar.host.url=http://XX.XX.XXX.XXX -Dsonar.login=XX.XX.XXX.XXX -X
I assume that you copied that command from a format like this:
mvn sonar:sonar \
-Dsonar.projectKey=XX.XX.XXX.XXX \
-Dsonar.host.url=http://XX.XX.XXX.XXX \
-Dsonar.login=XX.XX.XXX.XXX -X
In bash the \ must be at the end of the line if the command contains a linebreak so it's recognized as one command.
But you need to remove it if you convert it to one line.
Related
I am using Sonarqube to optimize my code quality in my project and I tried adding it to my CI cycle on GitLab but I am getting an error. When I just run mvn sonar:sonar in my IntelliJ terminal it works, but it somehow throws an error while executed in my GitLab CI. The error is:
Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.7.0.1746:sonar (default-cli) on project rlstop: Your project contains .java files, please provide compiled classes with sonar.java.binaries property, or exclude them from the analysis with sonar.exclusions property.
My gitlab-ci.yml:
stages:
- build
- test
- sonarqube
build:
script: "mvn compile"
test:
script: "mvn test"
stage: test
sonarqube:
script: "mvn sonar:sonar"
stage: sonarqube
application.properties:
spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.datasource.url=...
spring.datasource.username=...
spring.datasource.password=...
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.format_sql = true
## Hibernate Properties
# The SQL dialect makes Hibernate generate better SQL for the chosen database
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.SQLServer2012Dialect
# Hibernate ddl auto (create, create-drop, validate, update)
spring.jpa.hibernate.ddl-auto = update
sonar.java.binaries=bin
Does anybody know what this error means and how to resolve it?
The error mentioned that you need to provide the compiled classes of java using sonar.java.binaries
In your .gitlab-ci.yml file:
stages:
- build
- test
- sonarqube
build:
stage: build
script: "mvn compile"
test:
stage: test
script: "mvn test"
sonarqube:
stage: sonarqube
script: "mvn -U install sonar:sonar -Dsonar.projectKey=Gitlab-CI -Dsonar.projectName=Gitlab-CI -Dsonar.sources=. -Dsonar.java.binaries=**/* -Dsonar.language=java"
If in case you have added <sonar-java-binaries>target/classes</sonar-java-binaries> in your pom.xml, please remove. Only, have the entry of maven-sonar-plugin in the plugins section of pom.xml
Hope this might help you.
I have a problem similar to Run (Docker) Test Container in gitlab with Maven. The difference is that rather than my script running mvn directly it runs a docker multistage build that runs the test inside of the docker image. Unfortunately this doesn't appear to work for the PostgreSQL Test Container.
Dockerfile
#############
### build ###
#############
# base image
FROM maven:3-jdk-11 as build
# set working directory
WORKDIR /app
# add app
COPY . .
RUN export MAVEN_OPTS="-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true" && export MAVEN_CLI_OPTS="-B -U --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
RUN mvn $MAVEN_CLI_OPTS clean install
############
### prod ###
############
# Yea this isn't right, but it crashes before it gets to this point. This is for example purposes only.
FROM openjdk:15-jdk-alpine
COPY --from=build /app/reproducer-testcontainer/target/reproducer-testcontainer.jar /reproducer-testcontainer.jar
CMD java -jar reproducer-testcontainer.jar
When I run mvn clean install it works properly and runs my test using the PostgreSQL Test Container. However, when I run docker build . it fails at the mvn clean install step with the below stack trace.
Stack trace:
13:05:01.250 [main] ERROR org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - ping failed with configuration Environment variables, system properties and defaults. Resolved:
dockerHost=unix:///var/run/docker.sock
apiVersion='{UNKNOWN_VERSION}'
registryUrl='https://index.docker.io/v1/'
registryUsername='root'
registryPassword='null'
registryEmail='null'
dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'
due to org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:51)
<snip>
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocket.<init>(UnixDomainSocket.java:62)
<snip>
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.sun.jna.LastErrorException: [2] No such file or directory
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocketLibrary.connect(Native Method)
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocket.<init>(UnixDomainSocket.java:57)
... 35 common frames omitted
In my CI pipeline I'd like to only run docker build . and not worry about having another stage that does the mvn clean install.
How do I fix the configuration to get the java PostgreSQL Testcontainers to work inside of a Docker build so that I can use it in a multi-stage build?
Full Code example: https://gitlab.com/raymondcg/reproducer-testcontainer
Not really Testcontainers related.
Testcontainers requires a valid Docker daemon. When you build images, there is no daemon mounted into the image build context.
You can easily verify that by doing:
RUN curl --unix-socket /var/run/docker.sock http:/_/_ping
Make this command return "OK" (no need to run the Testcontainers code), and your tests will pass as well.
You can overwrite testcontainers default docker host by adding:
ENV DOCKER_HOST=tcp://host.docker.internal:2375
to your build stage.
I just created a Jenkins Pipeline DSL job where I cloned a Java code from SCM and tried to run mvn clean. But the pipeline continuously throwing an error saying:
mvn clean install -Dmaven.test.skip=true -Dfindbugs.skip=true
/var/lib/jenkins/workspace/<project>#tmp/durable-77d8d13c/script.sh: 2:
/var/lib/jenkins/workspace/<project>#tmp/durable-77d8d13c/script.sh: mvn: not found
Seems like it tries to find pom.xml inside the <project>#tmp directory which is empty. Actual code is cloned successfully inside the <project> directory. Below is my Jenkinsfile:
node {
stage ("Clean Workspace") {
echo "${WORKSPACE}"
cleanWs()
}
stage ("Get Code") {
git branch: "${params.branch}", url: 'git#bitbucket.org:xx/xxxxxxx.git'
}
stage ("mvn clean") {
sh "mvn clean install -Dmaven.test.skip=true -Dfindbugs.skip=true"
}
}
I also tried with ${WORKSPACE} env variable but still does not work.
The issue was resolved, I modified the Pipeline DSL jobs by including Maven tool step as:
stage ("mvn clean") {
withEnv( ["PATH+MAVEN=${tool name: 'mvn', type: 'maven'}/bin"] ) {
sh "mvn clean install -Dmaven.test.skip=true -Dfindbugs.skip=true"
}
}
Instead of doing like that you can directly add mvn to path itself. for that, open your ~/.bash_profile and add these lines.
MAVEN_HOME="<path to maven folder>"
export MAVEN_HOME
PATH=$PATH:$MAVEN_HOME/bin
Don't forget to do source ~/.bash_profile to get the changes. This will provide mvn to path so that you can access mvn directly.
So if you do that your code can be now like...
stage ("mvn clean") {
sh "mvn clean install -Dmaven.test.skip=true -Dfindbugs.skip=true"
}
The error message mvn: not found means that mvn command could not be found by Jenkins pipeline.
You should use a withMaven pipeline step to provide a valid maven environnement.
For example :
node {
stage ("Clean Workspace") {
echo "${WORKSPACE}"
cleanWs()
}
stage ("Get Code") {
git branch: "${params.branch}", url: 'git#bitbucket.org:xx/xxxxxxx.git'
}
stage ("mvn clean") {
withMaven {
sh "mvn clean install -Dmaven.test.skip=true -Dfindbugs.skip=true"
}
}
}
Check that Jenkins is properly configured to use maven (Under 'Manage Jenkins > Configure System') and the 'Pipeline Maven Plugin' is installed (Under 'Manage Jenkins > Manage Plugins').
More information here : https://wiki.jenkins.io/display/JENKINS/Pipeline+Maven+Plugin
I have integrated CircleCI to run Espresso test on my app. I have taken the following circle.yml file from another online github repo and changed the android build tools and android version to 25. However when I run the build on the circleCI server, I get the following error. I have granted the execution permission in yml file.
My app Repo structure is
Action failed: gradle dependencies
export TERM="dumb"
if [ -e ./gradlew ]; then ./gradlew dependencies;else gradle dependencies;fi
bash: line 2: ./gradlew: Permission denied
export TERM="dumb"
if [ -e ./gradlew ]; then ./gradlew dependencies;else gradle dependencies;fi
returned exit code 126
Action failed: gradle dependencies
circle.yml:
general:
artifacts:
- /home/ubuntu/MyRideApp/app/build/outputs/apk/
machine:
environment:
ANDROID_HOME: /usr/local/android-sdk-linux
ADB_INSTALL_TIMEOUT: 240
GRADLE_OPTS: '-Dorg.gradle.jvmargs="-Xmx3072M -XX:+HeapDumpOnOutOfMemoryError"'
dependencies:
pre:
- chmod +x gradlew
- touch app/google-services.json
- echo y | android update sdk --no-ui --all --filter "tools,android-25,build-tools-25.0.2,platform-tools,extra-android-m2repository,extra-android-support,extra-google-m2repository,extra-google-google_play_services"
cache_directories:
- /usr/local/android-sdk-linux/tools
- /usr/local/android-sdk-linux/build-tools/25.0.2
override:
- ANDROID_HOME=/usr/local/android-sdk-linux ./gradlew dependencies
# Comment the test stuff out (or remove it) if you don't need it.
test:
pre:
- emulator -avd circleci-android23 -no-audio -no-window:
background: true
parallel: true
- circle-android wait-for-boot
# unlock emulator
- sleep 30
- adb shell input keyevent 82
override:
# - ./gradlew clean assemble
# This will run the tests:
- ./gradlew assemble connectedDebugAndroidTest -PdisablePreDex --console=plain --info
post:
- cp -r app/build/outputs $CIRCLE_ARTIFACTS
- cp -r app/build/outputs/androidTest-results/connected/ $CIRCLE_TEST_REPORTS
gradle/wrapper/gradle-wrapper.properties
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-3.3-all.zip
/gradle/wrapper does contain gradle-wrapper.jar
A solution is on the circleCI forum.
You have to add execution right like the following : chmod +x gradlew
It works for me.
Here is an example of my circleci.yml :
machine:
java:
version: oraclejdk8
dependencies:
override:
- chmod +x gradlew
- ./gradlew dependencies
test:
override:
- chmod +x grailsw
- ./grailsw test-app --non-interactive
post:
- mkdir -p $CIRCLE_TEST_REPORTS/junit/
- find . -type f -regex ".*/target/test-reports/.*xml" -exec cp {} $CIRCLE_TEST_REPORTS/junit/ \;
I've successfully setup a project which uses Travis CI to for builds and tests. Now I'm trying to add Coverity Scan.
I created a branch called coverity_scan and set it be used for coverity builds. After I push a commit to this branch I can see in Travis CI build console that Coverity tool starts doing its job:
Coverity Scan analysis selected for branch coverity_scan.
Coverity Scan analysis authorized per quota.
...
Running Coverity Scan Analysis Tool...
The Travis build succeeds and in Coverity build-log.txt file I see this:
2016-10-06T21:02:39.132946Z|cov-build|2665|info|>
2016-10-06T21:02:39.132946Z|cov-build|2665|info|> Build time (cov-build overall): 00:01:36.812431
2016-10-06T21:02:39.132946Z|cov-build|2665|info|>
2016-10-06T21:02:39.134719Z|cov-build|2665|info|> Build time (Java emits total): 00:01:07.595656
2016-10-06T21:02:39.134719Z|cov-build|2665|info|>
2016-10-06T21:02:39.134719Z|cov-build|2665|info|>
2016-10-06T21:02:39.134763Z|cov-build|2665|info|> Emitted 30 Java compilation units (100%) successfully
2016-10-06T21:02:39.134763Z|cov-build|2665|info|> [WARNING] Recoverable errors were encountered during 1 of these Java compilation units.
2016-10-06T21:02:39.134763Z|cov-build|2665|info|>
2016-10-06T21:02:39.134763Z|cov-build|2665|info|> 30 Java compilation units (100%) are ready for analysis
2016-10-06T21:02:39.134763Z|cov-build|2665|info|> For more details, please look at:
2016-10-06T21:02:39.134763Z|cov-build|2665|info|> /home/travis/build/Edvinas01/chat-rooms/server/cov-int/build-log.txt
However after this finishes, I do not see any submitted builds or changes in projects Coverity dashboard. The project status stays on pending.
I've followed this guide and setup my .travis.yml file like this:
language: java
jdk:
- oraclejdk8
before_script:
- cd server
- chmod +x gradlew
script:
# Run tests when not on coverity branch.
- if [ ${COVERITY_SCAN_BRANCH} != 1 ]; then
./gradlew check;
fi
cache:
directories:
- ~/.gradle
after_success:
# Upload coveralls when not on coverity branch.
- if [ ${COVERITY_SCAN_BRANCH} != 1 ]; then
./gradlew cobertura coveralls;
else
cat cov-int/build-log.txt;
fi
notifications:
email:
on_success: change
env:
matrix:
- TERM=dumb
global:
# COVERITY_SCAN_TOKEN
- secure: "<TOKEN>"
before_install:
- echo -n | openssl s_client -connect scan.coverity.com:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | sudo tee -a /etc/ssl/certs/ca-
addons:
coverity_scan:
project:
name: "Edvinas01/chat-rooms"
description: "Build submitted via Travis CI"
notification_email: "<EMAIL>"
build_command_prepend: "./gradlew clean"
build_command: "./gradlew build"
branch_pattern: coverity_scan
Do I have to specify some additional configuration so that my Coverity builds get published?
Got some time and created a virtual machine with java and the coverity analysis tool. After pulling my project and running the tool I noticed this in the logs:
[WARNING] No files were emitted. This may be due to a problem with your configuration
or because no files were actually compiled by your build command.
After fiddling quite a bit and looking at other projects, I found out that this was due to Gradle version. My project was using 3.0 so I downgraded to 2.14.1 and it finally seems to be working.
For what is worth, there is no issue with using Coverity with any Gradle version, as long as you make sure you are not using the daemon (just to be sure you may specify --no-daemon on the command line).
That said, there are a number of other easy to miss gotchas, resulting in not-quite-obvious error messages.
For useful background, see Caleb's answer here:
Can't get Coverity Scan to work (Java/Kotlin/Gradle 3.3 on Windows and Travis)
For working example, you may refer to this project:
https://github.com/ddimtirov/nuggets