I want to setup a CI pipeline in GitLab for my Java project managed with Maven.
This is my gitlab-ci.yml
image: maven:3-jdk-9
variables:
MAVEN_CLI_OPTS: "--batch-mode"
stages:
- build
compile:
stage: build
script:
- mvn $MAVEN_CLI_OPTS compile
I always get the following exception:
I tried many things like changing versions of the plugins, various docker images, including a settings.xml and local repository in the project itself, but nothing works.
Thanks in advance for any help!
UPDATE:
Using the latest docker image everything works.
It seems like the CI server has no connection to the internet. Check this using the curl command in your .gitlab-ci.ymlfile.
But I'm pretty sure you guys at daimler have a local mirror, something like Artifactory.
In that case you have to use a settings.xml file.
Here is the official tutorial of Gitlab
Related
I would like to use the gitlab package registry globally.
My situation is as follows:
I' working on several maven projects for which I use a nexus server for my packages.
In my team, we would like to get rid of this nexus server because of maintenance effort in some migrations.
I recently found the gitlab package registry tool which works well so far when we want to push our packages.
Now we have the want to configure a settings.xml in our .m2 folder so that the dependencies are getting fetched from gitlab ci globally.
Is there please a way to do that? So far we just found the possibility with applying the parameter -s to the mvn command line which does not fit to our processes.
A template of a settings.xml would be appreciated.
Thanks in advance.
Edit 3:
I also tried to set maven proxy through java option parameters mentioned at this thread.
Edit 2:
I'm sure intellij idea are using same settings.xml, same maven binary and the same local repository as system maven.
Edit 1:
I tried to check build log of each workload, the main difference is about how to invoke maven at the very beginning of build log.
For intellij idea, it's like below:
C:\Program Files\Java\jdk1.8.0_101\bin\java.exe" -Dmaven.multiModuleProjectDirectory=C:\Users\eugene\IdeaProjects\alluxio -Dmaven.home=C:\apache-maven-3.5.4-bin\apache-maven-3.5.4 -Dclassworlds.conf=C:\apache-maven-3.5.4-bin\apache-maven-3.5.4\bin\m2.conf "-Dmaven.ext.class.path=C:\Program Files\JetBrains\IntelliJ IDEA\plugins\maven\lib\maven-event-listener.jar" -Dfile.encoding=UTF-8 -classpath C:\apache-maven-3.5.4-bin\apache-maven-3.5.4\boot\plexus-classworlds-2.5.2.jar org.codehaus.classworlds.Launcher -Didea.version2019.2.4 -DskipTests=true -T 2C clean install -DskipTests -Dmaven.javadoc.skip -Dfindbugs.skip -Dcheckstyle.skip -Dlicense.skip
For system shell, maven just launched without this invoking info.
I'm using maven to build a project Alluxio from source code.
I tried both Windows 10 and Ubuntu and found same issue when using intellij idea, let me clarify it into details.
OS: windows 10/ Ubuntu 18.04
Maven: 3.5.4
Build command: mvn -T 2C clean install -DskipTests -Dmaven.javadoc.skip -Dfindbugs.skip -Dcheckstyle.skip -Dlicense.skip
The Alluxio can be built successfully using maven 3.5.4 directly but failed with several errors using intellij idea. What makes me confused is that I configured intellij idea to use system maven 3.5.4 and used exactly same build command. Why errors happened here but not in system shell.
The error I met is like:
Failure to find com.google.guava:listenablefuture:jar:sources:9999.0-empty-to-avoid-conflict-with-guava in https://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
And also like:
sourceFile C:\Users\eugene\IdeaProjects\alluxio\table\server\underdb\target\alluxio-table-server-underdb-2.2.0-SNAPSHOT-jar-with-dependencies.jar does not exist
The method used to trigger maven build from intellij idea:
Click maven icon at right top corner
Click Execute Maven Goal
Input mvn -T 2C clean install -DskipTests -Dmaven.javadoc.skip -Dfindbugs.skip -Dcheckstyle.skip -Dlicense.skip and enter to launch build
The maven settings in intellij idea is as default except change the maven binary from build-in binary to system maven 3.5.4.
Thanks for your help in advance.
I have seen two issues that will produce an error message like you see and it has nothing to do with IntelliJ.
The first occurs when Maven fails to successfully download an artifact from a repository (e.g., network interruption). It will mark the artifact as failed and will refuse to retry until some period lapses. Cleaning your local Maven cache or removing that artifact's folder will fix this issue.
The second occurs when two separate Maven builds attempt to download the same artifact from different repositories. Many years ago, Maven had a problem with people building modified versions of open-source projects (e.g., Apache Commons) and publishing them in a publicly available repository. I don't remember the details but this caused lots of issues. Maven now records the repository used to fetch an artifact. When two Maven projects use different repositories, the second one built will fail because the repository does not match. I had this occur when switching to a private repository, Artifactory, and not having all my projects migrated yet.
Since you are attempting to build the same project with two tools, your issue appears to be a variant of the second issue. I suspect that IntelliJ is using a different settings.xml than what you have available from the command line and that IntelliJ is using different repositories. Repositories can be specified in the settings.xml as well as the project's POM.
Try deleting your local cache and building from IntelliJ first. If it succeeds and then the shell build fails, this is your problem.
Update: See this answer for more details on why Maven started tracking the repository but note the the tracking file is now called _remote.repositories.
https://stackoverflow.com/a/16870552/252344
I was testing Skaffod and It is a great tool for microservices development.
But I do not find any tutorial on how to use it with Java. Is there any support to Maven builds?
There is a discussion going on about adding support for Java apps here, you can very much use Docker multistage build with Skaffold. A probably working example is available here
Your build portion of Skaffold file will look something like:
apiVersion: skaffold/v1alpha2
kind: Config
build:
tagPolicy:
dateTime:
format: 2006-01-02_15-04-05.999_MST
timezone: Local
artifacts:
- imageName: <repo>/<image>
workspace: ./appdir
In the appdir - you can use a multistage Dockerfile and integrate with rest of the workflow. Multistage Dockefile will build artefact in one stage and create a container using the artefact of the first stage.
Skaffold now supports JIB out of the box which will be more efficient than multistage Dockerfile building! Check out the JIB Maven example in Skaffold.
I haven't tried it yet, but alternatively now it allows you to specify a custom build script: Custom Build Script
I'm assuming you could try as below, or source a build.sh as their documentation exemplifies.
build:
artifacts:
- image: my-image
custom:
buildCommand: mvn package && docker build .
I have several java components (WARs), all of them expose webservices, and they happen to use the samemessaging objects (DTOs).
This components all share a common maven dependency for the DTOs, let's call it "messaging-dtos.jar". This common dependency has a version number, for example messaging-dtos-1.2.3.jar, where 1.2.3 is the maven version for that artifact, which is published in a nexus repository and the like.
In the maven world, docker aside, it can get tedious to work with closed version dependencies. The solution for that is maven SNAPSHOTS. When you use for example Eclipse IDE, and you set a dependency to a SNAPSHOT version, this will cause the IDE to take the version from your current workspace instead of nexus, saving time by not having to close a version each time you make a small change.
Now, I don't know how to make this development cycle to work with docker and docker-compose. I have "Component A" which lives in its own git repo, and messaging-dtos.jar, which lives in another git repo, and it's published in nexus.
My Dockerfile simpy does a RUN mvn clean install at some point, bringing the closed version for this dependency (we are using Dockerfiles for the actual deployments, but for local environments we use docker-compose). This works for closed versions, but not for SNAPSHOTS (at least not for local SNAPSHOTs, I could publish the SNAPSHOT in nexus, but that creates another set of problems, with different content overwriting the same SNAPSHOT and such, been there and I would like to not come back).
I've been thinking about using docker-compose volumes at some point, maybe to mount whatever is in my local .m2 so ComponentA can find the snapshot dependency when it builds, but this doesn't feel "clean" enough, the build would depend partially on whatever is specified in the Dockerfile and partially on things build locally. I'm not sure that'd be the correct way.
Any ideas? Thanks!
I propose maintain two approaches: one for your local development environment (i.e. your machine) and another for building in your current CI tool.
For your local dev environment:
A Dockerfile that provides the system needs for your War application (i.e. Tomcat)
docker-compose to mount a volume with the built war app, from Eclipse or whatever IDE.
For CI (not your dev environment):
A very similar Dockerfile but one that can build your application (with maven installed)
A practical example
I use the docker feature: multi stage build.
A single Dockerfile for both Dev and CI envs that might be splited but I prefer to maintain only one:
FROM maven as build
ARG LOCAL_ENV=false
COPY ./src /app/
RUN mkdir /app/target/
RUN touch /app/target/app.war
WORKDIR /app
# Run the following only if we are not in Dev Environment:
RUN test $LOCAL_ENV = "false" && mvn clean install
FROM tomcat
COPY --from=build /app/target/app.war /usr/local/tomcat/webapps
The multi-stage build saves a lot of disk space discarding everything from the build, except what is being COPY --from='ed.
Then, docker-compose.yml used in Dev env:
version: "3"
services:
app:
build:
context: .
args:
LOCAL_ENV: true
volumes:
- ./target/app.war:/usr/local/tomcat/webapps/app.war
Build in CI (not your local machine):
# Will run `mvn clean install`, fetching whatever it needs from Nexus and so on.
docker build .
Run in local env (your local machine):
# Will inject the war that should be available after a build from your IDE
docker-compose up
I would like to build a test environment with Docker, where I can remotely send JUnit test classes (including the code that is tested), execute the tests and retrieve the results.
I found some articles which explained how to use docker for testing databaseconntection/writing inside a redis, but not how i can simple let my tests perform on docker and retrieve the results.
Do you have any recommendations how You would actually achieve this?
I don't know much about Jenkins, but would this might solve my problem?
Is there any good framework outside for this?
In a dockerfile, checkout your code and do a "maven test" command, redirect the result in a file that is on a mounted directory.
Each time you build the dockerfile, you do a unit test on your project.
With docker you also have a "docker test" command. I dont know if there is a plugin to use it on jenkins.
One way I found that works (using Gradle) is as follows. I know you are specifically referencing JUnit as your testing framework, but I actually think something similar to this could work.
Dockerfile (I called mine Dockerfile.UnitTests):
FROM gradle:jdk8 AS test-stage
WORKDIR /app
COPY . ./
RUN gradle clean
RUN gradle test
FROM scratch AS export-stage
COPY --from=test-stage /app/build/reports/tests/test/* /
I then run this with (in Gitbash on Windows 10):
> DOCKER_BUILDKIT=1 docker build -f Dockerfile.UnitTests --output type=tar,dest=UnitTests.tar .
This results in a tar file containing the test results displayed in an html file.
I executed the above in a Gitlab CI/CD pipeline and then sent the results to a web API for analysis.
A couple of assumptions:
My project is set up for Gradle builds so I have the structure from the root of my project src/test/java/groupname/projectname/testfile.java
I am working in Windows 10 targeting Linux containers and using Gitbash.