Develop Reference

Installing maven depencies from pom.xml in docker

I am trying to run a spring-boot maven project inside a docker environment. So the setup is as follows:
Docker is set up and installs Java, etc. (done only once)
App is run (can be any number of times)
What I am experiencing
Every time I run the spring-boot project by mvn spring-boot:run, it installs all the required libraries (every time I run the project) from the pom.xml (Java, Maven, etc. are preinstalled from the docker) and then runs the project.
What I am trying to do
This process of reinstalling every time is redundant and time-consuming, so I want to delegate this installation thing to the docker as well. Ideally, using the pom.xml to do the installations, though alternative ways are also welcome.
What I have tried so far
Install npm using a good tutorial, but it fails in Docker as we can't restart the terminal during docker build, while source ~/.bash_profile doesn't seem to work either.
Tried to build that project directly in docker (by RUN mvn clean install --fail-never) and copying both npm and node folders to the directory where I run the app. But it doesn't seem to work either as it's installing them every time without any change.
Can anyone please help me there? This problem has stuck the project. Thanks a lot!

From your question I understand that, in the Dockerfile you just install java, maven, etc. but does not build your project using mvn clean package install before executing mvn spring-boot:run (and that is redundant as well because mvn spring-boot:run does the build for you before staring the application).
You cannot skip installing maven dependency while running on containers as they are spun as they run. So it will be installed either while you call mvn clean install or mvn spring-boot:run.
What the max you can do is, using your devops pipeline, build the jar previously and in the Dockerfile just copy the build jar and execute.
Example Dockerfile in this case:
FROM openjdk:8-jdk-alpine
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
Here the previously build artifact is already available at target/

Slow gradle build in Docker. Caching gradle build

I am doing university project where we need to run multiple Spring Boot applications at once.
I had already configured multi-stage build with gradle docker image and then run app in openjdk:jre image.
Here is my Dockerfile:
FROM gradle:5.3.0-jdk11-slim as builder
USER root
WORKDIR /usr/src/java-code
COPY . /usr/src/java-code/
RUN gradle bootJar
FROM openjdk:11-jre-slim
EXPOSE 8080
WORKDIR /usr/src/java-app
COPY --from=builder /usr/src/java-code/build/libs/*.jar ./app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]
I am building and running everything with docker-compose. Part of docker-compose:
website_server:
build: website-server
image: website-server:latest
container_name: "website-server"
ports:
- "81:8080"
Of course first build take ages. Docker is pulling all it's dependencies. And I am okay with that.
Everything is working ok for now but every little change in code causes around 1 min build time for one app.
Part of build log: docker-compose up --build
Step 1/10 : FROM gradle:5.3.0-jdk11-slim as builder
---> 668e92a5b906
Step 2/10 : USER root
---> Using cache
---> dac9a962d8b6
Step 3/10 : WORKDIR /usr/src/java-code
---> Using cache
---> e3f4528347f1
Step 4/10 : COPY . /usr/src/java-code/
---> Using cache
---> 52b136a280a2
Step 5/10 : RUN gradle bootJar
---> Running in 88a5ac812ac8
Welcome to Gradle 5.3!
Here are the highlights of this release:
- Feature variants AKA "optional dependencies"
- Type-safe accessors in Kotlin precompiled script plugins
- Gradle Module Metadata 1.0
For more details see https://docs.gradle.org/5.3/release-notes.html
Starting a Gradle Daemon (subsequent builds will be faster)
> Task :compileJava
> Task :processResources
> Task :classes
> Task :bootJar
BUILD SUCCESSFUL in 48s
3 actionable tasks: 3 executed
Removing intermediate container 88a5ac812ac8
---> 4f9beba838ed
Step 6/10 : FROM openjdk:11-jre-slim
---> 0e452dba629c
Step 7/10 : EXPOSE 8080
---> Using cache
---> d5519e55d690
Step 8/10 : WORKDIR /usr/src/java-app
---> Using cache
---> 196f1321db2c
Step 9/10 : COPY --from=builder /usr/src/java-code/build/libs/*.jar ./app.jar
---> d101eefa2487
Step 10/10 : ENTRYPOINT ["java", "-jar", "app.jar"]
---> Running in ad02f0497c8f
Removing intermediate container ad02f0497c8f
---> 0c63eeef8c8e
Successfully built 0c63eeef8c8e
Successfully tagged website-server:latest
Every time it freezes after Starting a Gradle Daemon (subsequent builds will be faster)
I was thinking about adding volume with cached gradle dependencies but I don't know if that is core of the problem. Also i could't find good examples for that.
Is there any way to speed up the build?

Build takes a lot of time because Gradle every time the Docker image is built downloads all the plugins and dependencies.
There is no way to mount a volume at the image build time. But it is possible to introduce new stage that will download all dependencies and will be cached as Docker image layer.
FROM gradle:5.6.4-jdk11 as cache
RUN mkdir -p /home/gradle/cache_home
ENV GRADLE_USER_HOME /home/gradle/cache_home
COPY build.gradle /home/gradle/java-code/
WORKDIR /home/gradle/java-code
RUN gradle clean build -i --stacktrace
FROM gradle:5.6.4-jdk11 as builder
COPY --from=cache /home/gradle/cache_home /home/gradle/.gradle
COPY . /usr/src/java-code/
WORKDIR /usr/src/java-code
RUN gradle bootJar -i --stacktrace
FROM openjdk:11-jre-slim
EXPOSE 8080
USER root
WORKDIR /usr/src/java-app
COPY --from=builder /usr/src/java-code/build/libs/*.jar ./app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]
Gradle plugin and dependency cache is located in $GRADLE_USER_HOME/caches. GRADLE_USER_HOME must be set to something different than /home/gradle/.gradle. /home/gradle/.gradle in parent Gradle Docker image is defined as volume and is erased after each image layer.
In the sample code GRADLE_USER_HOME is set to /home/gradle/cache_home.
In the builder stage Gradle cache is copied to avoid downloading the dependencies again: COPY --from=cache /home/gradle/cache_home /home/gradle/.gradle.
The stage cache will be rebuilt only when build.gradle is changed.
When Java classes are changes, cached image layer with all dependencies is reused.
This modifications can reduce the build time but more clean way of building Docker images with Java applications is Jib by Google.
There is a Jib Gradle plugin that allows to build container images for Java applications without manually creating Dockerfile.
Building image with application and running the container is similar to:
gradle clean build jib
docker-compose up

Docker caches its images in "layers." Each command that you run is a layer. Each change that is detected in a given layer invalidates the layers that come after it. If the cache is invalidated, then the invalidated layers must be built from scratch, including dependencies.
I would suggest splitting your build steps. Have a previous layer which only copies the dependency specification into the image, then runs a command which will result in Gradle downloading the dependencies. After that's complete, copy your source into the same location where you just did that, and run the real build.
This way, the previous layers will be invalidated only when the gradle files change.
I haven't done this with Java/Gradle, but I have followed the same pattern with a Rust project, guided by this blog post.

You can try and use BuildKit (now activated by default in the latest docker-compose 1.25)
See "Speed up your java application Docker images build with BuildKit!" from
Aboullaite Med.
(This was for maven, but the same idea applies to gradle)
let's consider the following Dockerfile:
FROM maven:3.6.1-jdk-11-slim AS build
USER MYUSER
RUN mvn clean package
Modifying the second line always invalidate maven cache due to false dependency, which exposes inefficient caching issue.
BuildKit solves this limitation by introducing the concurrent build graph solver, which can run build steps in parallel and optimize out commands that don’t have an impact on the final result.
Additionally, Buildkit tracks only the updates made to the files between repeated build invocations that optimize the access to the local source files. Thus, there is no need to wait for local files to be read or uploaded before the work can begin.

As the other answers have mentioned, docker caches each step in a layer. If you could somehow get only the downloaded dependencies into a layer, then it would not have to be re downloaded each time, assuming the dependencies haven't changed.
Unfortunately, gradle doesn't have a built-in task to do this. But you can still work around it. Here's what I did:
# Only copy dependency-related files
COPY build.gradle gradle.properties settings.gradle /app/
# Only download dependencies
# Eat the expected build failure since no source code has been copied yet
RUN gradle clean build --no-daemon > /dev/null 2>&1 || true
# Copy all files
COPY ./ /app/
# Do the actual build
RUN gradle clean build --no-daemon
Also, make sure your .dockerignore file has at least these items, so that they're not sent in the docker build context when the image is built:
.gradle/
bin/
build/
gradle/

Just as an addition to other people answers, if your internet connection is slow, as it downloads dependencies every single time, you might want to set up sonatype nexus, in order to keep the dependencies already downloaded.

I used a slightly different idea. I scheduled a nightly build on my Jenkins building the entire Gradle project:
docker build -f Dockerfile.cache --tag=gradle-cache:latest .
# GRADLE BUILD CACHE
FROM gradle:6.7.1-jdk11
COPY build.gradle.kts /home/gradle/code/
COPY settings.gradle.kts /home/gradle/code/
COPY gradle.properties /home/gradle/code/
COPY ./src /home/gradle/code/src
WORKDIR /home/gradle/code
RUN gradle bootJar -i -s
Then I start my builds from this "cache image" so I can leverage all the Gradle goodness:
docker build --tag=my-app:$version .
# GRADLE BUILD
FROM gradle-cache:latest as gradle
COPY build.gradle.kts /home/gradle/code/
COPY settings.gradle.kts /home/gradle/code/
COPY gradle.properties /home/gradle/code/
RUN rm -rf /home/gradle/code/src
COPY ./src /home/gradle/code/src
WORKDIR /home/gradle/code
RUN gradle bootJar -i -s
# SPRING BOOT
FROM openjdk:11.0.9.1-jre
COPY --from=gradle /home/gradle/code/build/libs/app.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-Xmx2G", "-Djava.security.egd=file:/dev/./urandom", "-jar", "app.jar"]
Remember about pruning unused images every week or so.

I don't know much about docker internals, but I think that the problem is that each new docker build command, will copy all files and build them (if it detects changes in at least one file).
Then this will most likely change several jars and the second steps needs to run too.
My suggestion is to build on the terminal (outside of docker) and only docker build the app image.
This can even be automated with a gradle plugin:
https://github.com/Transmode/gradle-docker (one example, I did not search thoroughly)

How to cache maven dependencies in Docker

I'm working on a project with ~200MB dependencies and i'd like to avoid useless uploads due to my limited bandwidth.
When I push my Dockerfile (i'll attach it in a moment), I always have a ~200MB upload even if I didn't touch the pom.xml:
FROM maven:3.6.0-jdk-8-slim
WORKDIR /app
ADD pom.xml /app
RUN mvn verify clean --fail-never
COPY ./src /app/src
RUN mvn package
ENV CONFIG_FOLDER=/app/config
ENV DATA_FOLDER=/app/data
ENV GOLDENS_FOLDER=/app/goldens
ENV DEBUG_FOLDER=/app/debug
WORKDIR target
CMD ["java","-jar","-Dlogs=/app/logs", "myProject.jar"]
This Dockerfile should make a 200MB fatJAR including all the dependencies, that's why the ~200MB upload that occurs everytime. What i would like to achieve is building a Layer with all the dependencies and "tell" to the packaging phase to not include the dependencies JARs into the fatJAR but to search for them inside a given directory.
I was wondering to build a script that executes mvn dependency:copy-dependencies before the building process and then copying the directory to the container; then building a "non-fat"JAR that has all those dependencies only linked and not actually copied into it.
Is this possible?
EDIT:
I discovered that the Maven Local Repository of the container is located under /root/.m2. So I ended making a very simple script like this:
BuildDocker.sh
mvn verify -clean --fail-never
mv ~/.m2 ~/git/myProjectRepo/.m2
sudo docker build -t myName/myProject:"$1"
And edited Dockerfile like:
# Use an official Python runtime as a parent image
FROM maven:3.6.0-jdk-8-slim
# Copy my Mavne Local Repository into the container thus creating a new layer
COPY ./.m2 /root/.m2
# Set the working directory to /app
WORKDIR /app
# Copy the pom.xml
ADD pom.xml /app
# Resolve and Download all dependencies: this will be done only if the pom.xml has any changes
RUN mvn verify clean --fail-never
# Copy source code and configs
COPY ./src /app/src
# create a ThinJAR
RUN mvn package
# Run the jar
...
After the building process i stated that /root/.m2 has all the directories I but as soon as i launch the JAR i get:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Priority
at myProject.ThreeMeans.calculate(ThreeMeans.java:17)
at myProject.ClusteringStartup.main(ClusteringStartup.java:7)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Priority
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 2 more
Maybe i shouldn't run it through java -jar?

If I understand correctly what you'd like to achieve, the problem is to avoid creating a fat jar with all Maven dependencies at each Docker build (to alleviate the size of the Docker layers to be pushed after a rebuild).
If yes, you may be interested in the Spring Boot Thin Launcher, which is also applicable for non-Spring-Boot projects. Some comprehensive documentation is available in the README.md of the corresponding GitHub repo:
https://github.com/dsyer/spring-boot-thin-launcher#readme
To sum up, it should suffice to add the following plugin declaration in your pom.xml:
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<!--<version>${spring-boot.version}</version>-->
<dependencies>
<dependency>
<groupId>org.springframework.boot.experimental</groupId>
<artifactId>spring-boot-thin-layout</artifactId>
<version>1.0.19.RELEASE</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
Ideally, this solution should be combined with a standard Dockerfile setup to benefit from Docker's cache (see below for a typical example).
Leverage Docker's cache mechanism for a Java/Maven project
The archetype of a Dockerfile that avoids re-downloading all Maven dependencies at each build if only source code files (src/*) have been touched is given in the following reference:
https://whitfin.io/speeding-up-maven-docker-builds/
To be more precise, the proposed Dockerfile is as follows:
# our base build image
FROM maven:3.5-jdk-8 as maven
WORKDIR /app
# copy the Project Object Model file
COPY ./pom.xml ./pom.xml
# fetch all dependencies
RUN mvn dependency:go-offline -B
# copy your other files
COPY ./src ./src
# build for release
# NOTE: my-project-* should be replaced with the proper prefix
RUN mvn package && cp target/my-project-*.jar app.jar
# smaller, final base image
FROM openjdk:8u171-jre-alpine
# OPTIONAL: copy dependencies so the thin jar won't need to re-download them
# COPY --from=maven /root/.m2 /root/.m2
# set deployment directory
WORKDIR /app
# copy over the built artifact from the maven image
COPY --from=maven /app/app.jar ./app.jar
# set the startup command to run your binary
CMD ["java", "-jar", "/app/app.jar"]
Note that it relies on the so-called multi-stage build feature of Docker (presence of two FROM directives), implying the final image will be much smaller than the maven base image itself.
(If you are not interested in that feature during the development phase, you can remove the lines FROM openjdk:8u171-jre-alpine and COPY --from=maven /app/app.jar ./app.jar.)
In this approach, the Maven dependencies are fetched with RUN mvn dependency:go-offline -B before the line COPY ./src ./src (to benefit from Docker's cache).
Note however that the dependency:go-offline standard goal is not "perfect" as a few dynamic dependencies/plugins may still trigger some re-downloading at the mvn package step.
If this is an issue for you (e.g. if at some point you'd really want to work offline), you could take at look at that other SO answer that suggests using a dedicated plugin that provides the de.qaware.maven:go-offline-maven-plugin:resolve-dependencies goal.

In general Dockerfile container build, works in layers and each time you build these layers are available in catch and is used if there are no changes.
Ideally it should have worked same way.
Maven generally looks for dependencies by default in .m2 folder located in Home dir of User in Ubuntu /home/username/
If dependent jars are not available then it downloads those jars to .m2 and uses it.
Now you can zip and copy this .m2 folder after 1 successful build and move it inside Docker Container User's Home directory.
Do this before you run build command
Note: You might need to replace existing .m2 folder in docker
So your Docker file would be something like this
FROM maven:3.6.0-jdk-8-slim
WORKDIR /app
COPY .m2.zip /home/testuser/
ADD pom.xml /app
RUN mvn verify clean --fail-never
COPY ./src /app/src
RUN mvn package
...

The documentation of the official Maven Docker images also points out different ways to achieve better caching of dependencies.
Basically, they recommend to either mount the local maven repository as a volume and use it across Docker images or use a special local repository (/usr/share/maven/ref/) the contents of which will be copied on container startup.

What is the purpose of mvnw and mvnw.cmd files?

When I created a Spring Boot application I could see mvnw and mvnw.cmd files in the root of the project. What is the purpose of these two files?

These files are from Maven wrapper. It works similarly to the Gradle wrapper.
This allows you to run the Maven project without having Maven installed and present on the path. It downloads the correct Maven version if it's not found (as far as I know by default in your user home directory).
The mvnw file is for Linux (bash) and the mvnw.cmd is for the Windows environment.
To create or update all necessary Maven Wrapper files execute the following command:
mvn -N io.takari:maven:wrapper
To use a different version of maven you can specify the version as follows:
mvn -N io.takari:maven:wrapper -Dmaven=3.3.3
Both commands require maven on PATH (add the path to maven bin to Path on System Variables) if you already have mvnw in your project you can use ./mvnw instead of mvn in the commands.

Command mvnw uses Maven that is by default downloaded to ~/.m2/wrapper on the first use.
URL with Maven is specified in each project at .mvn/wrapper/maven-wrapper.properties:
distributionUrl=https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.3.9/apache-maven-3.3.9-bin.zip
To update or change Maven version invoke the following (remember about --non-recursive for multi-module projects):
./mvnw io.takari:maven:wrapper -Dmaven=3.3.9
or just modify .mvn/wrapper/maven-wrapper.properties manually.
To generate wrapper from scratch using Maven (you need to have it already in PATH run:
mvn io.takari:maven:wrapper -Dmaven=3.3.9

The Maven Wrapper is an excellent choice for projects that need a specific version of Maven (or for users that don't want to install Maven at all). Instead of installing many versions of it in the operating system, we can just use the project-specific wrapper script.
mvnw: it's an executable Unix shell script used in place of a fully installed Maven
mvnw.cmd: it's for Windows environment
Use Cases
The wrapper should work with different operating systems such as:
Linux
OSX
Windows
Solaris
After that, we can run our goals like this for the Unix system:
./mvnw clean install
And the following command for Batch:
./mvnw.cmd clean install
If we don't have the specified Maven in the wrapper properties, it'll be downloaded and installed in the folder $USER_HOME/.m2/wrapper/dists of the system.
Maven Wrapper plugin
Maven Wrapper plugin to make auto installation in a simple Spring Boot project.
First, we need to go in the main folder of the project and run this command:
mvn -N io.takari:maven:wrapper
We can also specify the version of Maven:
mvn -N io.takari:maven:wrapper -Dmaven=3.5.2
The option -N means –non-recursive so that the wrapper will only be applied to the main project of the current directory, not in any submodules.
Source 1 (further reading): https://www.baeldung.com/maven-wrapper

short answer: to run Maven and Gradle in the terminal without following manual installation processes.
Gradle example:
./gradlew clean build
./gradlew bootRun
Maven example:
./mvnw clean install
./mvnw spring-boot:run
"The recommended way to execute any Gradle build is with the help of the Gradle Wrapper (in short just “Wrapper”). The Wrapper is a script that invokes a declared version of Gradle, downloading it beforehand if necessary. As a result, developers can get up and running with a Gradle project quickly without having to follow manual installation processes saving your company time and money."
Gradle would also add some specific files corresponding to the Maven files Gradlew and Gradle.bat

In the windows OS, mvnw clean install is used for the maven clean and install activity, and mvnw spring-boot:run is used to run the Spring boot application from Command Prompt.
For Eaxmple:
C:\SamplesSpringBoot\demo>mvnw clean install
C:\SamplesSpringBoot\demo>mvnw spring-boot:run

By far the best option nowadays would be using a maven container as a builder tool. A mvn.sh script like this would be enough:
#!/bin/bash
docker run --rm -ti \
-v $(pwd):/opt/app \
-w /opt/app \
-e TERM=xterm \
-v $HOME/.m2:/root/.m2 \
maven mvn "$#"

Running main classes from a deployed artifact with maven

I don't get it. I've set up my pom.xml to use the Maven exec plugin so I can execute some of the classes in my project with the correct classpath, -D defines and -javaagent. So from a shell with the classes built in ./target/classes etc.. I can run the main() methods using
mvn exec:java -Dexec:mainClass=classWithAMainMethod
All good so far.
Now I want to ship my project(a jar artifact) and I still want to be able to use the configuration I've put in the pom.xml for running the classes with the correct arguments etc.. How do I do it? Is there some way of staying
mvn -artifactJar=MyArtifact.jar exec:java -Dexec:mainClass=classWithAMainMethod
when all I have is MyArtifact.jar(Or a maven repository with MyArtifact.jar in it)??
I've tried the following:
Get the jar with the dependency:get goal and unzip it. I can't do anything with it
as the pom.xml seems to end up in META-INF/maven in the artifact jar. Is there any way of using it?
Creating a dummy pom where I want to run my project with a single dependency on my projects artifact. I can then use exec:java to run the main classes but it's dosen't uses the configuration from my projects pom.
Thanks.

The AppAssembler plugin worked out quite well for me. I replaced the exec plugin config in my project's pom with something like this in the build section:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<version>1.2.2</version>
<configuration>
<repositoryLayout>flat</repositoryLayout>
<repositoryName>lib</repositoryName>
<extraJvmArguments>
-Djava.rmi.server.hostname=localhost
-javaagent:${spring.javaagent.jar}
</extraJvmArguments>
<programs>
<program>
<name>foo1</name>
<mainClass>net.foor.FooMain</mainClass>
</program>
...
</configuration>
</plugin>
In Eclipse I created an external tools launcher to run the resulting scripts from target/appassembler/bin
On the machine I wanted to deploy to(Assuming access to the internal Maven repository where my artifact+dependencies have been installed/deployed):
First use wget or mvn dependency:get to get a copy of my artifact jar.
Extract the pom. unzip -j artifact.jar */pom.xml*
Run mvn appassembler:assemble -DassembleDirectory=.
Move the artifact.jar into the ./lib directory
Set execute permissions on generated shell scripts in ./bin

Have you tried using something like onejar?
That sounds like what you're looking for.