I am trying to build a Java application and make a package using docker. This builds needs a maven repository which I don't want to include in the image, since it's very large. I wanted to try using volumes and mount my local maven repository to the maven repository in the image. I used apt-get install -y maven in order to have maven available, but I can't find the directory .m2 in the image $HOME.
I used ls -la $HOME, ls -la and ls -la /root to find the maven home, but there is no .m2 directory there.
EDIT 1:
I have these lines in Dockerfile:
FROM ubuntu
MAINTAINER Zeinab Abbasimazar
# Install and configure required packages
RUN apt-get update; \
apt-get install -y --no-install-recommends apt-utils; \
apt-get install -y dialog; \
apt-get install -y wget unzip curl maven; \
mkdir $HOME/.m2/; \
ls -la /usr/share/maven/conf/; \
echo \
"<settings xmlns='http://maven.apache.org/SETTINGS/1.0.0\' \
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' \
xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd'> \
<localRepository>/root/.m2/repository</localRepository> \
<interactiveMode>true</interactiveMode> \
<usePluginRegistry>false</usePluginRegistry> \
<offline>false</offline> \
</settings>" \
> /usr/share/maven/conf/settings.xml
VOLUME ["/home/zeinab/.m2/", "/root/.m2/"]
# Build
RUN mvn -X clean install -pl components -P profile
Which puts local repository configurations in image's maven configuration file, mounts my local maven repository to a directory in the image and finally performs the build. As I can see in the maven build log that it's using the local repository path I expected:
[DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml
[DEBUG] Reading user settings from /root/.m2/settings.xml
[DEBUG] Using local repository at /root/.m2/repository
But still can't detect dependencies.
I finally found the solution for mounting my local maven repository in docker. I changed my solution; I am mounting it in the run phase instead of build phase. This is my Dockerfile:
FROM ubuntu
MAINTAINER Zeinab Abbasimazar
ADD gwr $HOME
RUN apt-get update; \
apt-get install -y --no-install-recommends apt-utils; \
apt-get install -y wget unzip curl maven git; \
echo \
"<settings xmlns='http://maven.apache.org/SETTINGS/1.0.0\' \
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' \
xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd'> \
<localRepository>/root/.m2/repository</localRepository> \
<interactiveMode>true</interactiveMode> \
<usePluginRegistry>false</usePluginRegistry> \
<offline>false</offline> \
</settings>" \
> /usr/share/maven/conf/settings.xml; \
mkdir /root/.m2/; \
echo \
"<settings xmlns='http://maven.apache.org/SETTINGS/1.0.0\' \
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' \
xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd'> \
<localRepository>/root/.m2/repository</localRepository> \
<interactiveMode>true</interactiveMode> \
<usePluginRegistry>false</usePluginRegistry> \
<offline>false</offline> \
</settings>" \
> /root/.m2/settings.xml
WORKDIR .
CMD mvn -X clean install -pl components -P profile
At first, I build the image using above Dockerfile:
sudo docker build -t imageName:imageTag .
Then, I run a container as below:
sudo docker run -d -v /home/zeinab/.m2/:/root/.m2/ --name containerName imageName:imageTag
You don't find the ~/.m2 directory because it is created only when needed, i.e. when you store libraries in the local repository or when you add a config file.
You can create the ~/.m2 directory yourself and create your own settings.xml inside. There you can define the emplacement of the local repository:
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
https://maven.apache.org/xsd/settings-1.0.0.xsd">
...
<localRepository>/path/to/local/repo/</localRepository>
...
</settings>
Read the documentation for more details.
Related
i have been struggling to use my created eureka-server container in docker...
I have gone through previous solutions and am still not getting why i cant access the url: http://localhost:8761/
I have changed my properties file severally but no one seems to be working...
Firstly my application.properties file goes like this
eureka.client.register-with-eureka=false
eureka.client.fetch-registry=false
server.port=8761
spring.application.name=discovery-service
eureka.instance.prefer-ip-address=true
logging.level.org.springframework.cloud.commons.util.InetUtils=trace
spring.cloud.inetutils.timeout-seconds=10
And my dependecies tag of my pom.xml goes like this
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
</dependencies>
And i have also added the #EnableEurekaServer to my application class...
So when i created an image for it using the docker command docker build -t davidtega/eureka-layered -f Dockerfile.layered .
It worked perfectly, and i started a container using the docker command docker run -p 8761:8761 -t davidtega/eureka-layered
And this the log
But when i try to access http://localhost:8761/, this site cannot be reached is the response i get everytime...
So i noticed my app was running on 0.0.0.0:8761 not 127.0.0.1:8761
I was wondering how do i change it ???
I have two docker files, the first one is the DockerFile and the second one is the Dockerfile.layered
For my DockerFile, this is what is in it...
FROM openjdk:17
COPY target/*.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
And my DockerFile.layered file contains
FROM eclipse-temurin:17.0.4.1_1-jre as builder
WORKDIR extracted
ADD target/*.jar app.jar
RUN java -Djarmode=layertools -jar app.jar extract
FROM eclipse-temurin:17.0.4.1_1-jre
WORKDIR application
COPY --from=builder extracted/dependencies/ ./
COPY --from=builder extracted/spring-boot-loader/ ./
COPY --from=builder extracted/snapshot-dependencies/ ./
COPY --from=builder extracted/application/ ./
EXPOSE 8761
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]
Please and please help i request assistance, i am using spring cloud version 2.7 and java 17... Thanks
Add to the config eureka.hostname=localhost and eureka.client.serviceUrl.defaultZone=http://${eureka.instance.hostname}:${server.port}/eureka/
Make sure the port is mapped doing a docker run -p 8761 and then check that the port is correctly listening with lsof -i -P -n | grep LISTEN
Context: I'm able to dockerize and run my Springboot app locally just fine with the below, following the most common recommendations of using a generated jar file to run the app via Docker.
Question: Since it's bad practice to push the jar file to the repo as it's going to potentially contain secrets from the local application.yml files, and the docker file depends on the jar file, how can I have my app be dockerized not just locally but on the cloud in anywhich cicd pipeline? Would my next step be modifying the dockerfile to copy over the project directory, and handle generating the jar file itself? Or should I not be using a jar at all and copying over the directory and using a CMD [Some Spring Run command)]
DockerFile:
FROM maven:3.8.5-openjdk-17
ADD target/xyz.jar xyz.jar
ENV RESTFUL_PORT 8080
ENV PORT_POSTGRES 5432
EXPOSE $RESTFUL_PORT
EXPOSE $PORT_POSTGRES
ENTRYPOINT ["java", "-jar","/xyz.jar"]
The pom.xml plugin which generates the jar file:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.10.1</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
Docker Steps
Build the Jar:
mvn clean install
Build the Image:
docker build -t xyz -f Dockerfile
Run the Image:
docker run -d -p 8080:8080 -e "SPRING_PROFILES_ACTIVE=dev" xyz
The Dockerfile can be like this:
# Build Stage
FROM maven:3.8-openjdk-17-slim AS build
COPY src /home/app/src
COPY pom.xml /home/app
RUN mvn -f /home/app/pom.xml clean package
# Execution Stage
FROM openjdk:17-oracle
COPY --from=build /home/app/target/demo-0.1-SNAPSHOT.jar /usr/local/lib/demo.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","/usr/local/lib/demo.jar"]
To build docker image
$ docker build -t demo-app:1.0 .
Run docker image
$ docker run -p 8080:8080 demo-app:1.0
Simple build part of pom.xml will work
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
FROM maven:3.8-openjdk-17-slim AS build
WORKDIR /workspace/app
COPY mvnw .
COPY .mvn .mvn
COPY pom.xml .
COPY src src
RUN mvn -f /workspace/app/pom.xml install
RUN mkdir -p target/dependency && (cd target/dependency; jar -xf ../*.jar)
FROM openjdk:17-oracle
VOLUME /tmp
ARG DEPENDENCY=/workspace/app/target/dependency
COPY --from=build ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY --from=build ${DEPENDENCY}/META-INF /app/META-INF
COPY --from=build ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-cp","app:app/lib/*","com.demo.DemoServiceApplication"]
Got it working with this, only needing to change the last line.
Docker Command:
docker build -t demo -f Dockerfile .
docker run -d -p 8080:8080 -e "SPRING_PROFILES_ACTIVE=develop" demo:latest
This results in a DockerFile that is not dependent on an external jar file and can be run in isolation against a repo for a cicd pipeline.
Source: Toptal - Multi-Stage Build
Following is my script to kick off my spark job
#!/bin/bash
APP_DIR="/home/arvind/myApp"
JARS=$(echo $APP_DIR/lib/* | tr ' ' ',')
/home/arvind/spark3/bin/spark-submit \
--master spark://server4:7078 \
--verbose \
--jars $JARS \
--driver-class-path $APP_DIR/conf \
--class com.test.spark.MySparkApplication \
--conf spark.driver.extraJavaOptions="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Dlog4j.configuration=file:$APP_DIR/conf/log4j.properties" \
--conf spark.driver.memory="32g" \
--conf spark.cores.max="20" \
$APP_DIR/myApp.jar > $APP_DIR/logs/output.err 2> $APP_DIR/logs/output.txt &
The lib folder contains all the jar files, packaged with the application, including postgres dependencies
[arvind#server4 myApp]$ ls lib/postgresql-42.1.4.jar
lib/postgresql-42.1.4.jar
When I run the application on a spark3 cluster, using the above script, I get the following exception
265 Caused by: java.lang.ClassNotFoundException: org.postgresql.ds.PGSimpleDataSource
266 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
267 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
268 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
269 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
270 at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:77)
271 ... 25 more
But if I change the above script to submit it to a spark2.4.0 cluster, it works fine.
Could you please help me with this ?
If you add the jar to the --driver-class-path like so it should work:
#!/bin/bash
APP_DIR="/home/arvind/myApp"
JARS=$(echo $APP_DIR/lib/* | tr ' ' ',')
COLON_SEP_JARS=$(echo $APP_DIR/lib/* | tr ' ' ':')
/home/arvind/spark3/bin/spark-submit \
--master spark://server4:7078 \
--verbose \
--jars $JARS \
--driver-class-path $APP_DIR/conf:$COLON_SEP_JARS \
--class com.test.spark.MySparkApplication \
--conf spark.driver.extraJavaOptions="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Dlog4j.configuration=file:$APP_DIR/conf/log4j.properties" \
--conf spark.driver.memory="32g" \
--conf spark.cores.max="20" \
$APP_DIR/myApp.jar > $APP_DIR/logs/output.err 2> $APP_DIR/logs/output.txt &
My project has a dependency on oracle jar to connect to it's database. Since it's a external jar, I downloaded it from oracle website and installed it manually using
mvn install:install-file -Dfile=ojdbc8.jar -DgroupId=com.oracle -DartifactId=ojdbc8 -Dversion=12.2.0.1 -Dpackaging=jar
running maven clean package on local runs fine, since the ojdbc.jar has been installed.
However, now I am moving the build step to Docker image.
from alpine/git as clone
workdir /app
run git clone --single-branch -b master https://github.com/xxxxxxxx
from maven:3.5-jdk-8-alpine
workdir /app
copy --from=0 /app/idot/idot-backend /app
run mvn clean install -T 1C -DskipTests
from openjdk:8-jre-alpine
workdir /app
copy --from=build /app/target/idot-0.0.1-SNAPSHOT.jar /app
cmd ["java -jar idot-0.0.1-SNAPSHOT.jar"]
This step fails obviously during build since the external jar is not present. So to remove the dependency, I edited the pom.xml to include
<repositories>
<repository>
<id>local-maven-repo</id>
<url>file://${project.basedir}/libs</url>
</repository>
</repositories>
and below dependency remains as before
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc8</artifactId>
<version>12.2.0.1</version>
</dependency>
and then I deploy the jar file using below command
mvn deploy:deploy-file -Dfile=ojdbc8.jar -DgroupId=com.oracle -DartifactId=ojdbc8 -Dversion=12.2.0.1 -Durl=file:./libs/ -DrepositoryId=local-maven-repo
This creates a libs folder with ojdbc8-12.2.0.1.jar and other meta information. I check this folder in github
Now while trying to build the docker image it fails with the below error message.
...
...
.
.
.
Downloaded from central: https://repo.maven.apache.org/maven2/org/liquibase/liquibase-parent/3.6.3/liquibase-parent-3.6.3.pom (31 kB at 39 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/com/oracle/ojdbc8/12.2.0.1/ojdbc8-12.2.0.1.pom
[WARNING] The POM for com.oracle:ojdbc8:jar:12.2.0.1 is missing, no dependency information available
Downloading from central: https://repo.maven.apache.org/maven2/org/modelmapper/modelmapper/2.3.2/modelmapper-2.3.2.pom
.
.
.
Downloaded from central: https://repo.maven.apache.org/maven2/org/atteo/evo-inflector/1.2.2/evo-inflector-1.2.2.jar (13 kB at 138 B/s)
Downloading from central: https://repo.maven.apache.org/maven2/com/oracle/ojdbc8/12.2.0.1/ojdbc8-12.2.0.1.jar
Downloading from central: https://repo.maven.apache.org/maven2/org/modelmapper/modelmapper/2.3.2/modelmapper-2.3.2.jar
.
.
.
Downloaded from central: https://repo.maven.apache.org/maven2/com/google/guava/guava/20.0/guava-20.0.jar (2.4 MB at 12 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/io/springfox/springfox-swagger-ui/2.9.2/springfox-swagger-ui-2.9.2.jar (2.9 MB at 14 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.9.12/byte-buddy-1.9.12.jar (3.3 MB at 14 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:13 min (Wall Clock)
[INFO] Finished at: 2020-06-17T11:19:56Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project idot: Could not resolve dependencies for project com.novartis:idot:jar:0.0.1-SNAPSHOT: Could not find artifact com.oracle:ojdbc8:jar:12.2.0.1 in local-maven-repo (file:///app/libs) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
The command '/bin/sh -c mvn clean install -T 1C -DskipTests' returned a non-zero code: 1
Instead of all the headaches, I added the same install command in docker file also. It now works fine.
from alpine/git as clone
workdir /app
run git clone --single-branch -b master https://#github.com/xxxx
from maven:3.5-jdk-8-alpine
workdir /app
copy --from=0 /app/idot/idot-backend /app
run mvn install:install-file -Dfile=/app/ojdbc8.jar -DgroupId=com.oracle -DartifactId=ojdbc8 -Dversion=12.2.0.1 -Dpackaging=jar
run mvn clean install -T 1C -DskipTests
from openjdk:8-jre-alpine
workdir /app
copy --from=build /app/target/idot-0.0.1-SNAPSHOT.jar /app
cmd ["java -jar idot-0.0.1-SNAPSHOT.jar"]
I used same approach, Its works for me. You can see here..
<dependencies>
<dependency>
<groupId>org.java-websocket</groupId>
<artifactId>Java-WebSocket</artifactId>
<version>1.4.2-SNAPSHOT</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>maven-repository</id>
<url>file:///${project.basedir}/repo</url>
</repository>
</repositories>
mvn deploy:deploy-file -Durl="file:\\\<project-path>\repo" -Dfile=Java-WebSocket-1.4.2-SNAPSHOT.jar -DgroupId=org.java-websocket -DartifactId=Java-WebSocket -Dpackaging=jar -Dversion=1.4.2-SNAPSHOT
I am using Google Cloud Build to build a java project which has 100s of dependencies. By default the local maven repository cache will be empty and it downloads all dependencies each time there is a build.
The google documentation only suggests "Caching directories with Google Cloud Storage" https://cloud.google.com/cloud-build/docs/speeding-up-builds but it takes a long time to sync 7000 files (which means the build is slower)
just one dependency is 5 files
repository/org/mockito
repository/org/mockito/mockito-core
repository/org/mockito/mockito-core/2.15.0
repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.jar
repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.jar.sha1
repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.pom
repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.pom.sha1
repository/org/mockito/mockito-core/2.15.0/_remote.repositories
An example cloudbuild.yaml file
steps:
- name: gcr.io/cloud-builders/gsutil
args: ['rsync', '-r', 'gs://my-mavencache-bucket/repository', '.']
- name: 'gcr.io/$PROJECT_ID/mvn'
args: ['package']
...
I would like to mount gs://my-mavencache-bucket at as a volume - but I dont see an option to do that
After much experimentation, this solution seems to work quite well. google-storage-wagon. This maven plugin provides a mechanism to read and publish maven artifacts from a google
data bucket
Maven pom.xml contains
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
...
<repositories>
<repository>
<id>my-repo-bucket-release</id>
<url>gs://bucket-ave-build-artifact/external</url>
<releases>
<enabled>true</enabled>
<!-- TODO figure out why checksums do not match when artifact pulled from GCP -->
<checksumPolicy>ignore</checksumPolicy>
</releases>
</repository>
</repositories>
<distributionManagement>
<snapshotRepository>
<id>my-repo-bucket-snapshot</id>
<url>gs://my-build-artifact-bucket/snapshot</url>
</snapshotRepository>
<repository>
<id>my-repo-bucket-release</id>
<url>gs://my-build-artifact-bucket/release</url>
</repository>
</distributionManagement>
...
<extensions>
<extension>
<groupId>com.gkatzioura.maven.cloud</groupId>
<artifactId>google-storage-wagon</artifactId>
<!-- version 1.8 seems to produce exception, ticket logged -->
<version>1.7</version>
</extension>
</extensions>
</build>
and cloudbuild.yaml is simply
steps:
- name: 'gcr.io/cloud-builders/mvn'
# -X here simply for verbose maven debugging
args: ['deploy', '-X']
this will:
maven publish artifacts to a data bucket
gs://my-build-artifact-bucket/release
download external dependencies
from gs://my-build-artifact-bucket/external (if they exist in this directory)
I found the package google-storage-wagon very nice, but lacking in terms of authentication and timing of the synchronization.
I implemented it myself like follows. For more information about service accounts refer to this answer: https://stackoverflow.com/a/56610260/1236401
So assuming you have your service account key.json file handy and you have the name of your SERVICE_ACCOUNT as well as a storage bucket BUCKET_PATH, this is the basic Dockerfile:
FROM maven:3.6.1-jdk-12
ENV MAVEN_PATH="/root/.m2" \
BUCKET_PATH="gs://mugen-cache/maven"
COPY key.json /key.json
# install gcloud sdk
RUN mkdir -p $MAVEN_PATH && \
yum install -y curl which && \
curl https://sdk.cloud.google.com | bash > /dev/null
ENV PATH="${PATH}:/root/google-cloud-sdk/bin" \
SERVICE_ACCOUNT="mugen-build#mugen.iam.gserviceaccount.com"
# authenticate service account and install crcmod - https://cloud.google.com/storage/docs/gsutil/addlhelp/CRC32CandInstallingcrcmod
RUN gcloud auth activate-service-account $SERVICE_ACCOUNT --key-file=/key.json && \
yum install -y gcc python-devel python-setuptools redhat-rpm-config
RUN curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" && \
python get-pip.py && \
pip uninstall crcmod && \
pip install --no-cache-dir -U crcmod
RUN echo "Syncing m2 in..." && \
gsutil -q -m rsync -r $BUCKET_PATH $MAVEN_PATH && \
echo "Downloaded $(find $MAVEN_BUCKET -type f -name "*.pom" | wc -l) packages"
# ... build and stuff
RUN echo "Syncing m2 out..." && \
gsutil -q -m rsync -r $MAVEN_PATH $BUCKET_PATH
Some of the instructions here are specific to the base image (which is the REHL-based Oracle Linux Server), but you should be able to extract the important details in order to make it work in your case.