Develop Reference

How to build binaries outside of Dockerfile for docker-compose build

I am new to docker-compose and probably don't understand the ideology behind.
Short question -- I see there is docker-compose build command. I'd like to trigger a (gradle) script outside of the Dockerfile file as part of running it.
Let's say that I have a java web service inside Docker that I build in advance by gradle.
I don't add gradle agent to the Dockerfile (as I am expected to keep it small, right?), I only COPY binaries
FROM openjdk:17-jdk-alpine
RUN apk add --no-cache bash
VOLUME /var/log/app/
WORKDIR /app
ARG EXTRACTED
COPY ${EXTRACTED}/application/ ./
...
ENTRYPOINT ["java","org.springframework.boot.loader.JarLauncher"]
And so I build this image by a script
./gradlew build
java -Djarmode=layertools -jar build/libs/*.jar extract --destination build/libs-extracted/
docker build --build-arg EXTRACTED=build/libs-extracted -t my-server .
I can define the following compose.yml. How do I trigger gradle inside of it? Or, same as with my single Dockerfile, I am expected to wrap docker-compose build into a build script?
version: '3.1'
services:
db:
image: postgres:alpine
restart: always
my-server:
image: my-server
restart: always
depends_on:
- db
Maybe I am asking for a hack, but actually I am happy to take another approach if it's cleaner

I discovered multi-stage Dockerfile feature that addresses my task, and I can use it for both individual Dockerfile builds and docker-compose builds.
https://stackoverflow.com/a/61131308/1291049.
I will probably lose gradle daemon optimisations though.

Change your docker compose file:
version: '3.1'
services:
db:
image: postgres:alpine
restart: always
my-server:
build:
context: ./my-server
dockerfile: Dockerfile
restart: always
depends_on:
- db

How to do a Gradle composite build in Docker?

I am able to do a Gradle composite build locally.
Now I'm trying to build the docker image in a docker container instead of on my local laptop. And I got this error:
Included build '/home/yang-liu/eCommerceServiceDataAccessLayer' does not exist.
So it complains that above package doesn't exist.
I'm not sure how to build docker image inside docker container when multiple packages are involved. My two packages are in different GitHub repositories.
More Context
Project structure
settings.gradle files
settings.gradle for eCommerceService:
rootProject.name = 'eCommerceService'
includeBuild("../eCommerceServiceDataAccessLayer")
settings.gradle for eCommerceServiceDataAccessLayer:
rootProject.name = 'eCommerceServiceDataAccessLayer'
Dockerfile in eCommerceService
########################################################################################
# We are multi-stage builds here to build the docker image.
# https://docs.docker.com/develop/develop-images/multistage-build/
########################################################################################
########################################################################################
# Build Stage
########################################################################################
FROM openjdk:11 AS BUILD_ARTIFACT
RUN ls
ARG APP_NAME=ecommerce-service
ARG USER_NAME=yang-liu
ENV APP_HOME=/home/$USER_NAME/$APP_NAME
WORKDIR $APP_HOME
COPY build.gradle settings.gradle gradlew $APP_HOME
COPY gradle $APP_HOME/gradle
# https://stackoverflow.com/questions/25873971/docker-cache-gradle-dependencies
RUN ./gradlew build || return 0
COPY . .
RUN ./gradlew build
########################################################################################
# Run Stage
########################################################################################
FROM openjdk:11-jre AS RUN_ARTIFACT
# Run as a non-root user to mitigate security risks
# https://security.stackexchange.com/questions/106860/can-a-root-user-inside-a-docker-lxc-break-the-security-of-the-whole-system
ARG GROUP_NAME=ecommerce-service
ARG USER_NAME=yang-liu
ARG APP_NAME=ecommerce-service
ARG USER_NAME=yang-liu
ENV APP_HOME=/home/$USER_NAME/$APP_NAME
RUN addgroup $GROUP_NAME
ENV APP_HOME=/home/$USER_NAME
WORKDIR $APP_HOME
RUN adduser --ingroup $GROUP_NAME $USER_NAME --home $APP_HOME
USER $USER_NAME
# Copy the artifact from BUILD_ARTIFACT stage
COPY --from=BUILD_ARTIFACT $APP_HOME/build/libs/eCommerceService-0.0.1.jar eCommerceService-0.0.1.jar
# Set ENTRYPOINT in exec form to run the container as an executable
ENTRYPOINT ["java","-classpath", "-jar","eCommerceService-0.0.1.jar"]

docker error with multiple backend image install

Dockerfile
FROM python:3.6
ENV PYTHONUNBUFFERED 1
WORKDIR /usr/src/rango
COPY ./ /usr/src/rango
RUN pip install -r requirements.txt
here is my docker-compose file
services:
backend:
container_name: backend
build: ./
command: python manage.py runserver 0.0.0.0:8000
working_dir: /usr/src/rango
ports:
- "8000:8000"
tty: true
links:
- java
- elasticsearch
- node
#java
java:
image: openjdk:9-jre
#elastic search
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.0
ports:
- "9200:9200"
node:
image: node:10.13.0
Command i am using :
sudo docker-compose up
when i am running command i am getting error as
backend_node_1_26e7640d2fbb exited with code 0
backend_java_1_b1fbf7e151d7 exited with code 0
both node and java are not running .
i am using elastic search so i need java
plaese have look into my screenshot i have shared below.

A dockers images are self-reliant in terms of the language runtime that they run on, meaning that they include everything that's needed to run the particular process (excluding external dependencies, such as database or other services).
Therefore, ElasticSearch images does not require a Java container, and similarly the Node container is not needed. They are exiting with 0 exit status (indicating that they run successfully to their completion) as you haven't specified a command to execute (and nor is there a default one defined in the base image).
In summary, you can remove the java and node services from your compose file.

How to copy files from other location into Docker image

I am trying to copy files from other location (in windows from D drive that is shared drive also) to my image.
But it is giving me error.
Step 5/8 : ADD ${APP_PATH}/${ARTIFACT}-${VERSION} /opt/tomcat/webapps/${ARTIFACT}-${VERSION}
ADD failed: stat /var/lib/docker/tmp/docker-builder990684261/d:/Application_Build/spring-tournament-portal-0.0.1-SNAPSHOT: no such file or directory
Scenario:
I am trying to make test,build,release phase cycle with the help of docker.
So My Multistage docker file's first version looked like this
FROM maven as build
WORKDIR /usr/local
RUN mkdir app
COPY . /usr/local/app/spring-tournament-portal/
WORKDIR /usr/local/app/spring-tournament-portal
RUN mvn -e package
FROM tomcat
ENV PORT 8080
ENV VERSION 0.0.1-SNAPSHOT
ENV ARTIFACT spring-tournament-portal
ENV JPDA_ADDRESS 8000
ENV JPDA_TRANSPORT dt_socket
COPY --from=build /usr/local/app/${ARTIFACT}/target/${ARTIFACT}-${VERSION} /usr/local/tomcat/webapps/${ARTIFACT}-${VERSION}
WORKDIR /usr/local/tomcat/bin
CMD ["catalina.sh jpda run"]
EXPOSE ${PORT}
But on every build maven fetches its dependency. So it was taking too much time.
So i made another approach. I created two docker files(One for build and other for release).
Over here the role of build docker file does is copy all source code into an image only. So that when i will create container, I could easily create volume for maven dependency to avoid redownloading:-
FROM indiver/tournament-base
RUN apt-get update && \
apt-get install -y \
-o APT::Install-Recommend=false -o APT::Install-Suggests=false \
maven
WORKDIR /usr/local
COPY . /spring-tournament-portal/
WORKDIR /spring-tournament-portal
COPY ./docker/dev/entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["sh", "/usr/local/bin/entrypoint.sh"]
RUN ["chmod", "+x", "/usr/local/bin/entrypoint.sh"]
Here is entrypoint.sh file
#!/bin/bash
echo "$#" Phase is executing.....
mvn clean "$#"
When i run the image after building docker file with the help of docker-compose.xml. I set volume for dependency folder.
Here is docker compose.xml
version: "2"
services:
test-service:
build:
context: ../../
dockerfile: ./docker/dev/Dockerfile
container_name: "test-service"
command: 'test'
volumes:
- .m2:/root/.m2
build-service:
build:
context: ../../
dockerfile: ./docker/dev/Dockerfile
container_name: "build-service"
command: 'package'
env_file:
- ./EnvironmentConstant.env
volumes:
- .m2:/root/.m2
database-service:
container_name: "database-service"
env_file:
- ./EnvironmentConstant.env
image: mysql
ports:
- "3306:3306"
volumes:
- ./dev-mysql:/var/lib/mysql
networks:
- app-network
networks:
app-network:
driver: bridge
So it does not need to redownload the dependency again that is in .m2 volume.
After that i need to create release image having content of build prepared by maven. But container of build service is already stopped. So i need to use docker cp command to get my prepared build.
docker cp build-service:/spring-tournament-portal/target/spring-tournament-portal-0.0.1-SNAPSHOT D:/Application_Build
Now i want build that is in Application_Build folder should be copied into my release image.
So my release docker file looks like
FROM indiver/tournament-release
ENV APP_PATH=d:/Application_Build
ENV ARTIFACT=spring-tournament-portal
ENV VERSION=0.0.1-SNAPSHOT
ADD ${APP_PATH}/${ARTIFACT}-${VERSION} /opt/tomcat/webapps/${ARTIFACT}-${VERSION}
WORKDIR /opt/tomcat/bin
ENTRYPOINT ["catalina.sh", "jpda", "run"]
EXPOSE ${PORT}
But As i have mentioned above it is giving me error
Step 5/8 : ADD ${APP_PATH}/${ARTIFACT}-${VERSION} /opt/tomcat/webapps/${ARTIFACT}-${VERSION} ADD failed: stat /var/lib/docker/tmp/docker-builder990684261/d:/Application_Build/spring-tournament-portal-0.0.1-SNAPSHOT: no such file or directory.
I tried to copy by COPY and ADD command. But nothing is working. How can i achieve this.
If It can be achieved this with the help of other relatively easy flow. It would be helpful as well.

The Dockerfile reference says:
ADD obeys the following rules:
The path must be inside the context of the build.
...

Overall your approach seems to be too complicated to me and a very non micro-services way of doing things.
I would suggest that you copy your dependencies into the binary that you are creating to create a self contained fat (uber) jar, which you can copy to the docker image, the way spring boot does.
You approach for having separate docker files for different environments is also problematic and could result in unexpected conditions in production.

So I too faced this similar problem .
You can try the below listed command and replace the Square brackets with your path:
"docker cp /[Your Source Directory] [ContainerId]:/[Destination Path]"

Docker cache gradle dependencies

I'm trying to deploy our java web application to aws elastic beanstalk using docker, the idea is to be able to run the container locally for development and testing and eventually push it up to production using git.
I've created a base image that has tomcat8 and java8 installed, the image that performs the gradle builds inherit from this base image, speeding up build process.
All works well, except for the fact that the inheriting application container that gets built using docker doesn't seem to cache the gradle dependencies, it downloads it every time, including gradlew. We build our web application using the following command:
./gradlew war
Is there some way that i can cache the files in ~/.gradle this would speed my build up dramatically.
This isn't so much of an issue on beanstalk but is a big problem for devs trying to build and run locally as this does take a lot of time, as you can imagine.
The base image dockerfile:
FROM phusion/baseimage
EXPOSE 8080
RUN apt-get update
RUN add-apt-repository ppa:webupd8team/java
RUN apt-get update
RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections
RUN apt-get -y install oracle-java8-installer
RUN java -version
ENV TOMCAT_VERSION 8.0.9
RUN wget --quiet --no-cookies http://archive.apache.org/dist/tomcat/tomcat-8/v${TOMCAT_VERSION}/bin/apache-tomcat-${TOMCAT_VERSION}.tar.gz -O /tmp/catalina.tar.gz
# Unpack
RUN tar xzf /tmp/catalina.tar.gz -C /opt
RUN mv /opt/apache-tomcat-${TOMCAT_VERSION} /opt/tomcat
RUN ln -s /opt/tomcat/logs /var/log/tomcat
RUN rm /tmp/catalina.tar.gz
# Remove unneeded apps
RUN rm -rf /opt/tomcat/webapps/examples
RUN rm -rf /opt/tomcat/webapps/docs
RUN rm -rf /opt/tomcat/webapps/ROOT
ENV CATALINA_HOME /opt/tomcat
ENV PATH $PATH:$CATALINA_HOME/bin
ENV CATALINA_OPTS $PARAM1
# Start Tomcat
CMD ["/opt/tomcat/bin/catalina.sh", "run"]
The application dockerfile:
FROM <tag name here for base image>
RUN mkdir ~/.gradle
# run some extra stuff here to add things to gradle.properties file
# Add project Source
ADD . /var/app/myapp
# Compile and Deploy Application, this is what is downloading gradlew and all the maven dependencies every time, if only there was a way to take the changes it makes to ~/.gradle and persist it as a cache layer
RUN cd /var/app/myapp/ && ./gradlew war
RUN mv /var/app/myapp/build/libs/myapp.war /opt/tomcat/webapps/ROOT.war
# Start Tomcat
CMD ["/opt/tomcat/bin/catalina.sh", "run"]

I faced this issue. As you might agree, it is a best practice to download dependencies alone as a separate step while building the docker image. It becomes little tricky with gradle, since there is no direct support for downloading just dependencies.
Option 1 : Using docker-gradle Docker image
We can use pre-built gradle docker image to build the application. This ensures that it's not a local system build but a build done on a clean docker image.
docker volume create --name gradle-cache
docker run --rm -v gradle-cache:/home/gradle/.gradle -v "$PWD":/home/gradle/project -w /home/gradle/project gradle:4.7.0-jdk8-alpine gradle build
ls -ltrh ./build/libs
gradle cache is loaded here as a volume. So subsequent builds will reuse the downloaded dependencies.
After this, we could have a Dockerfile to take this artifact and generate application specific image to run the application.
This way, the builder image is not required. Application build flow and Application run flow is separated out.
Since the gradle-cache volume is mounted, we could reuse the downloaded dependencies across different gradle projects.
Option 2 : Multi-stage build
----- Dockerfile -----
FROM openjdk:8 AS TEMP_BUILD_IMAGE
ENV APP_HOME=/usr/app/
WORKDIR $APP_HOME
COPY build.gradle settings.gradle gradlew $APP_HOME
COPY gradle $APP_HOME/gradle
RUN ./gradlew build || return 0
COPY . .
RUN ./gradlew build
FROM openjdk:8
ENV ARTIFACT_NAME=your-application.jar
ENV APP_HOME=/usr/app/
WORKDIR $APP_HOME
COPY --from=TEMP_BUILD_IMAGE $APP_HOME/build/libs/$ARTIFACT_NAME .
EXPOSE 8080
CMD ["java","-jar",$ARTIFACT_NAME]
In the above Dockerfile
First we try to copy the project's gradle files alone, like
build.gradle, gradlew etc.,
Then we copy the gradle directory itself
And then we try to run the build. At this point, there is no other
source code files exists in the directory. So build will fail. But
before that it will download the dependencies. 
Since we expect the
build to fail, I have tried a simple technique to return 0 and allow
the docker to continue execution
this will speed up the subsequent build flows, since all the dependencies are downloaded and docker cached this layer. Comparatively, Volume mounting the gradle cache directory is still the best approach.
The above example also showcases multi-stage docker image building, which avoid multiple docker build files.

I
Add resolveDependencies task in build.gradle:
task resolveDependencies {
doLast {
project.rootProject.allprojects.each { subProject ->
subProject.buildscript.configurations.each { configuration ->
configuration.resolve()
}
subProject.configurations.each { configuration ->
configuration.resolve()
}
}
}
}
and update Dockerfile:
ADD build.gradle /opt/app/
WORKDIR /opt/app
RUN gradle resolveDependencies
ADD . .
RUN gradle build -x test --parallel && \
touch build/libs/api.jar
II
Bellow is what I do now:
build.gradle
ext {
speed = project.hasProperty('speed') ? project.getProperty('speed') : false
offlineCompile = new File("$buildDir/output/lib")
}
dependencies {
if (speed) {
compile fileTree(dir: offlineCompile, include: '*.jar')
} else {
// ...dependencies
}
}
task downloadRepos(type: Copy) {
from configurations.all
into offlineCompile
}
Dockerfile
ADD build.gradle /opt/app/
WORKDIR /opt/app
RUN gradle downloadRepos
ADD . /opt/app
RUN gradle build -Pspeed=true

You might want to consider splitting your application image to two images: one for building the myapp.war and the other for running your application. That way, you can use docker volumes during the actual build and bind the host's ~/.gradle folder into the container performing the build. Instead of only one step to run your application, you would have more steps, though. Example:
builder image
FROM <tag name here for base image including all build time dependencies>
# Add project Source
# -> you can use a project specific gradle.properties in your project root
# in order to override global/user gradle.properties
ADD . /var/app/myapp
RUN mkdir -p /root/.gradle
ENV HOME /root
# declare shared volume path
VOLUME /root/.gradle
WORKDIR /var/app/myapp/
# Compile only
CMD ["./gradlew", "war"]
application image
FROM <tag name here for application base image>
ADD ./ROOT.war /opt/tomcat/webapps/ROOT.war
# Start Tomcat
CMD ["/opt/tomcat/bin/catalina.sh", "run"]
How to use in your project root, assuming the builder Dockerfile is located there and the application Dockerfile is located at the webapp subfolder (or any other path you prefer):
$ docker build -t builder .
$ docker run --name=build-result -v ~/.gradle/:/root/.gradle/ builder
$ docker cp build-result:/var/app/myapp/myapp.war webapp/ROOT.war
$ cd webapp
$ docker build -t application .
$ docker run -d -P application
I haven't tested the shown code, but I hope you get the idea. The example might even be improved by using data volumes for the .gradle/ cache, see the Docker user guide for details.

The current version of Docker supports mounting a "cache" and it's local to the Docker environment (so it's not shared with your OS which is both good and bad; good in that there's nothing about your system in the build process, bad in that you have to download again)
This code is from my Spring Docker Swarm integration rework
FROM gradle:7.4-jdk17 AS builder
WORKDIR /w
COPY ./ /w
RUN --mount=type=cache,target=/home/gradle/.gradle/caches gradle build --no-daemon -x test
FROM openjdk:17-jdk as extractor
WORKDIR /w
COPY bin/extract.sh /w/extract.sh
COPY --from=builder /w/*/build/libs/*.jar /w/
RUN sh ./extract.sh
FROM openjdk:17-jdk as sample-service
WORKDIR /w
COPY --from=extractor /w/sample-service/* /w/
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=80", "org.springframework.boot.loader.JarLauncher"]
HEALTHCHECK --interval=5s --start-period=60s \
CMD curl -sfo /dev/null http://localhost:8080/actuator/health
USER 5000
EXPOSE 8080
What this does is from my current folder which is a multi-module gradle build I run the build. extractor stage unbundles the JAR file using extract.sh script below.
Then assembles the relevant component
The relevant contents of extract.sh
#!/bin/sh
set -e
set -x
# Remove support projects that won't be a Spring Boot
# rm buildSrc.jar
# rm gateway-common-*.jar
for jar in *.jar
do
DIR=$(basename $jar -0.0.1-SNAPSHOT.jar)
mkdir $DIR
java -Djarmode=layertools -jar $jar extract --destination $DIR
done

try changing the gradle user home directory
RUN mkdir -p /opt/gradle/.gradle
ENV GRADLE_USER_HOME=/opt/gradle/.gradle