I have a docker container which runs a springboot java application. Dockerfile:
# Create container with java preinstalled
FROM openjdk:8-jdk-alpine
# Create app directory
VOLUME /tmp
# Handle Arguments
ARG JAR_FILE
ARG ENV_NAME
ENV SPRING_PROFILES_ACTIVE=${ENV_NAME}
RUN echo ${ENV_NAME}
# Bundle app source
COPY ${JAR_FILE} app.jar
COPY application.yml application.yml
# Run the server
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-Dspring.config.location=application.yml","-jar","app.jar"]
Now, I have a custom library I need to install in that container. I'll need to copy the installation, extract it, run the install script and answer prompts (Y/n)
I understood the easiest way to do this is to connect to the container, install the package and commit the changes.
First - I start the container using:
docker run --name local-jdk8 -d openjdk:8-jdk-alpine
The next step is to copy the data and run the install script, but the container keeps on exiting since the run command is empty ("/bin/sh") which means I can't run
docker exec -it local-jdk8 bash
Any ideas on how I can modify such a container?
Solved it using expect library
My dockerfile :
# Create container with java preinstalled
FROM openjdk:8
# Create app directory
VOLUME /tmp
# Handle Arguments
ARG JAR_FILE
ARG ENV_NAME
ARG DRIVER_FILE
# Environment
ENV SPRING_PROFILES_ACTIVE=${ENV_NAME}
RUN echo ${ENV_NAME}
# Fingerprint Driver
RUN apt-get update -y
RUN apt-get install -y expect
COPY ${DRIVER_FILE} driver.tar.gz
COPY driver-install.exp driver-install.exp
RUN tar -xzf driver.tar.gz
RUN /driver-install.exp
# Copy app source
COPY ${JAR_FILE} app.jar
COPY application.yml application.yml
# Run the server
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-Dspring.config.location=application.yml","-jar","app.jar"]
driver-install.exp is the expect scripts that automatically interacts with the package installation
For what it's worth, here is a little trick that allows you to keep your container running to modify and commit it:
docker run --name local-jdk8 -d openjdk:8-jdk-alpine tail -f /dev/null
Furthermore, there is no bash installed on the container so sh will have to do:
docker exec -it local-jdk8 sh
Nevertheless, modifying Dockerfile is the better approach, since your change is persisted in code, rather than done on an potentially ephemeral container.
Related
I have written a small CLI using Java, Argparse4j, and packaged it in docker using this Dockerfile:
FROM openjdk:18
ENV JAR_NAME "my-jar-with-dependencies.jar"
ENV PROJECT_HOME /opt/app
RUN mkdir -p $PROJECT_HOME
WORKDIR $PROJECT_HOME
COPY run.sh $PROJECT_HOME/run.sh
RUN chmod +x $PROJECT_HOME/run.sh
COPY target/$JAR_NAME $PROJECT_HOME/cli.jar
ENTRYPOINT ["./run.sh"]
The last line of the Dockerfile then invokes a simple bash script:
#!/bin/bash
java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar ./cli.jar "$#"
The CLI I wrote has three main actions: upload, download and apply. Therefore argparse4j expects one of these actions to be passed as the first parameter, i.e.
java -jar cli.jar download #... whatever other argument
This works just fine when running the docker image locally, but completely fails when running in the CI pipeline:
download:
stage: download
image: <url>/my-image:<tag>
variables:
URL: <URL>
API_KEY: <API_KEY>
CI_DEBUG_TRACE: "true"
script:
- download -f zip -u true test-download.zip
This is the error that is returned:
Executing "step_script" stage of the job script 00:01
Using docker image sha256:<sha> for <url>/my-image:<tag> with digest <url>/my-image:<tag>#sha256:<sha> ...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
usage: tool [-h] ACTION ...
tool: error: invalid choice: 'sh' (choose from 'upload', 'download',
'apply')
I have tried following the suggestion in gitlab-runner doesn't run ENTRYPOINT scripts in Dockerfile but I can't seem to get the CI part to work correctly.
I would like to avoid using the entrypoint directive as it needs to be used on multiple files, so I rather fix the issue at the root.
Does anyone have an idea of what is happening or how to fix it?
I would like to avoid using the entrypoint directive as it needs to be used on multiple files, so I rather fix the issue at the root.
You can change your Dockerfile instead to keep default ENTRYPOINT (as openjdk:18 doesn't define any entrypoint, it will be empty):
FROM openjdk:18
# ...
# ENTRYPOINT ["./run.sh"] # remove this
# Add run.sh to path to be able to use `run.sh` from any directory
ENV PATH="${PATH}:/opt/app"
And update your run.sh to specify full path to jar:
#!/bin/bash
java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar $PROJECT_HOME/cli.jar "$#"
Now your container will start in Gitlab without having to specify entrypoint keyword for job. You can then setup something like this:
download:
stage: download
image: <url>/my-image:<tag>
variables:
URL: <URL>
API_KEY: <API_KEY>
CI_DEBUG_TRACE: "true"
script:
# Specify command using run.sh
# This command is run from within your container
# Note that script is not an argument passed on your image startup
# But independent commands run within your container using shell
- run.sh download -f zip -u true test-download.zip
Notes:
Gitlab won't run your script in Dockerfile's WORKDIR but in a dedicated directory where your project will be cloned.. Using ./ will look for script and jar in current directory at the moment your command is run, but they wouldn't be found if not run from /opt/app. Specyfing full path to jar and adding your run.sh script to PATH make sure they'll be found wherever your run.sh from. Alternatively you could run cd /opt/app in your job's script but it may cause unwanted side effects.
Without ENTRYPOINT you won't be able to run Docker commands like this
docker run " <url>/my-image:<tag>" download ...
You'll need to specify either COMMAND or --entrypoint such as
docker run "<url>/my-image:<tag>" run.sh download ...
docker run --entrypoint run.sh "<url>/my-image:<tag>" download ...
You specified not wanting to do this, but overriding image entrypoint on your job seems a much simpler and straightforward solution. Using multiple files you may leverage Gitlab's extends and include.
And now for the fun part
what is happening
When Gitlab run your container for a job it will use the entrypoint defined in your Dockerfile by default. From doc:
The runner starts a Docker container using the defined entrypoint. The default from Dockerfile that may be overridden in the
.gitlab-ci.yml file.
The runner attaches itself to a running container.
The runner prepares a script (the combination of before_script, script, and after_script).
The runner sends the script to the container’s shell stdin and receives the output.
And what the doc doesn't say is that Gitlab will try to use various form of sh as Docker command. In short for step 1. it's like running this Docker command:
# Gitlab tries to run container for your job
docker run -it "<url>/my-image:<tag>" sh
It doesn't work as Gitlab will use default entrypoint and the final command run in Docker is:
./run.sh sh
Where ./run.sh is the entrypoint from Dockerfile and sh is the command provided by Gitlab. It causes the error you see:
tool: error: invalid choice: 'sh' (choose from 'upload', 'download', 'apply')
You never reach your job's script (step 4). See ENTRYPOINT vs. CMD for details.
Furthermore, the script you define is a command itself. Even if your container started, it wouldn't work as the following command would be run inside your container:
download -f zip -u true test-download.zip
# 'download' command doesn't exists
# You probably want to run instead something like:
/opt/app/run.sh download -f zip -u true test-download.zip
So, after a bit of research, I have been able to find a solution that works for me.
From my research (and as Pierre B. pointed out in his answer), Gitlab essentially tries to inject a shell script that performs a check for which shell is available.
Now, my solution is in no way elegant, but does achieve what I wanted. I modified the Dockerfile like so:
FROM openjdk:18-bullseye
ENV JAR_NAME "my-jar.jar"
ENV PROJECT_HOME /opt/app
RUN mkdir -p $PROJECT_HOME
WORKDIR $PROJECT_HOME
COPY run.sh $PROJECT_HOME/run.sh
RUN chmod +x $PROJECT_HOME/run.sh
COPY target/$JAR_NAME $PROJECT_HOME/cli.jar
RUN echo '#!/bin/bash \njava $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar $PROJECT_HOME/cli.jar "$#"' > /usr/bin/i18n && \
chmod +x /usr/bin/i18n
ENTRYPOINT ["./run.sh"]
And also modified the run.sh script this way:
#!/bin/bash
if [[ -n "$CI" ]]; then
echo "this block will only execute in a CI environment"
exec /bin/bash
else
echo "Not in CI. Running the image normally"
java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar ./cli.jar "$#"
fi
This works because Gitlab, in its list of predefined variables, provides a CI env var that is set when the script is running on the CI. By doing so, I skip the java invocation but leave it in the case I need to use it when not on a CI.
Now when I need to use my image, all I need to specify in my .gitlab-ci.yml file is this:
download:
stage: download
image: <url>/my-image:<tag>
variables:
URL: <URL>
API_KEY: <API_KEY>
CI_DEBUG_TRACE: "true"
script:
- i18n download -f zip -u true test-download.zip # This calls the script that was injected in the Dockerfile
This way I essentially mimic an actual CLI, and can use it in all my projects that require this tool.
I am not sure though why I need to "echo" the script for the CLI, and I can't simply copy it. For some reason the env variables are not passed down and I couldn't spend any more time debugging it. So for now, it will stay like this.
If you have any recommendations on how to clean this up, please leave some comments and I will edit my answer!
Try to wrap your script in single quotes:
script:
- 'download -f zip -u true test-download.zip'
EDIT:
Oh, this open bug in gitlab could be relevant to you
I have Java app and want to generate docker image, I have shell script like this:
#!/bin/sh
java -version
export APPLICATION_DIR=$PWD
for rJarFile in `ls ${APPLICATION_DIR}/lib/*.jar`
do
export CLASSPATH=$rJarFile:$CLASSPATH
done
export CLASSPATH=$APPLICATION_DIR/classes:$CLASSPATH
java -Xverify:none -Xmx2048m -Djava.awt.headless=true -DFI_IS_CONFIGSER=N -DFICLIENT_APP_PATH=${APPLICATION_DIR} -DFI_APP_NAME=FIONLINE -DFI_BASE_INSTANCE_ID=1 -DPRODUCT_BOOTSTRAP_FILE=${APPLICATION_DIR}/data/BootstrapFile.properties -DFEBA_SYS_PATH=${APPLICATION_DIR}/data
And I try to convert it into Dockerfile like this
# FROM openjdk:8
FROM openjdk:11
RUN javac -version
# Create app directory
WORKDIR /usr/src/app
# Bundle app source
COPY . .
ENV APPLICATION_DIR=/usr/src/app
RUN echo $APPLICATION_DIR
RUN for rJarFile in `ls ${APPLICATION_DIR}/lib/*.jar`; do export CLASSPATH=$rJarFile:$CLASSPATH; done
RUN echo $CLASSPATH
ENV $CLASSPATH=$APPLICATION_DIR/classes:$CLASSPATH
# Run app
ENTRYPOINT ["java", "-Xverify:none", "-Xmx2048m", "-Djava.awt.headless=true", "-DFI_IS_CONFIGSER=N", "-DFICLIENT_APP_PATH=${APPLICATION_DIR} -DFI_APP_NAME=FIONLINE -DFI_BASE_INSTANCE_ID=1", "-DPRODUCT_BOOTSTRAP_FILE=${APPLICATION_DIR}/data/BootstrapFile.properties", "-DFEBA_SYS_PATH=${APPLICATION_DIR}/data"]
It can be generated, but there's an error when I try to run it like this:
Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: setenv: invalid argument: unknown
I've also changed this script RUN for rJarFile in `ls ${APPLICATION_DIR}/lib/*.jar`; do export into this RUN for rJarFile in ls ${APPLICATION_DIR}/lib/*.jar; do export CLASSPATH=$rJarFile:$CLASSPATH; done, but none of them working. I don't want to make Dockerfile execute the script. Below is logs when i generate and run it.
You cannot update the classpath as you do with:
ENV $CLASSPATH=$APPLICATION_DIR/classes:$CLASSPATH
instead you can do
ENV CLASSPATH=$APPLICATION_DIR/classes:$CLASSPATH
Also - please consider moving the script into a separate shell script and adding into the container. This would greatly simplify the Dockerfile, for example:
# FROM openjdk:8
FROM openjdk:11
RUN javac -version
# Create app directory
WORKDIR /usr/src/app
# Bundle app source
COPY . .
ENV APPLICATION_DIR=/usr/src/app
RUN echo $APPLICATION_DIR
ENTRYPOINT ["/usr/src/app/start_java.sh"]
and keep your existing script inside start_java.sh
I am trying to copy files from other location (in windows from D drive that is shared drive also) to my image.
But it is giving me error.
Step 5/8 : ADD ${APP_PATH}/${ARTIFACT}-${VERSION} /opt/tomcat/webapps/${ARTIFACT}-${VERSION}
ADD failed: stat /var/lib/docker/tmp/docker-builder990684261/d:/Application_Build/spring-tournament-portal-0.0.1-SNAPSHOT: no such file or directory
Scenario:
I am trying to make test,build,release phase cycle with the help of docker.
So My Multistage docker file's first version looked like this
FROM maven as build
WORKDIR /usr/local
RUN mkdir app
COPY . /usr/local/app/spring-tournament-portal/
WORKDIR /usr/local/app/spring-tournament-portal
RUN mvn -e package
FROM tomcat
ENV PORT 8080
ENV VERSION 0.0.1-SNAPSHOT
ENV ARTIFACT spring-tournament-portal
ENV JPDA_ADDRESS 8000
ENV JPDA_TRANSPORT dt_socket
COPY --from=build /usr/local/app/${ARTIFACT}/target/${ARTIFACT}-${VERSION} /usr/local/tomcat/webapps/${ARTIFACT}-${VERSION}
WORKDIR /usr/local/tomcat/bin
CMD ["catalina.sh jpda run"]
EXPOSE ${PORT}
But on every build maven fetches its dependency. So it was taking too much time.
So i made another approach. I created two docker files(One for build and other for release).
Over here the role of build docker file does is copy all source code into an image only. So that when i will create container, I could easily create volume for maven dependency to avoid redownloading:-
FROM indiver/tournament-base
RUN apt-get update && \
apt-get install -y \
-o APT::Install-Recommend=false -o APT::Install-Suggests=false \
maven
WORKDIR /usr/local
COPY . /spring-tournament-portal/
WORKDIR /spring-tournament-portal
COPY ./docker/dev/entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["sh", "/usr/local/bin/entrypoint.sh"]
RUN ["chmod", "+x", "/usr/local/bin/entrypoint.sh"]
Here is entrypoint.sh file
#!/bin/bash
echo "$#" Phase is executing.....
mvn clean "$#"
When i run the image after building docker file with the help of docker-compose.xml. I set volume for dependency folder.
Here is docker compose.xml
version: "2"
services:
test-service:
build:
context: ../../
dockerfile: ./docker/dev/Dockerfile
container_name: "test-service"
command: 'test'
volumes:
- .m2:/root/.m2
build-service:
build:
context: ../../
dockerfile: ./docker/dev/Dockerfile
container_name: "build-service"
command: 'package'
env_file:
- ./EnvironmentConstant.env
volumes:
- .m2:/root/.m2
database-service:
container_name: "database-service"
env_file:
- ./EnvironmentConstant.env
image: mysql
ports:
- "3306:3306"
volumes:
- ./dev-mysql:/var/lib/mysql
networks:
- app-network
networks:
app-network:
driver: bridge
So it does not need to redownload the dependency again that is in .m2 volume.
After that i need to create release image having content of build prepared by maven. But container of build service is already stopped. So i need to use docker cp command to get my prepared build.
docker cp build-service:/spring-tournament-portal/target/spring-tournament-portal-0.0.1-SNAPSHOT D:/Application_Build
Now i want build that is in Application_Build folder should be copied into my release image.
So my release docker file looks like
FROM indiver/tournament-release
ENV APP_PATH=d:/Application_Build
ENV ARTIFACT=spring-tournament-portal
ENV VERSION=0.0.1-SNAPSHOT
ADD ${APP_PATH}/${ARTIFACT}-${VERSION} /opt/tomcat/webapps/${ARTIFACT}-${VERSION}
WORKDIR /opt/tomcat/bin
ENTRYPOINT ["catalina.sh", "jpda", "run"]
EXPOSE ${PORT}
But As i have mentioned above it is giving me error
Step 5/8 : ADD ${APP_PATH}/${ARTIFACT}-${VERSION} /opt/tomcat/webapps/${ARTIFACT}-${VERSION} ADD failed: stat /var/lib/docker/tmp/docker-builder990684261/d:/Application_Build/spring-tournament-portal-0.0.1-SNAPSHOT: no such file or directory.
I tried to copy by COPY and ADD command. But nothing is working. How can i achieve this.
If It can be achieved this with the help of other relatively easy flow. It would be helpful as well.
The Dockerfile reference says:
ADD obeys the following rules:
The path must be inside the context of the build.
...
Overall your approach seems to be too complicated to me and a very non micro-services way of doing things.
I would suggest that you copy your dependencies into the binary that you are creating to create a self contained fat (uber) jar, which you can copy to the docker image, the way spring boot does.
You approach for having separate docker files for different environments is also problematic and could result in unexpected conditions in production.
So I too faced this similar problem .
You can try the below listed command and replace the Square brackets with your path:
"docker cp /[Your Source Directory] [ContainerId]:/[Destination Path]"
I have docker file
FROM java:8
# Install maven
RUN apt-get update
RUN apt-get install -y maven
WORKDIR /code/
# Prepare by downloading dependencies
#ADD pom.xml /mmt/CouchBaseClient/CB-RestAPI/CacheService/pom.xml
#RUN ["mvn", "dependency:resolve"]
#RUN ["mvn", "verify"]
ADD cacheService-0.0.1-SNAPSHOT.jar /code/cacheService-0.0.1-SNAPSHOT.jar
ADD couchclient-0.0.1-SNAPSHOT.jar /code/couchclient-0.0.1-SNAPSHOT.jar
EXPOSE 4567
CMD ["/usr/lib/jvm/java-8-openjdk-amd64/bin/java", "-jar", "couchclient-0.0.1-SNAPSHOT.jar server cacheService.yml" ]
When I build this file I get the following output
Sending build context to Docker daemon 35.46 MB
Step 1 : FROM java:8
---> 736600fd4ae5
Step 2 : RUN apt-get update
---> Using cache
---> a3466698c29d
Step 3 : RUN apt-get install -y maven
---> Using cache
---> d0fb8e77f89a
Step 4 : WORKDIR /code/
---> Using cache
---> 197735d2da02
Step 5 : ADD cacheService-0.0.1-SNAPSHOT.jar /code/cacheService-0.0.1-SNAPSHOT.jar
---> 9ba30f5a2144
Removing intermediate container bd3c072ebbc6
Step 6 : ADD couchclient-0.0.1-SNAPSHOT.jar /code/couchclient-0.0.1-SNAPSHOT.jar
---> ef59315ed7fe
Removing intermediate container 0da1a69bdb51
Step 7 : EXPOSE 4567
---> Running in a2b32799dd6c
---> 3fb2b534d7c5
Removing intermediate container a2b32799dd6c
Step 8 : CMD /usr/lib/jvm/java-8-openjdk-amd64/bin/java -jar couchclient-0.0.1-SNAPSHOT.jar server cacheService.yml
---> Running in efb44e2bcdb3
---> 56637dfacc0d
Removing intermediate container efb44e2bcdb3
Successfully built 56637dfacc0d
But no directory named code is being made and so no files are being added even though it is giving no compilation error
Used method suggested by #VonC
ENTRYPOINT ["/usr/lib/jvm/java-8-openjdk-amd64/bin/java", "-jar", "couchclient-0.0.1-SNAPSHOT.jar" ]
and then used this command to run the image
docker run <image> -d <arguments>
First, don't forget that ADD <src>... <dest> can invalidate the cache for all following instructions from the Dockerfile if the contents of <src> have changed. See "Best practices" and use COPY instead of ADD.
In both cases (ADD or COPY), if <dest> doesn’t exist, it is created along with all missing directories in its path.
So no mkdir necessary.
COPY cacheService-0.0.1-SNAPSHOT.jar /code/
COPY couchclient-0.0.1-SNAPSHOT.jar /code/
Otherwise, the file cacheService-0.0.1-SNAPSHOT.jar in the folder /code/cacheService-0.0.1-SNAPSHOT.jar/!
Finally, to be sure that the files are where they should be, open a bash:
docker run --rm -it <yourImage> bash
Or, if you have a running container:
docker exec -it <yourContainer> bash
And check what ls /code returns.
Also:
docker run --rm -it --entrypoint /bin/sh <yourImage> -c "ls -alrt"
The OP Legendary Hunter confirms in the comments the files are there.
The issue comes from the CMD which is not fully in exec form.
Each parameter should be in its own quotes:
CMD ["/usr/lib/jvm/java-8-openjdk-amd64/bin/java", "-jar", "couchclient-0.0.1-SNAPSHOT.jar", "server", "cacheService.yml" ]
If the last parameters are grouped together, CMD tries to access the jar file named "couchclient-0.0.1-SNAPSHOT.jar server cacheService.yml", which obvioulsy does not exist.
Hence the error message:
"Error: Unable to access jarfile couchclient-0.0.1-SNAPSHOT.jar server cacheService.yml"
Instead of using CMD, use ENTRYPOINT (with the sa me exec form, each arg in its own double-quotes), and leave CMD undefined.
That way, the arguments you will add to your docker run command will be passed to the ENTRYPOINT (which runs java -jar ...)
Since "server", "cacheService.yml" are the two arguments to be passed to the running container:
ENTRYPOINT ["/usr/lib/jvm/java-8-openjdk-amd64/bin/java", "-jar", "couchclient-0.0.1-SNAPSHOT.jar" ]
Build and then:
docker run --rm -it <image> server cacheService.yml
Once you know it is working, launch it in detached mode:
docker run -d <image> server cacheService.yml
Try this before WORKDIR line:
RUN mkdir /code
I have a java application (jar file) that I want to be able to run from a docker image.
I have created a Dockerfile to create an image using centos as base and install java as such:
Dockerfile
FROM centos
RUN yum install -y java-1.7.0-openjdk
I ran docker build -t me/java7 after to obtain the image me/java7
however I am stuck at some dead ends.
How do I copy the jar file from the host into the image/container
I require 2 parameters. 1 is a file, which needs to be copied into a directory into the container at runtime. The other is a number which needs to be passed to the jar file in the java -jar command automatically when the user runs docker run with the parameters
Extra Notes:
The jar file is a local file. Not hosted anywhere accessible via wget or anything. The closest I have at the moment is a windows share containing it. I could also access the source from a git repository but that would involve compiling everything and installing maven and git on the image so I'd rather avoid that.
any help is much appreciated.
In the Dockerfile, add a local file using ADD, e g
ADD your-local.jar /some-container-location
You could use volumes to put a file in the container in runtime, e g
VOLUME /copy-into-this-dir
And then you run using
docker run -v=/location/of/file/locally:/copy-into-this-dir -t me/java7
You can use ENTRYPOINT and CMD to pass arguments, e g
ENTRYPOINT ["java", "-jar", "/whatever/your.jar"]
CMD [""]
And then again run using
docker run -v=/location/of/file/locally:/copy-into-this-dir -t me/java7 --myNumber 42
(Have a look at the Dockerfile documentation.)
Suppose your file structure is as follow :
DockerTest
└── Dockerfile
└── local.jar
Dockerfile content will be :
FROM centos
RUN yum install -y java-1.7.0-openjdk
EXPOSE 8080
ADD /local.jar fatJar.jar
ENTRYPOINT ["java","-jar","fatJar.jar"]
Use following command :
$ cd DockerTest
$ docker build -f Dockerfile -t demo .