Develop Reference

gitlab-runner passes wrong arguments to custom image

I have written a small CLI using Java, Argparse4j, and packaged it in docker using this Dockerfile:
FROM openjdk:18
ENV JAR_NAME "my-jar-with-dependencies.jar"
ENV PROJECT_HOME /opt/app
RUN mkdir -p $PROJECT_HOME
WORKDIR $PROJECT_HOME
COPY run.sh $PROJECT_HOME/run.sh
RUN chmod +x $PROJECT_HOME/run.sh
COPY target/$JAR_NAME $PROJECT_HOME/cli.jar
ENTRYPOINT ["./run.sh"]
The last line of the Dockerfile then invokes a simple bash script:
#!/bin/bash
java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar ./cli.jar "$#"
The CLI I wrote has three main actions: upload, download and apply. Therefore argparse4j expects one of these actions to be passed as the first parameter, i.e.
java -jar cli.jar download #... whatever other argument
This works just fine when running the docker image locally, but completely fails when running in the CI pipeline:
download:
stage: download
image: <url>/my-image:<tag>
variables:
URL: <URL>
API_KEY: <API_KEY>
CI_DEBUG_TRACE: "true"
script:
- download -f zip -u true test-download.zip
This is the error that is returned:
Executing "step_script" stage of the job script 00:01
Using docker image sha256:<sha> for <url>/my-image:<tag> with digest <url>/my-image:<tag>#sha256:<sha> ...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
usage: tool [-h] ACTION ...
tool: error: invalid choice: 'sh' (choose from 'upload', 'download',
'apply')
I have tried following the suggestion in gitlab-runner doesn't run ENTRYPOINT scripts in Dockerfile but I can't seem to get the CI part to work correctly.
I would like to avoid using the entrypoint directive as it needs to be used on multiple files, so I rather fix the issue at the root.
Does anyone have an idea of what is happening or how to fix it?

I would like to avoid using the entrypoint directive as it needs to be used on multiple files, so I rather fix the issue at the root.
You can change your Dockerfile instead to keep default ENTRYPOINT (as openjdk:18 doesn't define any entrypoint, it will be empty):
FROM openjdk:18
# ...
# ENTRYPOINT ["./run.sh"] # remove this
# Add run.sh to path to be able to use `run.sh` from any directory
ENV PATH="${PATH}:/opt/app"
And update your run.sh to specify full path to jar:
#!/bin/bash
java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar $PROJECT_HOME/cli.jar "$#"
Now your container will start in Gitlab without having to specify entrypoint keyword for job. You can then setup something like this:
download:
stage: download
image: <url>/my-image:<tag>
variables:
URL: <URL>
API_KEY: <API_KEY>
CI_DEBUG_TRACE: "true"
script:
# Specify command using run.sh
# This command is run from within your container
# Note that script is not an argument passed on your image startup
# But independent commands run within your container using shell
- run.sh download -f zip -u true test-download.zip
Notes:
Gitlab won't run your script in Dockerfile's WORKDIR but in a dedicated directory where your project will be cloned.. Using ./ will look for script and jar in current directory at the moment your command is run, but they wouldn't be found if not run from /opt/app. Specyfing full path to jar and adding your run.sh script to PATH make sure they'll be found wherever your run.sh from. Alternatively you could run cd /opt/app in your job's script but it may cause unwanted side effects.
Without ENTRYPOINT you won't be able to run Docker commands like this
docker run " <url>/my-image:<tag>" download ...
You'll need to specify either COMMAND or --entrypoint such as
docker run "<url>/my-image:<tag>" run.sh download ...
docker run --entrypoint run.sh "<url>/my-image:<tag>" download ...
You specified not wanting to do this, but overriding image entrypoint on your job seems a much simpler and straightforward solution. Using multiple files you may leverage Gitlab's extends and include.
And now for the fun part
what is happening
When Gitlab run your container for a job it will use the entrypoint defined in your Dockerfile by default. From doc:
The runner starts a Docker container using the defined entrypoint. The default from Dockerfile that may be overridden in the
.gitlab-ci.yml file.
The runner attaches itself to a running container.
The runner prepares a script (the combination of before_script, script, and after_script).
The runner sends the script to the container’s shell stdin and receives the output.
And what the doc doesn't say is that Gitlab will try to use various form of sh as Docker command. In short for step 1. it's like running this Docker command:
# Gitlab tries to run container for your job
docker run -it "<url>/my-image:<tag>" sh
It doesn't work as Gitlab will use default entrypoint and the final command run in Docker is:
./run.sh sh
Where ./run.sh is the entrypoint from Dockerfile and sh is the command provided by Gitlab. It causes the error you see:
tool: error: invalid choice: 'sh' (choose from 'upload', 'download', 'apply')
You never reach your job's script (step 4). See ENTRYPOINT vs. CMD for details.
Furthermore, the script you define is a command itself. Even if your container started, it wouldn't work as the following command would be run inside your container:
download -f zip -u true test-download.zip
# 'download' command doesn't exists
# You probably want to run instead something like:
/opt/app/run.sh download -f zip -u true test-download.zip

So, after a bit of research, I have been able to find a solution that works for me.
From my research (and as Pierre B. pointed out in his answer), Gitlab essentially tries to inject a shell script that performs a check for which shell is available.
Now, my solution is in no way elegant, but does achieve what I wanted. I modified the Dockerfile like so:
FROM openjdk:18-bullseye
ENV JAR_NAME "my-jar.jar"
ENV PROJECT_HOME /opt/app
RUN mkdir -p $PROJECT_HOME
WORKDIR $PROJECT_HOME
COPY run.sh $PROJECT_HOME/run.sh
RUN chmod +x $PROJECT_HOME/run.sh
COPY target/$JAR_NAME $PROJECT_HOME/cli.jar
RUN echo '#!/bin/bash \njava $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar $PROJECT_HOME/cli.jar "$#"' > /usr/bin/i18n && \
chmod +x /usr/bin/i18n
ENTRYPOINT ["./run.sh"]
And also modified the run.sh script this way:
#!/bin/bash
if [[ -n "$CI" ]]; then
echo "this block will only execute in a CI environment"
exec /bin/bash
else
echo "Not in CI. Running the image normally"
java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar ./cli.jar "$#"
fi
This works because Gitlab, in its list of predefined variables, provides a CI env var that is set when the script is running on the CI. By doing so, I skip the java invocation but leave it in the case I need to use it when not on a CI.
Now when I need to use my image, all I need to specify in my .gitlab-ci.yml file is this:
download:
stage: download
image: <url>/my-image:<tag>
variables:
URL: <URL>
API_KEY: <API_KEY>
CI_DEBUG_TRACE: "true"
script:
- i18n download -f zip -u true test-download.zip # This calls the script that was injected in the Dockerfile
This way I essentially mimic an actual CLI, and can use it in all my projects that require this tool.
I am not sure though why I need to "echo" the script for the CLI, and I can't simply copy it. For some reason the env variables are not passed down and I couldn't spend any more time debugging it. So for now, it will stay like this.
If you have any recommendations on how to clean this up, please leave some comments and I will edit my answer!

Try to wrap your script in single quotes:
script:
- 'download -f zip -u true test-download.zip'
EDIT:
Oh, this open bug in gitlab could be relevant to you

Use cp in entrypoint for docker run

There is Dockerfile
FROM openjdk:11.0.12-jre-slim
COPY target/app.jar /app.jar
COPY configs configs
ENTRYPOINT ["java","-jar","/app.jar"]
In folder configs contains json configs for java application.
The build docker command is:
docker build --build-arg -f ~/IdeaProjects/app --no-cache -t app:latest
And the run command is:
docker run --entrypoint="cp configs var/opt/configs/ && java -jar app.jar" app:latest
Let's omit the ability to copy configs in the Dockerfile via COPY command. Unfortunately, this must be done using --entrypoint.
An error occurs when the docker run command was executed:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "cp configs var/opt/configs/ && java -jar app.jar": stat cp configs var/opt/configs/ && java -jar app.jar: no such file or directory: unknown.
Could you explain why the error occurred in this case?

I would do this with an entrypoint wrapper script. A Dockerfile can have both an ENTRYPOINT and a CMD; if you do, the CMD gets passed as arguments to the ENTRYPOINT. This means you can make the ENTRYPOINT a shell script that does first-time setup, then ends with exec "$#" to replace itself with the CMD.
#!/bin/sh
# docker-entrypoint.sh
# copy the configuration to the right place
cp configs var/opt/configs/
# run the main container command
exec "$#"
In the Dockerfile, make sure to COPY the script in (it should be checked in to source control as executable) and set it as the ENTRYPOINT.
...
COPY docker-entrypoint.sh .
ENTRYPOINT ["./docker-entrypoint.sh"] # must be JSON-array syntax
CMD ["java", "-jar", "/app.jar"] # what was previously ENTRYPOINT
When you run the container it's straightforward to replace the CMD, so you can double-check that this is doing the right thing by running an interactive shell in place of the java application.
docker run -v "$PWD/alt-configs:/configs" --rm -it my-image sh
If you do need to override the command like this at docker run time, the command you show uses && to run two commands consecutively. This needs to run a shell to be understood correctly, and in this context you need to manually provide a /bin/sh -c wrapper.
I would still recommend changing ENTRYPOINT to CMD in your Dockerfile; then you could run a relatively straightforward
docker run \
... \
-v "$PWD/alt-configs:/configs" \
my-image \
/bin/sh -c 'cp configs var/opt/configs && java -jar /app.jar'
If you use --entrypoint, it only takes the first word out of this command, and it is a Docker options so it needs to come before the image name. I'd recommend designing your image to avoid needing this awkward construct.
docker run \
... \
-v "$PWD/alt-configs:/configs" \
--entrypoint /bin/sh \
my-image \
-c 'cp configs var/opt/configs && java -jar /app.jar'
Your proposed command is having problems because it's trying to pass the entire command, including the embedded spaces and shell operators, as a single word, but that causes the OS-level process handling to try to look for an executable file with spaces and ampersands in the filename, hence the "no such file or directory" error.

Google App Engine: JAVA command not recognized by Python in Flex environment with Docker (`java` command is not found from this Python process)

I deployed a Python project on Google App Engine. As the project has a dependency on Java, I used a Docker container configuring two environments: Python + Java.
However, when I make a call to my Python service in GAE it's getting "java command is not found from this Python process. Please ensure Java is installed and PATH is set for java" error.
During the build process of the Docker file I am able to access Java after installing it. But during API execution it is not recognized by Python.
The "app.yaml" file used:
runtime: custom
env: flex
entrypoint: gunicorn -w 4 -k uvicorn.workers.UvicornWorker src.main:app
Below are the the Docker file used in Deploy:
### 1. Get Linux
FROM alpine:3.7
# Default to UTF-8 file.encoding
ENV LANG C.UTF-8
### 2. Get Java via the package manager
RUN apk update \
&& apk upgrade \
&& apk add --no-cache bash \
&& apk add --no-cache --virtual=build-dependencies unzip \
&& apk add --no-cache curl \
&& apk add --no-cache openjdk8-jre
#### OPTIONAL : 4. SET JAVA_HOME environment variable, uncomment the line below if you need it
ENV JAVA_HOME="/usr/lib/jvm/java-1.8-openjdk"
RUN export JAVA_HOME
ENV PATH $PATH:$JAVA_HOME/bin
RUN export PATH
RUN find / -name "java"
RUN java -version
FROM python:3.7
EXPOSE 8080
ENV APP_HOME /src
WORKDIR /src
COPY requirements.txt ./requirements.txt
RUN pip3 install -r requirements.txt
COPY . /src
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080"]
Here is a printout of the RUN java -version command during the deploy process:
RUN java -version command
Does anyone know why the error is happening even with Python and Java running on the same service on App Engine?
Are there any additional settings missing?

You are using two FROM in a Dockerfile, so you're effectively doing a multi-stage Docker build, but you are doing it wrong.
A multi-stage build should be something like this:
### 1. Get Linux
FROM alpine:3.7 as build
# here you install with apk and build your Java program
FROM python:3.7
EXPOSE 8080
...
# here you copy what you need from the previous Docker build, though
# since it was Java, which is not installed here anymore because it is
# a python image, you need to consider if you really need a multi-stage
# build
COPY --from=builder /src /src
Otherwise, just remove the second FROM python:3.7 and install it with apk.

Install a custom package in a docker container that keeps exiting

I have a docker container which runs a springboot java application. Dockerfile:
# Create container with java preinstalled
FROM openjdk:8-jdk-alpine
# Create app directory
VOLUME /tmp
# Handle Arguments
ARG JAR_FILE
ARG ENV_NAME
ENV SPRING_PROFILES_ACTIVE=${ENV_NAME}
RUN echo ${ENV_NAME}
# Bundle app source
COPY ${JAR_FILE} app.jar
COPY application.yml application.yml
# Run the server
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-Dspring.config.location=application.yml","-jar","app.jar"]
Now, I have a custom library I need to install in that container. I'll need to copy the installation, extract it, run the install script and answer prompts (Y/n)
I understood the easiest way to do this is to connect to the container, install the package and commit the changes.
First - I start the container using:
docker run --name local-jdk8 -d openjdk:8-jdk-alpine
The next step is to copy the data and run the install script, but the container keeps on exiting since the run command is empty ("/bin/sh") which means I can't run
docker exec -it local-jdk8 bash
Any ideas on how I can modify such a container?

Solved it using expect library
My dockerfile :
# Create container with java preinstalled
FROM openjdk:8
# Create app directory
VOLUME /tmp
# Handle Arguments
ARG JAR_FILE
ARG ENV_NAME
ARG DRIVER_FILE
# Environment
ENV SPRING_PROFILES_ACTIVE=${ENV_NAME}
RUN echo ${ENV_NAME}
# Fingerprint Driver
RUN apt-get update -y
RUN apt-get install -y expect
COPY ${DRIVER_FILE} driver.tar.gz
COPY driver-install.exp driver-install.exp
RUN tar -xzf driver.tar.gz
RUN /driver-install.exp
# Copy app source
COPY ${JAR_FILE} app.jar
COPY application.yml application.yml
# Run the server
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-Dspring.config.location=application.yml","-jar","app.jar"]
driver-install.exp is the expect scripts that automatically interacts with the package installation

For what it's worth, here is a little trick that allows you to keep your container running to modify and commit it:
docker run --name local-jdk8 -d openjdk:8-jdk-alpine tail -f /dev/null
Furthermore, there is no bash installed on the container so sh will have to do:
docker exec -it local-jdk8 sh
Nevertheless, modifying Dockerfile is the better approach, since your change is persisted in code, rather than done on an potentially ephemeral container.

How to pass System property to docker containers?

So I know you can pass Environment variables to a docker container using -e like:
docker run -it -e "var=var1" myDockerImage
But I need to pass a System Property to a docker container, because this is how I run my JAR:
java -Denvironment=dev -jar myjar.jar
So how can I pass a -D System property in Docker? Like:
docker run -it {INSERT Denvironment here} myDockerImage

Use the variable you passed into the container on the java command:
docker run -it -e "ENV=dev" myDockerImage
java -Denvironment=$ENV -jar myjar.jar

One more way to do it, if running under Tomcat, is setting your system variables in your Dockerfile using ENV JAVA_OPTS like this:
ENV JAVA_OPTS="-Djavax.net.ssl.trustStore=C:/tomcatDev.jks -D_WS_URL=http://some/url/"
Hope it helps!

One can also use the following start.sh ENTRYPOINT for the Docker container, make sure to use the array syntax, e.g.:
Dockerfile:
...
ENTRYPOINT ["/start.sh"]
The actual start.sh script:
#!/bin/bash
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"
exec $JAVA_HOME/bin/java -jar myjar.jar $#
Then you can just pass the Java system properties directly to your application as docker run container arguments:
docker run myDockerImage "-Dvar=var1"

Have a start.sh file, e.g.:
#!/usr/bin/env sh
exec java -Djava.security.egd=file:/dev/./urandom $* -jar /app.jar
In your Dockerfile:
...
COPY start.sh /start.sh
RUN chmod a+rx /start.sh
ENTRYPOINT ["/start.sh"]