How to install java in an airflow container using docker-compose.yaml - java

I am using this docker-compose.yaml file to run airflow on docker container.
https://airflow.apache.org/docs/apache-airflow/2.0.2/docker-compose.yaml
I need to install JRE in one of the containers. How do I add instruction to add java to the docker-compose.yaml file?

Try the following:
Create the following Dockerfile in the directory where you also have the docker-compose.yml:
FROM apache/airflow:2.0.2
USER root
# Install OpenJDK-11
RUN apt update && \
apt-get install -y openjdk-11-jdk && \
apt-get install -y ant && \
apt-get clean;
# Set JAVA_HOME
ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/
RUN export JAVA_HOME
USER airflow
WORKDIR /app
COPY requirements.txt /app
RUN pip install --trusted-host pypi.python.org -r requirements.txt
In order to install packages via apt, you have to switch to root user. Later, we switch back to user airflow. The specification of the working directory and the installation of python packages through requirements.txt might not apply, depending on your case.
Then, in your docker-compose.yml, add build: . after &airflow-common.
Finally, build your pipeline using docker-compose up -d --build.
For more information, look here: https://airflow.apache.org/docs/docker-stack/build.html#building-the-image

You have to build your own airflow image.
In your Dockerfile you have to apt install the airflow and export JAVA_HOME.
Dockerfile example:
FROM apache/airflow:2.5.1
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
openjdk-11-jre-headless \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
USER airflow
ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
RUN pip install --no-cache-dir apache-airflow-providers-apache-spark==2.1.3
Build command example:
docker build -t airflow-with-java .
After successfully image building edit your docker-compose by replacing airflow image:
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-airflow-with-java}
More information you can look here: https://airflow.apache.org/docs/docker-stack/build.html

Related

The command `bin/sh` returned a non-zero code: 100

So I am trying to install OpenJDK in a Dockerfile but I am having issues. It always errors with the following message: Sub-process /usr/bin/dpkg returned an error code (1) and then underneath The command bin/sh returned a non-zero code: 100. This is the command that failed to execute. Currently on Ubuntu 20.04 VM
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build-env
WORKDIR /app
# Copy csproj and restore as distinct layers
COPY Folder/*.csproj ./
RUN dotnet restore
# Copy everything else and build
COPY . ./
RUN dotnet build -c Release -o out
# Build runtime image
FROM mcr.microsoft.com/dotnet/runtime:5.0
# Install OpenJDK-14
RUN apt-get update && \
apt-get install -y default-jdk && \
apt-get install -y ant && \
apt-get clean;
# Fix certificate issues
RUN apt-get update && \
apt-get install ca-certificates-java && \
apt-get clean && \
update-ca-certificates -f;
# Setup JAVA_HOME -- useful for docker commandline
ENV JAVA_HOME /usr/lib/jvm/default/
RUN export JAVA_HOME
RUN apt-get install -y supervisor # Installing supervisord
ADD supervisord.conf /etc/supervisor/conf.d/supervisord.conf
WORKDIR /app
COPY Folder/Lavalink/* ./
COPY --from=build-env /app/out .
#ENTRYPOINT ["dotnet", "Application.dll"]
ENTRYPOINT ["/usr/bin/supervisord"]
Here is the supervisord as well
[supervisord]
nodaemon=true
[program:folder]
command=dotnet /app/Application.dll
[program:lavalink]
command=java -jar /app/Lavalink.jar
This is a Visual Studio project written in 5.0 with a .jar file that needs to be executed.
These didn't seem to help:
apt-get update' returned a non-zero code: 100, Docker File Non-Zero Code 100 Error When Building Basically what I am trying to achieve is to install java within a container. Preferably java 13 but this issue prevents me from doing so. Last, it is important to let you know that the same commands works on another container.
Add this before installing the jdk :
RUN mkdir -p /usr/share/man/man1/
This is a problem in the debian slim images and this image is based on buster-slim. Alternatively you can try to use one of the dotnet/runtime images based on Ubuntu (5.0-focal) or Alpine (5.0-alpine).

How to install java 8 using dockerfile in python:3.8-slim-buster base image

Below is my Dockerfile
FROM python:3.8-slim-buster
WORKDIR /app
RUN python --version
RUN apt-get install java-1.8.0-openjdk-devel
RUN python -m pip install --upgrade pip
RUN pip install --default-timeout=100 pyspark
I want to install java 8 and set JAVA_HOME variables. But when I am trying to build above image I am getting below error:
E: Unable to locate package java-1.8.0-openjdk-devel
E: Couldn't find any package by glob 'java-1.8.0-openjdk-devel'
E: Couldn't find any package by regex 'java-1.8.0-openjdk-devel'
This is my first attempt in creating a docker image. Please suggest what is wrong with above Dockerfile. I am working on centos7.
Another approach is that you can build your Dockerfile based on FROM ubuntu:20.04 where Python 3.8 is set as default (here). Then, install java and pip later.
FROM ubuntu:20.04
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN apt-get update -y \
&& apt-get install -y software-properties-common \
&& add-apt-repository ppa:deadsnakes/ppa \
&& apt-get install openjdk-8-jdk -y \
&& apt-get install python3-pip -y \
&& export JAVA_HOME \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

Error in JDK installation in ubuntu docker

I have a docker file like below.
FROM ubuntu
FROM python:3.6
RUN apt-get update --fix-missing
RUN apt-get install wget curl software-properties-common -y
RUN apt-get install g++ gcc mercurial -y
RUN apt-get update && \
apt-get install -y openjdk-8-jdk && \
apt-get install -y ant && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /var/cache/oracle-jdk8-installer;
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN export JAVA_HOME
When i try to build the docker using sudo docker build -t test_dock . command, there is an error saying
Unable to locate jdk package
So i have added
RUN add-apt-repository ppa:openjdk-r/ppa
before jdk installation command.
Now the build errors out saying
E: The repository 'http://ppa.launchpad.net/openjdk-r/ppa/ubuntu focal
Release' does not have a Release file.
What is the correct way to install jdk in ubuntu docker?
For me solved by replacing :
FROM ubuntu
FROM python:3.6
To:
FROM python:3-stretch

Docker – Compile maven project inside docker container & run it from there

I have a project running Vue & Spring Boot that I need to create a docker-compose.yml file to run mvn clean install to generate the .jar, and then build a "new" image from another Dockerfile with that said .jar inside the docker container.
This is the Dockerfile that needs to be run once the mvn clean install is completed:
FROM java:8
ENV WKHTML_VERSION 0.12.4
# Builds the wkhtmltopdf download URL based on version numbers above
ENV DOWNLOAD_URL "https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/${WKHTML_VERSION}/wkhtmltox-${WKHTML_VERSION}_linux-generic-amd64.tar.xz"
RUN apt-get update && \
apt-get install -y --no-install-recommends wget && \
wget $DOWNLOAD_URL && \
tar vxf wkhtmltox-${WKHTML_VERSION}_linux-generic-amd64.tar.xz && \
cp wkhtmltox/bin/wk* /usr/local/bin/ && \
cp wkhtmltox/lib/* /usr/local/lib/ && \
rm wkhtmltox-${WKHTML_VERSION}_linux-generic-amd64.tar.xz
# #see https://spring.io/guides/gs/spring-boot-docker/
COPY server/target/redo-server-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","app.jar"]
So the build steps need to be something like this:
Install node + maven / pull those images
Install postgresql & pull that image
Run mvn clean install & generate .jar
Build new image from abovementioned Dockerfile and run it
I am new to docker-compose so I am having troubles setting this up in the correct execution order.
The reason I need to do this is due to a problem with the production pipeline not having node or npm, which is needed to run the full maven application (Vue.js and Spring Boot app), which is why it needs to be compiled from inside the Docker container
It would be greatly appreciated if anyone could point me in the right direction, let alone – is this possible to do?
Solved by writing a multi-step build as my Dockerfile. I am installing node as a dependency in the client's pom.xml file.
# Install maven and copy project for compilation
FROM maven:latest as builder
COPY pom.xml /usr/local/pom.xml
COPY server /usr/local/server
COPY client /usr/local/client
WORKDIR /usr/local/
RUN mvn clean install
FROM openjdk:8
ENV WKHTML_VERSION 0.12.4
# Builds the wkhtmltopdf download URL based on version numbers above
ENV DOWNLOAD_URL "https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/${WKHTML_VERSION}/wkhtmltox-${WKHTML_VERSION}_linux-generic-amd64.tar.xz"
RUN apt-get update && \
apt-get install -y --no-install-recommends wget && \
wget $DOWNLOAD_URL && \
tar vxf wkhtmltox-${WKHTML_VERSION}_linux-generic-amd64.tar.xz && \
cp wkhtmltox/bin/wk* /usr/local/bin/ && \
cp wkhtmltox/lib/* /usr/local/lib/ && \
rm wkhtmltox-${WKHTML_VERSION}_linux-generic-amd64.tar.xz
COPY --from=builder /usr/local/redo/server/target/server-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar","app.jar"]

Create Docker container with both Java and Node.js

I am not sure why I expected this to work:
# Dockerfile
FROM node:6
FROM java:8
but it doesn't really work - looks like the first command is ignored, and second command works.
Is there a straightforward way to install both Node.js and Java in a Docker container?
Ultimately the problem I am trying to solve is that I am getting an ENOENT error when running Selenium Webdriver -
[20:38:50] W/start - Selenium Standalone server encountered an error: Error: spawn java ENOENT
And right now I assume it's because Java is not installed in the container.
The best way for you is to take java (which is officially deprecated and it suggests you use openjdk image) and install node in it.
So, start with
FROM openjdk:latest
This will use the latest openjdk image, which is 8u151 at this time. Then install node and other dependencies you might need:
RUN apt-get install -y curl \
&& curl -sL https://deb.nodesource.com/setup_9.x | bash - \
&& apt-get install -y nodejs \
&& curl -L https://www.npmjs.com/install.sh | sh
You might want to install things like grunt afterwards, so this might come in handy as well.
RUN npm install -g grunt grunt-cli
In total you will get the following Dockerfile:
FROM openjdk:latest
RUN apt-get install -y curl \
&& curl -sL https://deb.nodesource.com/setup_9.x | bash - \
&& apt-get install -y nodejs \
&& curl -L https://www.npmjs.com/install.sh | sh \
RUN npm install -g grunt grunt-cli
You may clone the Dockerfile from my gitlab repo here
You can use single FROM per generated image.
Try to use node as a base image and install java to it.
Dockerfile
FROM node:latest
RUN apt-get -y install default-jre
You can choose the version you need:
apt install default-jre
apt install openjdk-11-jre-headless
apt install openjdk-8-jre-headless
You can also use the node image and then install the default-jre:
# Dockerfile
FROM node:latest
RUN apt-get -y install default-jre
You can choose the version you need:
apt install default-jre
apt install openjdk-11-jre-headless
apt install openjdk-8-jre-headless
Perhaps, you might want to try this https://hub.docker.com/r/timbru31/java-node one to create the docker file.
This docker image comes with both Java and Node pre-installed. It comes in handy when both are required as a dependency.
Something like,
FROM timbru31/java-node:<tag>
The FROM inside your dockerfile simply tells docker from which image it should start the configuration. You can't simply concatenate multiple images together. There are already multiple container images available which offer preinstalled Java 8 and node JS. I don't want to recommend any image specifically but will direct you to docker-hub for you to go search on your own and use the container that suites your needs the best.
With version 14 of node it works perfectly for me:
FROM openjdk:latest
RUN apt-get install -y curl \
&& curl -sL https://deb.nodesource.com/setup_14.x | bash - \
&& apt-get install -y nodejs \
&& curl -L https://www.npmjs.com/install.sh | sh \
RUN npm install -g grunt grunt-cli
this worked for me:
FROM openjdk:16-slim-buster
RUN apt-get update; apt-get install -y curl \
&& curl -sL https://deb.nodesource.com/setup_14.x | bash - \
&& apt-get install -y nodejs \
&& curl -L https://www.npmjs.com/install.sh | sh
This works for me with node v16.15.0, be careful with the packages version of java, I'm using the latest by default. The packages used for java are:
Java Runtime Environment (jre)
Java Development Kit (jdk)
FROM node:16.15.0
RUN apt-get update \
&& apt-get install default-jre -y \
&& apt-get install default-jdk -y
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
RUN npm install --global nodemon
COPY . .
EXPOSE 3000
CMD ["npm", "run", "dev"]
I hope this works for you all

Categories

Resources