Develop Reference

Docker and Java - FontConfiguration issue

We've got a Java application that generates word documents using a 3rd party (Asposee but I don't think it matters here). The app is built from a simple Docker file:
FROM openjdk:10-jdk-slim
COPY target/*.jar /opt/
CMD $JAVA_HOME/bin/java $JAVA_OPTS -jar /opt/*.jar
When we build the application locally (mvn package then docker build) and run the application inside k8s it works well.
However, when we build the image in our CI/CD pipeline with Jenkins we get a runtime exception when running through a specific process which apparently requires additional fonts:
Caused by: java.lang.NullPointerException: null
at java.desktop/sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1288)
at java.desktop/sun.awt.FontConfiguration.readFontConfigFile(FontConfiguration.java:225)
at java.desktop/sun.awt.FontConfiguration.init(FontConfiguration.java:107)
at java.desktop/sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:765)
at java.desktop/sun.font.SunFontManager$2.run(SunFontManager.java:440)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.desktop/sun.font.SunFontManager.<init>(SunFontManager.java:385)
at java.desktop/sun.awt.FcFontManager.<init>(FcFontManager.java:35)
at java.desktop/sun.awt.X11FontManager.<init>(X11FontManager.java:56)
In that case the project is buit in Jenkins, compiled by the docker image maven:3.5.4-jdk-10-slim.
I've checked both jar files (locally and from jenkins) and the class files are the same (as expected).
In both cases it's the same base image so I don't understand what could be the difference. Is something different in Docker when building locally vs inside another Docker container?
EDIT
We've looked into both docker images and found the following difference.
Since locally built image ls -l /usr/lib returns:
drwxr-xr-x 2 root root 4096 May 3 2017 X11
drwxr-xr-x 5 root root 4096 Apr 26 00:00 apt
drwxr-xr-x 2 root root 4096 May 26 08:31 binfmt.d
drwxr-xr-x 2 root root 4096 Jun 6 01:50 cgmanager
drwxr-xr-x 2 root root 4096 Jun 6 01:50 dbus-1.0
drwxr-xr-x 2 root root 4096 Jun 6 01:51 dconf
drwxr-xr-x 3 root root 4096 Jun 6 01:51 debug
drwxr-xr-x 3 root root 4096 Apr 20 10:08 dpkg
drwxr-xr-x 2 root root 4096 Jun 6 01:50 environment.d
drwxr-xr-x 3 root root 4096 Apr 25 04:56 gcc
drwxr-xr-x 2 root root 4096 Jun 6 01:51 glib-networking
drwxr-xr-x 2 root root 4096 Apr 26 00:00 init
drwxr-xr-x 1 root root 4096 Jun 6 01:51 jvm
drwxr-xr-x 3 root root 4096 Jun 6 01:50 kernel
lrwxrwxrwx 1 root root 20 Mar 4 09:49 libnih-dbus.so.1 -> libnih-dbus.so.1.0.0
-rw-r--r-- 1 root root 34824 Mar 4 09:49 libnih-dbus.so.1.0.0
lrwxrwxrwx 1 root root 15 Mar 4 09:49 libnih.so.1 -> libnih.so.1.0.0
-rw-r--r-- 1 root root 92184 Mar 4 09:49 libnih.so.1.0.0
drwxr-xr-x 3 root root 4096 Mar 29 19:47 locale
drwxr-xr-x 3 root root 4096 Jun 6 01:50 lsb
drwxr-xr-x 1 root root 4096 Jul 21 2017 mime
drwxr-xr-x 2 root root 4096 Jun 6 01:50 modprobe.d
drwxr-xr-x 2 root root 4096 May 26 08:31 modules-load.d
-rw-r--r-- 1 root root 198 Jan 13 23:36 os-release
drwxr-xr-x 3 root root 4096 Jun 6 01:51 ssl
drwxr-xr-x 1 root root 4096 Jun 6 01:50 systemd
drwxr-xr-x 2 root root 4096 Jun 6 01:50 sysusers.d
drwxr-xr-x 2 root root 4096 Jul 21 2017 tar
drwxr-xr-x 15 root root 4096 Feb 11 20:06 terminfo
drwxr-xr-x 1 root root 4096 Jun 6 01:50 tmpfiles.d
drwxr-xr-x 1 root root 4096 Apr 26 00:00 udev
drwxr-xr-x 1 root root 16384 Jun 6 01:51 x86_64-linux-gnu
But inside Jenkins built image ls -l /usr/lib returns:
drwxr-xr-x 5 root root 4096 Jun 25 00:00 apt
drwxr-xr-x 3 root root 4096 Jul 3 01:00 debug
drwxr-xr-x 3 root root 4096 Apr 20 10:08 dpkg
drwxr-xr-x 3 root root 4096 Jun 17 03:36 gcc
drwxr-xr-x 2 root root 4096 Jun 25 00:00 init
drwxr-xr-x 1 root root 4096 Jul 3 01:00 jvm
drwxr-xr-x 1 root root 4096 Jul 12 11:00 locale
drwxr-xr-x 3 root root 4096 Jul 3 01:00 lsb
drwxr-xr-x 1 root root 4096 May 16 07:47 mime
-rw-r--r-- 1 root root 198 Jan 13 23:36 os-release
drwxr-xr-x 3 root root 4096 Jul 3 01:00 ssl
drwxr-xr-x 3 root root 4096 Apr 20 10:08 systemd
drwxr-xr-x 2 root root 4096 May 16 07:47 tar
drwxr-xr-x 15 root root 4096 May 21 08:54 terminfo
drwxr-xr-x 2 root root 4096 Jun 25 00:00 tmpfiles.d
drwxr-xr-x 3 root root 4096 Jun 25 00:00 udev
drwxr-xr-x 2 root root 4096 May 3 2017 X11
drwxr-xr-x 1 root root 4096 Jul 3 01:00 x86_64-linux-gnu
This is really puzzling as I thought Docker would always produce the same image from identical Dockerfiles

With openjdk:8u111-jdk-alpine, installing dejavu fix the problem:
For example:
Dockerfile:
FROM openjdk:8u111-jdk-alpine
# Needed to fix 'Fontconfig warning: ignoring C.UTF-8: not a valid language tag'
ENV LANG en_GB.UTF-8
# JRE fails to load fonts if there are no standard fonts in the image; DejaVu is a good choice,
# see https://github.com/docker-library/openjdk/issues/73#issuecomment-207816707
RUN apk add --update ttf-dejavu && rm -rf /var/cache/apk/*
VOLUME /tmp
COPY /target/*.jar app.jar
ENTRYPOINT ["java","-Xmx100m","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

Just add the following in your Dockerfile
RUN apk add --no-cache fontconfig ttf-dejavu

Installing libfontconfig1 solved the problem for me (source):
RUN apt-get install -y libfontconfig1 && rm -rf /var/lib/apt/lists/*

I think we found the problem.
When running on Jenkins we use the docker:dind (Docker inside docker) Docker image to provide the docker command in the build. This image is based on Alpine linux. When running docker info we get the following:
On Mac:
Kernel Version: 4.9.87-linuxkit-aufs
Operating System: Docker for Mac
On Jenkins:
Kernel Version: 4.4.115-k8s
Operating System: Alpine Linux v3.7 (containerized)
Alpine linux must be missing those fonts. We fixed the problem by manually installing them in the Dockerfile:
RUN apt-get update \
&& apt-get install --assume-yes apt-utils \
&& apt-get install --assume-yes software-properties-common \
&& apt-get install --assume-yes dbus \
&& apt-get install --assume-yes cgmanager \
&& apt-get install --assume-yes glib-networking \
&& apt-get install --assume-yes libnih-dbus-dev \
&& apt-get install --assume-yes dconf-cli \
&& apt-get install --assume-yes fontconfig
Not sure this is the minimum required libraries but those did the trick :D

If you trying to do in Java/Spring boot , adding following in Dockerfile worked for me in java 11.
# Font packages to be added for java 11
RUN apk update \
&& apk add fontconfig \
&& apk add ttf-dejavu
RUN ln -s /usr/lib/libfontconfig.so.1 /usr/lib/libfontconfig.so && \
ln -s /lib/libuuid.so.1 /usr/lib/libuuid.so.1 && \
ln -s /lib/libc.musl-x86_64.so.1 /usr/lib/libc.musl-x86_64.so.1
ENV LD_LIBRARY_PATH /usr/lib

If you are running it on local with Docker Desktop and using docker file to create the image.
Please make changes in Dockerfile from FROM openjdk:8-jdk-alpine to FROM adoptopenjdk/openjdk11:ubi

Problem NullPointerException with fonts occurs when you use openjdk docker image in alpine version for example :
adoptopenjdk/openjdk11:jre-11.0.11_9-alpine.
You need change image to full version: adoptopenjdk/openjdk11:jre-11.0.11_9

I also got the same issue while running below command and restart app server resolved this issue.
Yup install fontconfig

Updating java 6 cacerts with those from java 8

Rackspace replaced an outdated certificate, ive followed their instructions using keytool to update the cacerts for java 6 with no success (keytool).
What did work was just replacing the cacerts file from java 6 with one from java 8.
Is this "OK" to do? i assume formats between the two are the same and they are simply updated between version.
Apologies in advance if this is a stupid question

I initially did the same thing, and then symlinked to the java 8 file. But my problem was that the java 6 installation on macOs High Sierra (10.13.x).
# store path to java 6 home
tmp $ j6Security=$(/usr/libexec/java_home -v '1.6*')/lib/security;
# show pre-update state
tmp $ ls -la "$j6Security"
total 16
drwxr-xr-x 10 root wheel 320 Jan 20 19:39 .
drwxr-xr-x 41 root wheel 1312 Jan 20 19:39 ..
-rw-r--r-- 1 root wheel 2469 Jul 14 2015 US_export_policy.jar
lrwxr-xr-x 1 root wheel 79 Jan 20 19:39 blacklist -> /System/Library/Java/Support/Deploy.bundle/Contents/Home/lib/security/blacklist
lrwxr-xr-x 1 root wheel 81 Jan 20 19:39 cacerts -> /System/Library/Java/Support/CoreDeploy.bundle/Contents/Home/lib/security/cacerts
-rw-r--r-- 1 root wheel 3443 Jul 14 2015 java.policy
-rw-r--r-- 1 root wheel 13458 Jul 14 2015 java.security
-rw-r--r-- 1 root wheel 2486 Jul 14 2015 local_policy.jar
-rw-r--r-- 1 root wheel 347 Jul 14 2015 sunpkcs11-macosx.cfg
lrwxr-xr-x 1 root wheel 87 Jan 20 19:39 trusted.libraries -> /System/Library/Java/Support/Deploy.bundle/Contents/Home/lib/security/trusted.libraries
# store path to current (i.e., switcher) home
tmp $ jXSecurity=/Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security;
# replace (broken) j6 files with symlink to current files
tmp $ for file in blacklist cacerts trusted.libraries; do src="${jXSecurity}/${file}" tgt="${j6Security}/${file}"; test -f $tgt && sudo rm $tgt; sudo ln -s "$src" "$tgt"; done
# show post-update state
tmp $ ls -la "$j6Security"
total 16
drwxr-xr-x 10 root wheel 320 Jan 20 20:33 .
drwxr-xr-x 41 root wheel 1312 Jan 20 19:39 ..
-rw-r--r-- 1 root wheel 2469 Jul 14 2015 US_export_policy.jar
lrwxr-xr-x 1 root wheel 87 Jan 20 20:33 blacklist -> /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/blacklist
lrwxr-xr-x 1 root wheel 85 Jan 20 20:33 cacerts -> /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/cacerts
-rw-r--r-- 1 root wheel 3443 Jul 14 2015 java.policy
-rw-r--r-- 1 root wheel 13458 Jul 14 2015 java.security
-rw-r--r-- 1 root wheel 2486 Jul 14 2015 local_policy.jar
-rw-r--r-- 1 root wheel 347 Jul 14 2015 sunpkcs11-macosx.cfg
lrwxr-xr-x 1 root wheel 95 Jan 20 20:33 trusted.libraries -> /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/trusted.libraries
If I wanted to pin j8, I would replace jXSecurity=/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/trusted.libraries with j8Security=$(/usr/libexec/java_home -v '1.8*'). Both this and copying the files (that I've symlinked in this example) worked fine.
(In my case the installed symlinks (in the "pre-update" output) pointed to nonexistent files. My maven downloads were failing with "peer not authenticated". After applying these changes to my jdk6 installation everything began working again.)

WebLogic Console - No application files exist when redeploying application

When stopping/starting a deployment via the WebLogic Admin console, sometimes we get the following error:
Unable to access application source information in '/opt/product/oracle/local/managedservers/mydomain/servers/serverA/stage/apputil/apputil.war'
for application 'apputil’.
The specific error is: [Deployer:149158] No application files
exist at '/opt/product/oracle/local/managedservers/mydomain/servers/serverA/stage/apputil/apputil.war'
Yet, if I stop and start the managed server, the deployment appears to come back.
My question is, WHY do these war files disappear from the managed server seemingly randomly, while the server is running? This typically happens after we haven't touched a deployment for some time (6 months). Other war files for other deployments are there. It does not affect the running of the app, until we try to stop/start it.
This is what the filesystem looks like before and after.
[oracle#serverA stage]$ pwd;ls -alstr
/opt/product/oracle/local/managedservers/mydomain/servers/serverA/stage
total 20
4 drwxr-x--- 8 oracle dba 4096 Mar 19 2014 ..
4 drwxr----- 3 oracle dba 4096 Mar 19 2014 app-crypto-util
4 drwxr----- 2 oracle dba 4096 Mar 19 2014 appadmin
4 drwxr----- 2 oracle dba 4096 Mar 19 2014 appsm
4 drwxr----- 5 oracle dba 4096 May 1 15:29 .
[oracle#serverA stage]$ ls
appadmin app-crypto-util appsm
Restart managed server here...
[oracle#serverA stage]$ ls
appdmin app-crypto-util appsm apputil
[oracle#serverA stage]$ ls -alstr
total 24
4 drwxr-x--- 8 oracle dba 4096 Mar 19 2014 ..
4 drwxr----- 3 oracle dba 4096 Mar 19 2014 app-crypto-util
4 drwxr----- 2 oracle dba 4096 Mar 19 2014 appadmin
4 drwxr----- 2 oracle dba 4096 Mar 19 2014 appsm
4 drwxr----- 2 oracle dba 4096 Jun 25 14:35 apputil
4 drwxr----- 6 oracle dba 4096 Jun 25 14:35 .
[oracle#serverA stage]$ ls -alstr apputil/apputil.war
28660 -rw-r----- 1 oracle dba 29347298 Jun 25 14:35 apputil/apputil.war

This may happen when AdminServer & Managed Server are in different machines, or the war is being sent from a different machine to AS.
use arguments: -remote -upload
i.e.:
java weblogic.Deployer -adminurl t3://200.10.10.125:7001 -verbose -username weblogic -password welcome1 -deploy -targets WLCluster -name sample -remote -source sample.war -upload

Referring to the stage directory modification date, it appears this dir is created/edited when an event occur, maybe by a script, and if it is the case, the problem should come from that script when copying apps wars.
So in my POV when stopping a deployment, maybe weblogic stop the apputil managed server, wich delete the war from th stage dir, and when starting a deployment after stopping it, it does not start the specified managed server before, and try to redeploy all apps wich cause the exception.

What are jartmp files?

I know it is probably a dumb question but I find no results in SO and no relevant results in google (here is the google search link where I searched 'what is jartmp').
I found there are many jartmp files in my folder and don't know why they exist:
-rw-rw-r-- 1 0 Jun 11 14:28 jartmp1089103248955132063.tmp
-rw-rw-r-- 1 54935 Jun 6 03:21 jartmp1258300977464933918.tmp
-rw-rw-r-- 1 118685 Jun 26 22:47 jartmp1388010323455694859.tmp
-rw-rw-r-- 1 15643 May 29 16:45 jartmp1819063406633422416.tmp
-rw-rw-r-- 1 0 Jun 11 16:03 jartmp2142600141373219701.tmp
-rw-rw-r-- 1 197964 Jun 6 03:19 jartmp3480763606864988668.tmp
-rw-rw-r-- 1 126386 Jun 26 22:47 jartmp3533722093029133854.tmp
-rw-rw-r-- 1 7830 Jun 6 03:19 jartmp3713382469327367468.tmp
-rw-rw-r-- 1 55872 Jun 21 15:14 jartmp3950308579438275722.tmp
-rw-rw-r-- 1 39716 Jun 11 16:03 jartmp4311759817318348544.tmp
-rw-rw-r-- 1 0 Jun 11 14:31 jartmp499113526131437419.tmp
I tried to use head *.tmp to see the content of the .tmp files, but it seems that they are all binary files. Can I know why these files are generated and whether it is safe to delete them?
And my java version is 1.6.0_24 for your reference.
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

I can't imagine a circumstance where a file with that name would contain valuable data.
But if you are really worried, try using the file command to attempt to identify the files' types ... based on their contents.
How are they generated?
Well the most likely culprit would be some command that creates or unpacks JAR files. Try correlating the timestamps of the files with what you are / were doing at the time. That would help you narrow it down a bit.

This could be result of jar -uvf operation on a jarred file to update a file to jar which is not present in actual path. This is just one of the reason I noticed where it shows this in shell/cmd o/p
: no such file or directory & ends up creating a tmp file (binary) file of pretty huge size.

I started running into this issue with the Java 8 jar.exe when supplying files to update in an archive where one or more of the files didn't exist. For example,
%ProgramFiles%\Java\jdk1.8.0_121\bin\jar.exe uvf0 ..\Desktop.jar Desktop\images\*.gif Desktop\reports\*.rep
where no file that matches the pattern Desktop\reports\*.rep exists, even though gif files in the other path do. In this case, the "jartmp" file remains and the jar file intended to be updated is left untouched. This is a change from Java 7, where jar.exe would happily replace the files it found and ignore the others.

Explain the JVM Directory Layout on Mac OSX Leopard

Here is the directory layout that was installed with Leopard. What is the "A" directory and why the "Current" directory in addition to the "CurrentJDK"?
It seems like you can easily switch the current JDK by move the CurrentJDK link, but then the contents under Current and A will be out of sync.
lrwxr-xr-x 1 root wheel 5 Jun 14 15:49 1.3 -> 1.3.1
drwxr-xr-x 3 root wheel 102 Jan 14 2008 1.3.1
lrwxr-xr-x 1 root wheel 5 Feb 21 2008 1.4 -> 1.4.2
lrwxr-xr-x 1 root wheel 3 Jun 14 15:49 1.4.1 -> 1.4
drwxr-xr-x 8 root wheel 272 Feb 21 2008 1.4.2
lrwxr-xr-x 1 root wheel 5 Feb 21 2008 1.5 -> 1.5.0
drwxr-xr-x 8 root wheel 272 Feb 21 2008 1.5.0
lrwxr-xr-x 1 root wheel 5 Jun 14 15:49 1.6 -> 1.6.0
drwxr-xr-x 8 root wheel 272 Jun 14 15:49 1.6.0
drwxr-xr-x 8 root wheel 272 Jun 14 15:49 A
lrwxr-xr-x 1 root wheel 1 Jun 14 15:49 Current -> A
lrwxr-xr-x 1 root wheel 3 Jun 14 15:49 CurrentJDK -> 1.5
steve-mbp /System/Library/Frameworks/JavaVM.framework/Versions $
and the contents of A
-rw-r--r-- 1 root wheel 1925 Feb 29 2008 CodeResources
drwxr-xr-x 34 root wheel 1156 Jun 14 15:49 Commands
drwxr-xr-x 3 root wheel 102 Mar 6 2008 Frameworks
drwxr-xr-x 16 root wheel 544 Jun 14 15:49 Headers
-rwxr-xr-x 1 root wheel 236080 Feb 29 2008 JavaVM
drwxr-xr-x 29 root wheel 986 Jun 14 15:49 Resources
steve-mbp /System/Library/Frameworks/JavaVM.framework/Versions/A $

The (A, Current symbolic-linked to A) is part of the structure of a Mac OS X framework, which JavaVM.framework is. This framework may have C or Objective-C code in it, in addition to the actual JVM installations. Thus it could potentially be linked against from some C or Objective-C code in addition to containing the JVM alongside that.
Note that you should not change the CurrentJDK link to point at anything but what it is set to by Mac OS X. Unlike on other platforms, the Java virtual machine is an operating system service on Mac OS X, and changing it in this way would put you in an unsupported (and potentially untested, unstable, etc.) configuration.

You should use the Java Preferences command to change the jvm version.
If you have spotlight on your Harddisk, you can just spotlight "Java Preferences"

If you want to revert to an older JVM (here, 1.5), you can put the following in your ~/.profile (or paste it into a specific Terminal window):
export JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home/"
export PATH=$JAVA_HOME/bin/:$PATH