Develop Reference

Keycloak - cannot start embedded server: WFLYEMB0022: Cannot invoke 'start' on embedded process: Parameter 'abstractPath' must not be empty

I have a Keycloak instance running standalone on a Heroku PM dyno via the container registry. Recently, this app has crashed with the logs below. I have not changed the code since May 25th at which point everything was working. The heroku dyno is not near any limits. My local version is fine. Therefore, I think one of the following must have happened.
Some corruption based on deployment data for the standalone setting
Change in the underlying image- keycloak/11.0.2
Corrupted image in my container registry that is overwriting things
Debugging I have tried:
Verified that the db settings are correct - they were fine
Verified that queries on the db work as expected - also fine
Restarted the app, deployed the app, rebuilt the image - all same error message
Redeployed the app with new image 12.0.0 - same error message
Deployed a simple version of this image (https://github.com/mieckert/keycloak-heroku) via a button - This worked on first deploy and connected to the database as expected. However when I repushed and rereleased the exact same code, the same error appeared.
What should I try next?
Logs:
Jun 09 08:50:28 Release v66 created by user [EMAIL]
Jun 09 08:50:28 Deployed web (c19fbf3aed30) by user [EMAIL]
Jun 09 08:50:39 Starting process with command `-b 0.0.0.0`
Jun 09 08:50:40 Found database configuration in [CORRECT DETAILS WERE HERE]
Jun 09 08:50:43 Added [EMAIL] to '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user
Jun 09 08:50:43 -b 0.0.0.0
Jun 09 08:50:43 =========================================================================
Jun 09 08:50:43 Using PostgreSQL database
Jun 09 08:50:43 =========================================================================
Jun 09 08:50:44 15:50:44,394 INFO [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.2.Final
Jun 09 08:50:44 15:50:44,451 INFO [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.12.Final
Jun 09 08:50:44 15:50:44,459 INFO [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
Jun 09 08:50:44 15:50:44,576 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 12.0.0 (WildFly Core 13.0.3.Final) starting
Jun 09 08:50:44 15:50:44,611 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.as: org.jboss.msc.service.StartException in service jboss.as: Failed to start service
Jun 09 08:50:44 at org.jboss.msc#1.4.12.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
Jun 09 08:50:44 at org.jboss.msc#1.4.12.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
Jun 09 08:50:44 at org.jboss.threads#2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
Jun 09 08:50:44 at org.jboss.threads#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
Jun 09 08:50:44 at org.jboss.threads#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
Jun 09 08:50:44 at org.jboss.threads#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
Jun 09 08:50:44 at java.base/java.lang.Thread.run(Thread.java:834)
Jun 09 08:50:44 Caused by: java.lang.IllegalArgumentException: COM00008: Parameter 'abstractPath' must not be empty
Jun 09 08:50:44 at org.wildfly.common#1.5.4.Final//org.wildfly.common.Assert.checkNotEmptyParam(Assert.java:104)
Jun 09 08:50:44 at org.jboss.as.controller#13.0.3.Final//org.jboss.as.controller.services.path.AbsolutePathService.convertPath(AbsolutePathService.java:70)
Jun 09 08:50:44 at org.jboss.as.controller#13.0.3.Final//org.jboss.as.controller.services.path.AbsolutePathService.<init>(AbsolutePathService.java:49)
Jun 09 08:50:44 at org.jboss.as.controller#13.0.3.Final//org.jboss.as.controller.services.path.AbsolutePathService.addService(AbsolutePathService.java:59)
Jun 09 08:50:44 at org.jboss.as.controller#13.0.3.Final//org.jboss.as.controller.services.path.AbsolutePathService.addService(AbsolutePathService.java:53)
Jun 09 08:50:44 at org.jboss.as.controller#13.0.3.Final//org.jboss.as.controller.services.path.PathManagerService.addAbsolutePathService(PathManagerService.java:259)
Jun 09 08:50:44 at org.jboss.as.controller#13.0.3.Final//org.jboss.as.controller.services.path.PathManagerService.addHardcodedAbsolutePath(PathManagerService.java:160)
Jun 09 08:50:44 at org.jboss.as.server#13.0.3.Final//org.jboss.as.server.ServerPathManagerService.addService(ServerPathManagerService.java:55)
Jun 09 08:50:44 at org.jboss.as.server#13.0.3.Final//org.jboss.as.server.ApplicationServerService.start(ApplicationServerService.java:179)
Jun 09 08:50:44 at org.jboss.msc#1.4.12.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
Jun 09 08:50:44 at org.jboss.msc#1.4.12.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
Jun 09 08:50:44 ... 6 more
Jun 09 08:50:44 Cannot start embedded server: WFLYEMB0022: Cannot invoke 'start' on embedded process: WFLYSRV0141: Cannot start server: JBTHR00005: Operation failed: Failed to start service: COM00008: Parameter 'abstractPath' must not be empty
My docker entry point file
# Set database config from Heroku DATABASE_URL or HEROKU_POSTGRESQL_ORANGE_URL
if [ "$DATABASE_URL" != "" ]; then
echo "Found database configuration in DATABASE_URL=$DATABASE_URL"
regex='^postgres://([a-zA-Z0-9_-]+):([a-zA-Z0-9]+)#([a-z0-9.-]+):([[:digit:]]+)/([a-zA-Z0-9_-]+)$'
if [[ $DATABASE_URL =~ $regex ]]; then
export DB_ADDR=${BASH_REMATCH[3]}
export DB_PORT=${BASH_REMATCH[4]}
export DB_DATABASE=${BASH_REMATCH[5]}
export DB_USER=${BASH_REMATCH[1]}
export DB_PASSWORD=${BASH_REMATCH[2]}
echo "DB_ADDR=$DB_ADDR, DB_PORT=$DB_PORT, DB_DATABASE=$DB_DATABASE, DB_USER=$DB_USER, DB_PASSWORD=$DB_PASSWORD"
export DB_VENDOR=postgres
fi
fi
# usage: file_env VAR [DEFAULT]
# ie: file_env 'XYZ_DB_PASSWORD' 'example'
# (will allow for "$XYZ_DB_PASSWORD_FILE" to fill in the value of
# "$XYZ_DB_PASSWORD" from a file, especially for Docker's secrets feature)
file_env() {
local var="$1"
local fileVar="${var}_FILE"
local def="${2:-}"
if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
exit 1
fi
local val="$def"
if [ "${!var:-}" ]; then
val="${!var}"
elif [ "${!fileVar:-}" ]; then
val="$(< "${!fileVar}")"
fi
export "$var"="$val"
unset "$fileVar"
}
##################
# Add admin user #
##################
file_env 'KEYCLOAK_USER'
file_env 'KEYCLOAK_PASSWORD'
if [ $KEYCLOAK_USER ] && [ $KEYCLOAK_PASSWORD ]; then
/opt/jboss/keycloak/bin/add-user-keycloak.sh --user $KEYCLOAK_USER --password $KEYCLOAK_PASSWORD || echo "User already exists."
fi
############
# Hostname #
############
if [ "$KEYCLOAK_HOSTNAME" != "" ]; then
SYS_PROPS="-Dkeycloak.hostname.provider=fixed -Dkeycloak.hostname.fixed.hostname=$KEYCLOAK_HOSTNAME"
if [ "$KEYCLOAK_HTTP_PORT" != "" ]; then
SYS_PROPS+=" -Dkeycloak.hostname.fixed.httpPort=$KEYCLOAK_HTTP_PORT"
fi
if [ "$KEYCLOAK_HTTPS_PORT" != "" ]; then
SYS_PROPS+=" -Dkeycloak.hostname.fixed.httpsPort=$KEYCLOAK_HTTPS_PORT"
fi
fi
################
# Realm import #
################
if [ "$KEYCLOAK_IMPORT" ]; then
SYS_PROPS+=" -Dkeycloak.import=$KEYCLOAK_IMPORT"
fi
########################
# JGroups bind options #
########################
if [ -z "$BIND" ]; then
BIND=$(hostname -i)
fi
if [ -z "$BIND_OPTS" ]; then
for BIND_IP in $BIND
do
BIND_OPTS+=" -Djboss.bind.address=$BIND_IP -Djboss.bind.address.private=$BIND_IP "
done
fi
SYS_PROPS+=" $BIND_OPTS"
#################
# Configuration #
#################
# If the server configuration parameter is not present, append the HA profile.
if echo "$#" | egrep -v -- '-c |-c=|--server-config |--server-config='; then
SYS_PROPS+=" -c=standalone-ha.xml"
fi
############
# DB setup #
############
file_env 'DB_USER'
file_env 'DB_PASSWORD'
# Lower case DB_VENDOR
DB_VENDOR=`echo $DB_VENDOR | tr A-Z a-z`
# Detect DB vendor from default host names
if [ "$DB_VENDOR" == "" ]; then
if (getent hosts postgres &>/dev/null); then
export DB_VENDOR="postgres"
elif (getent hosts mysql &>/dev/null); then
export DB_VENDOR="mysql"
elif (getent hosts mariadb &>/dev/null); then
export DB_VENDOR="mariadb"
fi
fi
# Detect DB vendor from legacy `*_ADDR` environment variables
if [ "$DB_VENDOR" == "" ]; then
if (printenv | grep '^POSTGRES_ADDR=' &>/dev/null); then
export DB_VENDOR="postgres"
elif (printenv | grep '^MYSQL_ADDR=' &>/dev/null); then
export DB_VENDOR="mysql"
elif (printenv | grep '^MARIADB_ADDR=' &>/dev/null); then
export DB_VENDOR="mariadb"
fi
fi
# Default to H2 if DB type not detected
if [ "$DB_VENDOR" == "" ]; then
export DB_VENDOR="h2"
fi
# Set DB name
case "$DB_VENDOR" in
postgres)
DB_NAME="PostgreSQL";;
mysql)
DB_NAME="MySQL";;
mariadb)
DB_NAME="MariaDB";;
h2)
DB_NAME="Embedded H2";;
*)
echo "Unknown DB vendor $DB_VENDOR"
exit 1
esac
# Append '?' in the beggining of the string if JDBC_PARAMS value isn't empty
export JDBC_PARAMS=$(echo ${JDBC_PARAMS} | sed '/^$/! s/^/?/')
# Convert deprecated DB specific variables
function set_legacy_vars() {
local suffixes=(ADDR DATABASE USER PASSWORD PORT)
for suffix in "${suffixes[#]}"; do
local varname="$1_$suffix"
if [ ${!varname} ]; then
echo WARNING: $varname variable name is DEPRECATED replace with DB_$suffix
export DB_$suffix=${!varname}
fi
done
}
set_legacy_vars `echo $DB_VENDOR | tr a-z A-Z`
# Configure DB
echo "========================================================================="
echo ""
echo " Using $DB_NAME database"
echo ""
echo "========================================================================="
echo ""
if [ "$DB_VENDOR" != "h2" ]; then
/bin/sh /opt/jboss/tools/databases/change-database.sh $DB_VENDOR
fi
/opt/jboss/tools/x509.sh
/opt/jboss/tools/jgroups.sh $JGROUPS_DISCOVERY_PROTOCOL $JGROUPS_DISCOVERY_PROPERTIES
/opt/jboss/tools/autorun.sh
##################
# Start Keycloak #
##################
exec /opt/jboss/keycloak/bin/standalone.sh $SYS_PROPS $# -Djboss.http.port=$PORT
exit $?
My Dockerfile:
COPY docker-entrypoint.sh /opt/jboss/tools
ENTRYPOINT [ "/opt/jboss/tools/docker-entrypoint.sh" ]
CMD ["-b", "0.0.0.0"]
RUN mkdir -p /opt/jboss/keycloak/themes/my_theme/
COPY /my_theme/ /opt/jboss/keycloak/themes/my_theme/

I ran into this exact same problem, and was able to resolve the issue by setting up an entirely new Keycloak Heroku app, redeploying my code there, then renaming my apps so that traffic from the old app when to the new app. I can give detailed information on this if you need, but this resulted in a fresh Keycloak server (dyno) that once scaled to Performance-M ran exactly the same as before (afaik, Keycloak configuration is stored in the underlying database, so if you attach your original database to the new dyno app then you should be fine)
I ran into the exact same problem in the past week or so. I was actually running a different version of Keycloak as well, v10, and initially attempted upgrading to v13 to resolve the problem, but that didn't do it. I think you are correct that there is some sort of corruption happening, as my instance was also untouched when the first failure occurred. I checked Heroku changelogs for anything that would affect my deployment, as the failure seemed to occur after a daily dyno restart, but I found nothing there)
I think you are correct that there is some sort of corruption or underlying bug here, likely a hard-to-reproduce one, possibly with WildFly code, JVM, Keycloak, or something related to Heroku

Using tcnative module on AIX for HTTP2 on Tomcat

Has anyone successfully got the HTTP/2 connector working in Tomcat9 on AIX (e.g. powerpc-ibm-aix7.2.5.0)?
I followed the instructions here to build the tcnative module (using tomcat-native-1.2.24-src that comes with Tomcat 9.0.37, APR 1.5.2, OpenSSL 1.0.2, IBM Java 1.8.0_261) i.e.
$ ./configure --with-apr=/opt/freeware/bin/apr-1-config --with-java-home=/app/java8_64/ --with-ssl=yes --prefix=/app/tomcat
followed by
make && make install
This creates the expected entries in /app/tomcat/lib, i.e.
-rw-r--r-- 1 usrxxx grpxxxx 3440287 Mar 03 16:47 libtcnative-1.a
-rwxr-xr-x 1 usrxxx grpxxxx 1057 Mar 03 16:47 libtcnative-1.la
lrwxrwxrwx 1 usrxxx grpxxxx 23 Mar 03 16:47 libtcnative-1.so -> libtcnative-1.so.0.2.24
lrwxrwxrwx 1 usrxxx grpxxxx 23 Mar 03 16:47 libtcnative-1.so.0 -> libtcnative-1.so.0.2.24
-rwxr-xr-x 1 usrxxx grpxxxx 1372146 Mar 03 16:47 libtcnative-1.so.0.2.24
but when Tomcat starts I get
04-Mar-2021 15:30:00.752 WARNING [main] org.apache.catalina.core.AprLifecycleListener.init The Apache Tomcat Native library failed to load. The error reported was [tcnative-1 (Not found in java.library.path)]
java.lang.UnsatisfiedLinkError: tcnative-1 (Not found in java.library.path)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1462)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1414)
at java.lang.System.loadLibrary(System.java:584)
at org.apache.tomcat.jni.Library.<init>(Library.java:69)
at org.apache.tomcat.jni.Library.initialize(Library.java:206)
at org.apache.catalina.core.AprLifecycleListener.init(AprLifecycleListener.java:198)
...
04-Mar-2021 15:30:01.096 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.library.path=/app/java8_64/jre/lib/ppc64/compressedrefs:/app/java8_64/jre/lib/ppc64:/app/java8_64/jre/lib/ppc64/j9vm:/app/java8_64/jre/lib/ppc64:/app/java8_64/jre/../lib/ppc64:/app/java8_64/jre/lib/icc:/opt/freeware/lib:/opt/freeware/lib64:/usr/lib:/usr/lib64:/app/tomcat/lib:/usr/lib64:/usr/lib
...
04-Mar-2021 15:30:02.233 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[org.apache.coyote.http11.Http11AprProtocol-8443]] org.apache.catalina.LifecycleException: The configured protocol [org.apache.coyote.http11.Http11AprProtocol] requires the APR/native library which is not available
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1024)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:533)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1057)
Edit - based on suggestions from Piotr and Lorinczy:
Tried adding tcnative-1.so as a symlink - same error
Copied libtcnative-1.* to the bin folder. Still failing but new error (progress!?). Possible 32bit v 64bit issue?
09-Mar-2021 10:10:07.116 WARNING [main] org.apache.catalina.core.AprLifecycleListener.init The Apache Tomcat Native library failed to load. The error reported was [/app/apache-tomcat-9.0.37/bin/libtcnative-1.a ( 0509-022 Cannot load module /app/apache-tomcat-9.0.37/bin/libtcnative-1.a.
0509-026 System error: Cannot run a file that does not have a valid format.)]
java.lang.UnsatisfiedLinkError: /app/apache-tomcat-9.0.37/bin/libtcnative-1.a ( 0509-022 Cannot load module /app/apache-tomcat-9.0.37/bin/libtcnative-1.a.
0509-026 System error: Cannot run a file that does not have a valid format.)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1462)
...
All the dependencies seem to be available
$ldd libtcnative-1.so.0.2.24
libtcnative-1.so.0.2.24 needs:
/usr/lib/libssl.so
/usr/lib/libcrypto.so
/opt/freeware/lib/libapr-1.so
/usr/lib/libpthread.a(shr_xpg5.o)
/usr/lib/libc.a(shr.o)
/opt/freeware/lib/libgcc_s.a(shr.o)
/usr/lib/libcrypto.a(libcrypto.so.1.0.2)
/unix
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libcrypt.a(shr.o)
but the dump command seems to support the 32 v 64 bit theory:
$ dump -H -X64 libtcnative-1.so.0.2.24
libtcnative-1.so.0.2.24:
dump: libtcnative-1.so.0.2.24: 0654-108 file is not valid in the current object file mode.
Use the -X option to specify the desired object mode.
$ dump -H -X32 libtcnative-1.so.0.2.24
libtcnative-1.so.0.2.24:
***Loader Section***
Loader Header Information
VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00000364 0x00000771 0x00000084
#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000007 0x0000aacc 0x00006406 0x0000ab50
***Import File Strings***
INDEX PATH BASE MEMBER
0 /opt/freeware/lib:/usr/lib:/lib
1 libssl.so
2 libcrypto.so
3 libapr-1.so
4 libpthread.a shr_xpg5.o
5 libc.a shr.o
6 libgcc_s.a shr.o
It looks like only the 32-bit version of APR is currently available on the server also. I will update once I can get the 64-bit version installed.
Further Updates
The commands I am trying now are:
$ export CFLAGS=-maix64
$ export OBJECT_MODE=64
$ ./configure --with-apr=/opt/freeware/bin/apr-1-config_64 --with-java-home=/app/java8_64/ --with-ssl=/usr/include/openssl --prefix=/app/tomcat
$ make && make install
No errors but same outcome however. This doesn't seem to build a 64 bit version of the tomcat native module (if that is the issue).

Can't start JIRA by source

OS: CentOS
Installed JIRA by official guide:
https://confluence.atlassian.com/adminjiraserver/installing-jira-applications-on-linux-from-archive-file-938846844.html
When start JIRA, got error:
[jira#node1 bin]$ ./start-jira.sh
To run JIRA in the foreground, start the server with start-jira.sh -fg
executing as current user
`sMMMMMMMMMMMMMM+
MMMMMMMMMMMMMM
:sdMMMMMMMMMMM
MMMMMM
`sMMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sMMMMMMMMMMM MMMMM
MMMMMM `UOJ
`sMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sdMMMMMMMMMM MMMMM
MMMMMM `UOJ
MMMMMM
+MMMMM
MMMMM
`UOJ
Atlassian Jira
Version : 8.5.1
If you encounter issues starting or stopping JIRA, please see the Troubleshooting guide at https://docs.atlassian.com/jira/jadm-docs-085/Troubleshooting+installation
Using JIRA_HOME: /home/jira/jirasoftware-home
Server startup logs are located in /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/logs/catalina.out
Using CATALINA_BASE: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone
Using CATALINA_HOME: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone
Using CATALINA_TMPDIR: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/temp
Using JRE_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/bin/java
Using CLASSPATH: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/bin/bootstrap.jar:/home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/bin/tomcat-juli.jar
Using CATALINA_PID: /home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/work/catalina.pid
/home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/bin/check-java.sh: line 31: [: -ne: unary operator expected
/home/jira/jirasoftware/atlassian-jira-software-8.5.1-standalone/bin/set-gc-params.sh: line 5: [: -ge: unary operator expected
Existing PID file found during start.
Unable to read PID file. Start aborted.
Check JAVA_HOME:
[jira#node1 bin]$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/bin/java
Check PID:
[jira#node1 bin]$ ps aux | grep jira
root 8185 0.0 0.4 241148 4552 pts/0 S 06:47 0:00 sudo su - jira
root 8186 0.0 0.2 191748 2300 pts/0 S 06:47 0:00 su - jira
jira 8187 0.0 0.2 116096 2812 pts/0 S 06:47 0:00 -bash
jira 8299 0.0 0.1 155364 1876 pts/0 R+ 07:04 0:00 ps aux
jira 8300 0.0 0.0 112712 960 pts/0 R+ 07:04 0:00 grep --color=auto jira
No PID exists in the jira user process. Why aborted?

Use script to stop the jira and then start it again. As you can see in the log you have already PID file:
Existing PID file found during start.
Or you can remove the PID file by hand and try to start again

API calls to Heroku web-app running in webapp-runner eventually fail with NoSuchMethodError then NoClassDefFoundError in google.common

I am running a war on Heroku using webapp-runner. I deploy the application using the heroku-maven-plugin version 1.2 via the following command: mvn heroku:deploy-war. Initially, the app works and all endpoints return valid responses. However, if I allow the app to idle long enough for Heroku to put it to sleep and then invoke an endpoint which calls into guava I receive a NoSuchMethodError:
2017-09-23T19:19:45.388865+00:00 app[web.1]: SEVERE: Servlet.service() for servlet [jersey-serlvet] in context with path [] threw exception [org.glassfish.jersey.server.ContainerException: java.lang.NoSuchMethodError: com.google.common.base.CharMatcher.ascii()Lcom/google/common/base/CharMatcher;] with root cause
2017-09-23T19:19:45.388866+00:00 app[web.1]: java.lang.NoSuchMethodError: com.google.common.base.CharMatcher.ascii()Lcom/google/common/base/CharMatcher;
2017-09-23T19:19:45.388867+00:00 app[web.1]: at com.google.common.io.BaseEncoding$Alphabet.<init>(BaseEncoding.java:453)
2017-09-23T19:19:45.388868+00:00 app[web.1]: at com.google.common.io.BaseEncoding$Base64Encoding.<init>(BaseEncoding.java:892)
2017-09-23T19:19:45.388869+00:00 app[web.1]: at com.google.common.io.BaseEncoding.<clinit>(BaseEncoding.java:317)
...application specific stack trace
All subsequent calls to the same API produce a NoClassDefFoundError at the same point
2017-09-23T19:22:24.454901+00:00 app[web.1]: SEVERE: Servlet.service() for servlet [jersey-serlvet] in context with path [] threw exception [org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class com.google.common.io.BaseEncoding] with root cause
2017-09-23T19:22:24.454903+00:00 app[web.1]: java.lang.NoClassDefFoundError: Could not initialize class com.google.common.io.BaseEncoding
...application specific stack trace
These issues seem to suggest that the guava jar is present at compile time but not present at runtime. However, I logged-in to the web dyno and verified that the guava jar was included in my warfile
my-mbp:TrickServer me$ heroku ps:exec
Establishing credentials... done
Connecting to web.1 on ⬢ myapp...
~ $ cd target/
~/target $ ls
MyApp.war dependency mvn-dependency-list.log tomcat.52079
~/target $ jar -tf MyApp.war
...lots of dependencies...
WEB-INF/lib/google-oauth-client-1.20.0.jar
WEB-INF/lib/gson-2.2.4.jar
WEB-INF/lib/guava-23.0.jar <---guava
WEB-INF/lib/guava-jdk5-13.0.jar
...lots more dependencies...
I am struggling to explain why the endpoints work immediately after the app is deployed but later produce these errors. To me this behavior seems to suggest that Heroku is potentially supplying a different classpath when my app wakes up from sleep than when it is initially run or that Heroku is moving/cleaning up the guava jarfile.
Contents of my Procfile:
web: java $JAVA_OPTS -jar target/dependency/webapp-runner.jar --port $PORT --expand-war target/MyApp.war
Java Processes runnning on my web dyno:
~/target $ ps -ef | grep java
u30439 4 1 0 18:50 ? 00:00:44 java -Xmx300m -Xss512k -Dfile.encoding=UTF-8 -Duser.timezone=UTC -jar target/dependency/webapp-runner.jar --port 52079 target/MyApp.war
u30439 27 4 0 18:50 ? 00:00:00 bash --login -c java $JAVA_OPTS -jar target/dependency/webapp-runner.jar $WEBAPP_RUNNER_OPTS --port 52079 target/MyApp.war
Update 1
Since I am invoking my webapp with the --expand-war argument I also checked the jarfiles in the expanded directory to verify that guava was present. It is:
~/target/tomcat.55320/webapps/expanded/WEB-INF/lib $ ls
...dependencies...
google-oauth-client-1.20.0.jar
gson-2.2.4.jar
guava-23.0.jar
guava-jdk5-13.0.jar
...more dependencies...
Update 2
I added the following logic to the problematic web service to printout the classpath and the resources on it:
logger.info("System Classpath: " + System.getProperty("java.class.path"));
logger.info("Runtime Classes...");
ClassLoader cl = UserService.class.getClassLoader();
URL[] urls = ((URLClassLoader) cl).getURLs();
for(URL url: urls){
logger.info(url.getFile());
}
The next time the error occurred I examined the logs and to my surprise found that the guava jar was present on the runtime classpath!
2017-09-24T12:07:40.843438+00:00 app[web.1]: [heroku-exec] ERROR: Could not connect to proxy:
2017-09-24T12:07:40.844145+00:00 app[web.1]: [heroku-exec] ERROR: Too many reconnect attempts. Waiting 30 seconds...
2017-09-24T12:07:52.671620+00:00 app[web.1]: Sep 24, 2017 12:07:52 PM org.myorg.server.web.services.MyService authenticate
2017-09-24T12:07:52.671631+00:00 app[web.1]: INFO: System Classpath: target/dependency/webapp-runner.jar
2017-09-24T12:07:52.671931+00:00 app[web.1]: Sep 24, 2017 12:07:52 PM org.myorg.server.web.services.MyService authenticate
2017-09-24T12:07:52.671932+00:00 app[web.1]: INFO: Runtime Classes...
2017-09-24T12:07:52.672277+00:00 app[web.1]: Sep 24, 2017 12:07:52 PM org.myorg.server.web.services.MyService authenticate
2017-09-24T12:07:52.672279+00:00 app[web.1]: INFO: /app/target/tomcat.28304/webapps/expanded/WEB-INF/classes/
....
2017-09-24T12:07:52.690304+00:00 app[web.1]: Sep 24, 2017 12:07:52 PM org.myorg.server.web.services.MyService authenticate
2017-09-24T12:07:52.690306+00:00 app[web.1]: INFO: /app/target/tomcat.28304/webapps/expanded/WEB-INF/lib/google-oauth-client-1.20.0.jar
2017-09-24T12:07:52.690501+00:00 app[web.1]: Sep 24, 2017 12:07:52 PM org.myorg.server.web.services.MyService authenticate
2017-09-24T12:07:52.690503+00:00 app[web.1]: INFO: /app/target/tomcat.28304/webapps/expanded/WEB-INF/lib/guava-23.0.jar <--- Guava!!!
....
What is going on here? How do I debug this?

You probably have two versions of guava or its related jars on your classpath. See NoSuchMethodError exception when using com.google.common.base.Splitter

After some debugging, I discovered that my program had two different versions of Guava on the classpath (guava-23.0.jar & guava-jdk5-13.0.jar). The debugging tip suggested here was necessary but not sufficient for me to get to the bottom of this.
When working with ClassLoaders it's important to remember that the getClassLoader method defined in the .class object returns a reference to the ClassLoader that originally loaded the class. To find the duplicate jar, it was crucial to invoke classLoader.getResource("/com/google/common/base/CharMatcher.class") on the same ClassLoader which loaded the class that later failed with the NoSuchMethodError.
For posterity, the specific dependency which caused the conflict was com.google.api-client. I resolved it by adding the following exclusion to the dependency in my pom.xml
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.22.0</version>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>

How do I start Jenkins on port 8000?

With Hudson, running java -jar hudson-3.0.0-bundled.war -httpPort=8000 worked. I've tried the same with Jenkins, but it still runs on port 8080:
$ java -jar jenkins.war -httpPort=8000
Running from: /home/user/jenkins/jenkins.war
webroot: $user.home/.jenkins
Sep 25, 2013 12:22:14 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Jenkins home directory: /home/user/.jenkins found at: $user.home/.jenkins
Sep 25, 2013 12:22:15 PM winstone.Logger logInternal
INFO: HTTP Listener started: port=8080

There is a '-' missing in your command line.
From jenkins wiki: "--httpPort=$HTTP_PORT"
So use java -jar jenkins.war --httpPort=8000 instead of java -jar jenkins.war -httpPort=8000

Checking my jenkins init file I think you need --httpPort=$JENKINS_PORT. Have you tried with two dashes before httpPort?
Whether port 8000 is a sensible choice is a whole other question!