Develop Reference

Hybris 6.7 server startup issue on upgrading from 6.3

While doing an prod deployment and starting the server with 6.7 after upgrading from 6.3 we receive the following error in the console
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | SEVERE: LifecycleException
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | java.lang.VerifyError: (class: de/hybris/bootstrap/loader/metrics/ClassLoaderMetricRegistry, method: incrementEventCounter, signature: (Lde/hybris/bootstrap/loader/metrics/ClassLoaderMetricEvent;)V, offset: 21) invokedynamic bytecode is not supported in this class file version
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | at de.hybris.bootstrap.loader.PlatformInPlaceClassLoader.(PlatformInPlaceClassLoader.java:32)
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | at de.hybris.tomcat.HybrisWebappLoader$HybrisWebappClassLoader$1.call(HybrisWebappLoader.java:127)
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | at de.hybris.tomcat.HybrisWebappLoader$HybrisWebappClassLoader$1.call(HybrisWebappLoader.java:1)
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | at de.hybris.tomcat.HybrisWebappLoader$HybrisWebappClassLoader.invokeWithLock(HybrisWebappLoader.java:190)
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | at de.hybris.tomcat.HybrisWebappLoader$HybrisWebappClassLoader.getPlatfromInPlaceClassLoader(HybrisWebappLoader.java:109)
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | at de.hybris.tomcat.HybrisWebappLoader$HybrisWebappClassLoader.(HybrisWebappLoader.java:104)
INFO | jvm 1 | main | 2019/08/13 00:46:11.308 | at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Evnironment details:-
Java : SAP JVM 8
Hybris : 6.7 patch 13
Tried reverting back to Java 8 instead of SAP_JVM 8 but still the same issue. Works fine in other environments with both java configs.

Actually this issue was due to one property in local.properties which is
tomcat.generaloptions=-XX:-UseSplitVerifier -Xms12g -Xmx12g -XX:+UseTLAB -XX:+PrintTenuringDistribution -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:+UseCompressedOops -Xloggc:${HYBRIS_LOG_DIR}/java_gc.log -Dcom.sun.management.jmxremote -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dorg.tanukisoftware.wrapper.WrapperManager.mbean=true -Djava.endorsed.dirs="%CATALINA_HOME%/lib/endorsed" -Dcatalina.base=%CATALINA_BASE% -Dcatalina.home=%CATALINA_HOME% -Dfile.encoding=UTF-8 -Dlog4j.configuration=log4j_init_tomcat.properties -Djava.util.logging.config.file=jdk_logging.properties -Djava.io.tmpdir="${HYBRIS_TEMP_DIR}" -Dcom.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.dump=true
This property includes -XX:-UseSplitVerifier which was causing the issue. -XX:-UseSplitVerifier is deprecated as of Java 8.

Spark Cannot assign requested address: Service Driver failed after 16 retries

I have a cluster of 3 workers Spark. (worker-1, worker-2, worker-3) that runs with Spark 2.0.2.
The Spark Master is started on worker-1.
I submit my application with the following script :
#!/bin/bash
sparkMaster=spark://worker-1:6066
mainClass=my.package.Main
jar=/path/to/my/jar-with-dependencies.jar
driverPort=7079
blockPort=7082
deployMode=cluster
$SPARK_HOME/bin/spark-submit \
--conf "spark.driver.port=${driverPort}"\
--conf "spark.blockManager.port=${blockPort}"\
--class $mainClass \
--master $sparkMaster \
--deploy-mode $deployMode \
$jar
When my driver is started on the worker-1 (Worker + Master), everything is ok, and my application is correctly executed using all workers
But when my driver start on another worker (worker-2 or worker-3), he fails with error :
Launch Command: "/usr/java/jdk1.8.0_181-amd64/jre/bin/java" "-cp" "/root/spark-2.0.2-bin-hadoop2.7/conf/:/root/spark-2.0.2-bin-hadoop2.7/jars/*" "-Xmx1024M" "-Dspark.submit.deployMode=cluster" "-Dspark.app.name=my.package.Main" "-Dspark.driver.port=7083" "-Dspark.blockManager.port=7082" "-Dspark.master=spark://worker-1:7077" "-Dspark.jars=file:/path/to/my/jar-with-dependencies.jar" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker#worker-2:7078" "/data/spark/work/driver-20181001132624-0001/jar-with-dependencies.jar" "my.package.Main"
========================================
org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66) | Service 'Driver' could not bind on port 0. Attempting port 1.
org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66) | Service 'Driver' could not bind on port 0. Attempting port 1.
...
org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66) | Service 'Driver' could not bind on port 0. Attempting port 1.
org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66) | Service 'Driver' could not bind on port 0. Attempting port 1.
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'Driver' failed after 16 retries! Consider explicitly setting the appropriate port for the service 'Driver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:748)
My 3 workers are configured as follow :
SPARK_LOCAL_IP=worker-[X]
SPARK_LOCAL_DIRS=/data/spark/tmp
SPARK_WORKER_PORT=7078
SPARK_WORKER_DIR=/data/spark/work
SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400 -Dspark.worker.cleanup.interval=1800"
After multiple attempts to solve this problem, I tried to force the start of the driver on the master machine by adding to my submit the following option :
--conf "spark.driver.host=worker-1"
But the driver still start on a random worker, so it does not solve my problem.
Edit :
When I submit with the spark.driver.host option, the option does not appear in the Launch Command log (but the spark.driver.port appear, so I don't understand why my option is not taken this time).
Edit 2 :
I have done some deeper tests :
I now have only one worker running on worker-2, still submitting from worker-1 where my master is running.
When I submit my application, I can see on my worker logs :
2018-10-04 11:27:39,794 | dispatcher-event-loop-6 | INFO | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) | Asked to launch driver driver-20181004112739-0003
2018-10-04 11:27:39,833 | DriverRunner for driver-20181004112739-0003 | INFO | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) | Copying user jar file:/path/to/myjar-with-depencies.jar to /data/spark/work/driver-20181004112739-0003/myjar-with-depencies.jar
2018-10-04 11:27:39,833 | DriverRunner for driver-20181004112739-0003 | INFO | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) | Copying /path/to/myjar-with-depencies.jar to /data/spark/work/driver-20181004112739-0003/myjar-with-depencies.jar
2018-10-04 11:27:40,243 | DriverRunner for driver-20181004112739-0003 | INFO | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) | Launch Command: "/usr/java/jdk1.8.0_181-amd64/jre/bin/java" "-cp" "/root/spark-2.0.0-bin-hadoop2.7/conf/:/root/spark-2.0.0-bin-hadoop2.7/jars/*" "-Xmx1024M" "-Dspark.driver.supervise=false" "-Dspark.history.fs.cleaner.interval=12h" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://worker-1:7077" "-Dspark.history.fs.cleaner.maxAge=1d" "-Dspark.app.name=my.package.Main" "-Dspark.jars=file:/path/to/myjar-with-depencies.jar" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker#worker-2:7078" "/data/spark/work/driver-20181004112739-0003/myjar-with-depencies.jar" "my.package.Main"
2018-10-04 11:27:42,692 | dispatcher-event-loop-8 | WARN | org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66) | Driver driver-20181004112739-0003 exited with failure
And I still have the same error in my driver logs.
I then tried to run manually the command that is launched by the DriverRunner :
"/usr/java/jdk1.8.0_181-amd64/jre/bin/java" "-cp" "/root/spark-2.0.0-bin-hadoop2.7/conf/:/root/spark-2.0.0-bin-hadoop2.7/jars/*" "-Xmx1024M" "-Dspark.driver.supervise=false" "-Dspark.history.fs.cleaner.interval=12h" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://worker-1:7077" "-Dspark.history.fs.cleaner.maxAge=1d" "-Dspark.app.name=my.package.Main" "-Dspark.jars=file:/path/to/myjar-with-depencies.jar" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker#worker-2:7078" "/data/spark/work/driver-20181004112739-0003/myjar-with-depencies.jar" "my.package.Main"
And when I do that, the application start correctly (surprisingly).
What is the difference between my manual start, and the one from the Driver-Runner that can cause my binding error ?
Note :
I have made no modification on the Driver-Runner command line to work
I manually launched my command line in root, and my spark runs in root too.
Had the same behavior on Spark 2.0.0 and Spark 2.0.2

So, I answer my own question as I found the reason of this weird behavior.
It does happend when I run the spark-submit from a machine where there is a spark-env.sh file. And more precisely, when the SPARK_LOCAL_IP is set in this machine.
To avoid this problem, I created a 4th machine, running only a Spark Master and with no spark-env.sh file and from which I run my spark submit.

SonarQube 5.4 won't start wrapper error

The code error here:
Starting SonarQube...
wrapper | Spawning intermediate process...
Started SonarQube.
System info here:
Linux xxxxx 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
And
The wrapper.properties here:
# Path to JVM executable. By default it must be available in PATH.
# Can be an absolute path, for example:
#wrapper.java.command=/path/to/my/jdk/bin/java
wrapper.java.command=/usr/lib/jvm/java-8-oracle/jre/bin/java
#
# DO NOT EDIT THE FOLLOWING SECTIONS
#
#********************************************************************
# Wrapper Java
#********************************************************************
wrapper.java.additional.1=-Djava.awt.headless=true
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
wrapper.java.classpath.1=../../lib/jsw/*.jar
wrapper.java.classpath.2=../../lib/*.jar
wrapper.java.library.path.1=./lib
wrapper.app.parameter.1=org.sonar.application.App
wrapper.java.initmemory=3
wrapper.java.maxmemory=3
#********************************************************************
# Wrapper Logs
#********************************************************************
wrapper.console.format=PM
wrapper.console.loglevel=DEBUG
wrapper.logfile=../../logs/sonar.log
wrapper.logfile.format=M
wrapper.logfile.loglevel=INFO
# Maximum size that the log file will be allowed to grow to before
# the log is rolled. Size is specified in bytes. The default value
# of 0, disables log rolling. May abbreviate with the 'k' (kb) or
# 'm' (mb) suffix. For example: 10m = 10 megabytes.
#wrapper.logfile.maxsize=0
# Maximum number of rolled log files which will be allowed before old
# files are deleted. The default value of 0 implies no limit.
#wrapper.logfile.maxfiles=0
# Log Level for sys/event log output. (See docs for log levels)
wrapper.syslog.loglevel=NONE
#********************************************************************
# Wrapper Windows Properties
#********************************************************************
# Title to use when running as a console
wrapper.console.title=SonarQube
# Disallow start of multiple instances of an application at the same time on Windows
wrapper.single_invocation=true
#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
# using this configuration file has been installed as a service.
# Please uninstall the service before modifying this section. The
# service can then be reinstalled.
# Name of the service
wrapper.ntservice.name=SonarQube
# Display name of the service
wrapper.ntservice.displayname=SonarQube
# Description of the service
wrapper.ntservice.description=SonarQube
# Service dependencies. Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1=
# Mode in which the service is installed. AUTO_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START
# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=false
#********************************************************************
# Forking Properties
#********************************************************************
wrapper.disable_restarts=TRUE
wrapper.ping.timeout=0
wrapper.shutdown.timeout=3000
wrapper.jvm_exit.timeout=3000
if need more information, sonar.sh here:
#! /bin/sh
#
# rc file for SonarQube
#
# chkconfig: 345 96 10
# description: SonarQube system (www.sonarsource.org)
#
### BEGIN INIT INFO
# Provides: sonar
# Required-Start: $network
# Required-Stop: $network
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: SonarQube system (www.sonarsource.org)
# Description: SonarQube system (www.sonarsource.org)
### END INIT INFO
#
# Copyright (c) 1999, 2006 Tanuki Software Inc.
#
# Java Service Wrapper sh script. Suitable for starting and stopping
# wrapped Java applications on UNIX platforms.
#
#-----------------------------------------------------------------------------
# These settings can be modified to fit the needs of your application
# Default values for the Application variables, below.
#
# NOTE: The build for specific applications may override this during the resource-copying
# phase, to fill in a concrete name and avoid the use of the defaults specified here.
DEF_APP_NAME="SonarQube"
DEF_APP_LONG_NAME="SonarQube"
SONAR_HOME=/opt/sonar
PLATFORM=linux-x86-64
# Application
APP_NAME="${DEF_APP_NAME}"
APP_LONG_NAME="${DEF_APP_LONG_NAME}"
# Wrapper
WRAPPER_CMD="${SONAR_HOME}/bin/${PLATFORM}/wrapper"
WRAPPER_CONF="${SONAR_HOME}/conf/wrapper.conf"
# Priority at which to run the wrapper. See "man nice" for valid priorities.
# nice is only used if a priority is specified.
PRIORITY=
# Location of the pid file.
PIDDIR="/var/run"
More information about log error:
[~]:/linux-x86-64/sonar.sh console
Running SonarQube...
wrapper | --> Wrapper Started as Console
wrapper | Using tick timer.
wrapperp | server listening on port 32000.
wrapper | Command[0] : /usr/lib/jvm/java-8-oracle/jre/bin/java
wrapper | Command[1] : -Djava.awt.headless=true
wrapper | Command[2] : -Xms3m
wrapper | Command[3] : -Xmx3m
wrapper | Command[4] : -Djava.library.path=./lib
wrapper | Command[5] : -classpath
wrapper | Command[6] : ../../lib/jsw/wrapper-3.2.3.jar:../../lib/sonar-application-5.4.jar
wrapper | Command[7] : -Dwrapper.key=Z4yQiQf29fGXQCJP
wrapper | Command[8] : -Dwrapper.port=32000
wrapper | Command[9] : -Dwrapper.jvm.port.min=31000
wrapper | Command[10] : -Dwrapper.jvm.port.max=31999
wrapper | Command[11] : -Dwrapper.debug=TRUE
wrapper | Command[12] : -Dwrapper.pid=16048
wrapper | Command[13] : -Dwrapper.version=3.2.3
wrapper | Command[14] : -Dwrapper.native_library=wrapper
wrapper | Command[15] : -Dwrapper.cpu.timeout=10
wrapper | Command[16] : -Dwrapper.jvmid=1
wrapper | Command[17] : org.tanukisoftware.wrapper.WrapperSimpleApp
wrapper | Command[18] : org.sonar.application.App
wrapper | Launching a JVM...
jvm 1 | WrapperManager class initialized by thread: main Using classloader: sun.misc.Launcher$AppClassLoader#5c647e05
jvm 1 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
jvm 1 | Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
jvm 1 |
jvm 1 | Wrapper Manager: JVM #1
jvm 1 | Running a 64-bit JVM.
jvm 1 | Wrapper Manager: Registering shutdown hook
jvm 1 | Wrapper Manager: Using wrapper
jvm 1 | Load native library. One or more attempts may fail if platform specific libraries do not exist.
jvm 1 | Loaded native library: libwrapper-linux-x86-64.so
jvm 1 | Calling native initialization method.
jvm 1 | Inside native WrapperManager initialization method
jvm 1 | Java Version : 1.8.0_77-b03 Java HotSpot(TM) 64-Bit Server VM
jvm 1 | Java VM Vendor : Oracle Corporation
jvm 1 |
jvm 1 | Control event monitor thread started.
jvm 1 | Startup runner thread started.
jvm 1 | WrapperManager.start(org.tanukisoftware.wrapper.WrapperSimpleApp#4f023edb, args[]) called by thread: main
jvm 1 | Communications runner thread started.
jvm 1 | Open socket to wrapper...Wrapper-Connection
jvm 1 | Opened Socket from 31000 to 32000
jvm 1 | Send a packet KEY : Z4yQiQf29fGXQCJP
jvm 1 | handleSocket(Socket[addr=/127.0.0.1,port=32000,localport=31000])
wrapperp | accepted a socket from 127.0.0.1 on port 31000
wrapperp | read a packet KEY : Z4yQiQf29fGXQCJP
wrapper | Got key from JVM: Z4yQiQf29fGXQCJP
wrapperp | send a packet LOW_LOG_LEVEL : 1
wrapperp | send a packet PING_TIMEOUT : 0
wrapperp | send a packet PROPERTIES : (Property Values)
wrapper | Start Application.
wrapperp | send a packet START : start
jvm 1 | Received a packet LOW_LOG_LEVEL : 1
jvm 1 | Wrapper Manager: LowLogLevel from Wrapper is 1
jvm 1 | Received a packet PING_TIMEOUT : 0
jvm 1 | PingTimeout from Wrapper is 0
jvm 1 | Received a packet PROPERTIES : (Property Values)
jvm 1 | Received a packet START : start
jvm 1 | calling WrapperListener.start()
jvm 1 | Waiting for WrapperListener.start runner thread to complete.
jvm 1 | WrapperListener.start runner thread started.
jvm 1 | WrapperSimpleApp: start(args) Will wait up to 2 seconds for the main method to complete.
jvm 1 | WrapperSimpleApp: invoking main method
jvm 1 | 2016.04.10 04:51:23 INFO app[o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonar/temp
jvm 1 | 2016.04.10 04:51:23 INFO app[o.s.p.m.JavaProcessLauncher] Launch process[search]: /usr/lib/jvm/java-8-oracle/jre/bin/java -Djava.awt.headless=true -Xmx1G -Xms256m -Xss256k -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonar/temp -cp ./lib/common/*:./lib/search/* org.sonar.search.SearchServer /tmp/sq-process5756979811171898411properties
jvm 1 | Send a packet START_PENDING : 5000
jvm 1 | Wrapper Manager: ShutdownHook started
jvm 1 | WrapperManager.stop(0) called by thread: Wrapper-Shutdown-Hook
jvm 1 | Send a packet STOP : 0
wrapperp | read a packet START_PENDING : 5000
wrapper | JVM signalled a start pending with waitHint of 5000 millis.
wrapperp | read a packet STOP : 0
wrapper | JVM requested a shutdown. (0)
wrapper | wrapperStopProcess(0) called.
wrapper | Sending stop signal to JVM
wrapperp | send a packet STOP : NULL
jvm 1 | Startup runner thread stopped.
jvm 1 | Send a packet START_PENDING : 5000
jvm 1 | WrapperSimpleApp: start(args) end. Main Completed=false, exitCode=null
jvm 1 | WrapperListener.start runner thread stopped.
jvm 1 | returned from WrapperListener.start()
jvm 1 | Send a packet STARTED :
jvm 1 | Received a packet STOP :
jvm 1 | Thread, Wrapper-Shutdown-Hook, handling the shutdown process.
jvm 1 | calling listener.stop()
jvm 1 | WrapperSimpleApp: stop(0)
jvm 1 | returned from listener.stop() -> 0
jvm 1 | shutdownJVM(0) Thread:Wrapper-Shutdown-Hook
jvm 1 | Send a packet STOPPED : 0
wrapperp | read a packet START_PENDING : 5000
wrapper | JVM signalled a start pending with waitHint of 5000 millis.
wrapperp | read a packet STARTED :
wrapper | JVM signalled that it was started.
wrapperp | send a packet STOP : NULL
wrapperp | read a packet STOPPED : 0
wrapper | JVM signalled that it was stopped.
jvm 1 | Received a packet STOP :
jvm 1 | Closing socket.
wrapperp | socket read no code (closed?).
wrapperp | server listening on port 32001.
jvm 1 | Server daemon shut down
jvm 1 | Wrapper Manager: ShutdownHook complete
wrapper | JVM exited normally.
wrapper | Signal trapped. Details:
wrapper | signal number=17 (SIGCHLD), source="unknown"
wrapper | Received SIGCHLD, checking JVM process status.
wrapper | JVM process exited with a code of 0, leaving the wrapper exit code set to 0.
wrapper | <-- Wrapper Stopped

As discussed, this error occurs when the JVM can not start. In your case the minimum required amount of memory can not be allocated.

Red 5 media server version 1.0 finding correct Java version

I have installed Red 5 media server version 1.0 rc2 to a windows 2008 server, 64 bit.
I am trying to start the Red 5 service, that fails and the log is as below:
STATUS | wrapper | 2014/02/13 15:19:16 | --> Wrapper Started as Service
STATUS | wrapper | 2014/02/13 15:19:16 | Java Service Wrapper Community Edition 32-bit 3.3.6
STATUS | wrapper | 2014/02/13 15:19:16 | Copyright (C) 1999-2009 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2014/02/13 15:19:16 | http://wrapper.tanukisoftware.org
STATUS | wrapper | 2014/02/13 15:19:16 |
STATUS | wrapper | 2014/02/13 15:19:16 | Launching a JVM...
INFO | jvm 1 | 2014/02/13 15:19:16 | WrapperManager: Initializing...
INFO | jvm 1 | 2014/02/13 15:19:16 | WrapperSimpleApp: Class
org.red5.server.Bootstrap found but could not be initialized:
java.lang.UnsupportedClassVersionError: org/red5/server/Bootstrap :
Unsupported major.minor version 51.0
ERROR | wrapper | 2014/02/13 15:19:18 | JVM exited while loading the application.
...
FATAL | wrapper | 2014/02/13 15:19:45 | There were 5 failed launches in a row, each lasting less than 300 seconds. Giving up.
FATAL | wrapper | 2014/02/13 15:19:45 | There may be a configuration problem: please check the logs.
STATUS | wrapper | 2014/02/13 15:19:45 | <-- Wrapper Stopped
This seems from googling something to do with java version so I uninstalled the currect JDK and reinstalled JDK 6 Update 18 as described here: http://red5guide.com/install_jdk_jre_for_red5.html
I have set the windows PATH environment variable to C:\Program Files (x86)\Java\jre6\bin;C:\Program Files (x86)\Java\jdk1.6.0_18\bin
When i run java -version I get:
java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Client VM (build 16.0-b13, mixed mode, sharing)
However I still can't start the Red 5 service with the same log given as shown above.
Any ideas what could be wrong?

Unsupported major.minor version 51.0 indicates that the compiled version should be using JDK version 7. Install that version and run it again.

How can I get Jsvc to find the DaemonLoader?

I have the commons-daemon-x.jar in the classpath. The Jsvc is successfully launching the Java VM. But jsvc is reporting that it cannot find the daemon loader class even though it is in the classpath.
Does anyone know how to solve this problem? I’m running Mac OS X 10.8.3. Thanks.
Here's an excerpt of the jsvc debug output:
...
+-- DUMPING JAVA VM CREATION ARGUMENTS -----------------
| Version: 0x010004
| Ignore Unrecognized Arguments: False
| Extra options: 1
| "-Djava.class.path=commons-daemon-1.0.15.jar:./api-monitor.jar" (0x00000000)
+———————————————————————————
...
Java VM created successfully
Cannot find daemon loader org/apache/commons/daemon/support/DaemonLoader
java_init failed
Service exit with a return value of 1
...
Here’s the full debug output if it helps:
sudo jsvc -jvm server -debug -cp commons-daemon-1.0.15.jar:./api-monitor.jar ApiMonitorDaemon
+-- DUMPING PARSED COMMAND LINE ARGUMENTS --------------
| Detach: True
| Show Version: No
| Show Help: No
| Check Only: Disabled
| Stop: False
| Wait: 0
| Run as service: No
| Install service: No
| Remove service: No
| JVM Name: "server"
| Java Home: "null"
| PID File: "/var/run/jsvc.pid"
| User Name: "null"
| Extra Options: 1
| "-Djava.class.path=commons-daemon-1.0.15.jar:./api-monitor.jar"
| Class Invoked: "ApiMonitorDaemon"
| Class Arguments: 0
+-------------------------------------------------------
Home not specified on command line, using environment
Home not on command line or in environment, searching
Attempting to locate Java Home in /System/Library/Frameworks/JavaVM.framework/Home
Attempting to locate VM configuration file /System/Library/Frameworks/JavaVM.framework/Home/jre/lib/jvm.cfg
Attempting to locate VM configuration file /System/Library/Frameworks/JavaVM.framework/Home/lib/jvm.cfg
Found VM configuration file at /System/Library/Frameworks/JavaVM.framework/Home/lib/jvm.cfg
Found VM client definition in configuration
Checking library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libclient.dylib
Found VM jvm definition in configuration
Checking library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libjvm.dylib
Found VM hotspot definition in configuration
Checking library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libhotspot.dylib
Found VM server definition in configuration
Checking library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libserver.dylib
Found VM classic definition in configuration
Checking library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libclassic.dylib
Cannot locate library for VM classic (skipping)
Java Home located in /System/Library/Frameworks/JavaVM.framework/Home
+-- DUMPING JAVA HOME STRUCTURE ------------------------
| Java Home: "/System/Library/Frameworks/JavaVM.framework/Home"
| Java VM Config.: "/System/Library/Frameworks/JavaVM.framework/Home/lib/jvm.cfg"
| Found JVMs: 4
| JVM Name: "client"
| "/System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libclient.dylib"
| JVM Name: "jvm"
| "/System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libjvm.dylib"
| JVM Name: "hotspot"
| "/System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libhotspot.dylib"
| JVM Name: "server"
| "/System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libserver.dylib"
+-------------------------------------------------------
my-server:Applications developer$ redirecting stdout to /dev/null and stderr to /dev/null
Switching umask back to 022 from 077
Using specific JVM in /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libserver.dylib
Attemtping to load library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libserver.dylib
JVM library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libserver.dylib loaded
Attemtping to load library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libverify.dylib
Shell library /System/Library/Frameworks/JavaVM.framework/Home/../Libraries/libverify.dylib loaded
JVM library entry point found (0x02A49762)
+-- DUMPING JAVA VM CREATION ARGUMENTS -----------------
| Version: 0x010004
| Ignore Unrecognized Arguments: False
| Extra options: 1
| "-Djava.class.path=commons-daemon-1.0.15.jar:./api-monitor.jar" (0x00000000)
+-------------------------------------------------------
| Internal options: 4
| "-Dcommons.daemon.process.id=8919" (0x00000000)
| "-Dcommons.daemon.process.parent=8918" (0x00000000)
| "-Dcommons.daemon.version=1.0.15-dev" (0x00000000)
| "abort" (0x022bef30)
+-------------------------------------------------------
Java VM created successfully
Cannot find daemon loader org/apache/commons/daemon/support/DaemonLoader
java_init failed
Service exit with a return value of 1

You probably found the solution by now, but just in case someone stumble over this in the future, what I had missed in mine is that current directory defaults to '/', so you have to specify it if you want to use relative paths.
sudo jsvc -jvm server -cwd /path/to/my/daemon/ -cp commons-daemon-1.0.15.jar:./api-monitor.jar ApiMonitorDaemon
Note: my OS is ubuntu 12.10

Check the permissions of the commons-daemon-1.0.15.jar file, make sure your current user can read it.