I was having problems configuring my Raspberry Pi 4, as I explained in my previous question (JavaFX in RaspberryPi 4 with Liberica (Cannot open display ,Cannot create resource)), so I decided to start over.
I followed the steps on the Gluon tutorial (https://docs.gluonhq.com/#_javafx_on_embedded), (thanks to #José Pereda) but in the end, when I have to execute the line:
sudo -E java -Dglass.platform=gtk --module-path /opt/javafx-sdk-17/lib --add-modules javafx.controls -cp dist/. hellofx.HelloFX
It shows me this error:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:1051)
Caused by: java.lang.UnsupportedOperationException: Unable to open DISPLAY
at javafx.graphics/com.sun.glass.ui.gtk.GtkApplication.lambda$new$6(GtkApplication.java:173)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at javafx.graphics/com.sun.glass.ui.gtk.GtkApplication.<init>(GtkApplication.java:171)
at javafx.graphics/com.sun.glass.ui.gtk.GtkPlatformFactory.createApplication(GtkPlatformFactory.java:41)
at javafx.graphics/com.sun.glass.ui.Application.run(Application.java:145)
at javafx.graphics/com.sun.javafx.tk.quantum.QuantumToolkit.startup(QuantumToolkit.java:281)
at javafx.graphics/com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:288)
at javafx.graphics/com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:160)
at javafx.graphics/com.sun.javafx.application.LauncherImpl.startToolkit(LauncherImpl.java:658)
at javafx.graphics/com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:409)
at javafx.graphics/com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:363)
... 5 more
I have activated X11Forwarding in /etc/ssh/sshd_config.
I remember this error from other Attempts to work with JavaFX, so I tried to change from -Dglass.platform=gtk to -Dglass.platform=Monocle, but then It shows this fail:
Exception in thread "Event Thread" java.lang.UnsatisfiedLinkError: 'void com.sun.glass.ui.monocle.DispmanScreen.wrapNativeSymbols()'
at javafx.graphics/com.sun.glass.ui.monocle.DispmanScreen.wrapNativeSymbols(Native Method)
at javafx.graphics/com.sun.glass.ui.monocle.DispmanScreen.<init>(DispmanScreen.java:34)
at javafx.graphics/com.sun.glass.ui.monocle.DispmanPlatform.createScreen(DispmanPlatform.java:38)
at javafx.graphics/com.sun.glass.ui.monocle.NativePlatform.getScreen(NativePlatform.java:129)
at javafx.graphics/com.sun.glass.ui.monocle.MonocleApplication.staticScreen_getScreens(MonocleApplication.java:222)
at javafx.graphics/com.sun.glass.ui.Screen.initScreens(Screen.java:412)
at javafx.graphics/com.sun.glass.ui.Application.lambda$run$1(Application.java:153)
at javafx.graphics/com.sun.glass.ui.monocle.RunnableProcessor.runLoop(RunnableProcessor.java:92)
at javafx.graphics/com.sun.glass.ui.monocle.RunnableProcessor.run(RunnableProcessor.java:51)
at java.base/java.lang.Thread.run(Thread.java:834)
I'm using a 5.0 inch screen (http://www.lcdwiki.com/5inch_HDMI_Display-B) with an adapter from hdmi to microhdmi.
After Comments things
Enabling X11 Forwarding,same errors trying with sudo -E java -Dglass.platform=gtk --module-path /opt/javafx-sdk-17/lib --add-modules javafx.controls export DISPLAY='IP:0.0' -cp dist/. hellofx.HelloFX
then trying with:
sudo -E java -Degl.displayid=/dev/dri/card1 -Dembedded=monocle -Dglass.platform=Monocle -Dmonocle.platform=EGL -Dmonocle.egl.lib=/opt/javafx-sdk-17/lib/libgluon_drm-1.1.0.so -Dprism.verbose=true --module-path /opt/javafx-sdk-17/lib --add-modules javafx.controls -cp dist/. hellofx.HelloFX
I get:
Prism pipeline init order: es2 sw
Using Double Precision Marlin Rasterizer
Using dirty region optimizations
Using system sized mask for primitives
Not forcing power of 2 sizes for textures
Using hardware CLAMP_TO_ZERO mode
Opting in for HiDPI pixel scaling
Prism pipeline name = com.sun.prism.es2.ES2Pipeline
Loading ES2 native library ... prism_es2_monocle
succeeded.
GLFactory using com.sun.prism.es2.MonocleGLFactory
[GluonDRM] use GPU at /dev/dri/card1 and display id -1
S0 NOT defined:
[GluonDRM] Device /dev/dri/card1 could be opened, but has no drm capabilities.eglinit failed
eglbind ok
eglCreateWindowSurface called
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xb6dbd830, pid=904, tid=921
#
# JRE version: OpenJDK Runtime Environment (11.0.9.1+1) (build 11.0.9.1+1-post-Raspbian-1deb10u2)
# Java VM: OpenJDK Server VM (11.0.9.1+1-post-Raspbian-1deb10u2, mixed mode, g1 gc, linux-)
# Problematic frame:
# C [libc.so.6+0x77830] strlen+0x0
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/pi/Downloads/samples/CommandLine/Modular/CLI/hellofx/hs_err_pid904.log
#
# If you would like to submit a bug report, please visit:
# Unknown
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abortado
I used ulimit -c unlimited, then used same command, getting:
Prism pipeline init order: es2 sw
Using Double Precision Marlin Rasterizer
Using dirty region optimizations
Using system sized mask for primitives
Not forcing power of 2 sizes for textures
Using hardware CLAMP_TO_ZERO mode
Opting in for HiDPI pixel scaling
Prism pipeline name = com.sun.prism.es2.ES2Pipeline
Loading ES2 native library ... prism_es2_monocle
succeeded.
GLFactory using com.sun.prism.es2.MonocleGLFactory
[GluonDRM] use GPU at /dev/dri/card1 and display id -1
S0 NOT defined:
[GluonDRM] Device /dev/dri/card1 could be opened, but has no drm capabilities.eglinit failed
eglbind ok
eglCreateWindowSurface called
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xb6dcc830, pid=923, tid=940
#
# JRE version: OpenJDK Runtime Environment (11.0.9.1+1) (build 11.0.9.1+1-post-Raspbian-1deb10u2)
# Java VM: OpenJDK Server VM (11.0.9.1+1-post-Raspbian-1deb10u2, mixed mode, g1 gc, linux-)
# Problematic frame:
# C [libc.so.6+0x77830] strlen+0x0
#
# Core dump will be written. Default location: /home/pi/Downloads/samples/CommandLine/Modular/CLI/hellofx/core
#
# An error report file with more information is saved as:
# /home/pi/Downloads/samples/CommandLine/Modular/CLI/hellofx/hs_err_pid923.log
#
# If you would like to submit a bug report, please visit:
# Unknown
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abortado
I encountered these problems when I building openjdk9.
os: macos high sierra(10.13.5)
xcode： 9.4
freetype: 2.9.1
I encountered these problems when I building openjdk9
Building target 'default (exploded-image)' in configuration 'macosx-
x86_64-normal-server-slowdebug'
Warning: No mercurial configuration present and no .src-rev
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x000000010394aca8, pid=13189, tid=8451
#
# JRE version: OpenJDK Runtime Environment (9.0.4) (slowdebug build
9.0.4-internal+0-adhoc.apple.jdk9u)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 9.0.4-internal+0-adhoc.apple.jdk9u, mixed mode, tiered, compressed oops, serial gc, bsd-amd64)
# Problematic frame:
# V [libjvm.dylib+0xd4aca8] PerfData::~PerfData()+0x8
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/apple/SourceCode/jdk9u/make/hs_err_pid13189.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
[error occurred during error reporting (), id 0x4]
make[3]: *** [/Users/apple/SourceCode/jdk9u/build/macosx-x86_64-normal-server-slowdebug/jdk/_packages_attribute.done] Abort trap: 6
make[2]: *** [exploded-image-optimize] Error 2
ERROR: Build failed for target 'default (exploded-image)' in configuration 'macosx-x86_64-normal-server-slowdebug' (exit code 2)
No indication of failed target found.
Hint: Try searching the build log for '] Error'.
Hint: See common/doc/building.html#troubleshooting for assistance.
make[1]: *** [main] Error 2
make: *** [default] Error
I try to set slowdebug and fastdebug, always failure.
For the past few weeks, we have been having an issue where the ElasticSearch server dies. I cannot tell what the problem is and I'm not really sure where to even start?
We can restart the server and it will run ok for random lengths of time. Sometimes for the rest of the day, sometimes minutes, but it always crashes again eventually.
Here are some details that I hope someone will be able to process and point me in the right direction:
ElasticSearch Server info:
{
"name" : "WIuGVV9",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "T2Vvt3hzQhSJa4ZFWtdMKA",
"version" : {
"number" : "5.5.1",
"build_hash" : "19c13d0",
"build_date" : "2017-07-18T20:44:24.823Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}
$ cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)
$ java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)
$ free
total used free shared buff/cache available
Mem: 7915072 3650556 2827472 378156 1437044 3802152
Swap: 0 0 0
elasticsearch.log:
[2017-08-02T14:14:37,927][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna3117985363123958860.tmp: /tmp/jna--1985354563/jna3117985363123958860.tmp: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_141]
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_141]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_141]
at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_141]
at java.lang.System.load(System.java:1086) ~[?:1.8.0_141]
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0.jar:4.4.0 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0.jar:4.4.0 (b0)]
at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0.jar:4.4.0 (b0)]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_141]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_141]
at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.5.1.jar:5.5.1]
[2017-08-02T14:14:37,936][WARN ][o.e.b.Natives ] cannot check if running as root because JNA is not available
[2017-08-02T14:14:37,937][WARN ][o.e.b.Natives ] cannot install system call filter because JNA is not available
[2017-08-02T14:14:37,937][WARN ][o.e.b.Natives ] cannot register console handler because JNA is not available
[2017-08-02T14:14:37,941][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2017-08-02T14:14:37,941][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2017-08-02T14:14:38,150][INFO ][o.e.n.Node ] [] initializing ...
[2017-08-02T14:14:38,341][INFO ][o.e.e.NodeEnvironment ] [WIuGVV9] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [7.8gb], net total_space [38.7gb], spins? [unknown], types [rootfs]
[2017-08-02T14:14:38,341][INFO ][o.e.e.NodeEnvironment ] [WIuGVV9] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-08-02T14:14:38,452][INFO ][o.e.n.Node ] node name [WIuGVV9] derived from node ID [WIuGVV9sS0mlLPTxRVKn0w]; set [node.name] to override
[2017-08-02T14:14:38,453][INFO ][o.e.n.Node ] version[5.5.1], pid[21555], build[19c13d0/2017-07-18T20:44:24.823Z], OS[Linux/3.10.0-327.4.4.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
[2017-08-02T14:14:38,453][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-08-02T14:14:40,409][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [aggs-matrix-stats]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [ingest-common]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [lang-expression]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [lang-groovy]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [lang-mustache]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [lang-painless]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [parent-join]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [percolator]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [reindex]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [transport-netty3]
[2017-08-02T14:14:40,410][INFO ][o.e.p.PluginsService ] [WIuGVV9] loaded module [transport-netty4]
[2017-08-02T14:14:40,411][INFO ][o.e.p.PluginsService ] [WIuGVV9] no plugins loaded
[2017-08-02T14:14:42,797][INFO ][o.e.d.DiscoveryModule ] [WIuGVV9] using discovery type [zen]
[2017-08-02T14:14:43,759][INFO ][o.e.n.Node ] initialized
[2017-08-02T14:14:43,760][INFO ][o.e.n.Node ] [WIuGVV9] starting ...
[2017-08-02T14:14:44,061][INFO ][o.e.t.TransportService ] [WIuGVV9] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2017-08-02T14:14:44,088][WARN ][o.e.b.BootstrapChecks ] [WIuGVV9] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-08-02T14:14:47,174][INFO ][o.e.c.s.ClusterService ] [WIuGVV9] new_master {WIuGVV9}{WIuGVV9sS0mlLPTxRVKn0w}{uZIN-61JT4KLP1xUSVTdLQ}{localhost}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-08-02T14:14:47,204][INFO ][o.e.h.n.Netty4HttpServerTransport] [WIuGVV9] publish_address {66.55.80.152:9200}, bound_addresses {[::]:9200}
[2017-08-02T14:14:47,204][INFO ][o.e.n.Node ] [WIuGVV9] started
[2017-08-02T14:14:47,589][INFO ][o.e.g.GatewayService ] [WIuGVV9] recovered [3] indices into cluster_state
[2017-08-02T14:14:48,268][INFO ][o.e.c.r.a.AllocationService] [WIuGVV9] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[orgs][4], [orgs][0]] ...]).
[2017-08-02T14:18:01,316][INFO ][o.e.c.m.MetaDataDeleteIndexService] [WIuGVV9] [orgs/ZcizOIAsRWqSXZY8ZiR0BA] deleting index
/etc/elasticsearch/elasticsearch.yml:
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
transport.host: localhost
transport.tcp.port: 9300
#network.bind_host: "0.0.0.0"
#network.publish_host: _non_loopback:ipv4_
#network.host: _local_
network.host: 0.0.0.0
#network.bind_host:
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
/etc/elasticsearch/jvm.options :
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms2g
-Xmx2g
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## optimizations
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# force the server VM (remove on 32-bit client JVMs)
-server
# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# use old-style file permissions on JDK9
-Djdk.io.permissionsUseCanonicalPath=true
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${heap.dump.path}
## GC logging
#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime
# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${loggc}
# By default, the GC log file will not rotate.
# By uncommenting the lines below, the GC log file
# will be rotated every 128MB at most 32 times.
#-XX:+UseGCLogFileRotation
#-XX:NumberOfGCLogFiles=32
#-XX:GCLogFileSize=128M
# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
# If documents were already indexed with unquoted fields in a previous version
# of Elasticsearch, some operations may throw errors.
#
# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
# only for migration purposes.
#-Delasticsearch.json.allow_unquoted_field_names=true
/etc/sysconfig/elasticsearch :
################################
# Elasticsearch
################################
# Elasticsearch home directory
#ES_HOME=/usr/share/elasticsearch
# Elasticsearch Java path
#JAVA_HOME=
# Elasticsearch configuration directory
#CONF_DIR=/etc/elasticsearch
# Elasticsearch data directory
#DATA_DIR=/var/lib/elasticsearch
# Elasticsearch logs directory
#LOG_DIR=/var/log/elasticsearch
# Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch
# Additional Java OPTS
#ES_JAVA_OPTS=
# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true
################################
# Elasticsearch service
################################
# SysV init.d
#
# When executing the init script, this user will be used to run the elasticsearch service.
# The default value is 'elasticsearch' and is declared in the init.d file.
# Note that this setting is only used by the init script. If changed, make sure that
# the configured user can read and write into the data, work, plugins and log directories.
# For systemd service, the user is usually configured in file /usr/lib/systemd/system/elasticsearch.service
#ES_USER=elasticsearch
#ES_GROUP=elasticsearch
# The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process
ES_STARTUP_SLEEP_TIME=5
################################
# System properties
################################
# Specifies the maximum file descriptor number that can be opened by this process
# When using Systemd, this setting is ignored and the LimitNOFILE defined in
# /usr/lib/systemd/system/elasticsearch.service takes precedence
#MAX_OPEN_FILES=65536
# The maximum number of bytes of memory that may be locked into RAM
# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option
# in elasticsearch.yml.
# When using Systemd, the LimitMEMLOCK property must be set
# in /usr/lib/systemd/system/elasticsearch.service
#MAX_LOCKED_MEMORY=unlimited
# Maximum number of VMA (Virtual Memory Areas) a process can own
# When using Systemd, this setting is ignored and the 'vm.max_map_count'
# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
#MAX_MAP_COUNT=262144
Can anyone determine what my error is? I can't tell and my best best is either a permissions issue, or a memory issue. We do have 8 gb, which is the barely make it mark.
Thoughts?
Thanks a million.
[EDIT: New Details] :
dmesg outputs a lot. I'm not sure if there is a way to get timestamps out of the output, but over and over I see:
[10131180.901171] Out of memory: Kill process 13777 (java) score 295 or sacrifice child
[10131180.901186] Killed process 13777 (java) total-vm:5800372kB, anon-rss:2334928kB, file-rss:80kB
[10137088.438235] exim[7581]: segfault at 58 ip 000000000046bee7 sp 00007ffdd0dc63e0 error 4 in exim[400000+fa000]
[10138765.544389] exim[16401]: segfault at 58 ip 000000000046bee7 sp 00007ffede7cc3f0 error 4 in exim[400000+fa000]
[10162265.107101] exim[28217]: segfault at 58 ip 000000000046bee7 sp 00007fff84afb6e0 error 4 in exim[400000+fa000]
I think this is the root cause of our issue and when the boss gets in we will see about upgrading the server as it only has 8gb of ram.
[EDIT: even newer details] :
Running this command:
ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5
After the ES server is killed:
6.3 0.0 767376 560 /usr/local/cpanel/3rdparty/bin/clamd
2.7 25.5 310612 5060 dovecot/lmtp
2.4 0.3 860920 24850 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/lib/mysql/gigenet.pwi.com.err --open-files-limit=10000 --pid-file=/var/lib/mysql/gigenet.pwi.com.pid
0.3 0.3 75216 17701 tailwatchd
0.3 0.0 393280 5384 /usr/sbin/named -u named
After the ES server is restarted
9.3 45.5 4834304 6815 /bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Xms4g -Xmx4g -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
6.3 0.0 767376 560 /usr/local/cpanel/3rdparty/bin/clamd
2.4 0.3 860920 24850 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/lib/mysql/gigenet.pwi.com.err --open-files-limit=10000 --pid-file=/var/lib/mysql/gigenet.pwi.com.pid
1.2 1.1 243444 23536 /usr/local/cpanel/3rdparty/perl/524/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --max-spare=1 --max-children=3 --allowed-ips=127.0.0.1,::1 --pidfile=/var/run/spamd.pid --listen=5 --listen=6
1.2 0.5 244292 6618 spamd child
Based on the results in the answers to the question, I found this post
Prevent elasticsearch from being killed by OOM killer
While trying to figure out why OOM is killing java. I have since changed my config to match the recommendations in the first answer and the server still shuts down.
based on dmesg, the processes being killed are:
Out of memory: Kill process 24532 (java) score 293
Out of memory: Kill process 3408 (clamd) score 60
Out of memory: Kill process 1970 (lmtp) score 28
Out of memory: Kill process 17806 (mysqld) score 27
Is there anything in the logs of Elasticsearch that indicates are clean shutdown? Can you paste the lines before the restart?
Can you check with dmesg if the kernel OOM killer is stopping elasticsearch? This happens when the operating system is advised to free memory at all cost. The OOM killer then usually picks the process with the most memory and kills it. Most of the time this is Elasticsearch.
Are there any other services running on that machine, that might cause triggering the OOM killer?
Have you monitoring the system over time? Is memory eaten up? If the process is killed, are certain resources back to normal?
I am getting below error while restarting Jenkins.
INFO: Ready to receive data from Gerrit: Gerrit-QA
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007b1480000, 246939648, 0) failed; error='C
annot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 246939648 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/share/tomcat6/hs_err_pid29653.log
How I resolve this ?
We are periodically runnign jmap -heap command to monitor a tomcat application. However this is first time when we got OutOfMemoryError.
When we monitor the memory at server level there is around 110MB available. Since it is just monitoring can we ignore this time assuming no harm has been done to the application and the server was low on memory when this command run?
$ ./jmap -heap 13511
Attaching to process ID 13511, please wait...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.<init>(LinuxDebuggerLocal.java:210)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupDebuggerLinux(BugSpotAgent.java:816)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupDebugger(BugSpotAgent.java:518)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:39)
UPDATE: We tried to run jmap -heap again and got java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
So, the problem seems to be at OS resource level.
$ ./jmap -heap 13511
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
#
# Internal Error (gcTaskThread.cpp:38), pid=13162, tid=1104300352
# Error: Cannot create GC thread. Out of system resources.
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
# An error report file with more information is saved as:
# /data01/home/s618199/JAVA/jdk1.6.0_18/bin/hs_err_pid13162.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
Aborted
This is not a memory problem (even though it is reported as one) but an operating system resource problem: You can not create any more threads. And it seems to affect the jmap program and not Tomcat.