I cant seem to run java at all in a Docker container on my server. Even when issuing java -version, I get the following error.
root#86088d679103:/# java -version
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000035ce1000000, 2555904, 1) failed; error='Operation not permitted' (errno=1)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2555904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid17.log
According to this, java can't map 2.5Mb of space for reserved memory? This does not seem right...
I have the full log included at the end, but for the sake of some extra information, my system is reporting the following:
root#86088d679103:/# uname -m
x86_64
root#86088d679103:/# free -mh
total used free shared buffers cached
Mem: 15G 9.7G 5.8G 912K 148M 8.9G
-/+ buffers/cache: 639M 14G
Swap: 15G 0B 15G
Can anyone point me in the right direction?
Full Log: https://gist.github.com/KayoticSully/e206c44681ce261674ba
Update
#Yobert nailed the problem and I highly suggest you read through the comments and chat log. Good info in there.
For those who want the final command that made Java work: setfattr -n user.pax.flags -v "mr" /usr/bin/java
If your distro does not have setfattr installed by default it should be included in the installable package attr through paceman, apt-get, etc.
I had this same problem when using a Grsec enabled kernel. For java to play nice, I had to disable MPROTECT on the java binary. You can use the paxctl utility for this:
paxctl -m /usr/lib/jvm/java-7-openjdk/jre/bin/java
You'll need to do paxctl -c on the binary first if you've never used it on that binary before:
paxctl -c /usr/lib/jvm/java-7-openjdk/jre/bin/java
More information about paxctl can be found at: http://en.wikibooks.org/wiki/Grsecurity/Additional_Utilities
I had the same problem when running Docker on Alpine Linux, after enabling PaX soft mode it worked:
sysctl -w kernel.pax.softmode=1
Soft mode will disable most PaX features by default, therefore it is not recommended to enable it. The proper way is to use paxctl, as already mentioned above.
Also have a look here:
https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Support_soft_mode
This happened to me as well ,
We reduced the RAM size on our VM and after a couple of days started getting this error and service did not came up for ever.
Solution :: We reduced the heap size of the application or service having this issue and the service came up fine again.
Related
I've been reading multiple posts regarding limiting the container's JVM memory, nothing has worked so far, I don't know where I am messing up.
I made a simple "Hello World" in Spring Boot, using a REST controller, it has nothing else.
Such app was exported as a WAR file, running it with the Tomcat JDK 11 image, I can also run it using the JDK image with a FatJar but the problem persists either way.
Expected
Have my Java container not take more than 25 MB memory (for the sake of a number, could be more)
Actual
Such a simple application is taking 200 - 250 MB memory in docker stats
Dockerfile
FROM tomcat:9.0.30-jdk11-corretto
COPY tomcat.war /usr/local/tomcat/webapps/ROOT.war
CMD ["catalina.sh","run"]
docker-compose.yml
version: '3.7'
services:
hello:
build:
context: .
dockerfile: Dockerfile
image: app-test
environment:
- JVM_OPTS=-Xms13m -Xmx25m
ports:
- 8080:8080
I have tried
-Xms13m -Xmx25m
-XX:PermSize=13m -XX:MaxPermSize=25m
Among other flags that I already deleted and forgot about
To restrict memory for docker containers, you need to add following in your docker-compose.yml file
environment:
- <name>=<value>
deploy:
resources:
memory: 256M
Once you add the above code, then containers would be restricted to given memory limit and you can see the result in docker stats output. For more details about the resource restriction refer this docker documentation.
Note: If the container crosses the given memory limit, the kernel will kill the container.
The environment variable that tomcat's catalina.sh script depends on for java options is : JAVA_OPTS.
If you change the compose file to use the following env variable , it should work.
environment:
- JAVA_OPTS=-Xmx128m -Xms128m
Apart from the fact that 25m seems a way too less memory for a JVM with tomcat running.
Regardless of how little memory you set for JVM heap (-Xmx) there are lots of other types of non-heap memory that JVM uses: Java using much more memory than heap size (or size correctly Docker memory limit) - you need to take that into account.
You can also avoid setting -Xmx altogether and instead leverage -XX:MaxRAMPercentage: Clarification of meaning new JVM memory parameters InitialRAMPercentage and MinRAMPercentage
I am on a Mac OS and I run cassandra -f and immediately this happens:
[0.002s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/usr/local/apache-cassandra-3.0.10/logs/gc.log instead.
Unrecognized VM option 'UseParNewGC'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.```
I have no idea why this is happening. I did the proper
export CASSANDRA_HOME=/usr/local/apache-cassandra-3.0.10
export PATH=$PATH:$CASSANDRA_HOME/bin
But still it isnt working properly.
Is it something with my Java version? How can I do a complete clean install of Cassandra/get this to work?
In that version of Cassandra, the UseParNewGC setting is defined in the jvm.options file. It is the first setting in the block of CMS GC JVM settings.
#################
# GC SETTINGS #
#################
### CMS Settings
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
I suspect one of two things are going on.
It's possible that the -XX:+UseParNewGC setting is not appropriately specified. Double check this in your jvm.options file.
The more-likely scenario, is that a previous, erroneous edit made to the jmv.options file above the block I have shown above, is causing the issue. As the -XX:+UseParNewGC line is the first line in this block, the error appears to be here. The section above is where the heap sizing parameters are set, so I would check to see if something was uncommented or perhaps a quote was not properly closed.
Check your Java version with a java -version. Newer versions of Java (like 10 or 11 and higher) do not support the parallel garbage collector. Also, Cassandra 3.x only runs on Java 8, so you really don't have a reason to be on a recent JVM like that.
Apache Cassandra 3.11.10 runs with Java 8 only.
In PowerShell set the execution policy to Unrestricted
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted
Set-ExecutionPolicy Unrestricted
And if you get error related to Sigar then comment that line in cassandra-env.ps1 File under Conf folder.
Comment the below line
# $env:JVM_OPTS = "$env:JVM_OPTS -Djava.library.path=""$env:CASSANDRA_HOME\lib\sigar-bin"""
Is there a way to create a heap dump on a remote machine without JDK installed?
I can't change the installation / settings and it's running on Windows.
So I have pnly access to commandline tools.
Problem is that a Java app on a remote machine freezes (no out of memory exception so -XX:-HeapDumpOnOutOfMemoryError is useless) and we need to create a dump.
-XX:+HeapDumpOnCtrlBreak
is no option too, because it's not supported anymore on JDK6+.
JMX is not allowed due to security reasons.
Any Ideas? Thank you for your help!
Edit:
Windows
No JDK
No JMX
I think I solved the problem.
You have to "patch" your JRE with some files of the JDK (the same version of course - if you are running jre6uXX you need the corresponding files from jdk6uXX )
Copy the following files:
\JDK6uXX\bin\attach.dll --> %JAVAJRE_HOME%\bin\
\JDK6uXX\bin\jmap.exe --> %JAVAJRE_HOME%\bin\
\JDK6uXX\lib\tools.jar --> %JAVAJRE_HOME%\lib\
No files are overwritten, JRE shouldn't be affected by this.
Now you can use jmap just fine to take dumps ;-)
I appreciate your help! Bye
The simplest solution is to use jmap -dump:liv,format=b,file=app.dump on the command line. You can use jps -lvm to find the process id.
An alternative is to connect to it to jvisualvm This will take the dump and analyse it for you. You can also use this tool to read a dump written by jmap so you may end up using it anyway.
Where jvisualvm struggles is for large heap dumps i.e. more than about half you main memory size. I have found using YourKit to handle larger dumps and also give more useful information. An evaluation license might be all you need to diagnose this.
jmx is not allowed due to security reasons
In that case, you can't do this remotely, unless you use YourKit or some other commercial profiler.
You have start your application with jmx console enabled in a port to debug your application. Execute jconsole and connect to the port which you have enabled for debugging. You can also use of jmap to collect heapdump.
JProfiler has a command line utility bin/jpdump that can take an HPROF heap dump. There is no need to install JDK. There is also no need to run the GUI installer of JProfiler, just extract the ZIP distribution and execute jpdump on the command line.
Disclaimer: My company develops JProfiler.
Update 2016-06-23
As of JProfiler 9.2, jpdump and jpenable run with Java 6 as well.
You could use jvisualvm, just enable jmx port and connect to your application, then you will be able to generate a heap file.
You can do that by adding the following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=8484
-Dcom.sun.management.jmxremote.ssl=false
Then You need to add your tomcat process manually, So right click on you localhost node -> Add JMX Connection -> type your port -> OK.
Your tomcat process will be listed in under localhost node.
jmap -dump:format=b,file=snapshot.jmap
process-pid
Regardless of how the Java VM was started, the jmap tool will produce a head dump snapshot, in the above example in a file called snapshot.jmap. The jmap output files should contain all the primitive data, but will not include any stack traces showing where the objects have been created.
On starting JbossAS 5.1 server on Linux:
26204 jboss 20 0 4874m 1.3g 12m S 144.0 11.4 1:45.50 java
This is before any class-loading.
It starts with minimum 1g (RES) memory. How can i reduce this?
Is there any-way we can suppress memory usages?
inside your %JAVA_HOME%\bin directory (the linux equivelant)
Check the run.conf file for:
if [ "x$JAVA_OPTS" = "x" ]; then
JAVA_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
fi
I am running 5.1.0.GA on a very old PC, and having has lots of memory errors during start up i removed this specification from the JAVA_OPTS spec. I did this in windows so the batch syntax is different, but essentially, i just removed this option completely. It stopped the server from moaning about memory, but i don't know if you can use these options to restrict the memory usage further.
Not really an answer, but you might find it helps
I have the following problem:
$ java -jar program.jar
Error occurred during initialization of VM
Could not reserve enough space for code cache
The amount of memory that is seen by the system seems sufficient:
$ free -m
total used free shared buffers cached
Mem: 5959 640 5318 0 0 390
-/+ buffers/cache: 249 5710
Swap: 4099 0 4099
I tried lowering heap settings as low as 16mb, but it didn't help:
$ java -Xmx16m -Xms16m -jar program.jar
Error occurred during initialization of VM
Could not reserve enough space for code cache
What can be wrong? How can I debug this?
EDIT:
Forgot to include version - I use Sun Java 7u15, 64 bit. I actually can't get it to display a version string, because "java -version" fails with the same error.
Also, for some reason, if I use OpenJDK 6u27, it starts up - but since the app expects java 7, it doesn't work.
Seems that my java was running on Gentoo Hardened Linux, and kernel had PaX enabled. To disable pax memory control, I used the following command:
paxctl -c -m /path/to/your/java/bin/java
After that, everything was smooth.
You can set these sysctl variables (in BSD it is in /etc/sysctl.conf)
security.pax.mprotect.enabled=0
security.pax.mprotect.global=0
security.pax.mprotect.ptrace=0
than.... /etc/rc.d/sysctl restart
do the "trick"