I've been reading multiple posts regarding limiting the container's JVM memory, nothing has worked so far, I don't know where I am messing up.
I made a simple "Hello World" in Spring Boot, using a REST controller, it has nothing else.
Such app was exported as a WAR file, running it with the Tomcat JDK 11 image, I can also run it using the JDK image with a FatJar but the problem persists either way.
Expected
Have my Java container not take more than 25 MB memory (for the sake of a number, could be more)
Actual
Such a simple application is taking 200 - 250 MB memory in docker stats
Dockerfile
FROM tomcat:9.0.30-jdk11-corretto
COPY tomcat.war /usr/local/tomcat/webapps/ROOT.war
CMD ["catalina.sh","run"]
docker-compose.yml
version: '3.7'
services:
hello:
build:
context: .
dockerfile: Dockerfile
image: app-test
environment:
- JVM_OPTS=-Xms13m -Xmx25m
ports:
- 8080:8080
I have tried
-Xms13m -Xmx25m
-XX:PermSize=13m -XX:MaxPermSize=25m
Among other flags that I already deleted and forgot about
To restrict memory for docker containers, you need to add following in your docker-compose.yml file
environment:
- <name>=<value>
deploy:
resources:
memory: 256M
Once you add the above code, then containers would be restricted to given memory limit and you can see the result in docker stats output. For more details about the resource restriction refer this docker documentation.
Note: If the container crosses the given memory limit, the kernel will kill the container.
The environment variable that tomcat's catalina.sh script depends on for java options is : JAVA_OPTS.
If you change the compose file to use the following env variable , it should work.
environment:
- JAVA_OPTS=-Xmx128m -Xms128m
Apart from the fact that 25m seems a way too less memory for a JVM with tomcat running.
Regardless of how little memory you set for JVM heap (-Xmx) there are lots of other types of non-heap memory that JVM uses: Java using much more memory than heap size (or size correctly Docker memory limit) - you need to take that into account.
You can also avoid setting -Xmx altogether and instead leverage -XX:MaxRAMPercentage: Clarification of meaning new JVM memory parameters InitialRAMPercentage and MinRAMPercentage
Related
I'm trying to deploy a simple Spring + Spring Boot App to IBM Cloud using Cloud Foundry. My Docker file looks as follows:
# IBM Java SDK UBI is not available on public docker yet. Use regular
# base as builder until this is ready. For reference:
# https://github.com/ibmruntimes/ci.docker/tree/master/ibmjava/8/sdk/ubi-min
FROM ibmjava:8-sdk AS builder
LABEL maintainer="IBM Java Engineering at IBM Cloud"
WORKDIR /app
RUN apt-get update && apt-get install -y maven
COPY pom.xml .
RUN mvn -N io.takari:maven:wrapper -Dmaven=3.5.0
COPY . /app
RUN ./mvnw install
ARG bx_dev_user=root
ARG bx_dev_userid=1000
RUN BX_DEV_USER=$bx_dev_user
RUN BX_DEV_USERID=$bx_dev_userid
RUN if [ $bx_dev_user != "root" ]; then useradd -ms /bin/bash -u $bx_dev_userid $bx_dev_user; fi
# Multi-stage build. New build stage that uses the UBI as the base image.
# In the short term, we are using the OpenJDK for UBI. Long term, we will use
# the IBM Java Small Footprint JVM (SFJ) for UBI, but that is not in public
# Docker at the moment.
# (https://github.com/ibmruntimes/ci.docker/tree/master/ibmjava/8/sfj/ubi-min)
FROM adoptopenjdk/openjdk8:ubi-jre
# Copy over app from builder image into the runtime image.
RUN mkdir /opt/app
COPY --from=builder /app/target/javaspringapp-1.0-SNAPSHOT.jar /opt/app/app.jar
ENTRYPOINT [ "sh", "-c", "java -jar -XX:MaxRAM=10m /opt/app/app.jar" ]
As you can see, I've limited my app to use a max of 10MB and my instance has 256MB available. However, everytime I deploy the app, I get the following error:
Also, when I run the ps command to check the running processes, there's nothing that would consume all my memory.
I've ran the same app locally and it only requires 25MB max, so the 256MB of my instance should be enough. Any ideas on how to solve this problem?
NOTE: Unfortunately I am not allowed to increase the capacity of the instance at this moment of time.
[UPDATE] So I decided to start from a scratch and create a new Java Spring App in IBM Cloud. IBM automatically sets it to start with this template, which is cloned to your repo. The app was deployed normally and I could access it.
Now, I removed one line of code in the HTML and tried to deploy again. To my surprise, I got:
FAILED
Error restarting application: Server error, status code: 400, error code: 100005, message: You have exceeded your organization's memory limit: app requested more memory than available
How is this even possible?!
I found a solution; which is rather unexpected and counterintuitive. Here's what I did:
I reduced the memory of the instance of the application to 64MB.
In the manifest.yml, I set the memory of the application to 192M. My manifest looks as follows:
applications:
- instances: 1
timeout: 180
name: appname
buildpack: java_buildpack
path: ./target/appname-1.0-SNAPSHOT.jar
disk_quota: 1G
memory: 192M
domain: eu-gb.mybluemix.net
host: appHost
env:
JAVA_OPTS: '-XX:ReservedCodeCacheSize=32M -XX:MaxDirectMemorySize=32M'
JBP_CONFIG_OPEN_JDK_JRE: '[memory_calculator: {stack_threads: 2}]'
I pushed the changes in the manifest.yml to the repo, starting a new deployment. This time the application is deployed successfully and runs smoothly.
NOTE: After deploying, the memory of the instance is overwritten and 192MB are assigned. In order to perform any further changes in the code, I need to set the memory back to 64MB before pushing the changes and deploying the app.
For Lite accounts, memory is limited to 256 MB across all apps deployed for an account:
https://cloud.ibm.com/docs/account?topic=account-noruntimemem
You will either need to stop any Cloud Foundry applications that are not in use or upgrade your account.
In Cloud Foundry, your manifest.yml in the application code will specify the memory.
Note: If you are using a Toolchain, the deployment will create a new application and switch routes on every redeployment in the pipeline, so temporarily you will have two applications running to avoid downtime. This is another scenario where you could potentially run into this memory problem (i.e. your application specifies 256MB of memory but temporarily there is 512MB of usage while both applications are running). To fix this, either manually stop the CF application that's running before running the pipeline (will incur downtime while stopped) or upgrade your account.
I have simple site and want to deploy it at VPS with 512M ram.
At all, I have Cent OS + nginx + Postgresql and it takes 150M.
And, JRE8 + my project (including app server - Tomcat) tskes 254M.
htop show that I have at server 466M, so free only 62M but its yellow color in htop.
I run Java process via: nohup java -Xmx64m -jar some.jar &
Also, at application.properties I define: server.tomcat.max-threads = 10
SO QUESTION: how can I setup my site for use 180M instead 250M, so that I have at least a little memory left? What can I do? Change app server? Any settings?
Note that you are using Java 8 which has slightly different options for configuring memory usage.
Here is an example of memory consumption options in Java 8 (taken from here):
-XX:InitialCodeCacheSize=64M -XX:CodeCacheExpansionSize=1M -XX:CodeCacheMinimumFreeSpace=1M -XX:ReservedCodeCacheSize=200M
-XX:MinMetaspaceExpansion=1M -XX:MaxMetaspaceExpansion=8M -XX:MaxMetaspaceSize=200M
-XX:MaxDirectMemorySize=96M
-XX:CompressedClassSpaceSize=256M
-Xss1024K
As of raising this question, Docker looks to be new enough to not have answers to this question on the net. The only place I found is this article in which the author is saying it is hard, and that's it.
Note that in a docker-compose.yml file - you'll need to leave out the double-quotes:
environment:
- JVM_OPTS=-Xmx12g -Xms12g -XX:MaxPermSize=1024m
or
environment:
- CATALINA_OPTS=-Xmx12g -Xms12g -XX:MaxPermSize=1024m
I agree that it depends on what container you're using. If you are using the official Tomcat image, it looks like it's simple enough, you will need to pass the JAVA_OPTS environment variable with your heap settings:
docker run --rm -e JAVA_OPTS='-Xmx1g' tomcat
See How to set JVM parameters?
Update: Regarding this discussion, Java has upped there game regarding container support. Nowadays (or since JVM version 10 to be more exact), the JVM is smart enough to figure out whether it is running in a container, and if yes, how much memory it is limited to.
So, rather than setting fixed limits when starting your JVM, which you then have to change in line with changes to your container limits (resource limits in the K8s world), simply do nothing and let the JVM work out limits for itself.
Without any extra configuration, the JVM will set the maximum heap size to 25% of the allocated memory. Since this is frugal, you might want to ramp that up a bit by setting the -XX:MaxRAMPercentage attribute. Also, there is -XX:InitialRAMPercentage for initial heap size and -XX:MinRAMPercentage for containers with less than 96MB RAM.
For more information on the topic, here is an excellent overview.
You can also just place those settings in your image so something like the following would exist in your Dockerfile:
ENV JAVA_OPTS="-XX:PermSize=1024m -XX:MaxPermSize=512m"
you can do it by specifying java options environment in docker compose file
env:
- name: _JAVA_OPTIONS
value: "-Xmx1g"
it will change the heap size.
It all depends how your Java application is packaged and how it's configuration files are exposed using Docker.
For example the official tomcat image states that the configuration file is available in the default location: /usr/local/tomcat/conf/
So easy to override entire directory or just one configuration file:
docker run -it --rm -p 8080:8080 -v $PWD/catalina.properties:/usr/local/tomcat/conf/catalina.properties tomcat:8.0
I cant seem to run java at all in a Docker container on my server. Even when issuing java -version, I get the following error.
root#86088d679103:/# java -version
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000035ce1000000, 2555904, 1) failed; error='Operation not permitted' (errno=1)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2555904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid17.log
According to this, java can't map 2.5Mb of space for reserved memory? This does not seem right...
I have the full log included at the end, but for the sake of some extra information, my system is reporting the following:
root#86088d679103:/# uname -m
x86_64
root#86088d679103:/# free -mh
total used free shared buffers cached
Mem: 15G 9.7G 5.8G 912K 148M 8.9G
-/+ buffers/cache: 639M 14G
Swap: 15G 0B 15G
Can anyone point me in the right direction?
Full Log: https://gist.github.com/KayoticSully/e206c44681ce261674ba
Update
#Yobert nailed the problem and I highly suggest you read through the comments and chat log. Good info in there.
For those who want the final command that made Java work: setfattr -n user.pax.flags -v "mr" /usr/bin/java
If your distro does not have setfattr installed by default it should be included in the installable package attr through paceman, apt-get, etc.
I had this same problem when using a Grsec enabled kernel. For java to play nice, I had to disable MPROTECT on the java binary. You can use the paxctl utility for this:
paxctl -m /usr/lib/jvm/java-7-openjdk/jre/bin/java
You'll need to do paxctl -c on the binary first if you've never used it on that binary before:
paxctl -c /usr/lib/jvm/java-7-openjdk/jre/bin/java
More information about paxctl can be found at: http://en.wikibooks.org/wiki/Grsecurity/Additional_Utilities
I had the same problem when running Docker on Alpine Linux, after enabling PaX soft mode it worked:
sysctl -w kernel.pax.softmode=1
Soft mode will disable most PaX features by default, therefore it is not recommended to enable it. The proper way is to use paxctl, as already mentioned above.
Also have a look here:
https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Support_soft_mode
This happened to me as well ,
We reduced the RAM size on our VM and after a couple of days started getting this error and service did not came up for ever.
Solution :: We reduced the heap size of the application or service having this issue and the service came up fine again.
I am using the Apache Solr powered by BitNami EC2 AMI. Solr is running, but I'd like to change the startup configuration to increase the amount of memory allocated to JVM.
I have tried modifying the startup script at at /opt/bitnami/apache-solr/scripts/ctl.sh by modifying the following line:
SOLR="$JAVABIN -Dsolr.solr.home=$SOLR_HOME
-Djetty.logs=$INSTALL_PATH/logs/ -Djetty.home=$INSTALL_PATH/ -jar $INSTALL_PATH/start.jar $INSTALL_PATH/etc/jetty.xml"
I've tried different permutations for the memory flags and none of them work (some of them cause the Solr server to fail to start at all, while others allow it to start but have no effect on the JVM memory allocated). This is what I've tried adding to the line:
-Xmx 1000 -Xms 8000
-Xms1000m -Xmx8000m
-Xms1000 -Xmx8000
-Xms 1000m -Xmx 8000m
What is the correct way of going about this?
It turns out that the arguments needed to be at the start of the line. The following works:
SOLR="$JAVABIN -Xmx7168m -Xms1024m -Dsolr.solr.home=$SOLR_HOME
-Djetty.logs=$INSTALL_PATH/logs/ -Djetty.home=$INSTALL_PATH/ -jar $INSTALL_PATH/start.jar $INSTALL_PATH/etc/jetty.xml"