Develop Reference

Subgit "Out of Memory" and "GC Overhead Limit Exceeded"

I’m running a conversion project from svn to git. As the application is single threaded, I’m moving the project to a Faster PC.
So without any options bar httpSpooling = true; It runs OK on a VM – 4 CPU's, 20 Gb of Ram.
RAM Usage with two separate instances is 8GB, hitting a max of 9.8Gb.
Jobs Paused, Zipped & SCP'd to new machine – Bare Metal build of Deb9 (same as VM) i7 (8 CPUs(effective)) 16GB ram.
However when starting just one instance of SubGit; I get either Java out of memory or GC Overhead Limit Exceeded.
I’ve tried adding the following permutations to repo.git/subgit/config to [daemon]
javaOptions = -noverify -client -Djava.awt.headless=true -Xmx8g -XX:+UseParallelGC -XX:-UseGCOverheadLimit – This gives GC Overhead Limit Exceeded Error
#javaOptions = -noverify -client -Djava.awt.headless=true -Xmx8g -XX:+UseParallelGC -XX:-UseGCOverheadLimit – (OPS Disabled) Gives an out of memory error.
javaOptions = -noverify -client -Djava.awt.headless=true –Xmx12g -XX:-UseGCOverheadLimit – this gives out of memory errors.
I’ve tried other settings too, including changing –client for –server, but that appears to be more two way conversion, which is not something I’m trying to do.
There should be plenty of RAM based on the application usage on a system running successfully, so unless SubGit is ignoring some values, I can’t tell.

The 'javaOptions' in the [daemon] section may indeed be ignored depending on the operation you run: those java options affect SubGit daemon, but not the 'subgit install' or 'subgit fetch' operation. Since you've mentioned that repositories were moved to another machine, I believe, you have invoked either of those two commands to restart the mirror and that's why that 'daemon.javaOptions' is ignored. To tune SubGit's java options edit it right in the SubGit launching script (EXTRA_JVM_ARGUMENTS line):
EXTRA_JVM_ARGUMENTS="-Dsun.io.useCanonCaches=false -Djava.awt.headless=true -Djna.nosys=true -Dsvnkit.http.methods=Digest,Basic,NTLM,Negotiate -Xmx512m"
As for the memory consumption itself, it depends on which operations are being run. It's not completely clear how did you pause the jobs on the virtual machine (by 'subgit shutdown' or in another way?), which operations were running at that time (initial translation or regular fetches) and how did you restart the jobs on the new machine.

Coldfusion.exe v/s Java.exe

Has anyone experienced or can someone explain the circumstances wherein the following occurs (and ideally help in resolving it):
What normally happens:
We execute a large user query/build report process
Java.exe memory steadily climbs as does CPU usage
Report Renders, Java.exe CPU utilization drops back down
What, I assume, should not be happening:
We execute a large user report query/build report process
ColdFusion.exe memory steadily climbs as does CPU usage
Java.exe memory and CPU usage do not budge
I should note I do not believe that ColdFusion.exe existed, or at least wasn't present in the computer processes., prior to this behavior. Basically instead of the server running off of Java.exe it's running off ColdFusion.exe. My only theory is that ColdFusion can't find the default java.
Thanks in advance.
Additional Details
Server is, and has been, running on ColdFusion 11 Standard
I, foolishly and absent mindlessly, hit the 'Update' button without taking a snap-shot. Unfortunately, I believe it was patching CF and updating to the most recent version of Java. CF would not restart so used Adobes Un-install and removed the patches. It restarted and decided to run the update at a later date.
My nightly backups had rolled off of the server, by a day, when I realized the server was running so slowly and that Coldfusion.exe was running instead of Java.exe so a simple restore is off the table.
The thing is, the server is running pretty well but not as well as it was. Additionally, I can not access the debug information by enabling it in the admin and adding my ip. I can access an error with a cftry and catch but the debug info gives me this and only this:
Debugging Information
ColdFusion Server Standard 11,0,07,296330
Template /FBI/witsec/locations/map/index.cfm
Time Stamp 31-Jan-16 01:47 PM
Locale English (US)
User Agent Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
Remote IP 10.14.7.128
Host Name 10.14.7.128
------
Execution Time
It should be noted the the Execution Time is included in the above snippet, but obviously nothing follows.
Environment Settings - User:
Environment Settings - System:
Java and JVM CF 11 Admin Page
Java Virtual Machine Path:
C:\ColdFusion11\jre
Minimum JVM Heap Size: 4096 Maximum JVM Heap Size: 4096
JVM Arguments:
-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=49325
-server
-XX:NewRatio=3
-XX:SurvivorRatio=7
-XX:+UseCompressedOops
-Xss768k
-XX:MaxPermSize=256m
-XX:PermSize=128m
-XX:+DisableExplicitGC
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+CMSClassUnloadingEnabled
-XX:+CMSScavengeBeforeRemark
-XX:CMSInitiatingOccupancyFraction=68
-Dcoldfusion.home={application.home}
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog
-Duser.language=en
-Dcoldfusion.rootDir={application.home}
-Dcoldfusion.libPath={application.home}/lib
-Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true
-Dcoldfusion.jsafe.defaultalgo=FIPS186Random
-javaagent:C:/FusionReactor/instance/ColdFusionOnSGI/fusionreactor.jar=name=ColdFusionOnSGI,address=8088

JVM cant map reserved memory when running in Docker container

I cant seem to run java at all in a Docker container on my server. Even when issuing java -version, I get the following error.
root#86088d679103:/# java -version
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000035ce1000000, 2555904, 1) failed; error='Operation not permitted' (errno=1)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2555904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid17.log
According to this, java can't map 2.5Mb of space for reserved memory? This does not seem right...
I have the full log included at the end, but for the sake of some extra information, my system is reporting the following:
root#86088d679103:/# uname -m
x86_64
root#86088d679103:/# free -mh
total used free shared buffers cached
Mem: 15G 9.7G 5.8G 912K 148M 8.9G
-/+ buffers/cache: 639M 14G
Swap: 15G 0B 15G
Can anyone point me in the right direction?
Full Log: https://gist.github.com/KayoticSully/e206c44681ce261674ba
Update
#Yobert nailed the problem and I highly suggest you read through the comments and chat log. Good info in there.
For those who want the final command that made Java work: setfattr -n user.pax.flags -v "mr" /usr/bin/java
If your distro does not have setfattr installed by default it should be included in the installable package attr through paceman, apt-get, etc.

I had this same problem when using a Grsec enabled kernel. For java to play nice, I had to disable MPROTECT on the java binary. You can use the paxctl utility for this:
paxctl -m /usr/lib/jvm/java-7-openjdk/jre/bin/java
You'll need to do paxctl -c on the binary first if you've never used it on that binary before:
paxctl -c /usr/lib/jvm/java-7-openjdk/jre/bin/java
More information about paxctl can be found at: http://en.wikibooks.org/wiki/Grsecurity/Additional_Utilities

I had the same problem when running Docker on Alpine Linux, after enabling PaX soft mode it worked:
sysctl -w kernel.pax.softmode=1
Soft mode will disable most PaX features by default, therefore it is not recommended to enable it. The proper way is to use paxctl, as already mentioned above.
Also have a look here:
https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Support_soft_mode

This happened to me as well ,
We reduced the RAM size on our VM and after a couple of days started getting this error and service did not came up for ever.
Solution :: We reduced the heap size of the application or service having this issue and the service came up fine again.

MaxHeapSize in Java

I run command in window machine
java -XX:+PrintFlagsFinal -version | findstr /i "HeapSize PermSize ThreadStackSize
and got below value for MaxHeapSize
2118123520
which is 2020MB however, when I call Runtime.getRuntime().maxMemory() , it gives me 259522560 only i.e. 247MB .
Can anyone please suggest why I'm getting this mismatch ?

As per your last comment
I'm using eclipse IDE and didn't change any setting . do you know how to change maximum heap size in eclipse IDE ?
Set the VM arguments found under Run configuration.
VM arguments are typically values that change the behaviour of the Java Virtual Machine (JVM). For example, the -Xmx256M argument allows the Java heap to grow to 256MB.
For more info have a look at Eclipse launching program and VM arguments

Can anyone please suggest why I'm getting this mismatch ?
The Runtime.getRuntime().maxMemory() call is returning the maximum heap size for the current JVM. This is determined by either the -Xmx command line option (e.g. for the java command) or a platform specific default.
The command you are running is giving the maximum allowed value for the -Xmx commandline option on your platform.
Clearly these are different things. Furthermore, the numbers are usually different.

Reduce Jboss Memory Usage

On starting JbossAS 5.1 server on Linux:
26204 jboss 20 0 4874m 1.3g 12m S 144.0 11.4 1:45.50 java
This is before any class-loading.
It starts with minimum 1g (RES) memory. How can i reduce this?
Is there any-way we can suppress memory usages?

inside your %JAVA_HOME%\bin directory (the linux equivelant)
Check the run.conf file for:
if [ "x$JAVA_OPTS" = "x" ]; then
JAVA_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
fi
I am running 5.1.0.GA on a very old PC, and having has lots of memory errors during start up i removed this specification from the JAVA_OPTS spec. I did this in windows so the batch syntax is different, but essentially, i just removed this option completely. It stopped the server from moaning about memory, but i don't know if you can use these options to restrict the memory usage further.
Not really an answer, but you might find it helps