Has anyone experienced or can someone explain the circumstances wherein the following occurs (and ideally help in resolving it):
What normally happens:
We execute a large user query/build report process
Java.exe memory steadily climbs as does CPU usage
Report Renders, Java.exe CPU utilization drops back down
What, I assume, should not be happening:
We execute a large user report query/build report process
ColdFusion.exe memory steadily climbs as does CPU usage
Java.exe memory and CPU usage do not budge
I should note I do not believe that ColdFusion.exe existed, or at least wasn't present in the computer processes., prior to this behavior. Basically instead of the server running off of Java.exe it's running off ColdFusion.exe. My only theory is that ColdFusion can't find the default java.
Thanks in advance.
Additional Details
Server is, and has been, running on ColdFusion 11 Standard
I, foolishly and absent mindlessly, hit the 'Update' button without taking a snap-shot. Unfortunately, I believe it was patching CF and updating to the most recent version of Java. CF would not restart so used Adobes Un-install and removed the patches. It restarted and decided to run the update at a later date.
My nightly backups had rolled off of the server, by a day, when I realized the server was running so slowly and that Coldfusion.exe was running instead of Java.exe so a simple restore is off the table.
The thing is, the server is running pretty well but not as well as it was. Additionally, I can not access the debug information by enabling it in the admin and adding my ip. I can access an error with a cftry and catch but the debug info gives me this and only this:
Debugging Information
ColdFusion Server Standard 11,0,07,296330
Template /FBI/witsec/locations/map/index.cfm
Time Stamp 31-Jan-16 01:47 PM
Locale English (US)
User Agent Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko
Remote IP 10.14.7.128
Host Name 10.14.7.128
------
Execution Time
It should be noted the the Execution Time is included in the above snippet, but obviously nothing follows.
Environment Settings - User:
Environment Settings - System:
Java and JVM CF 11 Admin Page
Java Virtual Machine Path:
C:\ColdFusion11\jre
Minimum JVM Heap Size: 4096 Maximum JVM Heap Size: 4096
JVM Arguments:
-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=49325
-server
-XX:NewRatio=3
-XX:SurvivorRatio=7
-XX:+UseCompressedOops
-Xss768k
-XX:MaxPermSize=256m
-XX:PermSize=128m
-XX:+DisableExplicitGC
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+CMSClassUnloadingEnabled
-XX:+CMSScavengeBeforeRemark
-XX:CMSInitiatingOccupancyFraction=68
-Dcoldfusion.home={application.home}
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog
-Duser.language=en
-Dcoldfusion.rootDir={application.home}
-Dcoldfusion.libPath={application.home}/lib
-Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true
-Dcoldfusion.jsafe.defaultalgo=FIPS186Random
-javaagent:C:/FusionReactor/instance/ColdFusionOnSGI/fusionreactor.jar=name=ColdFusionOnSGI,address=8088
Related
I am on a Mac OS and I run cassandra -f and immediately this happens:
[0.002s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/usr/local/apache-cassandra-3.0.10/logs/gc.log instead.
Unrecognized VM option 'UseParNewGC'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.```
I have no idea why this is happening. I did the proper
export CASSANDRA_HOME=/usr/local/apache-cassandra-3.0.10
export PATH=$PATH:$CASSANDRA_HOME/bin
But still it isnt working properly.
Is it something with my Java version? How can I do a complete clean install of Cassandra/get this to work?
In that version of Cassandra, the UseParNewGC setting is defined in the jvm.options file. It is the first setting in the block of CMS GC JVM settings.
#################
# GC SETTINGS #
#################
### CMS Settings
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
I suspect one of two things are going on.
It's possible that the -XX:+UseParNewGC setting is not appropriately specified. Double check this in your jvm.options file.
The more-likely scenario, is that a previous, erroneous edit made to the jmv.options file above the block I have shown above, is causing the issue. As the -XX:+UseParNewGC line is the first line in this block, the error appears to be here. The section above is where the heap sizing parameters are set, so I would check to see if something was uncommented or perhaps a quote was not properly closed.
Check your Java version with a java -version. Newer versions of Java (like 10 or 11 and higher) do not support the parallel garbage collector. Also, Cassandra 3.x only runs on Java 8, so you really don't have a reason to be on a recent JVM like that.
Apache Cassandra 3.11.10 runs with Java 8 only.
In PowerShell set the execution policy to Unrestricted
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted
Set-ExecutionPolicy Unrestricted
And if you get error related to Sigar then comment that line in cassandra-env.ps1 File under Conf folder.
Comment the below line
# $env:JVM_OPTS = "$env:JVM_OPTS -Djava.library.path=""$env:CASSANDRA_HOME\lib\sigar-bin"""
I’m running a conversion project from svn to git. As the application is single threaded, I’m moving the project to a Faster PC.
So without any options bar httpSpooling = true; It runs OK on a VM – 4 CPU's, 20 Gb of Ram.
RAM Usage with two separate instances is 8GB, hitting a max of 9.8Gb.
Jobs Paused, Zipped & SCP'd to new machine – Bare Metal build of Deb9 (same as VM) i7 (8 CPUs(effective)) 16GB ram.
However when starting just one instance of SubGit; I get either Java out of memory or GC Overhead Limit Exceeded.
I’ve tried adding the following permutations to repo.git/subgit/config to [daemon]
javaOptions = -noverify -client -Djava.awt.headless=true -Xmx8g -XX:+UseParallelGC -XX:-UseGCOverheadLimit – This gives GC Overhead Limit Exceeded Error
#javaOptions = -noverify -client -Djava.awt.headless=true -Xmx8g -XX:+UseParallelGC -XX:-UseGCOverheadLimit – (OPS Disabled) Gives an out of memory error.
javaOptions = -noverify -client -Djava.awt.headless=true –Xmx12g -XX:-UseGCOverheadLimit – this gives out of memory errors.
I’ve tried other settings too, including changing –client for –server, but that appears to be more two way conversion, which is not something I’m trying to do.
There should be plenty of RAM based on the application usage on a system running successfully, so unless SubGit is ignoring some values, I can’t tell.
The 'javaOptions' in the [daemon] section may indeed be ignored depending on the operation you run: those java options affect SubGit daemon, but not the 'subgit install' or 'subgit fetch' operation. Since you've mentioned that repositories were moved to another machine, I believe, you have invoked either of those two commands to restart the mirror and that's why that 'daemon.javaOptions' is ignored. To tune SubGit's java options edit it right in the SubGit launching script (EXTRA_JVM_ARGUMENTS line):
EXTRA_JVM_ARGUMENTS="-Dsun.io.useCanonCaches=false -Djava.awt.headless=true -Djna.nosys=true -Dsvnkit.http.methods=Digest,Basic,NTLM,Negotiate -Xmx512m"
As for the memory consumption itself, it depends on which operations are being run. It's not completely clear how did you pause the jobs on the virtual machine (by 'subgit shutdown' or in another way?), which operations were running at that time (initial translation or regular fetches) and how did you restart the jobs on the new machine.
I cant seem to run java at all in a Docker container on my server. Even when issuing java -version, I get the following error.
root#86088d679103:/# java -version
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000035ce1000000, 2555904, 1) failed; error='Operation not permitted' (errno=1)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2555904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid17.log
According to this, java can't map 2.5Mb of space for reserved memory? This does not seem right...
I have the full log included at the end, but for the sake of some extra information, my system is reporting the following:
root#86088d679103:/# uname -m
x86_64
root#86088d679103:/# free -mh
total used free shared buffers cached
Mem: 15G 9.7G 5.8G 912K 148M 8.9G
-/+ buffers/cache: 639M 14G
Swap: 15G 0B 15G
Can anyone point me in the right direction?
Full Log: https://gist.github.com/KayoticSully/e206c44681ce261674ba
Update
#Yobert nailed the problem and I highly suggest you read through the comments and chat log. Good info in there.
For those who want the final command that made Java work: setfattr -n user.pax.flags -v "mr" /usr/bin/java
If your distro does not have setfattr installed by default it should be included in the installable package attr through paceman, apt-get, etc.
I had this same problem when using a Grsec enabled kernel. For java to play nice, I had to disable MPROTECT on the java binary. You can use the paxctl utility for this:
paxctl -m /usr/lib/jvm/java-7-openjdk/jre/bin/java
You'll need to do paxctl -c on the binary first if you've never used it on that binary before:
paxctl -c /usr/lib/jvm/java-7-openjdk/jre/bin/java
More information about paxctl can be found at: http://en.wikibooks.org/wiki/Grsecurity/Additional_Utilities
I had the same problem when running Docker on Alpine Linux, after enabling PaX soft mode it worked:
sysctl -w kernel.pax.softmode=1
Soft mode will disable most PaX features by default, therefore it is not recommended to enable it. The proper way is to use paxctl, as already mentioned above.
Also have a look here:
https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Support_soft_mode
This happened to me as well ,
We reduced the RAM size on our VM and after a couple of days started getting this error and service did not came up for ever.
Solution :: We reduced the heap size of the application or service having this issue and the service came up fine again.
On starting JbossAS 5.1 server on Linux:
26204 jboss 20 0 4874m 1.3g 12m S 144.0 11.4 1:45.50 java
This is before any class-loading.
It starts with minimum 1g (RES) memory. How can i reduce this?
Is there any-way we can suppress memory usages?
inside your %JAVA_HOME%\bin directory (the linux equivelant)
Check the run.conf file for:
if [ "x$JAVA_OPTS" = "x" ]; then
JAVA_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
fi
I am running 5.1.0.GA on a very old PC, and having has lots of memory errors during start up i removed this specification from the JAVA_OPTS spec. I did this in windows so the batch syntax is different, but essentially, i just removed this option completely. It stopped the server from moaning about memory, but i don't know if you can use these options to restrict the memory usage further.
Not really an answer, but you might find it helps
This is the error I get when I run my web application in an instance of the Tomcat servlet container started by NetBeans. To fix this I even changed the heap size in netbeans.conf, but still it shows the same error. How can I keep this from happening?
HTTP Status 500 -
--------------------------------------------------------------------------------
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: Servlet execution threw an exception
org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:362)
root cause
java.lang.OutOfMemoryError: Java heap space
note The full stack trace of the root cause is available in the Apache Tomcat/5.5.9 logs.
Changing the heap size in netbeans.conf only changes the heap for NetBeans itself, not for applications run through NetBeans.
The correct way is to right-click on the project and select "Properties" and then "Run"; there you can set the VM options appropriately (-Xmx256m, for instance). It should look something like this:
(Thanks to VonC for finding this picture.)
Stop Tomcat server, set environment variable CATALINA_OPTS, and then restart Tomcat. Look at the file tomcat-install/bin/catalina.sh or catalina.bat for how this variable is used. For example,
set CATALINA_OPTS="-Xms512m -Xmx512m" (Windows)export CATALINA_OPTS="-Xms512m -Xmx512m" (ksh/bash)setenv CATALINA_OPTS "-Xms512m -Xmx512m" (tcsh/csh)
In catalina.bat or catallina.sh, you may have noticed CATALINA_OPTS, JAVA_OPTS, or both can be used to specify Tomcat JVM options.
What is the difference between CATALINA_OPTS and JAVA_OPTS?
The name CATALINA_OPTS is specific for Tomcat servlet container, whereas JAVA_OPTS may be used by other java applications (e.g., JBoss). Since environment variables are shared by all applications, we don't want Tomcat to inadvertently pick up the JVM options intended for other apps. I prefer to use CATALINA_OPTS.
How to set java heap size in JBoss?
Stop JBoss server, edit $JBOSS_HOME/bin/run.conf, and then restart JBoss server. You can change the line with JAVA_OPTS to something like:
JAVA_OPTS="-server -Xms128m -Xmx128m"
How to set java heap size in Eclipse?
You have 2 options:
Edit eclipse-home/eclipse.ini to be something like the following and
restart Eclipse.
-vmargs-Xms64m-Xmx256m
Or, you can just run eclipse command with additional options at the
very end. Anything after -vmargs will be treated as JVM options and
passed directly to the JVM. JVM options specified in the command
line this way will always override those in eclipse.ini. For
example,
eclipse -vmargs -Xms64m -Xmx256m
How to set java heap size in NetBeans?
Exit NetBeans, edit the file netbeans-install/etc/netbeans.conf. For example,
netbeans_default_options="-J-Xms512m -J-Xmx512m -J-XX:PermSize=32m -J-XX:MaxPermSize=128m -J-Xverify:none
How to set java heap size in Apache Ant?
Set environment variable ANT_OPTS. Look at the file $ANT_HOME/bin/ant or %ANT_HOME%\bin\ant.bat, for how this variable is used by Ant runtime.
set ANT_OPTS="-Xms512m -Xmx512m" (Windows)export ANT_OPTS="-Xms512m -Xmx512m" (ksh/bash)setenv ANT_OPTS "-Xms512m -Xmx512m" (tcsh/csh)
If you increase the virtual memory of your Tomcat server then it will be OK.
Steps:
In NB go through the windows menu and add Services
You will find Tomcat in the services. Right click on Tomcat server and select Properties
Go to the platform in the properties and write -Xms512m in VM options field
I'm guessing that increasing the memory won't fix the problem. What is that MonitorFilter doing? What's eating up all that memory?
Your best bet is to figure that out. If this is a web app, see if you can turn off that filter and run without it. If you have success, you know that the MonitorFilter is causing your to fail.
This has nothing to do with NetBeans (well, perhaps), rather it has to do with Tomcat. Tomcat is the process that is running out of heap, not NetBeans. Track down the startup process for your Tomcat. If it's bundled with NB, then Tomcat is buried within the NB installation, check for an "enterpriseN" directory, N being a number, Tomcat is probably in there and it's a rather generic distribution of it.
As to why the monitor is run OOM, that's hard to say, it's a pretty simple process when you think about it. You can also try disabling HTTP monitoring to see if it's a problem with the Monitoring itself or something with your application.