I would like to stop Cassandra from dumping hprof files as I do not require the use of them.
I also have very limited disk space (50GB out of 100 GB is used for data), and these files swallow up all the disk space before I can say "stop".
How should I go about it?
Is there a shell script that I could use to erase these files from time to time?
It happens because Cassandra starts with -XX:+HeapDumpOnOutOfMemoryError Java option. Which is good stuff if you want to analyze. Also, if you are getting lots of heap-dump that indicate that you should probably tune the memory available to Cassandra.
I haven't tried it. But to block this option, comment the following line in $CASSANDRA_HOME/conf/cassandra-env.sh
JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
Optionally, you may comment this block as well, but not really required, I think. This block is available in 1.0+ version I guess. I can't find this in 0.7.3.
# set jvm HeapDumpPath with CASSANDRA_HEAPDUMP_DIR
if [ "x$CASSANDRA_HEAPDUMP_DIR" != "x" ]; then
JVM_OPTS="$JVM_OPTS -XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date +%s`-pid$$.hprof"
fi
Let me know if this worked.
Update
...I guess it is JVM throwing it out when Cassandra crashes / shuts down. Any way to prevent that one from happening?
If you want to disable JVM heap-dump altogether, see here how to disable creating java heap dump after VM crashes?
I'll admit i haven't used Cassandra, but from what i can tell, it shouldn't be dumping any hprof files unless you enable it at compile time, or the program experiences an OutofMemoryException. So try looking there.
in terms of a shell script, if the files are being dumped to a specific location you can use this command to delete all *.hprof files.
find /my/location/ -name *.hprof -delete
this is using the -delete directive from find that deletes all files that match the search. Look at the man page for find for more search options if you need to narrow it down more.
You can use cron to run a script at a given time, which would satisfy your "time to time" requirement, most linux distros have a cron installed, and work off of a crontab file. You can find out more about the crontab by using man crontab
Even if you update cassandra-env.sh to point to the heapdump path it will still not work. The reason was that from the upstart script /etc/init.d/cassandra there is this line which creates the default HeapDump path
start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -b -p "$PIDFILE" -- \
-p "$PIDFILE" -H "$heap_dump_f" -E "$error_log_f" >/dev/null || return 2
I'm not an upstart expert but what I did was just removed the param which creates the duplicate. Another weird observation also when checking cassandra process via ps aux you'll notice that you'll see some parameters being written twice. If you source cassandra-env.sh and print $JVM_OPTS you'll notice those variables okay.
Related
This is getting when i tried to download 150mb size file. i have JAVA_OPTS as following.
if [ -f /root/.rs850 -o -f /root/.rs851 -o -f /root/.rs950 -o -f /root/.rs951 ]
JAVA_OPTS="$JAVA_OPTS -Xmx1024m -XX:MaxPermSize=128m"
else
JAVA_OPTS="$JAVA_OPTS -Xmx512m -XX:MaxPermSize=128m"
It would appear that something in your webapp is buffering the big file in memory prior to it being downloaded by the user's web browser. (You can easily consume more memory than you would think, depending on the nature of the file and how it is being read and buffered in memory.)
There are two ways to address this:
Modify the code in your webapp that is responsible for serving up the file so that it doesn't need to buffer it in memory before sending it.
Increase the max heap size; e.g. change the -Xmx512 option to (say) -Xmx1024.
Unfortunately, the second solution means that your Tomcat instance will use more memory. It is also a bandaid. If you need to download a larger file, you are liable to run into the same problem again.
Another possibility is that your webapp has a memory leak, and it was just a coincidence that the 150 Mb download triggered the OOME.
My java process stopped responding. I tried to jstack but failed with below error.
21039: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
Then I used -F option, but "No deadlocks found."
Other info:
java version: java version
jmap: jmap
jstat: jstat
jinfo: jinfo
Can anyone help have a look and share some links on troubleshooting this kind of java "not responding" issue?
Possible reasons of Unable to open socket file problem:
The target PID is not a HotSpot JVM process.
This is obviously not your case, since jinfo PID works fine.
JVM is started with -XX:+DisableAttachMechanism option.
This can be also verified by jinfo PID.
Attach socket /tmp/.java_pidNNN has been deleted.
There is a common practice to clean /tmp automatically with some scheduled script. In this case you should configure the cleanup software not to delete .java_pid* files.
How to check: run lsof -p PID | grep java_pid
If it lists a socket file, but the file does not exist, then this is exactly the described problem.
Credentials of the current user (euid/egid) do not match the owner of the attach socket. Make sure you run jstack by the same user as JVM. Attach won't work if you run jstack by a different user, even if this user is root.
/tmp directory of the target process is not the same as /tmp of your shell. This may happen in the following cases:
JVM is started in a different mount namespace. Typically this happens when JVM runs in a Docker container. Running jstack from within the same container will help.
JVM is started in chroot environment. For example, LXC containers may use chroot.
How to check: run readlink -f /proc/PID/root/tmp to see if it points to /tmp or to some other directory.
Current working directory of the target JVM belongs to a file system that does not allow to change permissions. CIFS and DrvFs (WSL) are examples of such file systems.
How to check: run umask 077; touch /proc/PID/cwd/somefile.tmp, then verify that file owner is yourself, and file permissions are 600.
JVM is busy and cannot reach a safepoint. For instance, JVM is in the middle of long-running garbage collection.
How to check: run kill -3 PID. JVM should print a thread dump and heap info in its console. If JVM does not dump anything, but the process consumes almost 100% CPU or shows high I/O utilization, then this looks like the described problem.
JVM process is suspended.
How to check: run ps PID. The STAT column of alive JVM process should be Sl.
More about the internals of jstack.
There is also jattach project which is a better alternative to jstack / jmap. It can automatically handle credentials issue, it works with Docker containers, supports chroot'ed JVMs and handles uncommon file systems.
I have a .jar file I want to run all the time on a Debian server.
Currently I have figured out how to access the server via ssh on Cygwin and start the .jar. But when I close the Cygwin window on my development machine, it kills the process on the server (I think, as it's not responding any longer).
Currently I start it like this:
java -jar myjar.jar packageName.fileNameOfFileWithMainMethod
I need to make this file run automatically and continually on the server (it is an integral part of the system I am developing).
Unfortunately I know next to nothing about server management, or non-windows operating systems in general (wasn't me who chose or made the server), so I really don't know what to do, nor what to search for (apparently, since my searching gave no usable results).
I have read (and edited because the text was a mess) this question, but although I feel it might be hinting in the right direction, I didn't get much help from it. I also tried my best googlefu, but it got me a lot of only tangentially related results.
I guess I'll have to make some kind of script (possibly containing the code line above), do some stuff to it and put it somewhere specific on the server to accomplish what I want to do.
Would someone be so kind as to explain how this is done?
Shell hooks are good for configuring user environment variables.
Cron is for scheduled jobs, mostly related maintenance, such as creating backups, managing log-files etc ...
Background processes with nohup as advised by Николай Митропольский, or ssh with "screen" application (which let you detach/reattach to a "session"), will be useful in development time.
But can not handle server shutdown cleanups, or respond to restarts.
Init scripts mentioned above is the standard way to start/stop services.
There is an application named init, which is the first application started when a Unix-like system boots.
Init, according to runlevel, starts some scripts, and those scripts manages daemons (services in Windows).
So for services, you write "hooks" for runlevels,
In Debian, /etc/init.d/ where you put your init scripts,
you can read the scripts inside this folder to get the idea,
they are text files (bash scripts).
Those scripts are called with an argument
(a standard keywords, such as "start", "stop" etc..).
/etc/rc?.d/ (where ? is one of runlevels), where the init finds the scripts to run.
But those scripts are just "automatically created" symbolic links to the scripts in /etc/init.d/.
You do not need to touch anything inside /etc/rc?.d/ folder.
*After putting your script into /etc/init.d/,
you only need to call to create symbolic links *:
sudo update-rc.d "your-scripts-name" defaults
As you see there are some prefixes attached to names of scripts;
for example /etc/rc1.d/K10apache2 which is a symbolic link to /etc/init.d/apache2.
So a simple "ordered by name execution" is possible here.
Automatically creating those prefixes (so the execution order),
dependency information is required.
So init scripts includes this information.
Also information required when (at which runlevel) those scripts should be called (with "start" or "stop").
This dependency information is placed as comments in those scripts.
Forexample apache server init script (/etc/init.d/apache2) includes those lines;
# Provides: apache2
# Required-Start: $local_fs $remote_fs $network $syslog $named
# Required-Stop: $local_fs $remote_fs $network $syslog $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
Detailed information exists in Debian policy;
https://www.debian.org/doc/debian-policy/ch-opersys.html#s-sysvinit
also this will be usefull;
https://refspecs.linuxbase.org/LSB_2.1.0/LSB-generic/LSB-generic/iniscrptfunc.html
NOTE:
There is a huge transition and debates/fragmentations in Unix world.
Init and init scripts, tradationally used in Unix/Unix-like systems, nowadays becoming obsolete on many systems.
https://en.wikipedia.org/wiki/Init#Replacements_for_init
Debian currently uses systemd, but init scripts still works with systemd (systemd provides the compatibility).
One simple suggestion would be to run the jar file using Linux's CRON.
This article from unix stack exchange should get you going the correct direction for running a jar file using cron.
Alternatively, this article from mkyong.com is also clear and concise.
For example:
Connect using Cygwin
Run crontab -e
Enter 0 * * * * java -jar myjar.jar packageName.fileNameOfFileWithMainMethod to run the jar file every hour at the top of the hour. Or, to start it once on server startup enter #reboot java -jar myjar.jar packageName.fileNameOfFileWithMainMethod
Simplest solution is to detach process by using nohup with &:
nohup java -jar myjar.jar packageName.fileNameOfFileWithMainMethod &
to stop process will be possible with kill <process-id> command
process id could be found by ps -ef | grep packageName.fileNameOfFileWithMainMethod
But if you are developing serious application that is long-running on server you have to deal with initialization system like systemd, upstart or something like that.
On Linux when using -XX+HeapDumpOnOutOfMemoryError the hprof file produced is owned by the user under which the java process is running and has permissions of 600.
I understand that these permissions are best security wise but is it possible to override them?
You can start the JVM with
java -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="chmod g+r java_pid*.hprof" {mainclass} {args}
The command runs after the heap dump is created. This will allow group read access to all heap dump files in the current directory for example.
The -XX:OnOutOfMemoryError parameter doesn't work for me with spaces in the command on JRE 7(1.7.0_72). But pointing to a shell script (without spaces) does. Example:
-XX:OnOutOfMemoryError="/path/to/shell/script.sh"
I have a java application and I want to run a script whenever it experiences and OutOfMemoryException
This works great:
$ java -server -XX:OnOutOfMemoryError="./oom_script %p" TestOOMClass
Unfortunately my application is run by a bash script in production. The script boils down to this:
cmd='java -server -XX:OnOutOfMemoryError="./oom_script %p" TestOOMClass'
##does a lot of checking and cmd building here
exec -a app ${cmd}
When run like this java never respects the double quotes and thinks %p is the class. how do I prevent this? I've tried double escaping but that doesn't work.
Since your program is run as a shell script, I would suggest putting this as the first line in your shell script after the shebang:
set -xv
Then, in the crontab, put 2>&1 at the end of the command line, so STDERR and STDOUT are merged. Crontab usually emails out the STDOUT of a command to root, so you can see what the output is. If not, then apend the following to the end of the command in your crontab:
> /somedir/output.$$ 2>&1
Make sure somedir exists, and after crontab runs your command, you'll see the verbose and debug output. Each line in your shell script will be displayed before it is executed -- both as written and as the shell actually interprets it.
The set -xv becomes very useful in debugging any sell script. There could be all sorts of environmental issues involved between the cronjob and the script running under your login. You might even find a shell issue. For example, crontab usually executes shell scripts in Bourne shell and you probably have Bash or Kornshell as your default shell. Whatever it is, you'll usually find out the issue very quickly when you turn on verbose/debug mode.
You don't even have to do this to the entire script. You can put set -xv anywhere in your script to turn on verbose/debug mode, and set +xv to turn it off.
I could make several pious high minded recommendations (use quotes, don't assume environment things, prefix your command line with "bash -c" to make sure you're using the right shell, etc.), but this would be guessing what could be wrong. In order to really debug this issue, I would need to see the machine, know the OS, see your entire shell script, and understand the entire environment. And, the first thing I would do is add set -xv in your shell script.
Quotes and escaping is an art. I would suggest you add echo ${cmd} before calling exec so you can see what it looks like then.
I would suggest using
cmd='java -server -XX:OnOutOfMemoryError=\\"./oom_script %p\\" TestOOMClass'
instead (untested). You need it to look like \" when being echoed.
an alternative i suggest (to bypass the problem, not solve it indeed) is to rung and bash script and access the $PPID:
PPID The process ID of the shell's parent. This variable is readonly.
then kill the process with that ID (please bare in mind that is an untested suggestion)