I have been working on a large java application. It is quite parallel, and uses several fixedThreadPools (each with 8 threads). I am running it on a computer with 2 cores, each with 4 processors. My program is analyzing large sets of data, and the analysis is saved (serialized) after every set, though it works across data sets, and so is re-loaded every time I run a new one (and then saved).
My problem is this: after running 4-5 data sets (takes about 2 days, and I'm pretty happy with my coding efficiency) it will crash, after exactly the same amount of time on the 5th set (no matter which data set I use). The program is repetitive, and so there is nothing new in the code going on at this time. It is reproducible, and I am not sure what to do. I can post the full error log if that would help... I understand that this problem is ambiguous without a lot more detailed information, but if there are any go-to suggestions, it would be greatly appreciated.
I have been testing different settings to see if anything helps, and right now I am running with the following arguments.
-Xmx6g -Xmx12g -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
Thanks,
Joe
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000000, pid=18454, tid=140120548144896
#
# JRE version: 7.0_03-b147
# Java VM: OpenJDK 64-Bit Server VM (22.0-b10 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea7 2.1.1pre
# Distribution: Ubuntu precise (development branch), package 7~u3-2.1.1~pre1-1ubuntu2
# Problematic frame:
# C 0x0000000000000000
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-7/
#
Just a hard guess...
Might be it is not able to create more files
if you are running this in linux Try running
ulimit -c unlimited
Before you run your java program... This should help in two ways
It should increase the file creation limit
If any error occurs it will create the Core dump.
See how many file IO it is using while the program is running.
I'd instrument it with something like Visual VM. It'll show what's happening in memory, threads, CPU, objects created, etc. in real time as your app runs.
The nice version that I have is for Oracle/Sun JVMs only. There's one that ships with the JDK, but I don't believe it shows as much detail as the version 1.6.3 with all plugins installed.
Just add -Dorg.eclipse.swt.browser.DefaultType=mozilla in eclipse.ini file .
Related
I was going to start Eclipse from the terminal, but ran into this error:
A fatal error has been detected by the Java Runtime Environment: SIGSEGV (0xb) at pc=0x00000038d6e08e83, pid=6018, tid=47814740359488 JRE version: Java(TM) SE Runtime Environment (7.0_76-b13) (build
1.7.0_76-b13) Java VM: Java HotSpot(TM) 64-Bit Server VM (24.76-b04 mixed mode linux-amd64 compressed oops) Problematic frame: C [ld-linux-x86-64.so.2+0x8e83] Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
I'd like to fully understand why this happens, not simply to run this command that it shows me.
I know there's a question: Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
but nobody answered yet.
So please shed some light on this, your help and time is deeply appreciated!
Three questions:
why the core is dumped?
why core dumps are disabled on this machine
what the effects of ulimit -c unlimited are?
From the information you gave I assume it couldn't write a core dump because it would have been too big in terms of file size, at least beyond your ulimit limits. Ld-Linux is the shared library loader, its job is to load other shared libraries. If you type ulimit at the command line it will tell you what limits are set
$ ulimit
Maybe 2 Gigabytes and your eclipse is taking more than that, possible with a 64 bit JVM.
The sigsegv is eclipse or at least the JVM trying to do something with memory it shouldn't. That is accessing protected memory. This could be writing to read only memory or a pointer falling off the end of an allocated buffer. Do you get the same error if you restart eclipse?
If you get a core dump you can use a debugger such as gdb to analyze what went wrong.
gdb <executable_path> <coredump_file_path>
The core dump will be a .pid file. That would tell you more about the crash, maybe a strcpy() that has overflowed a buffer.
Abruptly last afternoon, my Java (Groovy, actually) started hanging in ClassLoader.load(String, boolean) while trying to call ImageIO.read(file). The stack trace indicated it was trying to load AWT related code.
In case it was an IntelliJ or Gradle issue I tried the command-line, where it still hung, but successfully ran past that point if I specified -Djava.awt.headless=true, apparently related to the -XstartOnFirstThread SWT 2013 bug, but of course that meant I was unable to display windows from the program.
This obtained even with a trivial 'only load an image' program.
Contrarily, an existing Java-based application ran happily, and a brand new user on the same machine had no problems.
Moving all my account's . files and logging out/in, and checking my environment variables for suspicious things compared with the brand new user had no effect.
I don't know precisely WHY this was happening, but it relates directly to having previously suffered a native crash:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000131099b1b, pid=11165, tid=4867
#
# JRE version: Java(TM) SE Runtime Environment (8.0_31-b13) (build 1.8.0_31-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C [libtesseract.dylib+0x156b1b] ELIST::length() const+0x5
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/tim/git/t4jhack/hs_err_pid11165.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
because I just triggered it again, and the issue resurrected immediately.
The app crash says
AppMain quit unexpectedly.
Click reopen to open the application again.
This report will be sent to Apple automatically
[show details] [OK] [Reopen]
I clicked 'OK' and 'Reopen' variously - both cause the problem.
The 'fix' I stumbled across:
In System Preferences, open the Java Control Panel, where it will say:
The last time you opened Java, it unexpectedly quit while reopening windows.
Do you want to try to reopen its windows again?
If you choose not to reopen windows, you may have to open and
position the windows yourself.
[don't reopen] [reopen]
You want 'Reopen', then just OK the resulting window.
I have followed the installation instructionrs http://bendemott.blogspot.de/2013/11/installing-pylucene-4-451.html for pylucene using the latest pylucene-4.9.0.0.
And when i tried to to lucene.initVM(), I get the following error:
alvas#ubi:~$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lucene
>>> lucene.initVM()
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007ffba22808b8, pid=5189, tid=140718811092800
#
# JRE version: OpenJDK Runtime Environment (7.0_65-b32) (build 1.7.0_65-b32)
# Java VM: OpenJDK 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 2.5.3
# Distribution: Ubuntu 14.04 LTS, package 7u71-2.5.3-0ubuntu0.14.04.1
# Problematic frame:
# V [libjvm.so+0x6088b8] jni_RegisterNatives+0x58
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/alvas/hs_err_pid5189.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
Aborted (core dumped)
And the file http://pastebin.com/6B8FyC4Z
Is there something wrong with my IceTea configuration? or my JDK? or JRE?
How should I resolve the problem?
So I took a look at your stack trace, and I don't think the issue was specifically pyLucene. In the stack trace, you see this error:
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000000
If you look at the first part, SIGSEGV, that means you have a segmentation fault somewhere in your system. SEGV_MAPERR is the specific error, which means that OpenJDK was trying to map memory to an object and failed. This could've been caused by not enough memory, a bad pagefile/virtual memory, bad address space, or even a bad library. Why it worked on another machine could be anything. Core dumps are really useful, so if you can run
ulimit -c unlimited
that will help give you something to look at. Was this in a VM or on a physical machine? I've seen random sigsegv in my Ubuntu VMs if they don't have enough memory allocated for various Java tasks. I saw this on my ESXi hypervisors specifically, and I noticed it the most was when ESXi started to perform memory swapping. I was able to resolve this by increasing memory, rebooting the VM, and making sure my hypervisor wasn't swapping memory. Let me know if that helps. :)
Edit: I also noticed that if the underlying storage provider had poor performance, that would impact with swap data and I feel that was also am impact with sigsegv issues.
I ran a script on MATLAB and it worked fine, when i want to run the script again, then MATLAB stuck in busy! i found a file "hs_err_pid1124" in the directory i works in it contain the following:
A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 16384000 bytes for GrET in
C:\BUILD_AREA\jdk6_17\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap
space?
#
# Internal Error (allocation.inline.hpp:39), pid=1124, tid=1380
# Error: GrET in
C:\BUILD_AREA\jdk6_17\hotspot\src\share\vm\utilities\growableArray.cpp
#
# JRE version: 6.0_17-b04
# Java VM: Java HotSpot(TM) Client VM (14.3-b01 mixed mode windows-x86 )
.
.
.
My computer RAM is 4G, i increased the System Swap Space, but still the problem not solved!!
Thanks,
The most likely suspect here is your code. I would expect you to do something strange (opening a file, and not closing it later?! Reading each file into a continiously growing variable?!).
However, without code this is hard to diagnose.
Here is what you can do:
Evaluate the visible memory usage: Put a breakpoint somewhere halfway through, and inspect the size of the largest variables. Also check the total size. (If the error is a regular matlab error, you could also use dbstop if error)
Persuade matlab to release memory: If step 1 yields nothing, you may actually be doing things right, but perhaps matlab does not manage its memory properly. This is rare, but occurs sometimes when repeating simple tasks a lot of times. In this case you can place the pack command somewhere in your code. Probably it will help.
This is the first time I am in this situation with Java.
Java just core dumps with the following error:
#
# A fatal error has been detected by the Java Runtime Environment:
#
[thread 140213457409792 also had an error]# Internal Error (safepoint.cpp:300), pid=4327
, tid=140213211031296
# guarantee(PageArmed == 0) failed: invariant
#
# JRE version: 6.0_24-b24
# Java VM: OpenJDK 64-Bit Server VM (20.0-b12 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea6 1.11.4
# Distribution: Ubuntu 12.04 LTS, package 6b24-1.11.4-1ubuntu0.12.04.1
# An error report file with more information is saved as:
# /tmp/hs_err_pid4327.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-6/
when I tried running it on a mac os, it core dumps at the same place (the JREs must be different)... so it must be something related to the code. I have no idea how to debug this, this is not an exception, and the log file specified up there does not give me much information. Any ideas what I can do about it to find the bug?
The /tmp/hs_err_pid4327.log file should contain a stack trace of where the core occurred. Unless you are making a JNI call, it is probably a Java bug.
The core dump is telling what you should do...
If you would like to submit a bug report, please include
instructions how to reproduce the bug and visit:
https://bugs.launchpad.net/ubuntu/+source/openjdk-6/
A quick look makes me think this is already reported.
The bug probably isn't in your code, per se. It's most likely an environmental issue - perhaps a JVM bug, perhaps some unusual condition, and most likely, both - a bug that occurs rarely, under an odd circumstance.
Google for the key elements in the message (e.g. "safepoint.cpp:100"), look at the other reports, and look for things you have in common or workarounds that may apply. In this case, one set of reports suggests that heavy multithreading may contribute to the problem.
Check if you have a hprof file in your application directory. Optionally you could dump at will using
jmap -dump:file=<file_name> <pid>
and then analyze the dump using MAT http://www.eclipse.org/mat/
You could also consider other tools quoted here :
Tool for analyzing java core dump