I am using Apache Ignite 2.6. I am using Ignite Filesystem, and when I write a specific file, which is about 25 MB, to IGFS, over and over, the data is not saved into the non-heap space. Instead, it goes into heap, which is subject to Garbage Collection, and it is relatively slow. How do I get IGFS to save a file into the large heap space I have allocated for it?
High level architecture--I have a client ignite node running inside of a tomcat for now, and a server ignite node, on which I intend this data to be stored. Scaling can occur once I get this working as expected--but it is very slow because of the aforementioned problem. It also OOMs when it runs out of heap space very quickly. Thing is, I want it to use the 30G of NON HEAP space I have allocated!
I intend this it to be an in memory cache. I am allocating 2 G of heap space and 30G of non heap space to the JVM. The non heap space never gets used and it runs out of memory as a result. I have confirmed that the non-heap space is not used using the JMX Console Memory tab--non heap space stays well below 100M, while heap space quickly balloons to 2G and then the JVM crashes.
The details: First, my ignite configuration (spring xml):
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
<property name="searchSystemEnvironment" value="true"/>
</bean>
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="marshaller">
<bean class="org.apache.ignite.internal.binary.BinaryMarshaller" />
</property>
<property name="fileSystemConfiguration">
<list>
<bean class="org.apache.ignite.configuration.FileSystemConfiguration">
<property name="name" value="igfs"/>
<property name="blockSize" value="#{128 * 1024}"/>
<property name="perNodeBatchSize" value="512"/>
<property name="perNodeParallelBatchCount" value="16"/>
<property name="prefetchBlocks" value="32"/>
</bean>
</list>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration" >
<!-- if I don't set this, the system region runs out of memory almost immediately -->
<property name="systemRegionMaxSize" value="#{6L * 1024 * 1024 * 1024"} />
<property name="systemRegionInitialSize" value="#{6L * 1024 * 1024 * 1024"} />
</bean>
</property>
</bean>
Here is the script I use to start up my ignite server process. It's a shell script running on a Linux machine with 64 G RAM and 40 G disk space.
IGNITE_HOME=/data/apache-ignite
export IGNITE_HOME
IGNITE_JMX_PORT=1234
export IGNITE_JMX_PORT
$IGNITE_HOME/bin/ignite.sh $IGNITE_HOME/ignite-media-server.xml -J-Xmx2G -J-Xms2G -J-XX::+HeapDumpOnOutOfMemoryError -J-XX:HeapDumpPath=$IGNITE_HOME -J-XX:+PrintGC -J-XX:+PrintGCTimeStamps -J-XX:+PrintGCDateStamps -J-Xloggc:$IGNITE_HOME/gc.log-$(date +%m%d-%H%M%S) -J-XX:+UseG1GC -J-XX:DisableExplicitGC -J-XX:MaxDirectMemorySize=30G
This is the code that creates my client igfs object, through which I save files to ignite. They tend to be on the large side.
public void init() throws Exception{
igniteInstanceName = "client-name=" + hostInfo.getLocalHost();
Ignition.setClientMode(true);
// reading in the same config file as the server uses to start up above. The big difference is the clientMode set to true here.
try(InputStream configFileInputStream = new FileInputStream(ResourceUtils.getFile("ignite-media-server.xml"));){
ignite = IgnitionEx.start(configFileInputStream, igniteInstanceName, null, null);
igfs = ignite.fileSystem("igfs");
}
catch(Throwable t){ /* do log */}
}
Here is a save method, that saves my files to ignite:
public saveStream(String cachePath, AudioInputStream toCache){
OutputStream os = null;
try{
IgfsPath cacheFile = new IgfsPath(cachePath);
os = igfs.create(cacheFile, true);
AudioSystem.write(toCache.getDataStream, AudioFileFormat.TYPE.WAVE, os);
}
finally{
// close streams
}
}
Why doesn't my data get saved to the speedy off-heap space? What am I missing? my server.config comes almost straight from the igfs provided example.
In other confusion, when I use ignitevisor.cmd to inspect memory usage on the server node before and after a shorter test (that doesn't make it crash) I see the following:
Look at memory allocation while ignite is empty in ignitevisor.cmd. See that my igfs region says:
Heap Memory Initialized: 2g
Heap Memory Used: 56mb
Non-heap memory initialized: 2mb
Non Heap memory used: 49 mb
Non heap memory maximum: 744mb
Create JUST SHY 2 G worth of files saved in IGFS--just short of an OOM since from bitter experience I know it will blow up shortly. Use ignitevisor.cmd to look at the memory allocation of the nodes. This is what ... – MeowCode 2 mins ago
Heap memory initialized: 2gb
Heap memory used: 1gb
Non Heap memory used 64 MB
Non heap memory maximum: 744mb
Why is there still almost nothing in non-heap? And why does ignitevisor think that the non-heap maximum is 744 MB when it should be 30 GB?
In other points of interest, if I increase my heap size to 6 GB, it runs longer, but still the server crashes with an "OutOfMemoryError:Java heap space". Interestingly, I can reproduce this even when I enable disk persistence. Inspecting the heap dump file reveals a lot of ConcurrentLinkedHashMap entries. The entries themselves are "org.apache.ignite.internal.GridTopic" objects. Each one has a uuid and most appear to be of type TOPIC_DATASTREAM.
Data is saved to Off-Heap all right, but you should be aware that a lot of transient objects involved in IGFS operation will still be briefly held on heap (and GCed after that).
"JMX Console Memory tab--non heap space" is the wrong metric. I don't think that there are any JVM metrics for Off-Heap. However, Ignite will print Off-Heap statistics at regular intervals.
Why you would run out of memory is not obvious. Have you tried collecting heap dump and analyzing it?
Related
I'm currently working on a project involving a distributed Apache Ignite DB on a cluster of Raspberry Pi.
I want to have 2 separate data regions including one with persistence enabled. Here is my custom config :
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration" id="ignite.cfg">
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="default_region"/>
<!-- Set max cache size to 50 MBytes-->
<property name="maxSize" value="#{50 * 1024 * 1024}"/>
<!-- Specifiying an eviction policy that evicts the latest used data when the data hits 90% of the max storage capacity-->
<property name="pageEvictionMode" value="RANDOM_LRU"/>
<property name="evictionThreshold" value="0.9"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="persistence_region"/>
<!-- Set max cache size to 100 MBytes-->
<property name="maxSize" value="#{100 * 1024 * 1024}"/>
<!-- Enable persistent data storage -->
<property name="persistenceEnabled" value="true"/>
</bean>
</list>
</property>
</bean>
</property>
<property name="authenticationEnabled" value="true"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="multicastGroup" value="228.10.10.157"/>
</bean>
</property>
</bean>
</property>
</bean>
I built my own Docker image with my custom conf based on the official Dockerfile except that I changed the base image with: FROM arm32v7/openjdk:8-jre-alpine
The troubles begin when I try to start my image... I have some warnings such as :
Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=1300MB, available=924MB]
And then several errors like :
[SEVERE][tcp-disco-msg-worker-[crd]-#2-#46][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=disco-notifier-worker, threadName=disco-notifier-worker-#45, blockedFor=11s]
After that, it's impossible for me to activate the cluster (through REST, or the control.sh script) and to query it.
If someone has a config file working on Rpi I'm really interested!
EDIT :
I have tried with #alamar 's suggestion (-Xmx384 and checkpoint page buffer size of 20M) but I still have theses errors when activating :
[SEVERE][rest-#68][GridTcpRestProtocol] Failed to process client request [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=90 lim=90 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=false, heartbeatTs=1607710531775, hashCode=9092883, interrupted=false, runner=grid-nio-worker-tcp-rest-1-#39]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:11211, rmtAddr=/127.0.0.1:59558, createTime=1607710503680, closeTime=1607710511719, bytesSent=2, bytesRcvd=96, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1607710533267, lastSndTime=1607710503680, lastRcvTime=1607710511719, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser [marsh=JdkMarshaller [clsFilter=o.a.i.marshaller.MarshallerUtils$1#1eb05], routerClient=false], directMode=false]], accepted=true, markedForClose=true]], msg=GridClientAuthenticationRequest [cred=SecurityCredentials [login=ignite], super=GridClientAbstractMessage [reqId=1, id=b2d7025f-e4a0-4ab7-8c3e-92e2c1c4aea9, destId=null, super=o.a.i.i.processors.rest.client.message.GridClientAuthenticationRequest#18dd064]]]
class org.apache.ignite.IgniteCheckedException: Failed to send message (connection was closed): GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=90 lim=90 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=false, heartbeatTs=1607710531775, hashCode=9092883, interrupted=false, runner=grid-nio-worker-tcp-rest-1-#39]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:11211, rmtAddr=/127.0.0.1:59558, createTime=1607710503680, closeTime=1607710511719, bytesSent=2, bytesRcvd=96, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1607710533267, lastSndTime=1607710503680, lastRcvTime=1607710511719, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser [marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1#1eb05], routerClient=false], directMode=false]], accepted=true, markedForClose=true]]
at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7589)
at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:172)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1$1.apply(GridTcpRestNioListener.java:296)
at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1$1.apply(GridTcpRestNioListener.java:293)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:399)
at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:354)
at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:293)
at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:261)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:399)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:511)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:490)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:467)
at org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:187)
at org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:184)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:399)
at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:354)
at org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:184)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to send message (connection was closed): GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=90 lim=90 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-rest-1, igniteInstanceName=null, finished=false, heartbeatTs=1607710531775, hashCode=9092883, interrupted=false, runner=grid-nio-worker-tcp-rest-1-#39]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:11211, rmtAddr=/127.0.0.1:59558, createTime=1607710503680, closeTime=1607710511719, bytesSent=2, bytesRcvd=96, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1607710533267, lastSndTime=1607710503680, lastRcvTime=1607710511719, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser [marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1#1eb05], routerClient=false], directMode=false]], accepted=true, markedForClose=true]]
at org.apache.ignite.internal.util.nio.GridNioServer.send0(GridNioServer.java:642)
at org.apache.ignite.internal.util.nio.GridNioServer.send(GridNioServer.java:583)
at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionWrite(GridNioServer.java:3693)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:121)
at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionWrite(GridNioCodecFilter.java:96)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:121)
at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionWrite(GridNioFilterChain.java:269)
at org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionWrite(GridNioFilterChain.java:192)
at org.apache.ignite.internal.util.nio.GridNioSessionImpl.send(GridNioSessionImpl.java:117)
at org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:290)
... 16 more
Result of "control.sh --set-state ACTIVE" :
Control utility [ver. 2.10.0-SNAPSHOT#20201013-sha1:a2fa7ec3]
2020 Copyright(C) Apache Software Foundation
User: root
Time: 2020-12-11T18:19:04.421
This cluster requires authentication.
Connection to cluster failed. Latest topology update failed.
Command [SET-STATE] finished with code: 2
Control utility has completed execution at: 2020-12-11T18:19:56.559
Execution time: 52138 ms
Thank you!
People did start it occasionally on Rpi.
In your case, please try decreasing JVM's Xmx (-Xmx384 will be OK) and also specify checkpoint page buffer size for persistent region explicitly (20M should be OK in your case).
If you still see "threads blocked" exceptions, please share complete log. You may use Apache Ignite userlist for that. Also, please describe what happens when you try to activate.
There are a couple of things here.
First, I don't think you needed to update the Docker file. A Pi3 has 64-bit cores, so the default should work. I created this one and it works fine on a Pi4.
Second, between your JVM and your data region, you're allocating more memory than your Pi has:
Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=1300MB, available=924MB]
You only appear to have 150Mb for off-heap, so I wonder how big you Java heap is? 8Gb should be plenty.
If your Pi is swapping to disk, that might be causing the blocked system threads.
Finally, since you enabled authentication, you need to supply a username and password when you try to activate the cluster. The default appears to be ignite/ignite.
I have installed jProfiler in my Linux machine and I am saving the data into .jps file. I am then loading this file into jProfiler UI in my local machine.
Here is my config file:
<?xml version="1.0" encoding="UTF-8"?>
<config>
<nextId id="104" />
<generalSettings setupHasRun="false">
<recordingProfiles>
<recordingProfile id="10" name="CPU recording">
<actionKey id="cpu"/>
</recordingProfile>
</recordingProfiles>
</generalSettings>
<templates>
<template id="50" name="Instrumentation, all features supported" startFrozen="false" recordCPUOnStartup="false" vmCannotExit="false" instrumentationType="1" samplingNoFilters="false" lineNumbers="false" samplingFrequency="5" timeType="1" disableCPUProfiling="false" recordAllocOnStartup="true" recordArrayAlloc="true" enableTriggersOnStartup="true" allocTreeRecordingType="1" disableMonitorContentions="false" componentDetection="true" chronoHeap="false" autoUpdatePeriodLong="5" autoUpdatePeriodShort="2" allUrls="false" payloadCap="50" eventCap="20000" showSystemThreads="false" utilConcurrentHandling="true" libraryDebugParameters="" exceptionalCap="5" exceptionalTimeType="4" autoTuneInstrumentation="true" autoTuneMaxAverage="100" autoTuneMinPerMille="10" samplingPayloadCallStacks="true" description="This is JProfiler's fully featured mode. In this setting, call stack information is accurate, but CPU overhead and distortion of measured call times may be high, depending on your filter settings. You should define inclusive filters for your own packages." system="true" />
<template id="51" name="Sampling for CPU profiling, some features not supported" startFrozen="false" recordCPUOnStartup="false" vmCannotExit="false" instrumentationType="3" samplingNoFilters="false" lineNumbers="false" samplingFrequency="5" timeType="1" disableCPUProfiling="false" recordAllocOnStartup="true" recordArrayAlloc="true" enableTriggersOnStartup="true" allocTreeRecordingType="1" disableMonitorContentions="false" componentDetection="true" chronoHeap="false" autoUpdatePeriodLong="5" autoUpdatePeriodShort="2" allUrls="false" payloadCap="50" eventCap="20000" showSystemThreads="false" utilConcurrentHandling="true" libraryDebugParameters="" exceptionalCap="5" exceptionalTimeType="4" autoTuneInstrumentation="true" autoTuneMaxAverage="100" autoTuneMinPerMille="10" samplingPayloadCallStacks="true" description="This template is particularly suitable for CPU profiling and for memory profiling when accurate allocation information is not important. Sampling has a very low overhead and does not distort measured call tines. Some views, like the method statistics are no available. JEE payloads cannot be annotated in the call tree, but payload hotspots without backtraces are available." system="true" />
</templates>
<sessions>
<session id="80" templateId="50" name="Animated Bezier Curve Demo" type="1" isStartupWorkingDirectory="true" mainClass="bezier.BezierAnim">
<filters>
<filter type="inclusive" name="com." />
</filters>
<exceptionalMethods/>
<classPath>
<classPathEntry path="demo/bezier/classes" />
</classPath>
<sourcePath>
<sourcePathEntry path="demo/bezier/src" />
</sourcePath>
<probes>
<probe name="com.jprofiler.agent.probe.interceptor.TrackingInterceptor" enabled="true" startProbeRecording="false" events="false" annotatePayloads="false">
<id value="3" />
</probe>
</probes>
</session>
The problem which I am facing is that I am not able to get any details regarding method statistics under the CPU views tab in jProfiler UI.
But I am able to get other fields in Telemetrics.
The version in use: JProfiler 9.1 and I have used sample config.xml to start with my test. DO i need to make any changed in my config file to get the method level statistics in my .jps file
Method statistics is recorded separately, because the overhead is too high to always be recorded together with CPU data.
When the session is live, go the method statistics view and enabled recording. For offline profiling, there is a trigger action that starts method statistics recording.
I have a very simple computation which produces letter matrices finds probably all the words in the matrix. The letters in the word are adjacent cells.
for (int i = 0; i < 500; i++) {
System.out.println(i);
Matrix matrix = new Matrix(4);
matrix.scanWordsRandomly(9);
matrix.printMatrix();
System.out.println(matrix.getSollSize());
matrix.write_to_db();
}
Here is the persisting code.
public void write_to_db() {
Session session = null;
try {
session = HibernateUtil.getSessionFactory().openSession();
session.beginTransaction();
Matrixtr onematrixtr = new Matrixtr();
onematrixtr.setDimension(dimension);
onematrixtr.setMatrixstr(this.toString());
onematrixtr.setSolsize(getSollSize());
session.save(onematrixtr);
for (Map.Entry<Kelimetr, List<Cell>> sollution : sollutions.entrySet()) {
Kelimetr kelimetr = sollution.getKey();
List<Cell> solpath = sollution.getValue();
Solstr onesol = new Solstr();
onesol.setKelimetr(kelimetr);
onesol.setMatrixtr(onematrixtr);
onesol.setSoltext(solpath.toString().replace("[", "").replace("]", "").replace("true", "").replace("false", ""));
session.save(onesol);
}
session.getTransaction().commit();
session.close();
}
catch (HibernateException he) {
System.out.println("DB Error : " + he.getMessage());
session.close();
}
catch (Exception ex) {
System.out.println("General Error : " + ex.getMessage());
}
}
Here is the hibernate configuration file.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD 3.0//EN" "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/kelimegame_db_dev?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password">!.Wlu9RrCA</property>
<property name="hibernate.show_sql">false</property>
<property name="hibernate.query.factory_class">org.hibernate.hql.classic.ClassicQueryTranslatorFactory</property>
<property name="hibernate.format_sql">false</property>
<!-- Use the C3P0 connection pool provider -->
<property name="hibernate.c3p0.acquire_increment">50</property>
<property name="hibernate.c3p0.min_size">10</property>
<property name="hibernate.c3p0.max_size">100</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">5</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
<mapping resource="kelimegame/entity/Progress.hbm.xml"/>
<mapping resource="kelimegame/entity/Solstr.hbm.xml"/>
<mapping resource="kelimegame/entity/Kelimetr.hbm.xml"/>
<mapping resource="kelimegame/entity/User.hbm.xml"/>
<mapping resource="kelimegame/entity/Achievement.hbm.xml"/>
<mapping resource="kelimegame/entity/Matrixtr.hbm.xml"/>
</session-factory>
</hibernate-configuration>
After finding all possible solutions I persist the matrix and the solutions using hibernate. I am also using c3pO library. I am not spawning any thread. All the work is being done in a very simple iterative way. But I am running the jar in separate processes.
From different terminals I am executing this :
java -jar NewDB.jar
I got a deadlock as follows :
Apr 25, 2013 8:38:05 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#7f0c09f9 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Apr 25, 2013 9:08:23 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#7f0c09f9 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#2933f261
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#1
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#116dd369
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#0
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#41529b6f
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#2
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#165ab5ea
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#1d5d211d
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#4d2905fa
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:662)
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:662)
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#2,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:662)
Apr 25, 2013 9:41:29 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#7f0c09f9 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Apr 25, 2013 9:55:18 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#7f0c09f9 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#5a337b7d
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#0
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#69f079ce
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#1
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#2accf9b8
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#2
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#771eb4fb
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#fc07d6
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#2266731b
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#740f0341
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#59edbee
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#78e924
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#2123aba
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#7acd8a65
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kfsx8uibeyqevbbapc|4045cf35]-HelperThread-#0,5,main]
java.text.NumberFormat.getInstance(NumberFormat.java:769)
java.text.NumberFormat.getInstance(NumberFormat.java:393)
java.text.MessageFormat.subformat(MessageFormat.java:1262)
java.text.MessageFormat.format(MessageFormat.java:860)
java.text.Format.format(Format.java:157)
java.text.MessageFormat.format(MessageFormat.java:836)
com.mysql.jdbc.Messages.getString(Messages.java:106)
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2552)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3002)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2991)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3532)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:943)
com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4113)
com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1308)
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2336)
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2176)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2158)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:525)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:183)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:172)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:188)Killed
caglar#ubuntu:~/NetBeansProjects/NewDB/dist$
My question is as follows :
Can this deadlock in c3po happen since I am running the program in
separate processes?
Should I use one process and multiple threads inside of this
process?
How can I trace this deadlock understand the cause of it? Is there a way to trace multiple JVMs causing deadlocks?
this is an interesting one.
you've published two distinct APPARENT DEADLOCKS. the first one is being caused by c3p0 attempting to close() Connections, and those close() operations are neither succeeding nor failing with an Exception in a timely manner. the second APPARENT DEADLOCK shows problems with Connection acquisition: c3p0 is attempting to acquire new Connections, and those attempts are neither succeeding nor failing with an Exception in a timely manner. the fact that very different operations are freezing suggests that it might be a more general problem with your dbms locking up under the stress of what you are doing or somesuch. it should be no problem to run multiple processes against your database, but you need to stay cognizant of limits.
there are a few interesting things about your configuration:
1) hibernate.c3p0.max_statements=5 is a very bad idea, on almost any pool and particularly on pools this large. you've got up to 100 Connections, and you're only allowing a total of 5 Statements to be cached between all of them. this might stress both the pool and the DBMS, as you will constantly be churning through PreparedStatements and the statement cache does a lot of bookkeeping about that. you may have meant that to be 5 cached statements per connection, but that's not what you have configured. you have set a global maximum for your pool. maybe try hibernate.c3p0.maxStatementsPerConnection=5 instead? or set max_statements to zero to turn statement caching off, at least until you resolve your deadlock. see http://www.mchange.com/projects/c3p0/#configuring_statement_pooling
2) if you are running your computation in multiple processes rather than multiple Threads, do you really need each process to hold 50 - 100 Connections? things may well be freezing up simply because you are stressing the dbms with too many Connections outstanding as each of your multiple processes acquire lots of resource-heavy Connections. you don't need more Connections in any process than you might have client Threads running concurrently within that process. i'd set hibernate.c3p0.acquire_increment and probably hibernate.c3p0.max_size to much smaller values.
3) if you really do need all those Connections running simultaneously, you can reduce the vulnerability of your pools to deadlock by increasing the config parameter numHelperThreads to some value greater than its default of 3. you probably want numHelperThreads to be something like twice the number of cores available on your machine. given that you are running multiple processes though, you might find that you are saturating your CPU, and that is freezing things up. so watch for that.
basically, try updating your configuration so that you are using resources -- file handles, network connections, CPU -- as efficiently as possible and so that you are not unnecessarily stressing the pool / statement cache / dbms more than you need to be.
if these suggestions don't resolve the problem, please post the fill config of your pools. c3p0 dumps its config at INFO level on pool initialization.
good luck!
I do a project merging Hibernate and Spring in a Java web application, using Tomcat under Linux environment. Due to the Mysql 8 hours timeout problem, we want to use C3P0 to manage a connection pool with our Mysql database.
But when we use it, we have numerous threads that are created. I figured it out beacause I did on each request a print of all of them with a memory status that show me the increasing memory and that kind of threads:
name: C3P0PooledConnectionPoolManager[identityToken->1hged7o8r13kpj7n1h3ycia|39c446]-HelperThread-#0 daemon: true group! main groupParent: system alive: true interrupted: false
name: C3P0PooledConnectionPoolManager[identityToken->1hged7o8r13kpj7n1h3ycia|17ec0e8]-AdminTaskTimer daemon: true group! main groupParent: system alive: true interrupted: false
It can produce more than 500 threads like these ones, after enough time.
Here is my Hibernate.cfg.xml:
<property name="connection.provider_class">
org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.idle_test_period">5</property>
<property name="hibernate.c3p0.max_size">100</property>
<property name="hibernate.c3p0.max_statements">100</property>
<property name="hibernate.c3p0.min_size">10</property>
<property name="hibernate.c3p0.timeout">5</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/myBase</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password"></property>
<property name="hibernate.hbm2ddl.auto">update</property>
<property name="hibernate.default_schema">myProject</property>
<property name="hibernate.query.factory_class">org.hibernate.hql.classic.ClassicQueryTranslatorFactory</property>
<property name="show_sql">false</property>
<property name="cache.provider_class">
org.hibernate.cache.NoCacheProvider
</property>
I also tried to add a C3P0 propeties file, but except reducing the helper thread number, it don't delete the unsused thread:
c3p0.maxStatements=5
c3p0.maxIdleTime=10
c3p0.numHelperThreads=1
c3p0.testConnectionOnCheckout=true
c3p0.preferredTestQuery=SELECT 1
c3p0.initialPoolSize=1
c3p0.minPoolSize=1
c3p0.maxPoolSize=10
c3p0.acquireIncrement=1
c3p0.idleConnectionTestPeriod=1
Does anyone have an idea of why this happen and how to solve this problem?
Thanks a lot.
if you are seeing a multiplication of c3p0 helper and timer threads, you are somehow creating a multitude of c3p0 DataSources when you want there to be just one. sometimes this happens if you are hot-reloading your app but forgetting to close() your old c3p0 DataSource when you recycle.
effectively it looks like you are "leaking" DataSources. you need to figure out why/where this is happening. for some clues, check out your logs for c3p0 DataSource initialization messages at INFO level. Search for the string "Initializing c3p0 pool", for example.
good luck!
Ok I found a combination of properties to solve my problem, keeping in mind that I don't need a lot of connection at a time:
c3p0.maxStatements=5
c3p0.maxIdleTime=10
c3p0.numHelperThreads=3
c3p0.testConnectionOnCheckout=true
c3p0.preferredTestQuery=SELECT 1
c3p0.initialPoolSize=1
c3p0.minPoolSize=1
c3p0.maxPoolSize=1
c3p0.acquireIncrement=1
c3p0.idleConnectionTestPeriod=1
c3p0.maxAdministrativeTaskTime=1
Thanks to everyone
I would expect it to create a number of threads proportional to c3p0.minPoolSize
and c3p0.maxPoolSize and your maximum is 10.
http://www.mchange.com/projects/c3p0/#other_ds_configuration
"numHelperThreads and maxAdministrativeTaskTime help to configure the behavior of DataSource thread pools. By default, each DataSource has only three associated helper threads. If performance seems to drag under heavy load, or if you observe via JMX or direct inspection of a PooledDataSource, that the number of "pending tasks" is usually greater than zero, try increasing numHelperThreads. maxAdministrativeTaskTime may be useful for users experiencing tasks that hang indefinitely and "APPARENT DEADLOCK" messages. (See Appendix A for more.) "
numHelperThreads defines how many threads per DataSource are used, therefore indeed you will have 10 threads with numHelperThreads=1.
The only way to make sure C3P0 consumes only one Thread is to set c3p0.minPoolSize
and c3p0.maxPoolSize to 1 but this defeats the purpose of connection pooling.
I'm running Tomcat7, the server is quite powerful, 8 GB RAM 8-core.
My problem is that the RES memory is geting higher and higher, until the server just doesn't respond anymore, not even calling OnOutOfMemoryError.
Tomcat configuration :
-Xms1024M
-Xmx2048M
-XX:PermSize=256m
-XX:MaxPermSize=512m
-XX:+UseConcMarkSweepGC
-XX:OnOutOfMemoryError='/var/tomcat/conf/restart_tomcat.sh'
Memory informations :
Memory: Non heap memory = 106 Mb (Perm Gen, Code Cache),
Loaded classes = 14,055,
Garbage collection time = 47,608 ms,
Process cpu time = 4,296,860 ms,
Committed virtual memory = 6,910 Mb,
Free physical memory = 4,906 Mb,
Total physical memory = 8,192 Mb,
Free swap space = 26,079 Mb,
Total swap space = 26,079 Mb
Perm Gen memory: 88 Mb / 512 Mb ++++++++++++
Free disk space: 89,341 Mb
The memory used by Tomcat doesn't look that high compared to the top command.
I also had java.net.SocketException: No buffer space available when trying to connect to SMTP server or when trying to connect to facebook servers.
I use Hibernate, with c3p0 connection pool with this configuration :
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://urldb/schema?autoReconnect=true</property>
<property name="hibernate.connection.username">username</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQL5InnoDBDialect</property>
<property name="hibernate.connection.password"></property>
<property name="connection.characterEncoding">UTF-8</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.idle_test_period">300</property>
<property name="hibernate.c3p0.timeout">5000</property>
<property name="hibernate.c3p0.max_size">50</property>
<property name="hibernate.c3p0.min_size">1</property>
<property name="hibernate.c3p0.max_statement">0</property>
<property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
I couldn't find anything... does someone have an hint of where I should be looking for ?
Thanks!
[UPDATE 1]HEAP DUMP :
HEAP HISTOGRAM :
class [C 269780 34210054
class [B 5600 33836661
class java.util.HashMap$Entry 221872 6212416
class [Ljava.util.HashMap$Entry; 23797 6032056
class java.lang.String 271170 5423400
class org.hibernate.hql.ast.tree.Node 103588 4972224
class net.bull.javamelody.CounterRequest 28809 2996136
class org.hibernate.hql.ast.tree.IdentNode 23461 2205334
class java.lang.Class 14677 2113488
class org.hibernate.hql.ast.tree.DotNode 13045 1852390
class [Ljava.lang.String; 48506 1335600
class [Ljava.lang.Object; 12997 1317016
Instance Counts for All Classes (excluding platform) :
103588 instances of class org.hibernate.hql.ast.tree.Node
33366 instances of class antlr.ANTLRHashString
28809 instances of class net.bull.javamelody.CounterRequest
24436 instances of class org.apache.tomcat.util.buf.ByteChunk
23461 instances of class org.hibernate.hql.ast.tree.IdentNode
22781 instances of class org.apache.tomcat.util.buf.CharChunk
22331 instances of class org.apache.tomcat.util.buf.MessageBytes
13045 instances of class org.hibernate.hql.ast.tree.DotNode
10024 instances of class net.bull.javamelody.JRobin
9084 instances of class org.apache.catalina.loader.ResourceEntry
7931 instances of class org.hibernate.hql.ast.tree.SqlNode
[UPDATE 2] server.xml :
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
URIEncoding="UTF-8"
maxThreads="150"
minSpareThreads="25"
maxSpareThreads="75"
enableLookups="false"
acceptCount="1024"
server="unknown"
address="public_ip"
/>
****[UPDATE 3] Output from log files : ****
2012-06-04 06:18:24,152 [http-bio-ip-8080-exec-3500] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/api].[Jersey REST Service]- Servlet.ser
vice() for servlet [Jersey REST Service] in context with path [/socialapi] threw exception
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563)
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118)
at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:326)
at org.apache.coyote.Request.doRead(Request.java:422)
[UPDATE 4] ServletContext
I use a ServletContextListenerin my application to instanciate controllers and keep a reference with event.getServletContext().setAttribute. Those controllers loads configurations and translations (the 88Mb in Perm).
Then to use the database i use :
SessionFactory sf = dbManager.getSessionFactory(DatabaseManager.DB_KEY_DEFAULT);
Session session = sf.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
//Do stuuf
tx.commit();
} catch (Exception e){
//Do something
} finally {
session.close();
}
Could this be the source of a leak ?
Why not to use Manual transaction/session, and how would you do then ?
Try with this parameter:
+XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpPath=dump.log
Also try with lower start memory parameters -Xms.
then you can inspect the dump to see if the problem was object allocation.
While running try
jps
That will output all java processes, lets say Tomcat is PID 4444:
jmap -dump:format=b,file=heapdump 4444
And
jhat heapdump
If you run out of memory while executing jhat just add more memory. From there you can inspect the heap of your application.
Another way to go is to enable Hibernate statistics to check that you are not retrieving more objects. Although it looks like a full garbage collection every hour should not be a problem (room for do it better there).
-verbose:gc -Xloggc:/opt/tomcat/logs/gc.out -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
And with GCViewer for example take a look at every space of memory (ternured, eden, survivors, perm).
Another handy tool:
jstack 4444 > stack.txt
That will retrieve a full stack trace of every thread running inside the java process with pid 4444.
Bear in mind that you need privileges if you started Tomcat as root or another user.
jps
won't output process which you have no privileges, therefore you cannot connect to it.
Since I don't know what your application is about (and therefore I don't know its requirements) 3 million instances looks like a lot.
With Hibernate statistics you can see which classes you instantiate the most.
Then tunning the proportions of your eden and ternured garbage recolection can be more efficient.
Newly instantiated objects goes to eden. When it fills up a minor gc triggers. What is not deleted goes to a survivor space. When this fills up it goes to ternured. Full gc will arise when ternured is full.
In this picture (which is inaccurate) I left aside String that become interned and Memory mapped files (that are not in heap). Take a look at which classes you instantiate most. Intensive use of String might lead to quickly fill up perm.
I guess you do so, but use a managed session factory, such as Spring (if in your stack) and avoid manually management of transactions and sessions.
Keep in mind that objects are deleted in the GC when no object refers to it. So as long as a object is reachable in your application the object remain.
If your ServletContextListener instantiate controllers and are stored in the event getServletContext. Make sure you completely remove the reference afterwards, if you keep a reference the objects won't be deleted, since they are still reachable.
If you manage your own transactions and session (which is fine if you cannot use a framework) then you must deal with code maintenance and bugs that Spring-tx for instance has solved and improved.
I personally would take advantage of FOSS. But of course sometime you cannot enlarge the stack.
If you are using Hibernate I would take a look at Spring-orm and Spring-tx to manage transactions and session. Also take a look at Hibernate patter Open Session In View.
I'd also recommend that you download Visual VM 1.3.3, install all the plugins, and attach it to the Tomcat PID so you can see what's happening in real time. Why wait for a thread dump? It'll also tell you CPU, threads, all heap generations, which objects consume the most memory, etc.