I am working on implementing Oracle Coherence replicated cache. The implementation is as follows:
<?xml version="1.0"?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>EntryList</cache-name>
<scheme-name>ENTRY_ITEMS</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<replicated-scheme>
<scheme-name>ENTRY_ITEMS</scheme-name>
<backing-map-scheme>
<local-scheme>
<scheme-name>ENTRY_ITEMS</scheme-name>
<unit-calculator>FIXED</unit-calculator>
<expiry-delay>60m</expiry-delay> <!-- expire after 60 minutes -->
<high-units>2000</high-units>
<eviction-policy>LFU</eviction-policy>
</local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</replicated-scheme>
</caching-schemes>
</cache-config>
tangasol-coherence-override.xml
<coherence xmlns:xsi="http://www.w4.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
coherence-operational-config.xsd">
<cluster-config>
<member-identity>
<cluster-name>clusterName</cluster-name>
<!-- Name of the first member of the cluster -->
<role-name>RoleName</role-name>
</member-identity>
<unicast-listener xml-override=coherence-environment.xml/>
</cluster-config>
</coherence>
coherence-environment.xml
<unicast-listener xmlns:xsi="http://www.w4.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
coherence-operational-config.xsd">
<well-known-addresses>
<socket-address id="1">
<address>member1</address>
<port>7777</port>
</socket-address>
</well-known-addresses>
<well-known-addresses>
<socket-address id="2">
<address>member2</address>
<port>7777</port>
</socket-address>
</well-known-addresses>
</unicast-listener>
This is implemented and tested to be working perfectly.
We were testing the eviction policy of the cache. To ease out testing I did the following:
I keep the size of cache as 4 by setting high-units as 4. Now add 4 entries in the cache. This should fill the cache completely.
Now if I make one more entry number 5 in the cache, I was expecting the lease frequently used entry to be kicked out of the cache to make room for the entry number 5.
The next time I access the cache for the new entry number 5, I should get a cache HIT.
But that's not happening, I always get a Cache MISS.
I ran my java code in debug mode and I see that the code PUT's entry number 5 in the cache but this PUT operation does not reflect on the cache.
Now I am definitely not the first person testing the coherence cache eviction policies. Am I missing anything in the configuration? Am I testing the eviction in a wrong way. Any inputs are welcome.
Thanks.
Try to isolate the problem:
change<expiry-delay>1</expiry-delay> (1ms)
add <low-units>0</low-units> (default value is 75% which is 3 entries).
try another policy <eviction-policy>LRU</eviction-policy>
If those won't help, try to add custom eviction policy class to see wheter eviction triggered. see here:
I have tried your example with 3 as High Units. My observation:
Eviction works as soon as I put 4th entry item. So, it works!!
You can start Coherence Server (coherence.sh) for command-line monitoring with same override & cache config file. See details, which gets printed when I put following command to see cache:
Map (?): cache EntryList
Cache Configuration: EntryList
SchemeName: ENTRY_ITEMS
AutoStart: true
ServiceName: ReplicatedCache
ServiceDependencies
EventDispatcherThreadPriority: 10
ThreadPriority: 10
WorkerThreadsMax: 2147483647
WorkerPriority: 5
EnsureCacheTimeout: 30000
BackingMapScheme
InnerScheme (LocalScheme)
SchemeName: ENTRY_ITEMS
UnitCalculatorBuilder
Calculator: FIXED
EvictionPolicyBuilder
Policy: LFU
ExpiryDelay: 1h
HighUnits
Units: 3
UnitFactor: 1
Related
I am using Google cloud managed redis cluster(v5) via redisson(3.12.5)
Following are my SingleServer configurations in yaml file
singleServerConfig:
idleConnectionTimeout: 10000
connectTimeout: 10000
timeout: 3000
retryAttempts: 3
retryInterval: 1500
password: null
subscriptionsPerConnection: 5
clientName: null
address: "redis://127.0.0.1:6379"
subscriptionConnectionMinimumIdleSize: 1
subscriptionConnectionPoolSize: 50
connectionMinimumIdleSize: 40
connectionPoolSize: 250
database: 0
dnsMonitoringInterval: 5000
threads: 0
nettyThreads: 0
codec: !<org.redisson.codec.JsonJacksonCodec> {}
I am getting following exceptions when I increase the load on my application
org.redisson.client.RedisTimeoutException: Unable to acquire connection! Increase connection pool size and/or retryInterval settings Node source: NodeSource
org.redisson.client.RedisTimeoutException: Command still hasn't been written into connection! Increase nettyThreads and/or retryInterval settings. Payload size in bytes: 34. Node source: NodeSource
It seems there is no issue on redis cluster and i think i need to make tweaking in my client side redis connection pooling confs(mentioned above) to make it work.
Please suggest me the changes i need to make in my confs
I am also curious if I should close the Redis connection after making get/set calls. I have tried finding this but found nothing conclusive on how to close Redis connections
One last thing that I want to ask is that is there any mechanism to get Redis connection pool stats(active connection, idle connection etc ) in Redisson
Edit1:
I have tried by changing values following values in 3 different iterations
Iteration 1:
idleConnectionTimeout: 30000
connectTimeout: 30000
timeout: 30000
Iteration 2:
nettyThreads: 0
Iteration 3:
connectionMinimumIdleSize: 100
connectionPoolSize: 750
I have tried these things but nothing has worked for me
Any help is appreciated.
Thanks in advance
Assuming you are getting low memory alerts on your cache JVM.
You may have to analyze the traffic and determine 2 things
Too many parallel cache persists.
Huge chunk of data being persisted.
Both can be determined by the traffic on your server.
For option 1 configuring pool-size would solve you issue, but for option 2 you may have to refactor your code to persist data in smaller chunks.
Try to set nettyThreads = 64 settings
I have a table where I am inserting records using record.insert() method. I believe this method is doing an insert and then a select but in a different transactions. At the same time I have another thread which pools this table for records processes them and then deletes them.
In some cases I am getting the below exception:
org.jooq.exception.NoDataFoundException: Exactly one row expected for refresh. Record does not exist in database.
at org.jooq.impl.UpdatableRecordImpl.refresh(UpdatableRecordImpl.java:345)
at org.jooq.impl.TableRecordImpl.getReturningIfNeeded(TableRecordImpl.java:232)
at org.jooq.impl.TableRecordImpl.storeInsert0(TableRecordImpl.java:208)
at org.jooq.impl.TableRecordImpl$1.operate(TableRecordImpl.java:169)
My solution was to use DSL.using(configuration()).insertInto instead of record.insert().
My question is shouldn't the insert and fetch be done in the same transaction?
UPDATE:
This is a dropwizard app that is using jooqbundle: com.bendb.dropwizard:dropwizard-jooq.
The configuration is injected in the DAO, the insert is as follows:
R object = // jooq record
object.attach(configuration);
object.insert();
On the second thread I am just selecting some records from this table, processing them and then deleting them
Jooq logs clearly shows that the 2 queries are not run in same transaction:
14:07:09.550 [main] DEBUG org.jooq.tools.LoggerListener - -> with bind values : insert into "queue"
....
14:07:09.083', 1)
14:07:09.589 [main] DEBUG org.jooq.tools.LoggerListener - Affected row(s) : 1
14:07:09.590 [main] DEBUG org.jooq.tools.StopWatch - Query executed : Total: 47.603ms
14:07:09.591 [main] DEBUG org.jooq.tools.StopWatch - Finishing : Total: 48.827ms, +1.223ms
14:07:09.632 [main] DEBUG org.jooq.tools.LoggerListener - Executing query : select "queue"."
I do not see the "autocommit off" or "savepoint" statements in the logs which are generally printed by jooq in case the queries are run in a transaction. I hope this helps, let me know if you need more info
UPDATE 2:
Jooq version is 3.9.1
mysql version 5.6.23
Database and jooq entry yml file:
database:
driverClass: com.mysql.jdbc.Driver
user: ***
password: ***
url: jdbc:mysql://localhost:3306/mySchema
properties:
charSet: UTF-8
characterEncoding: UTF-8
# the maximum amount of time to wait on an empty pool before throwing an exception
maxWaitForConnection: 1s
# the SQL query to run when validating a connection's liveness
validationQuery: "SELECT 1"
# the timeout before a connection validation queries fail
validationQueryTimeout: 3s
# initial number of connections
initialSize: 25
# the minimum number of connections to keep open
minSize: 25
# the maximum number of connections to keep open
maxSize: 25
# whether or not idle connections should be validated
checkConnectionWhileIdle: true
# the amount of time to sleep between runs of the idle connection validation, abandoned cleaner and idle pool resizing
evictionInterval: 10s
# the minimum amount of time an connection must sit idle in the pool before it is eligible for eviction
minIdleTime: 1 minute
jooq:
# The flavor of SQL to generate. If not specified, it will be inferred from the JDBC connection URL. (default: null)
dialect: MYSQL
# Whether to write generated SQL to a logger before execution. (default: no)
logExecutedSql: no
# Whether to include schema names in generated SQL. (default: yes)
renderSchema: yes
# How names should be rendered in generated SQL. One of QUOTED, AS_IS, LOWER, or UPPER. (default: QUOTED)
renderNameStyle: QUOTED
# How keywords should be rendered in generated SQL. One of LOWER, UPPER. (default: UPPER)
renderKeywordStyle: UPPER
# Whether generated SQL should be pretty-printed. (default: no)
renderFormatted: no
# How parameters should be represented. One of INDEXED, NAMED, or INLINE. (default: INDEXED)
paramType: INDEXED
# How statements should be generated; one of PREPARED_STATEMENT or STATIC_STATEMENT. (default: PREPARED_STATEMENT)
statementType: PREPARED_STATEMENT
# Whether internal jOOQ logging should be enabled. (default: no)
executeLogging: no
# Whether optimistic locking should be enabled. (default: no)
executeWithOptimisticLocking: yes
# Whether returned records should be 'attached' to the jOOQ context. (default: yes)
attachRecords: yes
# Whether primary-key fields should be updatable. (default: no)
updatablePrimaryKeys: no
Have included the jooq bundle in the Application class as described in https://github.com/benjamin-bader/droptools/tree/master/dropwizard-jooq.
Using https://github.com/xvik/dropwizard-guicey to inject the configuration into each DAO.
The guide module has the following binding:
bind(Configuration.class).toInstance(jooqBundle.getConfiguration());
I'm using EHCache 2.9 in my Spring Boot application and I've configured the cache to expire after 300 seconds (5 minutes).
When I run the application and request the element for the first time it gets cached and after that never expires.
However, when I do #CachePut it gets updated successfully and updated element is then returned.
What is wrong in my configuration?
Here is my ehcache.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<ehcache>
<defaultCache maxElementsInMemory="500" eternal="false"
overflowToDisk="false" memoryStoreEvictionPolicy="LFU" />
<diskStore path="java.io.tempdir"/>
<cache name="appointments"
maxElementsInMemory="5000"
eternal="false"
timeToIdleSeconds="0"
timeToLiveSeconds="300"
overflowToDisk="false"
memoryStoreEvictionPolicy="LFU" />
</ehcache>
And here is how I request the cache:
#Cacheable("appointments")
public List<Event> getEvents(String eventsForUser, Date startDate, Date endDate) throws Exception {
return fetchEventsFromTheServer(eventsForUser, startDate, endDate);
}
#CachePut("appointments")
public List<Event> refreshEventsCache(String eventsForUser, Date startDate, Date endDate) throws Exception {
return fetchEventsFromTheServer(eventsForUser, startDate, endDate);
}
Any suggestions?
Flush – To move a cache entry to a lower tier. Flushing is used to free up resources while still keeping data in the cluster. Entry E1 is shown to be flushed from the L1 off-heap store to the Terracotta Server Array (TSA).
Fault – To copy a cache entry from a lower tier to a higher tier. Faulting occurs when data is required at a higher tier but is not resident there. The entry is not deleted from the lower tiers after being faulted. Entry E2 is shown to be faulted from the TSA to L1 heap.
Eviction – To remove a cache entry from the cluster. The entry is deleted; it can only be reloaded from a source outside the cluster. Entries are evicted to free up resources. Entry E3, which exists only on the L2 disk, is shown to be evicted from the cluster.
Expiration – A status based on Time To Live and Time To Idle settings. To maintain cache performance, expired entries may not be immediately flushed or evicted. Entry E4 is shown to be expired but still in the L1 heap.
Pinning – To force data to remain in certain tiers. Pinning can be set on individual entries or an entire cache, and must be used with caution to avoid exhausting a resource such as heap. E5 is shown pinned to L1 heap.
http://www.ehcache.org/documentation/2.7/configuration/data-life.html
We're trying to get caching working in our app and are having a very hard time getting it to actually work. We don't have a very complex configuration, at least for this part of the application, so it's really frustrating and I finally have to throw myself on the tender mercies of the StackOverflow community.
First, for now, we're stuck at Spring 3.0 and Hibernate 3.6, so using the new Spring 3.1 #Cacheable annotations and all that other stuff isn't an option for us.
I've put the versions of our stack in the subject and what we have is a very typical configuration for that stack:
Our Hibernate entities are just defined with #Entity, DAOs with #Repository, and services with #Service
Transactional boundaries are set with #Transactional
I've added in caching by adding ehcache-core to the pom.xml and adding the following properties to the Hibernate properties map:
<prop key="hibernate.cache.use_second_level_cache">true</prop>
<prop key="hibernate.cache.use_query_cache">true</prop>
<prop key="hibernate.cache.region.factory_class">net.sf.ehcache.hibernate.SingletonEhCacheRegionFactory</prop>
I then added the #Cache annotation to my entity classes, something like this:
#Entity
#Cache(usage = CacheConcurrencyStrategy.READ_WRITE, region = "config")
public class Configuration extends AbstractHibernateEntity implements Serializable {
And to be honest, I thought I was done. Yay me for configuring caching so easily! And you could see it working, there were lots of messages in the logs as caching came online. That's great!
Only it turns out that caching has come online and nothing is getting cached. I've now been pounding on this for two days and have no clue why my objects are not getting cached. I've tried added <cache name="..."/> into my ehcache.xml, specifying regions on my #Cache annotations, checking the cache statistics, etc. I've added debug logging for both the net.sf.ehcache and org.hibernate.cache packages. I simply don't see what's keeping this from saving objects out to the cache and accessing them there.
After we realized our web application wasn't caching, I went back into the dependent libraries and realized that they weren't caching either (I added caching to the unit tests to make sure that caching didn't break the unit tests; it doesn't but maybe that's just because it's not caching). So I've been working on one of our lower-level dependencies, figuring if I could figure out how to make it work at that level, I could pull that up to the web app level. That's a great theory except that I can't even make it work at the lower level!
In my logs, I'm getting lots of messages. So here's a sample of the debug output from net.sf.ehcache:
14:03:20,795 INFO net.sf.ehcache: 284 - CACHE HITS: 0
14:03:20,796 INFO net.sf.ehcache: 284 - IN-MEM HITS: 0
14:03:20,796 INFO net.sf.ehcache: 284 - CACHE MISS: 0
14:03:20,796 INFO net.sf.ehcache: 284 - IN-MEM MISS: 0
14:03:20,796 INFO net.sf.ehcache: 284 - EVICTIONS: 0
14:03:20,797 INFO net.sf.ehcache: 284 - MEM OBJ CT: 0
14:03:20,886 DEBUG bernate.regions.EhcacheGeneralDataRegion: 211 - key: xhbm_configuration_data value: 5643835231789056
14:03:20,890 DEBUG bernate.regions.EhcacheGeneralDataRegion: 211 - key: xhbm_configuration value: 5643835231805440
14:03:20,891 DEBUG bernate.regions.EhcacheGeneralDataRegion: 211 - key: xhbm_configuration_data value: 5643834986045441
14:03:20,891 DEBUG bernate.regions.EhcacheGeneralDataRegion: 211 - key: xhbm_configuration value: 5643834986045441
14:03:20,897 DEBUG bernate.regions.EhcacheGeneralDataRegion: 211 - key: xhbm_configuration_data value: 5643835231834112
14:03:20,898 DEBUG bernate.regions.EhcacheGeneralDataRegion: 211 - key: xhbm_configuration value: 5643835231838208
14:03:20,898 DEBUG bernate.regions.EhcacheGeneralDataRegion: 211 - key: xhbm_configuration_data value: 5643834986078209
14:03:20,899 DEBUG bernate.regions.EhcacheGeneralDataRegion: 211 - key: xhbm_configuration value: 5643834986078209
And in the org.hibernate.cache log, I get lots of stuff like this:
14:47:26,077 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Pre-invalidating space [xhbm_configuration_data]
14:47:26,081 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Pre-invalidating space [xhbm_configuration]
14:47:26,082 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Invalidating space [xhbm_configuration_data], timestamp: 5643845820751872
14:47:26,082 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Invalidating space [xhbm_configuration], timestamp: 5643845820751872
14:47:26,089 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Pre-invalidating space [xhbm_configuration_data]
14:47:26,091 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Pre-invalidating space [xhbm_configuration]
14:47:26,092 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Invalidating space [xhbm_configuration_data], timestamp: 5643845820792832
14:47:26,092 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Invalidating space [xhbm_configuration], timestamp: 5643845820792832
14:47:26,125 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Pre-invalidating space [xhbm_configuration_data]
14:47:26,130 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Pre-invalidating space [xhbm_configuration]
14:47:26,131 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Invalidating space [xhbm_configuration_data], timestamp: 5643845820952576
14:47:26,132 DEBUG rg.hibernate.cache.UpdateTimestampsCache: 168 - Invalidating space [xhbm_configuration], timestamp: 5643845820952576
I really don't know what else to say about it from here. The full code is available both from my Bitbucket repo (it's Mercurial, not git) at https://bitbucket.org/rherrick/nrg_config or as a zip from Dropbox at https://dl.dropboxusercontent.com/u/42711610/nrg_config.zip. You should just be able to run the unit tests from Maven with:
mvn clean install
If anyone could help me figure out what the hell's going wrong here, I'd be ecstatically happy! And don't you want me to be ecstatically happy? I know I do :)
Seriously, many thanks in advance and many thanks will follow afterwards for any help with this issue.
As far as I understand second-level cache for objects will be used only when you call load get, list etc on Session. I can't see in your code such calls.
Query cache will be used only when you call setCacheable on Query or Criteria object. And again, you never call that method in your code.
Alexander,
You're on the right track, but that was only one factor among a few that caused the problem. There were really three factors going on:
Our code actually is calling load(), get(), list(), etc., but it's mostly wrapped up in our framework code. I went through and templatized a bunch of that code to explicitly force caching there. That started SOME caching.
JPA/Hibernate entities should NEVER initialize ANYTHING! We had a base class that set a couple of flags and timestamps on instantiation. When an object was retrieved, this ended up making the object look dirty and in need of a full run out to the database. This was really the big one for our framework.
Watching out for transactional boundaries when setting transient properties. When you go into and then out of a method marked #Transactional, any changes to your objects will be persisted, regardless of whether you called a session.save() on that object or not, as described here. This was causing objects to constantly fail to match against the cached version (although I'd have thought the modified version of the object would match caching, but that didn't seem to happen).
In the end, this ended up being really difficult to diagnose primarily because it wasn't one single factor, but caused by lots of different things together!
I'm running Tomcat7, the server is quite powerful, 8 GB RAM 8-core.
My problem is that the RES memory is geting higher and higher, until the server just doesn't respond anymore, not even calling OnOutOfMemoryError.
Tomcat configuration :
-Xms1024M
-Xmx2048M
-XX:PermSize=256m
-XX:MaxPermSize=512m
-XX:+UseConcMarkSweepGC
-XX:OnOutOfMemoryError='/var/tomcat/conf/restart_tomcat.sh'
Memory informations :
Memory: Non heap memory = 106 Mb (Perm Gen, Code Cache),
Loaded classes = 14,055,
Garbage collection time = 47,608 ms,
Process cpu time = 4,296,860 ms,
Committed virtual memory = 6,910 Mb,
Free physical memory = 4,906 Mb,
Total physical memory = 8,192 Mb,
Free swap space = 26,079 Mb,
Total swap space = 26,079 Mb
Perm Gen memory: 88 Mb / 512 Mb ++++++++++++
Free disk space: 89,341 Mb
The memory used by Tomcat doesn't look that high compared to the top command.
I also had java.net.SocketException: No buffer space available when trying to connect to SMTP server or when trying to connect to facebook servers.
I use Hibernate, with c3p0 connection pool with this configuration :
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://urldb/schema?autoReconnect=true</property>
<property name="hibernate.connection.username">username</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQL5InnoDBDialect</property>
<property name="hibernate.connection.password"></property>
<property name="connection.characterEncoding">UTF-8</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.idle_test_period">300</property>
<property name="hibernate.c3p0.timeout">5000</property>
<property name="hibernate.c3p0.max_size">50</property>
<property name="hibernate.c3p0.min_size">1</property>
<property name="hibernate.c3p0.max_statement">0</property>
<property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
I couldn't find anything... does someone have an hint of where I should be looking for ?
Thanks!
[UPDATE 1]HEAP DUMP :
HEAP HISTOGRAM :
class [C 269780 34210054
class [B 5600 33836661
class java.util.HashMap$Entry 221872 6212416
class [Ljava.util.HashMap$Entry; 23797 6032056
class java.lang.String 271170 5423400
class org.hibernate.hql.ast.tree.Node 103588 4972224
class net.bull.javamelody.CounterRequest 28809 2996136
class org.hibernate.hql.ast.tree.IdentNode 23461 2205334
class java.lang.Class 14677 2113488
class org.hibernate.hql.ast.tree.DotNode 13045 1852390
class [Ljava.lang.String; 48506 1335600
class [Ljava.lang.Object; 12997 1317016
Instance Counts for All Classes (excluding platform) :
103588 instances of class org.hibernate.hql.ast.tree.Node
33366 instances of class antlr.ANTLRHashString
28809 instances of class net.bull.javamelody.CounterRequest
24436 instances of class org.apache.tomcat.util.buf.ByteChunk
23461 instances of class org.hibernate.hql.ast.tree.IdentNode
22781 instances of class org.apache.tomcat.util.buf.CharChunk
22331 instances of class org.apache.tomcat.util.buf.MessageBytes
13045 instances of class org.hibernate.hql.ast.tree.DotNode
10024 instances of class net.bull.javamelody.JRobin
9084 instances of class org.apache.catalina.loader.ResourceEntry
7931 instances of class org.hibernate.hql.ast.tree.SqlNode
[UPDATE 2] server.xml :
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
URIEncoding="UTF-8"
maxThreads="150"
minSpareThreads="25"
maxSpareThreads="75"
enableLookups="false"
acceptCount="1024"
server="unknown"
address="public_ip"
/>
****[UPDATE 3] Output from log files : ****
2012-06-04 06:18:24,152 [http-bio-ip-8080-exec-3500] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/api].[Jersey REST Service]- Servlet.ser
vice() for servlet [Jersey REST Service] in context with path [/socialapi] threw exception
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563)
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118)
at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:326)
at org.apache.coyote.Request.doRead(Request.java:422)
[UPDATE 4] ServletContext
I use a ServletContextListenerin my application to instanciate controllers and keep a reference with event.getServletContext().setAttribute. Those controllers loads configurations and translations (the 88Mb in Perm).
Then to use the database i use :
SessionFactory sf = dbManager.getSessionFactory(DatabaseManager.DB_KEY_DEFAULT);
Session session = sf.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
//Do stuuf
tx.commit();
} catch (Exception e){
//Do something
} finally {
session.close();
}
Could this be the source of a leak ?
Why not to use Manual transaction/session, and how would you do then ?
Try with this parameter:
+XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpPath=dump.log
Also try with lower start memory parameters -Xms.
then you can inspect the dump to see if the problem was object allocation.
While running try
jps
That will output all java processes, lets say Tomcat is PID 4444:
jmap -dump:format=b,file=heapdump 4444
And
jhat heapdump
If you run out of memory while executing jhat just add more memory. From there you can inspect the heap of your application.
Another way to go is to enable Hibernate statistics to check that you are not retrieving more objects. Although it looks like a full garbage collection every hour should not be a problem (room for do it better there).
-verbose:gc -Xloggc:/opt/tomcat/logs/gc.out -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
And with GCViewer for example take a look at every space of memory (ternured, eden, survivors, perm).
Another handy tool:
jstack 4444 > stack.txt
That will retrieve a full stack trace of every thread running inside the java process with pid 4444.
Bear in mind that you need privileges if you started Tomcat as root or another user.
jps
won't output process which you have no privileges, therefore you cannot connect to it.
Since I don't know what your application is about (and therefore I don't know its requirements) 3 million instances looks like a lot.
With Hibernate statistics you can see which classes you instantiate the most.
Then tunning the proportions of your eden and ternured garbage recolection can be more efficient.
Newly instantiated objects goes to eden. When it fills up a minor gc triggers. What is not deleted goes to a survivor space. When this fills up it goes to ternured. Full gc will arise when ternured is full.
In this picture (which is inaccurate) I left aside String that become interned and Memory mapped files (that are not in heap). Take a look at which classes you instantiate most. Intensive use of String might lead to quickly fill up perm.
I guess you do so, but use a managed session factory, such as Spring (if in your stack) and avoid manually management of transactions and sessions.
Keep in mind that objects are deleted in the GC when no object refers to it. So as long as a object is reachable in your application the object remain.
If your ServletContextListener instantiate controllers and are stored in the event getServletContext. Make sure you completely remove the reference afterwards, if you keep a reference the objects won't be deleted, since they are still reachable.
If you manage your own transactions and session (which is fine if you cannot use a framework) then you must deal with code maintenance and bugs that Spring-tx for instance has solved and improved.
I personally would take advantage of FOSS. But of course sometime you cannot enlarge the stack.
If you are using Hibernate I would take a look at Spring-orm and Spring-tx to manage transactions and session. Also take a look at Hibernate patter Open Session In View.
I'd also recommend that you download Visual VM 1.3.3, install all the plugins, and attach it to the Tomcat PID so you can see what's happening in real time. Why wait for a thread dump? It'll also tell you CPU, threads, all heap generations, which objects consume the most memory, etc.