Application is using Hibernate 3 and needs to cache data while saving to the database.
Is it possible to cache only a few columns of an entity, instead of all?
Is it possible to set expiration time of an entity after a few days? For example, is it possible to set expiration time to 3 days, so that records can be auto-removed from the cache after 3 days?
I have started looking into caching process but not sure if the above two needs can be fulfilled.
No, it is not possible to cache only a subset of columns for an entity in the Hibernate second-level cache, as otherwise Hibernate would need to go to the database to fetch the remaining data when assembling such entity instances anyway, thus defeating the purpose of the cache.
Yes, it is possible to set expiration time for cached data. Hibernate leaves it to up to the L2 cache provider to manage expiration and eviction policy, so you need to configure it there (consult the documentation of the cache provider you use).
More details here and here.
1) YES, Make other fields Transient, so they will not take part in second level caching pr
Related
I am developing a distributed Java application that needs to check a list of blacklist userids on each request.
If request fails on some eligibility rules, system should add userid ( a parameter of request ) to blacklist.
I am trying to find a proper caching solution for blacklist implementation. My requirements are;
querying blacklist should be very fast
blacklist persistence technology should be scalable
all blacklist data should be persisted on a RDBMS also for fail over / reloading purposes.
They are two possible solutions;
Option 1: I can use redis for storing blacklist data. Whenever a request fails on eligibility rules I can add userid to redis cache easly.
- advantages: extremely fast query, easy to implement
- disadvantages: trusting on redis persistency although it works, it is a cache solution by design not a persistency layer.
Option 2: I can use redis for storing blacklist data meanwhile I can maintain db tables on RDBMS for blacklist. Whenever a request fails on eligibility rules I can add userid to redis cache and rdbms table together.
- advantages: extremely fast query, ability(possibility) to reload redis cache from db
- disadvantages: there is a consistency issue between redis and db table.
Option 3: I can use hazelcast as hibernate L2 cache and when I add any user id to blacklist it is both added to cache and db.
I have questions about option 3
Does hazelcast L2 cache is suitable for preserving such a list of blacklisted users?
Does hibernate manages consistency issue between cache and db?
When application restarted, how L2 cache is reloaded?
and a last question
- Do you have any other suggestion for such a use-case?
Edit:
There will be 100m records in blacklist and I have a couple smilar blacklist.
my read performance is important. I need to query existence of a key within blacklist ~100ms
Ygok,
Still waiting for clarification on the query requirements but I can assume it a lookup by key (since you mention Redis and Redis doesn't have a query language. Hazelcast does have Distributed Query / Predicate API).
Lookup by key is an extremely fast operation with Hazelcast.
In option 2 you need to maintain data consistency between your RDBMS and Redis cache. Using Hazelcast MapLoader / MapStore you can implement write-through- / read-through- cache concepts. All you need to do is put the entry to the cache, and Hazelcast persists it immediately or with configured delay (with batching) to the RDBMS.
In terms of performance, please, feel free to make yourself familiar with recent Hazelcast / Redis benchmark.
Let me know if you have any questions.
I had similar question before, first of all, how much data do you want to store and spend how much memory? how fast query per second do you need? what the data structure like, only userId as a key?
Hazelcast query not very fast on my testing(you can do it for yourself), but it can store large memory data. Hazelcast using Java
default serialize, it cost a lot of memory and IO.
Hazelcast provide hibernate L2 cache, cache data store on
Hazelcast(only query cache), so restart your application not affect
the cache.
Redis provide memory data persistence(DUMP and AOF), maybe a
bit of data will be lost when server crashed, but it very fast.
If you want to not lose any data, store on multi MySQL
server(split data by userId to different server, but you should
consider the problems when add new server), at the same time, you can
add local cache (e.g. Ehcache or google CacheBuilder) and set a
expire time, it can be promote performance.
It's possible to maintain consistency between Redis cache and RDBMS using Redisson framework. It provides write-through and read-through strategies for Map object using MapWriter and MapLoader objects which are required to use in your case.
Please read this documentation section
I have an app to retrieve data from Database, and I monitor the time my app takes to retrieve data.
But I have an issue when I use the same data input set to retrieve data with my app, the second time retrieving will take much less time.
I assume Java or Hibernate has some cache or temp file to save the data, so second time run will be fast, but I don't want it happen. I need monitor the time it actually takes, not the time retrieve from cache or temp file.
I tried to forbid the cache and temp file generate in Java control Panel, I tried to disable the hibernate cache(first level or second level). But these are still not solve my problem. The second time run still takes less time than it should take.
Any idea the reason caused the second time run faster? it just a simple app to retrieve data from DB
The Hibernate 1st level cache can not be disabled (see How to disable hibernate caching). You need to understand Hibernate's session cache if you want to force Hibernate querying to the database.
Lokesh Gupta has a good tutorial on http://howtodoinjava.com/2013/07/01/understanding-hibernate-first-level-cache-with-example/
First level cache is associated with “session” object and other
session objects in application can not see it.
The scope of cache objects is of session. Once session is closed,
cached objects are gone forever.
First level cache is enabled by default and you can not disable it.
When we query an entity first time, it is retrieved from database
and stored in first level cache associated with hibernate session.
If we query same object again with same session object, it will be
loaded from cache and no sql query will be executed.
The loaded entity can be removed from session using evict() method.
The next loading of this entity will again make a database call if
it has been removed using evict() method.
The whole session cache can be removed using clear() method. It will
remove all the entities stored in cache.
You should therefor either use the evict() or clear() method to force a query to the database.
In order to verify this, you can turn on SQL output using the hibernate.show_sql configuration property (see https://docs.jboss.org/hibernate/orm/5.0/manual/en-US/html/ch03.html#configuration-optional).
Have you tried disabling the cache in the database itself?
I believe that Hibernate first and second level caches are Hibernate specific, but the database will still cache under the hood.
MySQL - force not to use cache for testing speed of query
Instead of explaining my question, I will try to give an example:
Let's say I have a User entity and a Item entity. User entity has one-to-one relation to Item.
Let's say that at some point my server updates The table using a sql-update query.
and my question is: Next time I do something like:
Item item = user.getItem();
How can I make sure that the data is up-to-date ? and not the old data that was initially read from DB when User instance was first queried?
Hope my question is clear...
Thank you!
You can be certain about updated entities by flushing entity manager after the DML commands and then query the object again.
Regards
Himanshu
If you do it in the new session, then it will be up to date (if you don't use L2 cache for the entities in question).
If you use L2 cache, then it will not be up to date (if the data is updated in the database without Hibernate being aware of it). In this case, if it is ok for your use cases to use stale data for a specific time interval, you can configure expiration policy for User and Item entities, so that their lifespan in the cache is limited. After they expire from the cache, updated data will be fetched from the database.
If you can properly invalidate the affected second-level cache entries upon changing the data in the background, then you can avoid using the stale data entirely (or reduce the possible time interval in which they will be used as stale).
If you do it in the existing session instance, then both Item and User instances will be in the first-level cache, so you will always get the data that were initially fetched. This is almost always the desired behavior. However, you can manually evict an entity instance (session.evict(item); session.evict(user)) or clear the entire session (session.clear()) to evict all the instances from the current session and then re-read them again from the database.
We have an entire table of entities that we need to load during a hibernate session and the only way I know to load all entities is through an HQL query:
public <T> List<T> getAllEntities(final Class<T> entityClass) {
if (null == entityClass)
throw new IllegalArgumentException("entityClass can't be null");
List<T> list = castResultList(createQuery(
"select e from " + entityClass.getSimpleName() + " e ").list());
return list;
}
We use EHcache for 2nd level caching.
The problem is this gets called 100's of times in a given transaction session and takes up a considerable portion of the total time. Is there any way to load all entities of a given type (load an entire table) and still benefit from 1st level session cache or 2nd level ehcache.
We've been told to stay away from query caching because of their potential performance penalties relative to their gains.
* Hibernate Query Cache considered harmful
Although we're doing performance profiling right now so it might be time to try turning on query cache.
L1 and L2 cache can't help you much with the problem of "get an entire table."
The L1 cache is ill-equipped because if someone else inserted something, it's not there. (You may "know" that no one else would ever do so within the business rules of the system, but the Hibernate Session doesn't.) Hence you have to go look in the DB to be sure.
With the L2 cache, things may have been expired or flushed since the last time anybody put the table in there. This can be at the mercy of the cache provider or even done totally externally, maybe through a MBean. So Hibernate can't really know at any given time if what's in the cache for that type represents the entire contents of the table. Again, you have to look in the DB to be sure.
Since you have special knowledge about this Entity (new ones are never created) that there isn't a practical way to impart on the L1 or L2 caches, you need to either use the tool provided by Hibernate for when you have special business-rules-level knowledge about a result set, query cache, or cache the info yourself.
--
If you really really want it in the L2 cache, you could in theory make all entities in the table members of a collection on some other bogus entity, then enable caching the collection and manage it secretly in the DAO. I don't think it could possibly be worth having that kind of bizarreness in your code though :)
Query cache is considered harmful if and only if the underlying table changes often. In your case the table is changed once a day. So the query would stay in cache for 24 hours. Trust me: use the query cache for it. It is a perfect use case for a query cache.
Example of harmful query cache: if you have a user table and you use the query cache for "from User where username = ..." then this query will evict from cache each time the user table is modified (another user changes/deletes his account). So ANY modification of this table triggers cache eviction. The only way to improve this situation is querying by natural-id, but this is another story.
If you know your table will be modified only once a day as in your case, the query cache will only evict once a day!
But pay attention on your logic when modifying the table. If you do it via hibernate everything is fine. If you use a direct query you have to tell hibernate that you have modified the table (something like query.addSynchronizedEntity(..)). If you do it via shell script you need to adjust the time-to-live of the underlying cache region.
Your answer is by the way reimplementing the query cache as the query cache just caches the list of ids. The actual objects are looked up in L1/L2 cache. so you still need to cache the entities when you use the query cache.
Please mark this as the correct answer for further reference.
We ended up solving this by storing in memory the primary keys to all the entities in the table we needed to load (because they're template data and no new templates are added/removed).
Then we could use this list of primary keys to look up each entity and utilize Hibernates 1st and 2nd level cache.
My Java web application (tomcat) gets all of its data from an SQL database. However, large parts of this database are only updated once a day via a batchjob. Since queries on these tables tend do be rather slow, I want to cache the results.
Before rolling my own solution, I wanted to check out existing cache solutions for java. Obviously, I searched stackoverflow and found references and recommendations for ehcache.
But looking through the documentation it seems it only allows for setting the lifetime of cached objects as a duration (e.g. expire 1 hour after added), while I want an expiry based on a fixed time (e.g. expire at 0h30 am).
Does anyone know a cache library that allows such expiry behaviour? Or how to do this with ehcache if that's possible?
EhCache allows you programmatically set the expiry duration on an individual cache element when you create it. The values configured in ehcache.xml are just defaults.
If you know the specific absolute time that the element should expire, then you can calculate the difference in seconds between then and "now" (i.e. the time you add to the cache), and set that as the time-to-live duration, using Element.setTimeToLive()
Do you need a full blown cache solution? You use standard Maps and then have a job scheduled to clear them at the required time.