TLDR: Is there a way to periodically refresh keys in Memcached so that those keys are always hot (to minimize cache misses).
Longer version:
I'm using Memcached as a distributed cache. I'm storing big values in the cache (~1MB size of each entry).
My application needs consistent read latencies, but the access pattern of the keys vary greatly. Some keys are accessed every second, while others sometimes aren't accessed for hours.
I want my application to have no cache misses and want all the cache entries to be always hot. Since my access pattern isn't consistent, I'm wondering if there's a way to periodically refresh the cache? Any ideas about how this can be achieved?
PS: In a standard setup, the application would put the data in Memcached and then Memcached might evict it after the TTL. It can sometimes evict it sooner if its slab class is full and it can't assign any free pages. However, for this discussion, I'm assuming that TTL is the only mechanism for eviction. And I want to refresh the cache entry and update it's TTL before it can be evicted.
Related
One downside to distributed caching is that every cache query (hit or miss) is a network request which will obviously never be as fast as a local in memory cache. Often this is a worthy tradeoff to avoid cache duplication, data inconsistencies, and cache size constraints. In my particular case, it's only data inconsistency that I'm concerned about. The size of the cached data is fairly small and the number of application servers is small enough that the additional load on the database to populate the duplicated caches wouldn't be a big deal. I'd really like to have the speed (and lower complexity) of a local cache, but my data set does get updated by the same application around 50 times per day. That means that each of these application servers would need to know to invalidate their local caches when these writes occurred.
A simple approach would be to have a database table with a column for a cache key and a timestamp for when the data was last updated. The application could query this table to determine if it needs to expire it's local cache. Yes, this is a network request as well, but it would be much faster than transporting the entire cached data set over the network. Before I go and build something custom, is there an existing caching product for Java/Spring that can be configured to work this way? Is there a gotcha I'm not thinking about? Note that this isn't data that has to be transactionally consistent. If the application servers were out of sync by a few seconds, it wouldn't be a problem.
I don't know of any implementation that queries the database in the way you specify. What does exist are solutions where changes in local caches are distributed among the members in a group. JBossCache is an example where you also have the option to only distribute invalidation of objects. This might be the closest to what you are after.
https://access.redhat.com/documentation/en-us/jboss_enterprise_application_platform/4.3/html/cache_frequently_asked_questions/tree_cache#a19
JBossCache is not a spring component as such, but you create and use a cache as a spring bean should not be a problem.
Let say i have a array of memcache server, the memcache client will make sure the the cache entry is only on a single memcache server and all client will always ask that server for the cache entry... right ?
Now Consider two scenarios:
[1] web-server's are getting lots of different request(different urls) then the cache entry will be distributed among the memcache server and request will fan out to memcache cluster.
In this case the memcache strategy to keep single cache entry on a single server works.
[2] web-server's are getting lots of request for the same resource then all request from the web-server will land on a single memcache server which is not desired.
What i am looking for is the distributed cache in which:
[1] Each web-server can specify which cache node to use to cache stuff.
[2] If any web-server invalidate a cache then the cache server should invalidate it from all caching nodes.
Can memcache fulfill this usecase ?
PS: I dont have ton of resouces to cache , but i have small number of resource with a lots of traffic asking for a single resource at once.
Memcache is a great distributed cache. To understand where the value is stored, it's a good idea to think of the memcache cluster as a hashmap, with each memcached process being precisely one pigeon hole in the hashmap (of course each memcached is also an 'inner' hashmap, but that's not important for this point). For example, the memcache client determines the memcache node using this pseudocode:
index = hash(key) mod len(servers)
value = servers[index].get(key)
This is how the client can always find the correct server. It also highlights how important the hash function is, and how keys are generated - a bad hash function might not uniformly distribute keys over the different servers…. The default hash function should work well in almost any practical situation, though.
Now you bring up in issue [2] the condition where the requests for resources are non-random, specifically favouring one or a few servers. If this is the case, it is true that the respective nodes are probably going to get a lot more requests, but this is relative. In my experience, memcache will be able to handle a vastly higher number of requests per second than your web server. It easily handles 100's of thousands of requests per second on old hardware. So, unless you have 10-100x more web servers than memcache servers, you are unlikely to have issues. Even then, you could probably resolve the issue by upgrading the individual nodes to have more CPUs or more powerful CPUs.
But let us assume the worst case - you can still achieve this with memcache by:
Install each memcache as a single server (i.e. not as a distributed cache)
In your web server, you are now responsible for managing the connections to each of these servers
You are also responsible for determining which memcached process to pass each key/value to, achieving goal 1
If a web server detects a cache invalidation, it should loop over the servers invalidating the cache on each, thereby achieving goal 2
I personally have reservations about this - you are, by specification, disabling the distributed aspect of your cache, and the distribution is a key feature and benefit of the service. Also, your application code would start to need to know about the individual cache servers to be able to treat each differently which is undesirable architecturally and introduces a large number of new configuration points.
The idea of any distributed cache is to remove the ownership of the location(*) from the client. Because of this, distributed caches and DB do not allow the client to specify the server where the data is written.
In summary, unless your system is expecting 100,000k or more requests per second, it's doubtful that you will this specific problem in practice. If you do, scale the hardware. If that doesn't work, then you're going to be writing your own distribution logic, duplication, flushing and management layer over memcache. And I'd only do that if really, really necessary. There's an old saying in software development:
There are only two hard things in Computer Science: cache invalidation
and naming things.
--Phil Karlton
(*) Some distributed caches duplicate entries to improve performance and (additionally) resilience if a server fails, so data may be on multiple servers at the same time
The application I'm developing uses simple HashMaps as cache for certain objects that come from the DB. It's far from ideal, but the amount of data for these chached lists is really small (less than 100) and does not change often. This solution provides minimal overhead. When an item in one of these cached lists changes, its value is replaced in the HashMap.
We're nearing the launch date on production for this application. To provide a reasonably scalable solution, we've come with a load-balancing solution. The balancer switches between several Wildfly-nodes, which each hold the entire application, except for the DB.
The issue now is that when an cached item changes, it's only updated in one of the nodes. The change is not applied to the cache in other nodes. Possible solutions are:
Disable the caching. Not an option.
Use a cache server like Ehcache Server. In this way there would be one cache for all nodes. The problem however would be too much overhead due to REST calls.
A additional web service in every node. This web service would keep track of all load-balanced nodes. When a cached value changes in a node, the node would signal other nodes to evict their caches.
An off-the-shelf solution like Ehcache with signalling features. Does this exist?
My question is: Are there products that offer the last solution (free and with open license, commercially usable)? If not, I would implement the third solution. Are there any risks/mistakes I would have to look out for?
Risks/mistakes: Of course one major thing is data consistency. When caching data from a database I'll usually make sure I make use of transactions when updating. Usually I use a pattern like this:
begin transaction
invalidate cache entries in the transaction
update database
commit transaction
In case of a cache miss during the update happens the read needs to wait until the transaction is committed.
For your use case the typical choice is a clustered or distributed cache, like: HazelCast, Infinispan, Apache Ignite. However, somehow this seams really to heavy in your use case.
An alternative is to implement an own mechanism to publish invalidation events to all nodes. Still this is no easy task, since you may want to make sure that every node received the message, but also be fault tolerant if one nodes goes down at the same time. So you probably want to use a proper library for that, e.g. JGroups or the various MQ products.
I implemented it without JGroups or other signaling libraries. Each node has a REST endpoint to evict the cache. When a node starts up, it registers itself in a DB table with its IP, domain and a token. When it shuts down it removes its record.
When an object is updated in a node, the node evicts its cache and starts several threads that send a REST call (with its token and an object type) to all other nodes using Unirest, which in turn check the token and evict their caches. When an error is thrown, the called node is removed from the list.
It should be improved in terms of security and fault tolerance. The removal of nodes is really pessimistic now. Only after several failed attempts the node should be removed. For now, this simple solution does the job!
I need to keep some values in memory, sort of in-memory db. In terms of reliability, I am not affraid of system failure, I can live with that. However, I can not use memcache service, because the values can be evicted anytime. I need the values to be available on other machines, when application scales. I suppose that appengine will not make memory scale or will it (e.g. if I keep value in an ordinary Java collection)?
What I am trying to achieve here is a "pick a nickname" service. This works in two steps. First, user reserves a nickname. Then he registers the nickname. Nicknames are stored under one entity group (sic!). Therefore I need to avoid datastore contention.
As far as I understand from the https://developers.google.com/appengine/articles/scaling/memcache I can to a certain extent rely on that values in memcache should not be evicted on arbitrary resons. However, I have to count on that this will happen from time to time (e.g. on high memory levels). And this losses of value are very unpleasant to my users.
Your application shares a single instance of Memcache, it is not local to a "machine" (or rather instance of your application).
So if you are running 2 instances and they both retrieve the same value from memcache they will both get the same value.
Running an "in memory" database is not feasible in the cloud - what memory is it you were planning to use, the memory in the instance that's about to shut down?
https://developers.google.com/appengine/articles/scaling/memcache
When designing your application, take the time to consider which datasets can be cached for future reuse. These could be commonly viewed pages or often read datastore entities, just to name a few. There may also be some data in your application which you would like to have shared among all instances of your app but does not need to be persisted forever. In such cases, memcache can improve the scalability of your app by providing a fast and efficient distributed storage system for transient data. Adding memcache logic to your server side code is often well worth the few extra lines of code.
You can use app engine NDB, when you use Python27. NDb is a datastore with auto caching and much more.
Other Machines ? You mean shared between instances of the same app.
Why would one want to use an out of the box caching product like ehcache or memcached ?
Wont a simple hashmap do ? I understand this is a naive question but I would like to see some answers about when a simple hashmap will suffice and a thirdparty caching solution is overkill.
Some things Ehcache can give you, that you would have to manage yourself with a HashMap.
An eviction policy. If your data never grows, then no need to worry. But if you want to prevent a memory leak eventually breaking your app, then you need an eviction policy. With ehcache, you can configure the time to live, and time to idle of elements in your cache.
Clustered caching with Terracotta. If you have more than one tomcat for failover / scalability, then you can link Ehcache up to a Terracotta cluster, so that all instances can see the same data if needed.
Transparent disk overflow - be this on the tomcat server, or the terracotta cluster. When data doesn't fit into heap.
Off heap storage. New technologies such as BigMemory mean you have access to a much larger in-memory cache without GC overheads.
Concurrency. Ehcache can use a ConcurrentDistributedMap to give the optimal performance in a clustered configuration.
This is just the tip of the iceberg.
as Tom mentioned, requirements say everything. If all you need is a place to put in your data using key-value pairs, a hashmap will do.
But if you need overflow capabilities (writing to disk when the map is "full"), entry expiration (remove when an entry has not been "touched" in a while), clustered caches, redundant caches, you fall back on the don't reinvent the wheel paradigm, and use the third-party caching solution.
I've been using ehcache for almost 3 years now. I use just a slice of the total feature set, but the ones I do, work great.