I've generally implemented sequence number generation using database sequences in the past.
e.g. Using Postgres SERIAL type http://www.neilconway.org/docs/sequences/
I'm curious though as how to generate sequence numbers for large distributed systems where there is no database. Does anybody have any experience or suggestions of a best practice for achieving sequence number generation in a thread safe manner for multiple clients?
OK, this is a very old question, which I'm first seeing now.
You'll need to differentiate between sequence numbers and unique IDs that are (optionally) loosely sortable by a specific criteria (typically generation time). True sequence numbers imply knowledge of what all other workers have done, and as such require shared state. There is no easy way of doing this in a distributed, high-scale manner. You could look into things like network broadcasts, windowed ranges for each worker, and distributed hash tables for unique worker IDs, but it's a lot of work.
Unique IDs are another matter, there are several good ways of generating unique IDs in a decentralized manner:
a) You could use Twitter's Snowflake ID network service. Snowflake is a:
Networked service, i.e. you make a network call to get a unique ID;
which produces 64 bit unique IDs that are ordered by generation time;
and the service is highly scalable and (potentially) highly available; each instance can generate many thousand IDs per second, and you can run multiple instances on your LAN/WAN;
written in Scala, runs on the JVM.
b) You could generate the unique IDs on the clients themselves, using an approach derived from how UUIDs and Snowflake's IDs are made. There are multiple options, but something along the lines of:
The most significant 40 or so bits: A timestamp; the generation time of the ID. (We're using the most significant bits for the timestamp to make IDs sort-able by generation time.)
The next 14 or so bits: A per-generator counter, which each generator increments by one for each new ID generated. This ensures that IDs generated at the same moment (same timestamps) do not overlap.
The last 10 or so bits: A unique value for each generator. Using this, we don't need to do any synchronization between generators (which is extremely hard), as all generators produce non-overlapping IDs because of this value.
c) You could generate the IDs on the clients, using just a timestamp and random value. This avoids the need to know all generators, and assign each generator a unique value. On the flip side, such IDs are not guaranteed to be globally unique, they're only very highly likely to be unique. (To collide, one or more generators would have to create the same random value at the exact same time.) Something along the lines of:
The most significant 32 bits: Timestamp, the generation time of the ID.
The least significant 32 bits: 32-bits of randomness, generated anew for each ID.
d) The easy way out, use UUIDs / GUIDs.
You could have each node have a unique ID (which you may have anyway) and then prepend that to the sequence number.
For example, node 1 generates sequence 001-00001 001-00002 001-00003 etc. and node 5 generates 005-00001 005-00002
Unique :-)
Alternately if you want some sort of a centralized system, you could consider having your sequence server give out in blocks. This reduces the overhead significantly. For example, instead of requesting a new ID from the central server for each ID that must be assigned, you request IDs in blocks of 10,000 from the central server and then only have to do another network request when you run out.
Now there are more options.
Though this question is "old", I got here, so I think it might be useful to leave the options I know of (so far):
You could try Hazelcast. In it's 1.9 release it includes a Distributed implementation of java.util.concurrent.AtomicLong
You can also use Zookeeper. It provides methods for creating sequence nodes (appended to znode names, though I prefer using version numbers of the nodes). Be careful with this one though: if you don't want missed numbers in your sequence, it may not be what you want.
Cheers
It can be done with Redisson. It implements distributed and scalable version of AtomicLong. Here is example:
Config config = new Config();
config.addAddress("some.server.com:8291");
Redisson redisson = Redisson.create(config);
RAtomicLong atomicLong = redisson.getAtomicLong("anyAtomicLong");
atomicLong.incrementAndGet();
If it really has to be globally sequential, and not simply unique, then I would consider creating a single, simple service for dispensing these numbers.
Distributed systems rely on lots of little services interacting, and for this simple kind of task, do you really need or would you really benefit from some other complex, distributed solution?
There are a few strategies; but none that i know can be really distributed and give a real sequence.
have a central number generator. it doesn't have to be a big database. memcached has a fast atomic counter, in the vast majority of cases it's fast enough for your entire cluster.
separate an integer range for each node (like Steven Schlanskter's answer)
use random numbers or UUIDs
use some piece of data, together with the node's ID, and hash it all (or hmac it)
personally, i'd lean to UUIDs, or memcached if i want to have a mostly-contiguous space.
Why not use a (thread safe) UUID generator?
I should probably expand on this.
UUIDs are guaranteed to be globally unique (if you avoid the ones based on random numbers, where the uniqueness is just highly probable).
Your "distributed" requirement is met, regardless of how many UUID generators you use, by the global uniqueness of each UUID.
Your "thread safe" requirement can be met by choosing "thread safe" UUID generators.
Your "sequence number" requirement is assumed to be met by the guaranteed global uniqueness of each UUID.
Note that many database sequence number implementations (e.g. Oracle) do not guarantee either monotonically increasing, or (even) increasing sequence numbers (on a per "connection" basis). This is because a consecutive batch of sequence numbers gets allocated in "cached" blocks on a per connection basis. This guarantees global uniqueness and maintains adequate speed. But the sequence numbers actually allocated (over time) can be jumbled when there are being allocated by multiple connections!
Distributed ID generation can be archived with Redis and Lua. The implementation available in Github. It produces a distributed and k-sortable unique ids.
I know this is an old question but we were also facing the same need and was unable to find the solution that fulfills our need.
Our requirement was to get a unique sequence (0,1,2,3...n) of ids and hence snowflake did not help.
We created our own system to generate the ids using Redis. Redis is single threaded hence its list/queue mechanism would always give us 1 pop at a time.
What we do is, We create a buffer of ids, Initially, the queue will have 0 to 20 ids that are ready to be dispatched when requested. Multiple clients can request an id and redis will pop 1 id at a time, After every pop from left, we insert BUFFER + currentId to the right, Which keeps the buffer list going. Implementation here
I have written a simple service which can generate semi-unique non-sequential 64 bit long numbers. It can be deployed on multiple machines for redundancy and scalability. It use ZeroMQ for messaging. For more information on how it works look at github page: zUID
Using a database you can reach 1.000+ increments per second with a single core. It is pretty easy. You can use its own database as backend to generate that number (as it should be its own aggregate, in DDD terms).
I had what seems a similar problem. I had several partitions and I wanted to get an offset counter for each one. I implemented something like this:
CREATE DATABASE example;
USE example;
CREATE TABLE offsets (partition INTEGER, offset LONG, PRIMARY KEY (partition));
INSERT offsets VALUES (1,0);
Then executed the following statement:
SELECT #offset := offset from offsets WHERE partition=1 FOR UPDATE;
UPDATE offsets set offset=#offset+1 WHERE partition=1;
If your application allows you, you can allocate a block at once (that was my case).
SELECT #offset := offset from offsets WHERE partition=1 FOR UPDATE;
UPDATE offsets set offset=#offset+100 WHERE partition=1;
If you need further throughput an cannot allocate offsets in advance you can implement your own service using Flink for real time processing. I was able to get around 100K increments per partition.
Hope it helps!
The problem is similar to:
In iscsi world, where each luns/volumes have to be uniquely identifiable by the initiators running on the client side.
The iscsi standard says that the first few bits have to represent the Storage provider/manufacturer information, and the rest monotonically increasing.
Similarly, one can use the initial bits in the distributed system of nodes to represent the nodeID and the rest can be monotonically increasing.
One solution that is decent is to use a long time based generation.
It can be done with the backing of a distributed database.
My two cents for gcloud. Using storage file.
Implemented as cloud function, can easily be converted to a library.
https://github.com/zaky/sequential-counter
Related
We have an app server, which processes a large volume of incoming objects.
One of its functions, is to group those objects into groups, based on bespoke collection of grouping keys that depend on object type.
E.g, there is a grouping rule table that says:
Object type 1: grouping keys are col1, col2, col4, col5
Object type 2: grouping keys are col2, col3 ...
Originally, we had a singleton server, and the problem was solved by having an in-memory index, mapping an object type + grouping key string, to a group ID. Then, we had a synchronized code that would check if the index contained an entry for grouping keys of a given object. If so, the object got the group ID from the cache, otherwise, we assigned group ID of the object to be its object ID - and stored it in the cache.
This worked well... until the server was re-designed from a singleton, to several distributed server instances using Ignite cache to store the data (including the grouping cache).
Due to inherent slowness of the Ignite solution, a race condition was introduced, since the synchronization mechanism used in a singleton to prevent them could not sustain the slowness of Ignite (transactions are too slow).
What can be done to solve this problem in a distributed situation, avoiding either race conditions (which produce different group IDs for objects that should be in same group), OR even worse, false positive grouping (e.g. grouping 2 objects that should be in different groups)?
Constraints:
Pure hashing function cannot be used, due to a risk of hash key collisions. The grouping may not have false positives, ever (e.g. assigning same group ID to objects that should not be groupd together). Imagine that this could lead to loss of PII, or other high risk - so no matter how good the hashing function is and how rare collisions are, they are still unacceptable.
Solution must be realtime, since grouping data is used in other functionalities of the server within seconds or possibly fractions of a second of processing an object. So if there is some post-processing sweep to re-group things "correctly" is introduced with a latency of 30 seconds, that risks 30 seconds of group level updates being done to incorrect group membership.
Maintaining individual lists of group IDs synchronized between server instances via messaging system is not acceptable due to high volume of data (e.g. 5 servers * 1million objects would mean sending 4 million group ID updates). That was the whole point of having an Ignite cache.
Technical constraints: Java server instances, running on distinct Linux servers. They use a homegrown MQ like messaging system to talk to each other in general, and Ignite cache cluster to store shared data instead of local in-memory cache (which is the source of the problem).
The performance of Ignite can't cause a race condition. It doesn't matter whether an update takes a microsecond or a minute, a race condition is a synchronisation issue.
In any case, reading and writing a bunch of records in one unit says "transaction." Ignite supports distributed transactions.
try (Transaction tx = ignite.transactions().txStart()) {
cache1.get(...);
cache2.put();
cache2.put();
tx.commit();
}
catch (...) {
If you can't use transactions you need to either "manually" have locks (which is probably going to be slower) or have your Group ID be predictable. For example, your key for group one could be a concatenation of columns 1, 2, 4 and 5.
But really this is a data modelling question that may not be a good fit for Stack Overflow.
We are developing an application in which entity ids for tables must be in incremental order starting from 1 to so on, for each namespace.
We came across allocateIdRange, allocateIds methods in DatastoreService interface but these ids must be assigned manually and will not be assigned by DatastoreService itself. Assigning ids manually may leads to synchronization problems with multiple instances.
Can anyone provide me suggestions to overcome this problem?
We are using objectify 3.0 for DatastoreService operations.
I agree with Tim Hoffman and tx802 when they say you should reconsider your design regarding sequential ids. However a while ago i had to implement something very similar because the customer forced us to use sequential and uninterrupted numbers for order numbers (for unclear reasons). Regardless we complied with the customers wishes by using sharding counters(link contains full code sample) for the order numbers. Sharding counters work like this:
You create a couple of entities of the same kind in your datastore which are just counter values
The actual value is calculated by querying all entities of that kind and summarizing their values
When you wish to increase the value, one of the entities is randomly chosen and incremented
The current counter value may be cached in memcache for improved performance
Why does this work:
As you may know you have a restriction/limitation of 1 transaction per second and entity group in the datastore. Therefor you shard the counter into multiple entities and avoid this limitation. The more traffic you expect, the more shards you're going to need. Luckily you can increase the count of shards at any time.
We also know that writes are slow in comparison to reads. Therefor building the sum of all shards is a fast operation while increasing a single shard value (write) is slow, which doesn't bother us when using sharding counters because we have sufficient time.
Summarized:
You can use sharding counters for sequential ids. If you can avoid the whole sequential id dilemma it would be a better solution though.
We are working on a layered Java application that talks directly to Gemfire.
We need to be able to generate unique "long" sequence numbers, guaranteed unique across all nodes of the application. (Not all nodes are clustered)
Normally I would create a sequence in Oracle, but in this case, even though our Gemfire configuration has a connection to the relational database for write behind persistence, our application has no other knowledge of the database.
What would be the best way to generate those guaranteed unique long values, without going to the database?
The first question to ask yourself is do you really need a long sequence number (monotonically increasing long integer) or do you just need a globally unique identifier (like a UUID).
The most performant solution is going to be a globally unique id and I would just suggest using a GUID.
If you need a globally unique monotonically increasing long value (long sequence) then you will have to use some distributed locking and increment a value in the region. The method for this and performance depends on the type of region you are using.
Look at Region.replace(K, V, V). It can perform globally atomic updates to values under specific region definitions. You may need to consider a region that just has your sequences if your current region type is not sufficiently defined.
I am writing a software where my code runs on two different machines. Will my GUID be still unique across the whole cluster if I have generation logic on multiple JVM.what are chances of collision in my specific use case ?
If your "GUID" is a UUID.randomUUID() then the probabilities are quite low. Otherwise, it depends on how you are generating your GUID, but the general principle
behind them is that you have enough random bits so that a collision will be unlikely.
Just in case all instances of your distributed system use a common database, then you could create a sequence on that database and use values from that sequence to avoid duplicate IDs.
GLOBALLY unique ID, in this case global means universal.
Every GUID generated is guaranteed to be unique in the universe. That's the whole purpose of GUIDs.
I want generate unique ID just like auto increment in java . So previously i used current nano seconds but i end up with clash since two data comes with in same nano seconds ..
Does UUID solves the above problem ?
Note :: In my project i can even get 10000 rows of records for each and every minute and I will dump those records along with UIDS in to table .And there may be a situation where i would stop my product and restart it after some time ....So during that situation how could UUID class clarifies the previously generated Uids(which i stored in DB) with the new one going to created(Yet to be dumped in DB) ?
While the UUIDs are not guaranteed to be unique, the probability of a duplicate is extremely low. See Random UUID probability of duplicates.
For your application, it makes sense to use the UUID, but you may want to deal with the extremely rare condition, just in case.
I seriously doubt you get two records in the same nano-second as making the call System.nanoTime() takes over 100 ns. It is more likely your clock doesn't have nano second accuracy.
However, if you restart your server, you can get repeating nanoTime().
One way around this is to use
AtomicLong counter = new AtomicLong(System.currentTimeMillis()*1000);
long id = counter.incrementAndGet();
// something like ctz9yamgu8
String id = Long.toString(counter.incrementAndGet(), 36);
This will start a counter when the application restarts and they will not be overlap between restarts unless you sustain over one million ids per second. (Over the life of the instance)
Note: this only works for on a per instance basis. Multiple servers need to use a different approach.
There seems to be some confusion on this page about the nature of UUID.
Study the Wikipedia page. You will see there are different versions of UUID.
You asked:
Does UUID solves the above problem ?
Yes, UUID values do solve your problem.
A point in space and time
The original Version 1 represents a point in space and time, never to be repeated.
Version 1 does this by using the MAC address of the machine on which it is generated (a point in space). To this it combines the current moment. Add in an arbitrary number that increments when a change in the computer clock is noticed. The clock is not as much of an issue now that computers have built-in batteries and network connections to time servers. By combining these, there is no practical chance of collisions.
Because of concerns over the security and privacy issues involved in tracking and divulging the MAC address and moment, some people may not want to use this version. For example, Java omits generating Version 1 from its UUID class.
FYI, the more powerful database servers such as Postgres can generate UUID values including Version 1. You may choose to generate your UUIDs on the database server rather than in your app.
Random
One commonly used version is Version 4, in which 122 of the 128 bits are generated randomly. If a cryptographically-strong random generator is used, this is quite effective. This version has a much higher chance of collisions than in Version 1. But for most practical scenarios, the random-based UUID is entirely reliable.