Are UUIDs in Java interned like Strings? If not, should I be trying to recycle UUID objects to minimize RAM usage?
I use UUIDs as data type of database primary key & foreign key columns. So this means many rows repeating the use of UUID for shared foreign key value.
So when retrieving rows from the database, should I check to see if each UUID is a duplicate, and if duplicated, use the original object reference? Or is this being done on my behalf already, similar to how Strings are interned?
… // common JDBC code
UUID id = null ;
while (rs.next()) {
UUID idFresh = rs.getObject( 1 );
// Recycle the UUID object where possible.
id = ( ( null == id ) || idFresh.equals( id ) ) ? idFresh : id ; // If null or identical, use the existing object reference.
String name = rs.getString( 2 );
}
…
A quick look into the java runtime source code shows that UUIDs are not interned.
And it would probably be a bad idea to intern them, because if you were to traverse a large database, UUID interning could cause the JVM to run out of memory simply due to never foregtting any UUID it has seen.
Also, there is not much benefit to interning UUIDs, because
They don't occupy much space(basically just the UUID’s 128-bit value stored as a pair of long)
UUID comparison and hashcode computation is cheap.(One of the greatest benefits of String interning is that the hashcode of the string gets computed only once, which is a bit of a concern because its computation can be slightly expensive.)
UUIDs (and also strings) are not automatically deduplicated. In general, it would also be a bad idea, as newly created UUIDs should be unique, so sharing will not work.
When you refer to string interning, it is true that the JVM will share strings in specific case, for instance:
String x = "ab";
String y = "a" + "b";
assert x == y; // references are identical (x and y are shared)
These are strings, however, that can be resolved at compile time. If you create a string or UUID at runtime, it will always create a new object.
In your question, you describe a different scenario, though. Here, you are reading UUIDs from a database. Depending on the data, there could be good opportunities for sharing UUIDs, or there could be none (e.g., if the UUID is used as the primary key).
id | name | country
1 | A | <UUID-1>
2 | B | <UUID-1>
3 | C | <UUID-2>
4 | D | <UUID-1>
5 | E | <UUID-1>
(Note that when reading the UUIDs from the database or from the the network, you cannot assume that the UUIDs will be deduplicated. In general, you will receive copies of the same value.)
So, if your data looks like above, sharing of UUIDs can make sense. But will it reduce the memory usage?
An UUID is an object with two long variables. In a 64-bit JVM, this will take 32 bytes. If you share the UUID, then you will only pay the 32 bytes once, and afterwards pay only 8 bytes for the reference. If you use compressed pointers, the reference will fit in 4 bytes.
Is this gain significant enough? That depends on your specific application. In general, I would not share an UUID. I have worked on an application, however, where sharing the UUID was indeed an improvement. Reducing memory usage down was critical, and the reduction from a full object to a reference was an improvement.
Having said that, this type of optimization is rarely needed. As a rule of thumb, I would only do it if UUIDs are heavily shared and reducing memory at all costs is necessary. Otherwise, the CPU overhead of deduplicating them and the extra complexity in the code is often not worth it, or worse, could even slow down your application.
If you decide to deduplicate them, how will you do it? There is no built-in function like String#intern, but you can manually create a map to deduplicate. Depending on whether you want to deduplicate globally or only locally within in the current function call, you can use a ConcurrentHashMap or simply a (non-synchronized) HashMap.
As a side-note, not directly related to your question, I mentioned String#intern as it is part of the String API. However, I would strongly recommend against using it, as it is a huge performance bottleneck. Doing the deduplication yourself will be significantly faster.
Related
I want to generate an unique ID within my application which can be used as a primary key for my database. The ID should be created by the application (and not by the database). I am wondering which method should I better use for that:
version 1:
// maybe faster and more memory friendly than version 2
// but what about collisions?
var uniqueId = ThreadLocalRandom.current().nextInt()
or version 2:
var uniqueId = UUID.randomUUID().toString()
or version 3:
// maybe the same as version 1
var uniqueId = new SecureRandom().nextInt()
or anything better?
Both versions 1 and 3 will produce collisions, because there are only 2^32 different int's. You need to calculate the probability yourself.
Regular UUIDs (version 2) are also random, but there is 2^128 of them, which means the chances of a collision are negligible.
I would go for the UUID, that is what I normally use.
You will know the number of characters that all the IDs have.
But it's my opinion, I don't think that there is any significant difference that you should choose one besides the other.
I was under the impression that the UUID spec required a guaranteed, true, globally unique result, not unique 99.99999999999% of the time, but truly 100% of the time. From the spec:
A UUID is 128 bits long, and can guarantee
uniqueness across space and time.
It looks like java only support V3 and V4 of the UUID spec. V4 isn't truly unique. With the V3 implementation using nameUUIDFromBytes, the following results in duplicates, because the computer is too fast (edit: looping to 10 and called new Date().getTime() will produce duplicates because the computer loops faster than new Date().getTime() can produce a different value on each iteration):
String seed;
for (int i = 0; i < 10; i++) {
seed = "<hostname>" + new Date().getTime();
System.out.println(java.util.UUID.nameUUIDFromBytes(seed.getBytes()));
}
Am I mistaken in assuming that a UUID is 100% unique, and that it is only practically unique but not perfectly so? Is there anyway to do this in Java?
There are different methods of UUID generation. The kind you're using is behaving exactly as it should. You're using nameUUIDFromBytes, a "Static factory to retrieve a type 3 (name based) UUID based on the specified byte array."
This generates the same UUID if given the same name. As you've discovered, your loop is passing-in the same name every time, so you get the same UUID.
Have a look at Gabe's advice here: Which UUID version to use?
He recommends you use V4, which as others have pointed out is good enough for any realistic use case.
Because your entropy is limited to your memory, you can never ensure a UUID is "guaranteed, true, globally unique result". However, 99.99999999999% is already pretty good.
If you want to ensure unique values in your database, you could use a simple integer that's incremented to be sure it's unique. If you want to use UUIDs and be really sure they're unique, you just have to check that upon creation. If there's a duplicate, just create another one until it's unique.
Duplicates can happen, but IIRC, part of them is created dependent on your current time, so if you're just creating one every 5 minutes, you should be safe.
As others have pointed out, the type-4 UUID returned by UUID.randomUUID() is likely to be unique enough for any practical application. Cases where it's not are likely to be pathological: for example, rolling back a VM to a live snapshot, without restarting the Java process, so that the random-number generator goes back to an exact prior state.
By contrast, a type-3 or type-5 UUID is only as unique as what you put into it.
A type-1 UUID (time-based) should be very slightly "more" unique, under certain constraints. The Java platform does not include support for generating a type-1 UUID, but I've written code (possibly not published) to call a UUID generating library via JNI. It was 18 lines of C and 11 lines of Java.
If I generate a UUID from a "seed" string as follows, is there any way for someone to re-generate the original string?
UUID uuid = null;
try {
uuid = UUID.nameUUIDFromBytes(("seedString").getBytes("utf8"));
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
System.out.println("UUID: " + uuid.toString());
I would assume it isn't possible, as I believe this person found here: Convert UUID to bytes
However, I see that the same UUID is generated every time from a certain String/bytes, and since it has to be unique, simple "seed" values could just be guessed? For example, UUID of f is 8fa14cdd-754f-31cc-a554-c9e71929cce7 so if I see that I know it was generated from "f".
Since you are getting the UUID by casting bytes to a UUID, and you are always using the same starting bytes to cast from, the uuid would always be the same UUID across multiple runs.
I think you've confused a random seed with the "from bytes" method in the UUID routines. It is more like a cast than a seed initialization. And even if it was like a seed initialization, initializing with a constant seed would only mean that you always walk the "same" pseudo-random path (meaning that after walking it once, you can know the next step(s)).
aug also makes an excellent point, which I'll elaborate a bit on here. A UUID is an identifier, which is assumed to be unique only by virtue of there being so many to choose from; however, if you create a routine that returns the same one(s) repeatedly, it's not going to be unique due to your selection mechanism. The actual mechanism doesn't assure uniqueness; even less so when using a routine guaranteed to return identical values.
As they are not guaranteed to be unique (UUIDs have a fixed number of bits and eventually all combinations can be exhausted), one can imagine that there are more inputs than UUIDs (although there's a lot of UUIDs) so UUID collision is inevitable (even if it would theoretically take more time than the heat death of the universe). From a practical side of things, you probably have little to worry about; but, it could still (minuscule chance) happen.
This also means that one can (in theory) guarantee that some two inputs out there can wind up with the same UUID, and as a result, UUIDs are not generally reversible (however, in specific (limited) cases, perhaps they could be made reversible).
There are an infinite number of strings that may generate a given UUID, so even if somebody guesses the string you used to create a given UUID, they may never be sure.
My webapplication has a table in the database with an id column which will always be unique for each row. In addition to this I want to have another column called code that will have a 6 digit unique Alphanumeric code with numbers 0-9 and alphabets A-Z. Alphabets and number can be duplicate in a code. i.e. FFQ77J. I understand the uniqueness of this 6 digit alphanumeric code reduces over time as more rows are added but for now I am ok with this.
Requirement (update)
- The code should be at least of length 6
- Each code should be Alphanumeric
So I want to generate this Alphanumeric code.
Question
What is a good way to do this?
Should I generate the code and after the generation, run a query to the database and check if it already exists, and if so then generate a new one? To ensure the uniqueness, does this piece of code need to be synchronized so that only one thread runs it?
Is there something built-in to the database that will let me do this?
For the generation I will be using something like this which I saw in this answer
char[] symbols = new char[36];
char[] buf;
for (int idx = 0; idx < 10; ++idx)
symbols[idx] = (char) ('0' + idx);
for (int idx = 10; idx < 36; ++idx)
symbols[idx] = (char) ('A' + idx - 10);
public String nextString()
{
for (int idx = 0; idx < buf.length; ++idx)
buf[idx] = symbols[random.nextInt(symbols.length)];
return new String(buf);
}
Since it's a requirement for the shortcode to not be guessable, you don't want to tie it to your uniqueID row ID. Otherwise that means your rowID needs to be random, in addition to unique. Starting with a counter 0, and incrementing, makes it pretty obvious when your codes are: 000001, 000002, 000003, and so forth.
For your short code, generate a random 32bit int, omit the sign and convert to base36. Make a call to your database, to ensure it's available.
You haven't explicitly called out scalability, but I think it's important to understand the limitations of your design wrt to scale.
At 2^31 possible 6 char base36 values, you will have collisions at ~65k rows (see Birthday Paradox questions)
From your comment, modify your code:
public String nextString()
{
return Integer.toString(random.nextInt(),36);
}
I would simply do this:
String s = Integer.toString(i, 36).toUpperCase();
Choosing base-36 will use characters 0-9a-z for the digits. To get a string that uses uppercase letters (as per your question) you would need to fold the result to upper case.
If you use an auto increment column for your id, set the next value to at least 60,466,176, which when rendered to base 36 is 100000 - always giving you a 6 digit number.
I would start with 0 for an empty table and do a
SELECT MAX(ID) FROM table
to find the largest id so far. Store it in an AtmoicInteger and convert it using toString
AtomicInteger counter = new AtomicInteger(maxSoFar);
String nextId = Integer.toString(counter.incrementAndGet(), 36);
or for padding. 36 ^^ 6 = 2176782336L
String nextId = Long.toString(2176782336L + counter.incrementAndGet(), 36).substring(1);
This will give you uniqueness and no duplicates to worry about. (it's not random either)
Simply, you can use Integer.toString(int i, int radix). Since you have base 36(26 letters+10 digits) you set the radix to 36 and i to your integer. For example, to use 16501, do:
String identifier=Integer.toString(16501, 36);
You can uppercase it with .toUpperCase()
Now onto your other questions, yes, you should query the database first to ensure it doesn't exist. If depending on the database, it may need to be synchronized, or it may not be as it'll use its own locking system. In any case, you'd need to tell us which database.
On the question of whether there's a builtin, we'd need to know the DB type as well.
To create a random but unique value within a small range here are some ideas I know of:
Create a new random value and try to insert it.
Let a database constraint catch violations. This column should also likely be indexed. The DML may need to be tried several times until a unique ID is found. This will lead to more collisions as time progresses, as noted (see the birthday problem).
Create a "free IDs" table ahead of time and on usage mark the ID as being used (or delete it from the "free IDs" table). This is similar to #1 but shifts when the work is done.
This allows the work of finding "free IDs" to be done at another time, perhaps during a cron job, so that there will not be a contraint violation during the insert keeping the insert itself the "same speed" throughout the usage of said domain. Make sure to use transactions.
Create a 1-to-1/injective "mixer" function such that the output "appears random". The point is this function must be 1-to-1 to inherently avoid duplicates.
This output number would then be "base 36 encoded" (which is also injective); but it would be guaranteed unique as long as the input (say, an auto-increment PK) was unique. This would likely be less random than the other approaches, but should still create a nice-looking non-linear output.
A custom injective function can be created around an 8-bit lookup table fairly trivially - just process a byte at a time and shuffle the map appropriately. I really like this idea, but it can still lead to somewhat predictable output
To find free IDs, approaches #1 and #2 above can use "probing with IN" to minimize the number of SQL statements used. That is, generate a bunch of random values and query for them using IN (keeping in mind what sizes of IN your database likes) and then see which values were free (as having no results).
To create a unique ID not constained to such a small space, a GUID or even hashing (e.g. SHA1) might be useful. However, these only guarantee uniqueness because they have 126/160-bit spaces so that the chance of collision (for different input/time-space) is currently accepted as improbable.
I actually really like the idea of using an injective function. Bearing in mind that it is not good "random" output, consider this pseudo-code:
byte_map = [0..255]
map[0] = shuffle(byte_map, seed[0])
..
map[n] = shuffle(byte_map, seed[1])
output[0] = map[0][input[0]]
..
output[n] = map[n][input[n]]
output_str = base36_encode(output[0] .. output[n])
While a very simple setup, numbers like 0x200012 and 0x200054 will still share common output - e.g. 0x1942fe and 0x1942a9 - although the lines will be changed a bit due to the later application of the base-36 encoding. This could probably be further improved to "make it look more random".
For efficient usage, try caching generated code in a HashSet<String> in your application:
HashSet<String> codes = new HashSet<String>();
This way you don't have to make a db call every time to check whether the generated code is unique or not. All you have to do is:
codes.contains(newCode);
And, yes, you should synchronize your method which updates the cache
public synchronize String getCode ()
{
String newCode = "";
do {
newCode = nextString();
}
while(codes.contains(newCode));
codes.put(newCode);
}
You mentioned in your comments that the relationship between id and code should not be easily guessable. For this you basically need encryption; there are plenty of encryption programs and modules out there that will perform encryption for you, given a secret key that you initially generate. To employ this approach, I would recommend converting your id into ascii (i.e., representing as base-256, and then interpreting each base-256 digit as a character) and then running the encryption, and then converting the encrypted ascii (base-256) into base 36 so you get your alpha-numeric, and then using 6 randomly chosen locations in the base 36 representation to get your code. You can resolve collisions e.g. by just choosing the nearest unused 6-digit alpha-numeric code when a collision occurs, and noting the re-assigned alpha-numeric code for the id in a (code <-> id) table that you will have to maintain anyway since you cannot decrypt directly if you only store 6 base-36 digits of the encrypted id.
I have an application which takes data from a file and stores it for later use. Each line in the file corresponds to one object Foo, which contains n pairs of Bar objects which are made of a single character String, each with a distinct Name. So I store this data like so:
Foo extends HashMap<Name, Pair<Bar, Bar>>
where Pair<A, B> is my own class which just stores 2 values and provides some methods (equals, hashcode etc).
The problem I have encountered is that when I store n=114 (this just happens to be the number in my test data) Pair objects in my Foo it should have a retained size of not much more than 228 bytes, when in fact it is more like 25kbytes. This means when I have ~1000 Foo objects I need 25MB of memory rather than 228kB, which is not really acceptable. (Note: the keys for each Foo object are the same, fooOne.keySet().equals(fooTwo.keySet()))
I am using VisualVM to profile my application, and when I delve into an instance of Foo I see:
Field Type Retained
-
this Foo 24750
...
v table HashMap$Entry[] 24662
v [0] HashMap$Entry 200
v value Pair 156
v first Bar 60
...
> code String 36
v second Bar 60
...
> code String 36
v key Name 72
...
> name String 36
> [1] HashMap$Entry 200
> [2] <HashMap$Entry> -
...
> [233] HashMap$Entry 600
...
> [255] <HashMap$Entry> -
So as you can see all the useful information is being surrounded by lots of useless (to me) data. If I had fewer, larger objects with the same data in I can see my useful:useless ratio would be better, but I can't see how I can implement this in any other way. Is there some other way I can store my data, but still be as convenient and easy to use as this?
EDIT
My application will need to be scalable to upwards of 6000 Bar instances and maybe as many Foo instances.
I'm not entirely sure that I get your question right but in this situation using Flyweights may do the trick.
Flyweight pattern
I think a lot of your problem is just object oriented code in general, and Unicode conversion specifically.
In Java a character in a string requires two bytes to store. So at the very least you can expect to double your memory usage versus keeping a file on the drive.
Each object, each little string is going to require a word worth of information because of the pointer the JVM needs to point to your object. So each pair of data is a word for the key and a word for the value plus the actual size of each. Now these pointers get added to the hash, which uses a word to point to itself, and several words to point to the entryset. And so it goes. This is object oriented programming.
Now you could change your code to store the pair as a simple char[2]. This would cut down on your memory foot print. Then when you want to interact with it, you could wrap the array with a Pair object.
You can try to drop the Bar and Pair objects and store a pair of as simple String object ,e.g. "ab" (where "a","b" currently correspond to a Pair made of Bar("a") and Bar("b"))
Probably use the Flyweight patterns to share the common names of all Foo objects, since you have fooOne.keySet().equals(fooTwo.keySet())
You say:
I have an application which takes data from a file and stores it for
later use
and later (in a comment)
I've been asked to make it as memory efficient as possible
I suspect your most memory efficient solution is to store the file and parse it upon request, rather than parse and store in advance. But do you really want to do this and suffer the related performance costs ? I don't think your memory issues are particularly huge, but (as stated by others) I would investigate the flyweight pattern.
Take a look here. You'll see that you need quite a lot more bytes than you think to store a class (string or other) in the JVM's heap.
36 bytes for a 1 character string sounds quite right, as you need to store a lot of metadata for the object that holds the character (be sure to account for UTF encoding) plus the string class overhead.