Documentation about generatorId collisions when id structure changes about idgen

[RobThree]

How about this?

[RobThree]

What do you think about adding something about version bits,

I don't want to give people ideas; essentially all of the 'host' bits can be used for whatever, but I don't want to encourage 'abuse' of any of the bits. If you want to reserve some bits for versioning in your project then go ahead, but I don't want this to become a common practice.

and/or multiple id structures being active at the same time

Again, it's perfectly possible, but I don't think it's a good idea so I don't want to encourage it.

Technically you can have a generator per - say - entity. As long as your customer entities have a dedicated IdGenerator from, say, your products then there will be no collisions. As long as they're all on the same host/process/thread/... you can consider the id's not much different from an RDBMS using autoincremented id's; usually such tables have id's 1, 2, 3, ... and that's fine; they live in their own "namespace" (the table). Customer 1 doesn't "collide" with Product 1. But the scenario what IdGen (or Snowflake if you will) is designed for is if there are multiple hosts/processes/threads/... (i.e. "generators") generating id's for the same entity and you want to avoid collisions without having to coordinate these generators (other than assigning them a GeneratorId on instantiation).

I think there is a typo (comma), and I fixed it below. You've captured the gist though.

I literally had it there and then got rid of it 🤪 I've put it back 😉

[RobThree]

That should work fine. Did you consider putting the version bit last?

gggggggggggggv
\-----------/|
       |     +->  deployment bit
       v
       generator bits as usual
       13 bits

That should be "more correct" sortability-wise ('less impact'). Also it would have a nice even/uneven effect for blue/green. Though your option will work just as fine too.

[rmandvikar]

I'm fine with what you have. It brings attention to the issue, and I understand that being prescriptive here may not make sense. We can only so much. 🤷🏿

[rmandvikar]

What do you think about adding something about version bits, and/or multiple id structures being active at the same time?

I think there is a typo (comma), and I fixed it below. You've captured the gist though.

Although changing the structure at a later stage isn't impossible, careful consideration is needed to ensure no collisions will occur.

[rmandvikar]

Again, it's perfectly possible, but I don't think it's a good idea so I don't want to encourage it.

In my case it's blue/green deployments. So, it is unavoidable. Both id structures are only active for some time.

[RobThree]

@rmandvikar I've edited / updated my previous comment but I noticed you already 👍'ed it. Just making sure you're aware of it.

In my case it's blue/green deployments. So, it is unavoidable. Both id structures are only active for some time.

What's stopping you from reserving generator id's per (blue/green) deployment? Which is, essentially, what a 'version' bit does ofcourse.

[rmandvikar]

I have taken a bit of a practical approach considering complexity, maintainability, et. al. for generator_id. Where,

generator_id = 13-bit generator_id hash function based on string generator id

A hash function has pros/cons. It is easy to understand, and maintain as long as the string ids are known beforehand, which is always the case.

generator_id (14 bit):

vggggggggggggg
|\-----------/
|       v
|       generator bits as usual
v       13 bits
deployment bit
1 bits

[rmandvikar]

That should be "more correct" sortability-wise ('less impact'). Also it would have a nice even/uneven effect for blue/green. Though your option will work just as fine too.

I forgot to mention. One advantage of keeping the deployment version bit as MSB is that it won't shift as the id structure changes. The timestamp at 41 bits with 1 ms tick gives ~69 years, which suffices for me now, and I don't plan to change that. But, I do plan to increase the generator bits, and shrink the sequence bits in future, if needed. So, the fixed position of the version bit helps quite a bit (pun-intended 😄) in that case.

// 41/14/8
tttttttttttttttttttttttttttttttttttttttttvgggggggggggggsssssssss
\---------------------------------------/|\-----------/\-------/
              timestamp                  v  generator  sequence
                                         version

Just mentioning as it may be helpful to someone; a hack I have due to blue/green deployments, combined with the lack of version bit in the very first deploy, is to use the sequence MSB as an interim version bit (the version bit mentioned above in generator will also be added). 2^8 = 256 calls /ms/generator volume is very high, and the sequence MSB will always be 0, and it can be versioned to 1. The interim version bit in sequence MSB can be removed as a fast follow, then relying on the version bit in generator going forward.

So,

// note: deployments when changing id structure from 41/10/12 -> 41/14/8
     tttttttttttttttttttttttttttttttttttttttttggggggggggsssssssssssss
now: .........................................x.............0........ # sequence MSBs always 0 due to not as high traffic
     tttttttttttttttttttttttttttttttttttttttttvgggggggggggggsssssssss
 +1: .........................................0.............1........ # generator MSB versioned to 0, sequence MSB versioned to 1
 +2: .........................................1...................... # generator MSB versioned to 1, sequence MSB version removed
 +3: .........................................0...................... # generator MSB versioned to 0
 +4: .........................................1...................... # generator MSB versioned to 1
...: repeat 3, 4