robthree / idgen

Twitter Snowflake-alike ID generator for .Net

IdGen

Twitter Snowflake-alike ID generator for .Net. Available as Nuget package

Why

In certain situations you need a low-latency, distributed, uncoordinated, (roughly) time ordered, compact and highly available Id generation system. This project was inspired by Twitter's Snowflake project which has been retired. Note that this project was inspired by Snowflake but is not an exact implementation. This library provides a basis for Id generation; it does not provide a service for handing out these Id's nor does it provide generator-id ('worker-id') coordination.

How it works

IdGen generates, like Snowflake, 64 bit Id's. The Sign Bit is unused since this can cause incorrect ordering on some systems that cannot use unsigned types and/or make it hard to get correct ordering. So, in effect, IdGen generates 63 bit Id's. An Id consists of 3 parts:

• Timestamp
• Generator-id
• Sequence

An Id generated with a Default IdStructure is structured as follows:

However, using the IdStructure class you can tune the structure of the created Id's to your own needs; you can use 45 bits for the timestamp, 2 bits for the generator-id and 16 bits for the sequence if you prefer. As long as all 3 parts (timestamp, generator and sequence) add up to 63 bits you're good to go!

The timestamp-part of the Id should speak for itself; by default this is incremented every millisecond and represents the number of milliseconds since a certain epoch. However, IdGen relies on an ITimeSource which uses a 'tick' that can be defined to be anything; be it a millisecond (default), a second or even a day or nanosecond (hardware support etc. permitting). By default IdGen uses 2015-01-01 0:00:00Z as epoch, but you can specify a custom epoch too.

The generator-id-part of the Id is the part that you 'configure'; it could correspond to a host, thread, datacenter or continent: it's up to you. However, the generator-id should be unique in the system: if you have several hosts or threads generating Id's, each host or thread should have it's own generator-id. This could be based on the hostname, a config-file value or even be retrieved from an coordinating service. Remember: a generator-id should be unique within the entire system to avoid collisions!

The sequence-part is simply a value that is incremented each time a new Id is generated within the same tick (again, by default, a millisecond but can be anything); it is reset every time the tick changes.

System Clock Dependency

We recommend you use NTP to keep your system clock accurate. IdGen protects from non-monotonic clocks, i.e. clocks that run backwards. The DefaultTimeSource relies on a 64bit monotonic, increasing only, system counter. However, we still recommend you use NTP to keep your system clock accurate; this will prevent duplicate Id's between system restarts for example.

The DefaultTimeSource relies on a Stopwatch for calculating the 'ticks' but you can implement your own time source by simply implementing the ITimeSource interface.

Getting started

Install the Nuget package and write the following code:

using IdGen;
using System.Linq;

class Program
{
    static void Main(string[] args)
    {
        var generator = new IdGenerator(0);
        var id = generator.CreateId();
        // Example id: 862817670527975424
    }
}

Voila. You have created your first Id! Want to create 100 Id's? Instead of:

var id = generator.CreateId();

write:

var id = generator.Take(100);

This is because the IdGenerator() implements IEnumerable providing you with a never-ending stream of Id's (so you might want to be careful doing a .Select(...) or Count() on it!).

The above example creates a default IdGenerator with the GeneratorId (or: 'Worker Id') set to 0 and using a DefaultTimeSource. If you're using multiple generators (across machines or in separate threads or...) you'll want to make sure each generator is assigned it's own unique Id. One way of doing this is by simply storing a value in your configuration file for example, another way may involve a service handing out GeneratorId's to machines/threads. IdGen does not provide a solution for this since each project or setup may have different requirements or infrastructure to provide these generator-id's.

The below sample is a bit more complicated; we set a custom epoch, define our own id-structure for generated Id's and then display some information about the setup:

using IdGen;
using System;

class Program
{
    static void Main(string[] args)
    {
        // Let's say we take april 1st 2020 as our epoch
        var epoch = new DateTime(2020, 4, 1, 0, 0, 0, DateTimeKind.Utc);
            
        // Create an ID with 45 bits for timestamp, 2 for generator-id 
        // and 16 for sequence
        var structure = new IdStructure(45, 2, 16);
            
        // Prepare options
        var options = new IdGeneratorOptions(structure, new DefaultTimeSource(epoch));
            
        // Create an IdGenerator with it's generator-id set to 0, our custom epoch 
        // and id-structure
        var generator = new IdGenerator(0, options);

        // Let's ask the id-structure how many generators we could instantiate 
        // in this setup (2 bits)
        Console.WriteLine("Max. generators       : {0}", structure.MaxGenerators);

        // Let's ask the id-structure how many sequential Id's we could generate 
        // in a single ms in this setup (16 bits)
        Console.WriteLine("Id's/ms per generator : {0}", structure.MaxSequenceIds);

        // Let's calculate the number of Id's we could generate, per ms, should we use
        // the maximum number of generators
        Console.WriteLine("Id's/ms total         : {0}", structure.MaxGenerators * structure.MaxSequenceIds);


        // Let's ask the id-structure configuration for how long we could generate Id's before
        // we experience a 'wraparound' of the timestamp
        Console.WriteLine("Wraparound interval   : {0}", structure.WraparoundInterval(generator.Options.TimeSource));

        // And finally: let's ask the id-structure when this wraparound will happen
        // (we'll have to tell it the generator's epoch)
        Console.WriteLine("Wraparound date       : {0}", structure.WraparoundDate(generator.Options.TimeSource.Epoch, generator.Options.TimeSource).ToString("O"));
    }
}

Output:

Max. generators       : 4
Id's/ms per generator : 65536
Id's/ms total         : 262144
Wraparound interval   : 407226.12:41:28.8320000 (about 1114 years)
Wraparound date       : 3135-03-14T12:41:28.8320000+00:00

IdGen also provides an ITimeSouce interface; this can be handy for unittesting purposes or if you want to provide a time-source for the timestamp part of your Id's that is not based on the system time. For unittesting we use our own MockTimeSource.

<configuration>
  <configSections>
    <section name="idGenSection" type="IdGen.Configuration.IdGeneratorsSection, IdGen.Configuration" />
  </configSections>

  <idGenSection>
    <idGenerators>
      <idGenerator name="foo" id="123"  epoch="2016-01-02T12:34:56" timestampBits="39" generatorIdBits="11" sequenceBits="13" tickDuration="0:00:00.001" />
      <idGenerator name="bar" id="987"  epoch="2016-02-01 01:23:45" timestampBits="20" generatorIdBits="21" sequenceBits="22" />
      <idGenerator name="baz" id="2047" epoch="2016-02-29"          timestampBits="21" generatorIdBits="21" sequenceBits="21" sequenceOverflowStrategy="SpinWait" />
    </idGenerators>
  </idGenSection>

</configuration>

The attributes (name, id, epoch, timestampBits, generatorIdBits and sequenceBits) are required. The tickDuration is optional and defaults to the default tickduration from a DefaultTimeSource. The sequenceOverflowStrategy is optional too and defaults to Throw. Valid DateTime notations for the epoch are:

• yyyy-MM-ddTHH:mm:ss
• yyyy-MM-dd HH:mm:ss
• yyyy-MM-dd

You can get the IdGenerator from the config using the following code:

var generator = AppConfigFactory.GetFromConfig("foo");

Dependency Injection

There is an IdGen.DependencyInjection NuGet package available that allows for easy integration with the commonly used Microsoft.Extensions.DependencyInjection.

Usage is straightforward:

services.AddIdGen(123); // Where 123 is the generator-id

Or, when you want to use non-default options:

services.AddIdGen(123, () => new IdGeneratorOptions(...));  // Where 123 is the generator-id

This registers both an IdGenerator as well as an IIdGenerator<long>, both pointing to the same singleton generator.

Upgrading from 2.x to 3.x

Upgrading from 2.x to 3.x should be pretty straightforward. The following things have changed:

• Most of the constructor overloads for the IdGenerator have been replaced with a single constructor which accepts IdGeneratorOptions that contains the ITimeSource, IdStructure and SequenceOverflowStrategy
• The MaskConfig class is now more appropriately named IdStructure since it describes the structure of the generated ID's.
• The UseSpinWait property has moved to the IdGeneratorOptions and is now an enum of type SequenceOverflowStrategy instead of a boolean value. Note that this property has also been renamed in the config file (from useSpinWait to sequenceOverflowStrategy) and is no longer a boolean but requires one of the values from SequenceOverflowStrategy.
• ID is now Id (only used as return value by the FromId() method)

The generated 2.x ID's are still compatible with 3.x ID's. This release is mostly better and more consistent naming of objects.

FAQ

Q: Help, I'm getting duplicate ID's or collisions?

A: Then you're probably not using IdGen as intended: It should be a singleton (per thread/process/host/...), and if you insist on having multiple instances around they should all have their own unique GeneratorId.

A: Also: Don't change the structure; once you've picked an IdStructure and go into production commit to it, stick with it. This means that careful planning is needed to ensure enough ID's can be generated by enough generators for long enough. Although changing the structure at a later stage isn't impossible, careful consideration is needed to ensure no collisions will occur.

Q: I'm experiencing weird results when these ID's are used in Javascript?

A: Remember that generated ID's are 64 (actually 63) bits wide. Javascript uses floats to store all numbers and the maximum integer value you can safely store is 53 bits. If you need to handle these ID's in Javascript, treat them as strings.

---

Icon made by Freepik from www.flaticon.com is licensed by CC 3.0.

idgen's Issues

Duplicate Ids in a two instance microservice

Hello, I am using .Net 5 API and the nuget of the IdGen 3.0.0
I am using two instances of this API
In the logs, I noticed for some cases I have a pk violation, which pk is a long from IdGen .

Here are some pieces of my code

public static class UniqueIdGenerator
	{
		private static IdGenerator? _generator;

		public static void Configure(int generatorId)
		{
			if (_generator != null)
				throw new InvalidOperationException();

			_generator = new IdGenerator(generatorId);
		}

		public static long CreateId() => _generator?.CreateId() ?? throw new InvalidOperationException();
	}

Then I register it in program.cs class

public class Program
	{
		public static void Main(string[] args)
                {
                       UniqueIdGenerator.Configure(0);
                       //bla bla bla
                }
         }

and for example, if I want to create a new customer I call the static method like

public async Task AddCustomer(AddCustomerRequest request)
{
    var customer = new Customer
    {
        Id = UniqueIdGenerator.CreateId(),
        Name = request.Name,
        Phone = request.Phone
        //etc
    }
    //bla bla bla
}

Now I have some logs like that
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details.
---> Microsoft.Data.SqlClient.SqlException (0x80131904): Violation of PRIMARY KEY constraint 'PK_Customers'. Cannot insert duplicate key in object 'dbo.Customers'. The duplicate key value is (842512993898987520).

And the customer with id 842512993898987520 already exists.
Is something wrong with my configuration of IdGen?
Thanks in advance!

SpinWait drastically slows continous ID generation rate for small tick length

When using a small tick length together with SequenceOverflowStrategy.SpinWait, the continuous ID generation rate slows down much more than expected once you overflow the sequence number and start spinwaiting.

As an example, the default options use 12 sequence bits, allowing 4096 IDs to be generated during a 1 millisecond tick. On a fast enough machine that can actually generate these IDs in less than 1 millisecond, you'd then expect to be able to generate IDs continuously at a rate of 4096 per millisecond, spinwaiting until the start of the next tick after each overflow. However, in reality, I can only generate roughly 320 IDs per millisecond continuously.

The issue seems to be that System.Threading.SpinWait doesn't actually keep spinning, but instead initiates a context switch after a short while. It may then take 10 to 15 milliseconds before the thread is scheduled again, thereby skipping a lot of ticks. I have tested that the time required to generate (N > 4096) IDs roughly correspond with the time required to SpinWait for (N - 1) / 4096 milliseconds. See benchmark below.

A solution could be to switch from System.Threading.SpinWait to an implementation that doesn't initiate context switches. See example code below. This of course doesn't prevent the operating system from preempting the thread and context switching anyway, but it does seem to work in my benchmark.

A more durable solution is probably to switch to longer ticks, perhaps 10 ms, where the context switching matters less, but that would of course be on consumers. (Changing the default would be breaking.) But maybe the IdGen documentation could at least describe the performance penalty of short ticks together with spinwaiting.

Benchmark code

public class IdGeneratorBenchmarks
{
    private IdGenerator _generator;

    [Params(4096, 4097, 40960)]
    public int N;

    [IterationSetup(Target = nameof(IdGeneratorCreateId))]
    public void IdGeneratorSetup()
    {
        // This ensures the previous operation hasn't used up the available sequence numbers for the next one.
        _generator = new(0, new IdGeneratorOptions(sequenceOverflowStrategy: SequenceOverflowStrategy.SpinWait));
    }

    [Benchmark]
    public void IdGeneratorCreateId()
    {
        for (int i = 0; i < N; i++)
        {
            _generator.CreateId();
        }
    }

    [Benchmark]
    public void SystemThreadingSpinWait()
    {
        var millisecondsWaitNeeded = (N - 1) / 4096;
        var last = 0L;
        var stopwatch = Stopwatch.StartNew();
        for (int i = 0; i < millisecondsWaitNeeded; i++)
        {
            SpinWait.SpinUntil(() => stopwatch.ElapsedMilliseconds > last);
            last = stopwatch.ElapsedMilliseconds;
        }
    }

    [Benchmark]
    public void ManualSpinWait()
    {
        var millisecondsWaitNeeded = (N - 1) / 4096;
        var last = 0L;
        var stopwatch = Stopwatch.StartNew();
        for (int i = 0; i < millisecondsWaitNeeded; i++)
        {
            while (stopwatch.ElapsedMilliseconds <= last) ;
            last = stopwatch.ElapsedMilliseconds;
        }
    }
}

Benchmark results

|                  Method |        Job | InvocationCount | UnrollFactor |     N |              Mean |            Error |            StdDev |            Median |
|------------------------ |----------- |---------------- |------------- |------ |------------------:|-----------------:|------------------:|------------------:|
| SystemThreadingSpinWait | DefaultJob |         Default |           16 |  4096 |          30.01 ns |         1.225 ns |          3.434 ns |          29.24 ns |
|          ManualSpinWait | DefaultJob |         Default |           16 |  4096 |          30.62 ns |         0.645 ns |          1.227 ns |          30.54 ns |
|     IdGeneratorCreateId | Job-TRHHUL |               1 |            1 |  4096 |     298,785.71 ns |     1,712.429 ns |      1,518.024 ns |     298,200.00 ns |
| SystemThreadingSpinWait | DefaultJob |         Default |           16 |  4097 |  15,446,881.15 ns |    67,378.315 ns |     63,025.718 ns |  15,455,123.44 ns |
|          ManualSpinWait | DefaultJob |         Default |           16 |  4097 |   1,000,269.71 ns |       108.778 ns |         96.429 ns |   1,000,239.94 ns |
|     IdGeneratorCreateId | Job-TRHHUL |               1 |            1 |  4097 |  10,610,806.00 ns | 1,889,663.490 ns |  5,571,715.435 ns |  13,349,950.00 ns |
| SystemThreadingSpinWait | DefaultJob |         Default |           16 | 40960 | 136,070,392.11 ns | 2,665,274.841 ns |  2,962,445.923 ns | 136,095,775.00 ns |
|          ManualSpinWait | DefaultJob |         Default |           16 | 40960 |   9,000,772.81 ns |       141.551 ns |        132.407 ns |   9,000,756.25 ns |
|     IdGeneratorCreateId | Job-TRHHUL |               1 |            1 | 40960 | 128,078,597.00 ns | 4,144,345.151 ns | 12,219,695.183 ns | 135,176,300.00 ns |

Feature: CreateId that accepts generator ID as argument

We have a use case where we need to specify a custom generator ID for each request since we encode some other info in each ID. Since the library only generates IDs with three parts epoch + generator ID + sequence, our only option is to use custom generator ID - that we create using our logic.

https://github.com/RobThree/IdGen/blob/master/IdGen/IdGenerator.cs#L100

Can you create a method 'long CreateIdusingGenId(int generatorId)' and then update CreateIdImpl signature to have generator id as parameter (default value -1), In case of CreateId, pass -1 as generator Id and it will use member _generatorid value, otherwise it will use the provided one.

GeneratorId for Azure Function(s)

Hi, I would like to use the IdGen lib to generate UUIDs via an Azure Function. As the Function scales up, the GeneratorId should be unique to avoid collision as I understand the docs. I am having a real challenge with a identifying a unique GeneratorID between 0-1023 dynamically. I thought of using the Process.ID which should be unique enough. I thought of converting a GUID to an int (via a byte[]) but both are too large. Any thoughts or assistance would be appreciated.

Documentation about generatorId collisions when id structure changes

One issue that I bumped into is with blue/green deployments, and when the id structure changes, there is a possibility of generatorId collisions, which could lead to id collisions as both id structures are active at the same time. I was wondering if a note could be added about this in the documentation, as it was difficult to catch. To tackle this, I went with a "deployment bit" in the generatorId which I rev by toggling whenever the id structure changes, as just 1 bit suffices to differentiate blue/green deployments. What say? Thanks.

How to sync DefaultTimeSource's Elapsed

I have two nodes writing records to database, using this component to generate the ids. But this ids' ordering is different from creationtime's, I think it's related to DefaultTimeSource. I can not ensure this nodes start at same time, so the Stopwatch's Elapsed is different. How can I solve this problem? Remove the stopwatch's Elapsing? I don't know what it does.

Different host instances, run for a period of time, the KEY sequence is inconsistent

Hi
First of all, thank you for providing such a good package. I have some problems in use and would like to ask how to solve it. details as following.

I use this package on one application and deploy this application to two different hosts, Provide KEY generation service through Load Balance architecture. The system local time of the two hosts is calibrated through NTP and uses different GeneratorId.

Instance create method:

private static IIdGenerator _generator;

generator = new IdGenerator(_machineId, new IdGeneratorOptions(sequenceOverflowStrategy: SequenceOverflowStrategy.SpinWait));

But when my application runs for a period of time, the KEY sequence generated on the two hosts will be wrong, as follows.

1.Host A GenKey=700000000000100000
2.Host B GenKey=700000000000200000
3.Host A GenKey=700000000000300000
4.Host B GenKey=700000000000400000
5.Host A GenKey=700000000000500000
6.Host B GenKey=700000000000600000
after a while.......
1001.Host A GenKey=700000000001100000
1002.Host B GenKey=700000000001200000
1003.Host A GenKey=700000000001400000
1004.Host B GenKey=700000000001300000
1005.Host A GenKey=700000000001600000
1006.Host B GenKey=700000000001500000

I hope that the time generated by the two hosts will not be rolled back. How should I deal with it? Maybe implement ITimeSource the NTPTimeSource?

Thanks

Add support .Net Core

Hi, Do you add support for .net core ? Thanks.

Is there a way to generate id from the past

Im creating a feed and using IdGen to create time based ids to sort it and its working perfectly. But is there a way to pass in a date in history and calculate the id? Like 3 days ago or something?

Add to documentation

• JS gotcha: 53 bit floats. Use string representation
• Suggest singleton for DI

Generated Ids have timestamps from the past

The stopwatch is declared/instantiated as a static variable.

So it wont actually be started until it is referenced.

This results in timestamps that are marked back in time. The elapsed time is only measured after the stopwatch starts and the stopwatch only starts after the first call to CreateId().

Initializing with a call to CreateId on process startup as a workaround.

Confirm IdGen Generator Id Always Unique！

https://www.cnblogs.com/yushuo/p/9406906.html
This blog information, only to explain Twitter's Snowflake algorithm will create a lots of
same Id under the pressure test using Jmeter when test thread number >= 1000,

select DataId from IdTest GROUP BY DataId HAVING COUNT(DataId)>1
this SQL means how many data are duplicated.

so I hope that our IdGen can do a good stress test, and give us test result, to ensure that the generated Id is always unique.

Thanks！

Non-generic IIdGenerator

I'm curious why the IIdGenerator interface is generic – aren't the values always long?

Maybe there could be a non-generic interface using long internally?

generators clock running fast

Hi,

I have an issue with conflicting ID's being generated which i cannot wrap my head around,

I don't do any fancy configuration, just the default mask with a generator id. I do however run it inside an Orleans Grain that has the same key as the generator id meaning it should be impossible to have 2 different instances/activation of it.
Its also being run in a docker container.

i use the default DefaultTimeSource

generator gets created as such:
this.RoundIdGenerator = new IdGenerator(integrationId) ... Task.FromResult(RoundIdGenerator.CreateId()

On the same generator i got this the following sequences 2 days apart:

678847413238120448
678847436310986752
678847441495146496
...
678847431995047936
678847436310986752 Collision
678847526589186048

the system was rebooted in between which would have triggered a new generator and assume that also resets the drift where the generators clock runs

What is the best way to solve this? how do i avoid the generators stopwatch to run to fast?

Timestamp bits go ahead of time

DefaultTimeSource implementation uses static Stopwatch which starts on first usage of the class.
If you instantiate DefaultTimeSource later (as part of new IdGenerator for example) it counts ticks as current offset + elapsed time on static Stopwatch which may run for awhile. As a result timestamp bits show time in future. It may result non-unique identifiers in some scenarios.

IdGenerator.GetFromConfig("foo") is error!

IdGenerator.GetFromConfig("foo") is error!

System.Configuration.ConfigurationErrorsException

Move to better monotonous increasing timesource

We should move to using a better timesource with better resolution (MSDN states 10ms resolution) to prevent problems when the user changes the system's date.

One solution would be to use a Stopwatch and use some offset to calculate the time since epoch and the start of the stopwatch to get to the "current (monotonous increasing, not wall-clock) time".

This would, at least, guarantee time always moving forward (as long as the stopwatch is running; so it should probably be a static private so all instances rely on the same stopwatch); however: when the program would be restarted with an adjustment of time done in-between runs the Id's generated could still have some 'overlap'.

Environment.Tickcount is an int and thus will overflow every 24.9/49.8 days; a P/Invoke to GetTickCount64 could be used instead but then we'd still have the problem of adjusted times, however, now not in-between runs but in-between reboots.

Also: we should move away from milliseconds (and DateTime alltogether) and use a more abstract term like "ticks" (how they're defined, being milliseconds or nanoseconds or even seconds or whatever) would be up to whomever is implementing ITimeSource. GetTime() would have to be removed/dropped in favor of GetTicks() and this method should return a (u)long.

Drop support for netstandard 1.1

Hello,

Do you have any plan to discontinue support for netstandard 1.1 since it contains vulnerable nuget packages such as System.Drawing.Common, System.Net.Http, System.Net.Security ? I think that people shouldn't have any problem using netstandard 2.0 nowadays.

In production environment, Can be defined as static member?

public static class Program
{
    public static readonly IdGenerator generator = new IdGenerator(0);

    static void Main(string[] args)
    {
        var id = generator.CreateId();
        Console.WriteLine(id);

        Console.ReadKey();
    }
}

Public Method Comments

Sorry to be a style-douche, with this.
Was test-driving your id generator package. Nice work, by the way.

I was trying to identify the usage of methods on the generator class.
But, the Intellisense wasn't showing anything.

I peeked into the nupkg (v3.0.3 in case it matters), and saw that you include an xml doc file for the netstandard1.1 target.
But, there was not one for the netstandard2.0 target... probably the version my test code is linking to.

Nuspec's are annoying to deal with, and I've never understood why the documentation is so sparse.
And for multi-targeted and multi-runtime packaging, there's few working examples on the web that well-define how to organize the files node (for a sourcing nuspec file).
Here's the files node excerpt of what I do in sourcing nuspec files.
This example targets net5 and net6 and two runtimes of each: win64 and linux64.

Normally, for a simplex nuspec file, the xml doc is listed in the "lib" folder, next to the corresponding dll.
But for a nuspec that packages for multiple target framework versions, the xml doc gets listed in the "ref" folder of each target version.
It doesn't matter which compiled dll file you use for the ref file. But, list the xml doc from the same dll.

NOTE: Since multiple "lib" folders are required for multiple framework (net5.0, net6.0, etc), a separate "ref" is required.
But, multiple target runtimes (Win-x64, linux-x64, etc) can share the same "ref".

NOTE: I also include pdb files with my nuget packages.
Contrary to style-police guidelines, I do this, so any logging library can directly include file and line numbers as log message context, without resorting to hairy indirection of IL Offsets and pdb2xml mapping work (my OGA.SharedKernel library has a stacktrace class to help with this).

Hope it helps.

  <files>
    <file src=".\NETCore_Common_NET6\bin\DebugWin\net6.0\win-x64\NETCore_Common.dll" target="ref\net6.0" />
    <file src=".\NETCore_Common_NET6\bin\DebugWin\net6.0\win-x64\NETCore_Common.xml" target="ref\net6.0" />
    <file src=".\NETCore_Common_NET6\bin\DebugWin\net6.0\win-x64\NETCore_Common.dll" target="runtimes\win-x64\lib\net6.0" />
    <file src=".\NETCore_Common_NET6\bin\DebugWin\net6.0\win-x64\NETCore_Common.pdb" target="runtimes\win-x64\lib\net6.0" />
    <file src=".\NETCore_Common_NET6\bin\DebugLinux\net6.0\linux-x64\NETCore_Common.dll" target="runtimes\linux-x64\lib\net6.0" />
    <file src=".\NETCore_Common_NET6\bin\DebugLinux\net6.0\linux-x64\NETCore_Common.pdb" target="runtimes\linux-x64\lib\net6.0" />

    <file src=".\NETCore_Common_NET5\bin\DebugWin\net5.0\win-x64\NETCore_Common.dll" target="ref\net5.0" />
    <file src=".\NETCore_Common_NET5\bin\DebugWin\net5.0\win-x64\NETCore_Common.xml" target="ref\net5.0" />
    <file src=".\NETCore_Common_NET5\bin\DebugWin\net5.0\win-x64\NETCore_Common.dll" target="runtimes\win-x64\lib\net5.0" />
    <file src=".\NETCore_Common_NET5\bin\DebugWin\net5.0\win-x64\NETCore_Common.pdb" target="runtimes\win-x64\lib\net5.0" />
    <file src=".\NETCore_Common_NET5\bin\DebugLinux\net5.0\linux-x64\NETCore_Common.dll" target="runtimes\linux-x64\lib\net5.0" />
    <file src=".\NETCore_Common_NET5\bin\DebugLinux\net5.0\linux-x64\NETCore_Common.pdb" target="runtimes\linux-x64\lib\net5.0" />
  </files>

Invalid Tests?

Thanks for this code - I am having a play now.

I (well Resharper) noticed a few of the unit tests are using ReferenceEquals incorrectly:

Assert.ReferenceEquals(target1, target2);

should be

Assert.IsTrue(ReferenceEquals(target1, target2));

IdGen is generating 17bit Id instead of 16bit? What should we change in the method so that it generate 16 bit Id instead of 17 bit now and in the future it continuous to generate 16 bit Id?

Hello @RobThree,

We are using IdGen in one of our projects and it is generating 17bit Id instead of 16bit?
What should we change in the method so that it generate 16 bit Id instead of 17 bit now and in the future it continuous to generate 16 bit Id?
Is it because of date issue?
Please see below screen shot of our current implementation.

Looking for your prompt response!

Thanks

Is there a way to shorten the number of digits generated for an id?

The 19 digits are fine, but I don't need that level of precision so I'd like to lower the amount of digits to somewhere around 12. Is this possible?

Feature suggestion: (Option to) keep trying rather than throw SequenceOverflowException

SequenceOverflowException only occurs under very specific circumstances (hardware speed, unusually high demand, very tight loop etc.) which may not always be easy to test for or anticipate.

It's hard to imagine a use case where a user can simply give up if they can't have an ID, so there's no harm in simply waiting. The default settings allow for 4 million IDs per second, which ought to be fast enough for all intents and purposes, and much preferable to unexpected crashes.

Of course, it's easy to wrap the whole thing in a try/catch block, but my concern isn't usability so much as reliability.

Constructor mismatch

In the sample in README the IdGenerator() constructor takes 3 arguments, while in v3 it actually takes 2 -- generatorId and options. There's no argument for the epoch as shown in the sample. Either docs or the constructor needs to be updated.

Could we log how often we overflow sequence IDs and have to spinwait?

Somewhat related to #53, we would like to use SequenceOverflowStrategy.SpinWait, but to know how often we overflow the sequence number and have to spinwait. We could perhaps extract the sequence number from the returned ID and looks for those just before an overflow, but would it be possible for IdGen to somehow indicate that an overflow has happened? Maybe it's a bit much to integrate logging just for this, but perhaps there's be another way to do the signaling?

Handling uniqueness ID across multiple instance of same service

Hi,
Thanks for this useful package.
I have multiple instance of same service with using Kubernetes replicases, how can I handle uniqueness between instances, I cannot set different generatorId on each instance for example using Kubernetes environment variables. Do you have any solution for this in microservices world?

Concurrency issues with instances scale up

This is a question rather than an issue. Sorry about that. I want to use idGen for database table id generation in an app running on Fargate, or perhaps azure webapps. Given that the scaled out instances would use the same database is there a likelihood that there will be collisions when the apps or containers scale out (aspin up more fargate instances/webapp instances)? Would i have to somehow assign a different generator id for each appinstance as it starts up ? What would you do to ensure smooth id assignment in this scenario?

Using IdGen for migrating legacy data

Hello, I have a couple of questions about using IdGen for project with legacy data:

1. Is it correct that to avoid collisions between Id-s generated by two instances of an "IdGenerator" it is enough to use different "generator-id-part" values for those instances?

2. In case of having legacy data with DB-generated Id-s (auto-increment Identity), for migrating to client-generated Id-s using "IdGen" we should at least verify that maximum Id in existing data is less than minimum Id which will be generated for new records by "IdGen" (with particular epoch and "generator-id-part"). Maybe you have any other caveats/advices for such a use-case?

Thanks in advance

Int use for Generator Id

Just a quick question: The Generator Id is supposed to be 10 bits, however in the constructor(new IdGenerator(int myInt)) the passed in value - myInt, is 32 bits. Am I missing something?

Thank you very much,

Provide id into [timestamp ,generatorId ,sequence]?

var generator = new IdGenerator(0);
var id = generator.CreateId();
var genData = generator.FromId(id); //add this method?
I want to know when Id was generated。

Performance Tests

This is more of a question than an issue. Looking at the unit tests I don't see any performance tests, is it possible to add some to validate that the number of generated ids do not collide, i.e. they are unique for a given time interval within the mask resolution?

Can support.net core？

Id to datetime

Hello, thank you, your library is very useful

I have a question, can we get timestamp value from a ID , or can we know what Id value that start of each year ?

I am going to partition my table but I just have the Id value, I want to partition the table by year so I need to know what Id if start of a year

Does IdGen work across cluster nodes with no collisions?

Is IdGen works across cluster nodes with no collisions?

Use dependabot to keep references up to date

IdGen.DependencyInjection package referencing old IdGen and other old packages.

dependabot can help with that.

Here is one of my repos using dependabot

Sequential for Microsoft SQL Server or MySQL?

Is it designed to be sequential for Microsoft SQL Server or MySQL?

How to generate 12 digits

Hi, I want to batch generation12 digits Ids. max count about 100M. how to set the configs?
thank you!

Debugging InvalidSystemClockExceptions

We're running into sporadic InvalidSystemClockExceptions that end up requiring restarts, and we're running out of ideas trying to find where it goes wrong.

one error example is:
IdGen.InvalidSystemClockException: Clock moved backwards or wrapped around. Refusing to generate id for 4294446821 ticks
from reading the code, this implies the new timestamp is behind the last generated one by 50 days, which seems too big of a difference even if there was a clock sync issue (we've checked and our k8s nodes are running chrony and the recorded sync errors are nowhere near that much of a difference).

The issue seems to happen once a month or so, and forces us to manually restart pods as it blocks all writes, and waiting the ticks for it to restart would take days.

Our timesource implementation

public class ClockTimeSource : ITimeSource
{
    private long previousTimestamp = 0;

    public DateTimeOffset Epoch { get; }
    public TimeSpan TickDuration => TimeSpan.FromMilliseconds(1);

    public ClockTimeSource() { }
    public ClockTimeSource(DateTimeOffset epoch) => Epoch = epoch;

    public long GetTicks()
    {
        var timestamp = (long)(DateTimeOffset.UtcNow - Epoch).TotalMilliseconds;
        if (timestamp > previousTimestamp)
        {
            previousTimestamp = timestamp;
        }
        return previousTimestamp;
    }
}

Would appreciate any pointers on where to start debugging this.

Thread safety

Hi, on too many requests, the IdGen package has a duplicate ID problem, and some requests fail. You seem to have forgotten to define the static lock. Is shown in the image below.

Tag 3.0.3 is missing

Nuget 3.0.3 is out, but the git tag is missing in github repo (see), and so is the release (see). The nuget doesn't follow semver2 (with commit hash stamped) so I can't tell which commit I should be looking at. Please push the 3.0.3 version's tag to github.

I think this is the diff, but I can't be sure:
https://github.com/RobThree/IdGen/compare/3.0.2..0ef2be36db0489d246fde51b51ad27056978d83b

Avoid spending a sign bit by using ulongs

Hey, would it be possible to create a way to use AddIdGen which uses ulongs instead of longs instead to avoid wasting a single bit on the sign modifier?

NetStandard 2.0 support

Do you plan to release NuGet package with NetStandard 2.0 support?

Consider adding IdGenerator#CreateFromDateTime

When importing data, it's useful to create IDs for a specific date/time, but as far as I know, this requires constructing a custom time source, options, and generator for every generated ID (example).

A method like IdGenerator#CreateFromDateTime(DateTime dateTime) would make this a lot more efficient.

Sequence input for CreateId()

I would like to input my custom sequence id. Could this be supported? I know the implications, but I want to input sequence into the function in a similar setting how Instagram creates UIDs > https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c

IdGenerator does not contain a definition for 'Take'

In the documentation, it shows that the generator can create specified numbers of Ids by using generator.Take(100). However the .NET compiler throws an error "Compilation error (line 10, col 28): 'IdGenerator' does not contain a definition for 'Take' and no accessible extension method 'Take'..."

using System;
using IdGen;
					
public class Program
{
	public static void Main()
	{
		Console.WriteLine("Hello World");
		var generator = new IdGenerator(0);
               var id = generator.Take(100);
		
		
	}
}

Compilation error (line 10, col 28): 'IdGenerator' does not contain a definition for 'Take' and no accessible extension method 'Take' accepting a first argument of type 'IdGenerator' could be found (are you missing a using directive or an assembly reference?)

What's wrong with this code？

https://www.cnblogs.com/yushuo/p/9406906.html
Using Twitter's Snowflake to generator unique id,but there is a serious repetition

Creating ID for a specific point in past time

If I want my epoch time to be Jan 1st, 2018. And I need the ID that'd be generated for Jan 24, 2018 00:00.

var ts = new MockTimeSource(24, TimeSpan.FromDays(1), new DateTime(2018, 1, 1, 0, 0, 0, DateTimeKind.Utc));
var idGenerator = new IdGenerator(0, new IdGeneratorOptions(timeSource: ts));
var id = idGenerator.CreateId()

Is that the right way to get the id for some date in the past?

var epochDT = new DateTime(2018, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var msgDT = new DateTime(2018, 1, 24, 0, 0, 0, DateTimeKind.Utc);
TimeSpan diff = msgDT - epochDT;
var ts = new MockTimeSource((long)diff.TotalMilliseconds, TimeSpan.FromMilliseconds(1), new DateTime(2018, 1, 1, 0, 0, 0, DateTimeKind.Utc));
var idGenerator = new IdGenerator(0, new IdGeneratorOptions(timeSource: ts));

var ts2 = new MockTimeSource(23, TimeSpan.FromDays(1), new DateTime(2018, 1, 1, 0, 0, 0, DateTimeKind.Utc));
var idGenerator2 = new IdGenerator(0, new IdGeneratorOptions(timeSource: ts2));

For the example above results for ID are very different.

"id1": 833492090, - From ms
"id2": 96468992 - From days

What am I missing that causes such difference?

SQL Server equivalent possible?

Hello and thank you for providing this library!

I was curious if you had ever come across someone creating a SQL Server Function equivalent of this implementation. Are you aware of a technical challenge that might prevent attempting such a thing? I imagined the DB could reserve a GeneratorId for itself just like servers would and avoid any generation issues.

IIdGenerator is registered as a singleton, but duplicate IDs are still generated

//services.AddIdGen(258);
services.AddSingleton<IIdGenerator<long>>(c => new IdGenerator(258));

controller：

private readonly IdGenerator _idGen;
public LearnHistoryController( IdGenerator idGen)
{
            _idGen = idGen;
 }

action use code:

public ApiResponse GetSnowId(int count)
{
            return SuccessResult(_idGen.Take(count));
}

What went wrong? thanks