Dataset Updating Plan about ipfs-geoip

[Kubuxu]

Sorry (got a lot on my head).
The best structure for it would be ipld based radix tree, but it wouldn't be easy to implement.
We can do away with something simpler (pseudo radix tree). I will write something more on that tomorrow.

[lidel]

Figuring out new data structures will be time consuming, and given limited bandwidth I propose we simply convert the new CSV dataset to the format of old CVS source files:

ipfs-geoip/src/generate/index.js

Lines 15 to 22 in 00f72f9

	// All data is stored in an ipfs folder called data
	// this is the hash of that folder. It includes three files
	//
	// data
	// |- blocks.csv
	// |- countries.csv
	// |- locations.csv
	const DATA_HASH = 'QmTMh5Q1CnB9jV774aKCvPSqibwDy9sJmo7BCThD5f1oY3'

That way ipfs-geoip does not need to be rewritten, and we can at least update IPv4 info.

[hacdias]

Also cc @Kubuxu

[Kubuxu]

@hacdias how many entries there are in IPv6 dataset?

[hacdias]

@Kubuxu about 431 990 for the IPv6 dataset and 3 252 943 for the IPv4 one.

[hacdias]

Pinging @lidel and @Kubuxu 😅

[hacdias]

Thanks! Will be awaiting!

[hacdias]

Not accounting for IPv6 addresses, this is my current idea and plan. I am thinking of keeping the same binary tree structure as we have now, but I do have some doubts and questions to be answered. Please let me know what you think:

1. We have locations and IP blocks. There are ~120 000 location and ~3 115 100 IPv4 blocks, which means the relationship between locations and IP blocks is one to many.
2. Taking the last point in account, I thought about separating both things:
   1. Create a tree with the locations. It should be a really simple tree whose main purpose is to be pinned on the gateways and the locations should be CID addressed.
      1. Is there any limit of IPLD Links? Is it possible to just have one object with links to all locations and pin that?
   2. Create a second tree with the blocks. It is a binary tree as we have now. We address the location by their hash so we just need to get the object of the location after getting the rest of the information about a certain IP.
      1. We have 32 children now and I am thinking about increasing to 64.
      2. We have a lot of IPv4 blocks and this is gonna take tons of time to generate.

Feedback? @lidel @Kubuxu

[lidel]

Sadly I had no bandwidth, this needs another set of eyes.