Move Mongo DB to Azure about kernelci-project

[gctucker]

@mgalka How much progress has been made on this already?

[mgalka]

@mgalka How much progress has been made on this already?

Afaik, the issue with Azure accounts/credentials is being sorted out. While it's not quite there yet there was no real possibility to move MongoDB from the current location. Some intermittent actions have been taken.
Amount of RAM has been increased on kernel.org and the size of cache for MongoDB has been adjusted so it can handle the load at a right speed and not consume all the memory.
At the moment no OOM errors are observed.

[gctucker]

Right, but nothing really related to Atlas then?

I did have a look a while ago but with a free account it's hard to get some idea of how well it would work for KernelCI. One thing I've found is there are quite a few high-quality online tools provided by Atlas that can't be used with standalone MongoDB instances. For example, generating graphs and reports based on data directly queried in the database. Also I think it makes replication or load balancing a lot easier.

[gctucker]

@mgalka If there are still issues with the current MongoDB instance we have on a VM then we should create issues for that. This issue is just about creating a MongoDB instance in the cloud (Atlas...), not fixing our current instance.

[mgalka]

No, nothing really related to Atlas has been done. I simply still don't have access to the Azure admin panel and I believe the issue is still in progress. I guess it'd make sense to actually create a GitHub issue to track it and make a dependency on it here.

I thought that an update about current state of the MongoDB server is not entirely out of place as the issue description clearly states what was the reason behind the idea of moving MongoDB to Atlas. At the moment we're not observing issues with MongoDB being unresponsive. I'll take care of creating separate issues if we do.

[gctucker]

I thought that an update about current state of the MongoDB server is not entirely out of place

It's not out of place, it just doesn't tell whether any investigation has been done into moving Mongo DB to Azure / Atlas.

[gctucker]

Maybe the title of this issue should be changed and we should then close if there's no problem any more, and a new one should be created about investigating what it would take to move Mongo DB to Azure / Atlas.

[VinceHillier]

I put together a rough price list based on the current size of mongodb, it can be found here:

https://docs.google.com/spreadsheets/d/1J6PjztPgBVQq3B-8CKmoFESC0KIAug2zZAEC3zcR8sU/edit?usp=sharing

Mongo has not responded to me since November 9th on their custom Atlas pricing.

We can also reduce the size of the production deployment to use a lesser Cluster Tier. Click on 'View Piricng' under dedicated, then client the Azure tab: https://www.mongodb.com/pricing

[gctucker]

It would also be worth evaluating how much it would cost with just 3 months of data in Mongo, and everything older would be moved to an archive which we could host on a VM or with a cheaper Cloud solution.

[gctucker]

I think we can close this now, we're starting to get prepared for deprecating the old system and the new API will be using a new database which might be deployed directly in Atlas. Also we've solved the issue of mongod crashing by adding more RAM to the VM.