Migrate existing home dir data to staging about dandi-hub

[kabilar]

• Build migration script
• Run migration during downtime (when DoEKS deployment is pushed to production)

[asmacdo]

Option 1: Migration script
This has been more complex than I expected. The VPCs are isiolated from one another. To connect 2 VPCs, we use a "peering connection" https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html

Peering connections complexity:

• each VPC
  • local DNS
  • AWS DNS
• Security Groups
  • configuration must be changed in many objects, and they must be changed to complement one another
  • the dandihub.k8s.local (current ansible deployment) and eks-hub-staging (new) are configured differently.

Option 2: AWS DataSync
https://aws.amazon.com/datasync/
$0.0125 /GB
$0.0125 * 1024 * 12 = $158 per 12TB sync

Supposedly simpler and easier to automate.

[asmacdo]

It was tricky getting DataSync to connect, but what finally seemed to work was:

DataSync Task needs 2 Security Groups (SG)

• source location + SG for EFS1 in VPC1
• destination location + SG for EFS2 in VPC2

Each Security Group needs NFS SG Rules (8 total) granting:

1. Inbound port 2029 to itself
2. Outbound port 2029 to itself
3. Inbound port 2029 to the other SG
4. Outbound port 2029 to the other SG

[asmacdo]

So far so good, but I want to investigate the 70,000 (out of 6 million) files that were skipped (seems like all at the very beginning, maybe dotfile-type stuff?

On the next run ill make an s3 bucket for so we can get the logs.

[asmacdo]

Also on the next run, (as discussed in dandi standup) we'll change config from "no delete" so if any files are removed from production, they will also be removed from staging.

[kabilar]

Thanks for the update, Austin. Glad to hear that it is coming together.