Check clean up of EBS volumes about dandi-hub

[satra]

Is the /shared mount the same type of single-zone infrequent-access EFS?

yes

[asmacdo]

Is the /shared mount the same type of single-zone infrequent-access EFS?

yes

https://github.com/dandi/dandi-hub/blob/do-eks/helm/jupyterhub/dandihub.yaml#L183-L199

[asmacdo]

I deleted old staging-volumes that are no longer attached to resources. Next time we bring the whole thing down and backup we need to check if this is still happening, leaving open.

[asmacdo]

Not sure if I understand correctly-- does this mean we do not need persistent volumes for the users?

[satra]

in the current hub the user persistent volumes come from EFS not EBS. i believe EBS is more expensive than EFS.

[CodyCBakerPhD]

i believe EBS is more expensive than EFS.

EFS is more expensive in terms of amount per unit time (assuming standard billing at $0.30/GB-month), but is only billed for total instantaneous usage over time

EBS is technically cheaper and has a lot more options for optimizing I/O speeds (let's just use $0.08/GB-month as baseline), but it has to be pre-allocated at a certain size on spawn of the instance or mount of the volume (I don't know what/how the Hub handles that) and you get billed for that full amount regardless of how much disk space the user actually utilizes on the volume over that time

So which is more expensive depends on (i) Hub configuration settings and (ii) user behavior

The way Luiz and I usually answer the question of which is more expensive for a non-theoretical application is to look at the billing information after usage, which delivers total summary amounts for each approach - is that available in this case?

[satra]

@CodyCBakerPhD - just to make sure we are comparing the costs as it stands, in dandi hub right now, we use single zone, infrequent-access for the EFS volume, which ends up being lower than EBS.

[satra]

however EBS can have quota's since K8s can provision them on the fly and attach to a pod. while EFS, currently cannot have quota's at least in the traditional sense. one could technically run an NFS service to then map EFS with quotas to the pods.

[CodyCBakerPhD]

just to make sure we are comparing the costs as it stands, in dandi hub right now, we use single zone, infrequent-access for the EFS volume, which ends up being lower than EBS.

Ahh OK thanks for clarifying. Single zone definitely makes sense

Seems the biggest difference between the types as advertised is latency and throughput speeds. But I haven't run benchmarks for read/write speeds between standard and infrequent EFS, but hope to get answers on that sometime this year as a part of the NWB Benchmark projects

Is the /shared mount the same type of single-zone infrequent-access EFS?