streamer file cleanup in Tier0 about t0

[gowdy]

gowdy: I was pretty sure there was no GC in EOS.

I asked the EOS developer about it. He is interested in adding a feature to expire files after a certain amount of time. He wondered if empty directories should also be cleaned up, I thought yes.

I've asked for a times cale as I'm not sure what it would be.

[gowdy]

gowdy: So they expect it to be available at the start of February. We probably shouldn't expect it in production before March.

[gowdy]

gowdy: When this was brought up at a meeting that didn't have the developer but the operations team they were very negative about having GC in EOS... should I bring it up again with the developer? Or should we do the clean up ourselves like the EOS Ops team would like?

[hufnagel]

hufnagel: At the moment I assume we'll have to delete files ourself, at least the temporary files that are produced in the Tie0 For streamer files on a 30 day cycle this would be too long to track from within the Tie0though, so we would have to handle this externally.

Something like deleting older files every few weeks or every other month or so. For this use case, an automatic file expiration at the EOS level would probably still be useful, although I see how support for something like this can be dangerous. Maybe a cron job that deletes files older than 30 days would be sufficient.

[hufnagel]

hufnagel: The real question is, how hard wold it be to run such a cron job against an EOS directory. I am pretty sure the EOS operations team is also not going to be too happy about us running a daily 'find dir -ctime' against an EOS directory with millions of files in it. They can't have it both ways though...

[hufnagel]

hufnagel: Questionable if we ever do this in an automated fashion

[germanfgv]

This script has already been implemented and we've been using it for years.