irepl does not processing remaining existing data objects if it processes a nonexistent one about irods_client_icommands

[tedgin]

Matthew, thanks for taking the time to explain how you perform asynchronous replication at scale. You've given me food for thought.

[mcast]

On Thu, Jan 27, 2022 at 06:57:46AM -0800, Tony Edgin wrote: Matthew's good suggestion is not a workaround for us. We replicate to a data center 1400 km away (27 ms RTT), The time overhead of a single irepl call is 1-2 s. [...]

We're fortunate that even our off-side replication is still local-ish, down a wide pipe. The overhead per irepl command does limit throughput on small files. I haven't tried multiple files per irepl, but I will put a comment in the code in case we need it some time - thank you. For our setup, going over `irepl -N3` doesn't give significant speed improvement. We also know with `iget` that large values of `-N` can degrade performance - apparently by seeking the disk heads around instead of streaming. irods/irods#5701 I sort my irepls by size and issue them from both ends of the queue, with some randomisation to avoid synchronising start times. Then I limit the total number of job slots (to avoid pounding the iCAT) and number of large jobs (to avoid pounding the link). Keep well, -- Matthew (I'm away on Fridays)

…

-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

[mcast]

On Thu, Jan 20, 2022 at 01:08:53PM -0800, Tony Edgin wrote: When irepl is provided multiple data objects on its command line, [... some not existing ...] Before it processes the one that exists, it exits.

That does sound unhelpful! On the zone I work on, I only ever issue `irepl` for one file per command; but I do run multiple `irepl` or `itrim` in parallel.

[...] This behavior makes it hard to script bulk irepl operations. When a failure occurs, it's difficult to figure out which data objects weren't processed.

The approach I took is to scan the entire set of files, using `iquest` and partitioning into windows of `DATA_ID` to bring about 100k files (with replica info) per window. Then process that output to generate a list of operations within the window, to bring those files into the desired state. This has implications for the resource tree structure, in that we have to have two roots ("green" being on-site, "red" was nominally off-site but now mostly just... slightly further away). If you're running client-side `irepl` instead of using rules, I presume you have a similar arrangement? I know that by doing this I'm not getting the benefit of "integrated Rules" but what I have instead is more flexibility and visibility into the state of replication. At 10E6 files / 5.2 PiB we're not regretting this decision yet. 8-) iquest runs are serial: total 18 minutes (157q) to construct "windows" by binary chop + 20 minutes (75q) to list replicas for all windows. The first half of this is static enough to re-do once a week and most of the iquests could run in parallel, but there hasn't been a need for that. I realise the above isn't going to help get this bug fixed, but hopefully it helps you evaluate paths from "where you are" to "where you want to be". Keep well, -- Matthew

…

-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

[tedgin]

Matthew's good suggestion is not a workaround for us. We replicate to a data center 1400 km away (27 ms RTT), The time overhead of a single irepl call is 1-2 s. We typically replicate 20 - 30 TiB per week of files with sizes ranging from 0 B - 1 TiB with a median of ~20 kiB, so we primarily replicate smalll files. For a few hundred thousand small files, passing only one file at a time to irepl would add several days to the time it takes to replicate the data. We currently bin our files by size. For files under 100 MiB, we make 16 irepl calls in parallel passing in 512 files per call, as file size increases, we reduce the number of files passed to each irepl, and we reduce the number of irepl calls made at time. Over 100 MiB, multi-threaded transfer kicks in. We try to keep the total number of transfer threads to at least 16. We can usually complete a weeks worth of replication in 2 - 3 days.