Specifying multiple platforms about conda-mirror

[ericdill]

It is not possible right now with the conda-mirror CLI. To have this functionality internally we wrap the conda-mirror cli with this:

for platform in linux-64 win-64 osx-64;
do
  source "/opt/continuum/anaconda/bin/activate" "/opt/envs/conda-mirror" && \
    conda-mirror \
      --config /opt/tools/conda-mirror-config.yaml \
      --upstream-channel https://conda.anaconda.org/conda-forge \
      --target-directory /opt/data/conda/www/upstream-conda-forge \
      --num-threads 7 \
      --platform $platform -vv 2>> /var/log/condaserver/mirror_conda_conda-forge.log
done

[jakirkham]

Are PRs welcome for this sort of functionality?

[ericdill]

Short answer, yes.

Long answer, I'm at a fork in the road for how to maintain conda-mirror long term. I think it needs some re-architecting. There are a handful of people that I know of that use conda-mirror in different ways. All want conda-mirror to support their use case (#48: rapid validation, #64: incremental initial sync, skipping validation: #62, and this issue). I totally understand the desire here, but this code base has already grown the concurrent validation appendage and other unwieldy appendages are looming on the horizon.

The way that we use conda-mirror internally is to use this open source version as the starting point and fold in additional functionality like hipchat integration and sending metrics to graphite, to call out a couple. I think that this is a model that will make conda-mirror generically usable for most people while minimizing the maintenance burden of the open-source version. As such, I've been wanting to find some time for a few months now to implement v2 of the mirror strictly as:

1. A library of reusable functions for interacting with anaconda.org
2. A reference implementation of a CLI to support mirroring internally
3. Drop the notion of parallel-validation and don't support anything more complicated than simple, serial download/validation.

I think this let's our internal implementation remain just that: An internal, mostly gross wrapping of conda-mirror that solves our internal needs while keeping the annoying parts of interacting with anaconda.org out in the open. I think that other people that use conda-mirror should adopt the same philosophy: Use conda-mirror for interacting with anaconda.org and make any local optimizations that are necessary for your specific infrastructure.

Now, with that said, it is unclear if/when I will ever have time to reimplement a simplified version of conda-mirror as I just described above 😢. So, in the mean time, you're welcome to submit a PR that adds something like --platform linux-64,win-64,osx-64 (or whatever API you were thinking) and I'll merge it once we agree on the functionality/implementation 😀