Move away from CI with conda-build about cookiecutter-cms

[richardjgowers]

@jchodera I've only used conda-build periodically to build packages for either conda-forge or private servers, but each time it's felt like I've had to relearn how things like compilers etc are defined as it's continually tweaked/improved upstream*. I'm imagining (maybe wrong) that if I used conda-build for my travis setup it would occasionally break my CI. This proposed approach seems like it would randomly break CI less often.

*(This is where conda-forge is nice as the updating recipes is done by a bot).

[jchodera]

I'd love to see what an example of this might look like!

[dgasmith]

Conda-build (current)

    # Create test environment for package
  - conda create -n test python=$PYTHON_VER pip pytest pytest-cov
  - conda activate test

    # Install pip only modules
  - pip install codecov

    # Build and install package
  - conda build --python=$PYTHON_VER devtools/conda-recipe
  - conda install --use-local repo_name

meta.yaml

package:
  name: qcfractal
  version: "0.0.0" # Must manually bump

source:
  path: ../..

build:
  number: 0 # Must manually bump
  script: python setup.py install --single-version-externally-managed --record record.txt


requirements:
  build:
    - python
    - setuptools

  run:
    - python
    - numpy 
    - scipy
    - ...

test:
  requires:
    - python
    - pytest
  commands:
    - py.test --pyargs package
  imports:
    - package

about:
  home: https://github.com/MolSSI/QCFractal
  license: BSD-3-Clause License

New developers are required to pip install -e . to obtain all depends from pip or guess depends in a conda install ... or run a full conda-build to get up to speed.

Conda install explicit list

  - conda install python=$PYTHON_VER numpy scipy ...
  - python setup.py develop --no-deps

Conda environment

  - python devtools/scripts/conda_env.py -n=test -p=$PYTHON_VER devtools/conda-envs/base.yaml
  - python setup.py develop --no-deps

base.yaml

name: test
channels:
  - conda-forge
dependencies:
  - numpy
  - scipy
  - ...

New developers can also just run python devtools/scripts/conda_env.py -n=test -p=3.6 devtools/conda-envs/base.yaml to get a full development environment.

---

Just from a complexity standpoint, the last two options are much easier to teach than conda-build IMO.

[jchodera]

This does look very appealing!

[richardjgowers]

My 2 cents - Using conda-build is confusing (partly because it's continually improved) so having to repeatedly use it seems like a bad experience. Conversely using conda-forge has been great for the projects I've deployed there. So this seems like a good idea

The dependency duplication is annoying (but allows different install routes), if there's a solution to this I'd be interested!

[jchodera]

My 2 cents - Using conda-build is confusing (partly because it's continually improved) so having to repeatedly use it seems like a bad experience.

Just to clarify: Do you mean having to tweak things occasionally to keep the travis tests up-to-date is problematic?

My understanding and experience with the http://omnia.md is that maintaining a whole conda-forge like build infrastructure does require frustrating periodic updates, but that maintaining a conda-build based travis testing system does not require tweaking at all.

The dependency duplication is annoying (but allows different install routes), if there's a solution to this I'd be interested!

The dependencies could be extracted from the conda recipe's meta.yaml by taking a union of all dependencies listed in the recipe. That would also catch problems, and prevent situations where it would be impossible to package via conda-forge because dependencies are required from multiple channels or pip.

[dgasmith]

@richardjgowers your experience matches my own.

Maintaining conda-build based build system does require tweaks as conda-build is updated sometimes these are small, sometimes large. Sometimes conda-build is updated with a bug in the release, and everything fails randomly for a short period.

I would say the inability to pull from pip or other conda channels is negative not positive. There are plenty of cases where you want to pull nightly builds, specific GitHub hashes, or testing tools not on conda (codecov only appeared there very recently) during testing, but not during the build.

[jchodera]

Maintaining conda-build based build system does require tweaks as conda-build is updated sometimes these are small, sometimes large. Sometimes conda-build is updated with a bug in the release, and everything fails randomly for a short period.

That hasn't happened for quite some time now, and the CLI changes so rarely that I think these concerns are minimal. They have been good about not breaking the YAML syntax for some time now, though they often add to it in a way that does not break things. It's only the internal Python API that presents problems.

I would say the inability to pull from pip or other conda channels is negative not positive. There are plenty of cases where you want to pull nightly builds, specific GitHub hashes, or testing tools not on conda (codecov only appeared there very recently) during testing, but not during the build.

I still think that letting users mix pip and multiple conda channels is going to lead to packages that are hard to package and share, which is inimical to the idea of developing tools to aid the community in being more collaborative. We want to make it as easy as possible for people to develop, share, and reuse software components to foster collaboration.

[jchodera]

In particular, we're almost to the point where we can all say "hey, conda-forge is a great way to share our tools as part of a collaborative CMS community---let's try to use that" and make it as easy as possible for people to do this. I think we could either push things the right way and realize that future, or let more entropy creep in and be fighting for many more years to get something like a coherent, collaborative CMS community that can share or build on tools.

[loriab]

Sorry, I didn't see this when it was posted (now watching repo). Some unorganized thoughts having read through the issue.

• I use conda-build all the time, love the meta.yaml syntax, and I still never considered using c-b in travis.yaml, so while I appreciate that the current c-b+CI route (1) keeps the recipe up-to-date and (2) keeps the project ready for distribution, I've no objection to dropping c-b in CI and letting CI focus on testing the current repo master against some set of obtainable or buildable dependencies.
• Between "Conda install explicit list" and "Conda environment", I tend to prefer the former, as they're more flexible for scrabbling together just the right deps for the current state of master (e.g., from different channels, pip, installing a metapackage then removing one invalid pkg) as @dgasmith mentioned. However, I've used the "Conda environment" setup along the MolSSI model for a few projects (albeit the simpler more-py kind), and it's worked well. (Though this route does involve more pip when all packages aren't in the same channel.)
• to be continued

[Lnaden]

There are a few of things to have to balance here:

• The cookiecutter's users will likely be those with minimal CI and deployment expertise, or at least should be designed to target those users
• Learning Conda-build and CI at the same time can be tricky, especially if something goes wrong and debugging is required
• We want users to practice both CI testing and some form of packaging
  • BUT I think we do not want users deploying their package themselves in isolation. So we should push them to do something, either Pypi, Conda-forge, or Omnia
• On the last point, we want to offload the work of someone managing the deployment off the dev themselves. The dev should make sure their package can be deployed, but actual deployment should be handled by some other person/org.

the CLI changes so rarely that I think these concerns are minimal.

The CLI has not changed, that for the most part is true

They have been good about not breaking the YAML syntax for some time now, though they often add to it in a way that does not break things. It's only the internal Python API that presents problems.

This is a problem, but the issue I am finding is most of these changes sorely lack documentation (e.g. the difference between build, host, and run sections), and its less intuitive for a user to figure out what they should do for the meta.yaml file than for the env file.

As for the time factor. If you do a conda-build + install + testing on the same job, Conda has to resolve the dependencies at least 3 times: 1 for build/host, one for run, one for install. It can rely on the downloaded packages from the previous resolutions, but its still a slower step. For example, I tested YANK with conda-build vs setting up an env. + install (no tests). It was 13 min (c-b) vs 9 min (env). That may not seem like much, but when you are just trying to up and test on CI with limited time, it can add up.

I ran the same timing test on the output of the cookiecutter as it stands, 1 min (cb) vs 20 sec (env) for comparison.

Possible Plan of attack

• Set up the CI tests to use the Environment system and run tests
• Set up the CI to allow failures for building (conda-build or pip setup.py sdist/bdist) so those run in parallel just to make sure it can be packaged
  • Both Travis and Apveyor have this ability now. Personal experience is mixed but works for Travis, don't know about AppVeyor
• Recommend people use either Pypi or some Conda-based solution for proper distribution, i.e. let someone else handle the formal distribution
• Point users to the distribution solutions we recommend

Thoughts?

[loriab]

• As long as dealing with decently maintained python-only deps where cookiecutter project places no version constraints on the deps, mixing different conda channels and pip should be fairly robust. It's when constructing a dev/testing environment requires a consistent solution that pip becomes more dangerous and conda has advantages.
• The main danger to mixing multiple conda channels or pip, though, is compiled dependencies. I can definitely see forbidding pip for compiled deps. Rather than constraining the cookiecutter user into a single channel, it makes better sense to me to place the burden further up and encourage channels to be binary compatible with defaults. c-f and defaults compatibility is well on its way. and psi4 and defaults problems are an anomaly and should be reported to me. I haven't tried other channels lately.
• summary: source/install/release are so close in python that I agree it's tempting to do all phases at once. But I think it's better to relax a bit and let CI do CI, not check reproducible packaging as well. Can help speed development. I agree multiple channels is entropy. But putting off that constraint to the release (not development CI) stage and then strongly encouraging deps from a small number of binary compatible channels (perhaps including the users' own if they have a small ecosys of in-dev cookiecutters not ready for release) can achieve the "coherent, collaborative CMS community" that we all want without giving up emergency control.
• end of comments