Comments (12)
@jchodera I've only used conda-build periodically to build packages for either conda-forge or private servers, but each time it's felt like I've had to relearn how things like compilers etc are defined as it's continually tweaked/improved upstream*. I'm imagining (maybe wrong) that if I used conda-build for my travis setup it would occasionally break my CI. This proposed approach seems like it would randomly break CI less often.
*(This is where conda-forge is nice as the updating recipes is done by a bot).
from cookiecutter-cms.
I'd love to see what an example of this might look like!
from cookiecutter-cms.
Conda-build (current)
# Create test environment for package
- conda create -n test python=$PYTHON_VER pip pytest pytest-cov
- conda activate test
# Install pip only modules
- pip install codecov
# Build and install package
- conda build --python=$PYTHON_VER devtools/conda-recipe
- conda install --use-local repo_name
meta.yaml
package:
name: qcfractal
version: "0.0.0" # Must manually bump
source:
path: ../..
build:
number: 0 # Must manually bump
script: python setup.py install --single-version-externally-managed --record record.txt
requirements:
build:
- python
- setuptools
run:
- python
- numpy
- scipy
- ...
test:
requires:
- python
- pytest
commands:
- py.test --pyargs package
imports:
- package
about:
home: https://github.com/MolSSI/QCFractal
license: BSD-3-Clause License
New developers are required to pip install -e .
to obtain all depends from pip
or guess depends in a conda install ...
or run a full conda-build
to get up to speed.
Conda install explicit list
- conda install python=$PYTHON_VER numpy scipy ...
- python setup.py develop --no-deps
Conda environment
- python devtools/scripts/conda_env.py -n=test -p=$PYTHON_VER devtools/conda-envs/base.yaml
- python setup.py develop --no-deps
base.yaml
name: test
channels:
- conda-forge
dependencies:
- numpy
- scipy
- ...
New developers can also just run python devtools/scripts/conda_env.py -n=test -p=3.6 devtools/conda-envs/base.yaml
to get a full development environment.
Just from a complexity standpoint, the last two options are much easier to teach than conda-build IMO.
from cookiecutter-cms.
This does look very appealing!
from cookiecutter-cms.
My 2 cents - Using conda-build is confusing (partly because it's continually improved) so having to repeatedly use it seems like a bad experience. Conversely using conda-forge has been great for the projects I've deployed there. So this seems like a good idea
The dependency duplication is annoying (but allows different install routes), if there's a solution to this I'd be interested!
from cookiecutter-cms.
My 2 cents - Using conda-build is confusing (partly because it's continually improved) so having to repeatedly use it seems like a bad experience.
Just to clarify: Do you mean having to tweak things occasionally to keep the travis tests up-to-date is problematic?
My understanding and experience with the http://omnia.md is that maintaining a whole conda-forge like build infrastructure does require frustrating periodic updates, but that maintaining a conda-build based travis testing system does not require tweaking at all.
The dependency duplication is annoying (but allows different install routes), if there's a solution to this I'd be interested!
The dependencies could be extracted from the conda recipe's meta.yaml
by taking a union of all dependencies listed in the recipe. That would also catch problems, and prevent situations where it would be impossible to package via conda-forge
because dependencies are required from multiple channels or pip
.
from cookiecutter-cms.
@richardjgowers your experience matches my own.
Maintaining conda-build
based build system does require tweaks as conda-build
is updated sometimes these are small, sometimes large. Sometimes conda-build
is updated with a bug in the release, and everything fails randomly for a short period.
I would say the inability to pull from pip or other conda channels is negative not positive. There are plenty of cases where you want to pull nightly builds, specific GitHub hashes, or testing tools not on conda (codecov only appeared there very recently) during testing, but not during the build.
from cookiecutter-cms.
Maintaining conda-build based build system does require tweaks as conda-build is updated sometimes these are small, sometimes large. Sometimes conda-build is updated with a bug in the release, and everything fails randomly for a short period.
That hasn't happened for quite some time now, and the CLI changes so rarely that I think these concerns are minimal. They have been good about not breaking the YAML syntax for some time now, though they often add to it in a way that does not break things. It's only the internal Python API that presents problems.
I would say the inability to pull from pip or other conda channels is negative not positive. There are plenty of cases where you want to pull nightly builds, specific GitHub hashes, or testing tools not on conda (codecov only appeared there very recently) during testing, but not during the build.
I still think that letting users mix pip
and multiple conda
channels is going to lead to packages that are hard to package and share, which is inimical to the idea of developing tools to aid the community in being more collaborative. We want to make it as easy as possible for people to develop, share, and reuse software components to foster collaboration.
from cookiecutter-cms.
In particular, we're almost to the point where we can all say "hey, conda-forge
is a great way to share our tools as part of a collaborative CMS community---let's try to use that" and make it as easy as possible for people to do this. I think we could either push things the right way and realize that future, or let more entropy creep in and be fighting for many more years to get something like a coherent, collaborative CMS community that can share or build on tools.
from cookiecutter-cms.
Sorry, I didn't see this when it was posted (now watching repo). Some unorganized thoughts having read through the issue.
- I use conda-build all the time, love the
meta.yaml
syntax, and I still never considered using c-b intravis.yaml
, so while I appreciate that the current c-b+CI route (1) keeps the recipe up-to-date and (2) keeps the project ready for distribution, I've no objection to dropping c-b in CI and letting CI focus on testing the current repo master against some set of obtainable or buildable dependencies. - Between "Conda install explicit list" and "Conda environment", I tend to prefer the former, as they're more flexible for scrabbling together just the right deps for the current state of master (e.g., from different channels, pip, installing a metapackage then removing one invalid pkg) as @dgasmith mentioned. However, I've used the "Conda environment" setup along the MolSSI model for a few projects (albeit the simpler more-py kind), and it's worked well. (Though this route does involve more
pip
when all packages aren't in the same channel.) - to be continued
from cookiecutter-cms.
There are a few of things to have to balance here:
- The cookiecutter's users will likely be those with minimal CI and deployment expertise, or at least should be designed to target those users
- Learning Conda-build and CI at the same time can be tricky, especially if something goes wrong and debugging is required
- We want users to practice both CI testing and some form of packaging
- BUT I think we do not want users deploying their package themselves in isolation. So we should push them to do something, either Pypi, Conda-forge, or Omnia
- On the last point, we want to offload the work of someone managing the deployment off the dev themselves. The dev should make sure their package can be deployed, but actual deployment should be handled by some other person/org.
the CLI changes so rarely that I think these concerns are minimal.
The CLI has not changed, that for the most part is true
They have been good about not breaking the YAML syntax for some time now, though they often add to it in a way that does not break things. It's only the internal Python API that presents problems.
This is a problem, but the issue I am finding is most of these changes sorely lack documentation (e.g. the difference between build
, host
, and run
sections), and its less intuitive for a user to figure out what they should do for the meta.yaml
file than for the env file.
As for the time factor. If you do a conda-build + install + testing
on the same job, Conda has to resolve the dependencies at least 3 times: 1 for build/host, one for run, one for install. It can rely on the downloaded packages from the previous resolutions, but its still a slower step. For example, I tested YANK with conda-build
vs setting up an env. + install (no tests). It was 13 min (c-b) vs 9 min (env). That may not seem like much, but when you are just trying to up and test on CI with limited time, it can add up.
I ran the same timing test on the output of the cookiecutter as it stands, 1 min (cb) vs 20 sec (env) for comparison.
Possible Plan of attack
- Set up the CI tests to use the Environment system and run tests
- Set up the CI to allow failures for building (
conda-build
orpip setup.py sdist/bdist
) so those run in parallel just to make sure it can be packaged- Both Travis and Apveyor have this ability now. Personal experience is mixed but works for Travis, don't know about AppVeyor
- Recommend people use either Pypi or some Conda-based solution for proper distribution, i.e. let someone else handle the formal distribution
- Point users to the distribution solutions we recommend
Thoughts?
from cookiecutter-cms.
- As long as dealing with decently maintained python-only deps where cookiecutter project places no version constraints on the deps, mixing different conda channels and pip should be fairly robust. It's when constructing a dev/testing environment requires a consistent solution that
pip
becomes more dangerous andconda
has advantages. - The main danger to mixing multiple conda channels or
pip
, though, is compiled dependencies. I can definitely see forbiddingpip
for compiled deps. Rather than constraining the cookiecutter user into a single channel, it makes better sense to me to place the burden further up and encourage channels to be binary compatible with defaults.c-f
anddefaults
compatibility is well on its way. andpsi4
anddefaults
problems are an anomaly and should be reported to me. I haven't tried other channels lately. - summary: source/install/release are so close in python that I agree it's tempting to do all phases at once. But I think it's better to relax a bit and let CI do CI, not check reproducible packaging as well. Can help speed development. I agree multiple channels is entropy. But putting off that constraint to the release (not development CI) stage and then strongly encouraging deps from a small number of binary compatible channels (perhaps including the users' own if they have a small ecosys of in-dev cookiecutters not ready for release) can achieve the "coherent, collaborative CMS community" that we all want without giving up emergency control.
- end of comments
from cookiecutter-cms.
Related Issues (20)
- Switch to mamba + miniforge = mambaforge to speed up CI? HOT 14
- Revisit using src/package_name directory instead of package_name directory HOT 6
- [ALERT TO USERS] Travis CI Security Breach - REMOVE Travis - Cycle Secure Env Vars HOT 3
- Test API Issue Creation
- Where to write the dependency? HOT 4
- Tests folder outside Python package? HOT 1
- Add pre-commit.ci
- PEP 517/518 (pyproject.toml) HOT 8
- Default chron schedule for GHA
- Allow package name to be different from repository name HOT 1
- Update default branch to main instead of master HOT 3
- Remove default channel from conda-forge setting HOT 2
- cruft remains in setup.cfg HOT 2
- Use native config files for coverage, flake8, and yapf HOT 1
- Missing versioneer file?
- Deprecated use of pyyaml's yaml.load function without a Loader argument
- Multiple errors while using cookiecutter for initiating a new repository HOT 2
- `provision-with-micromamba` is deprecated HOT 3
- Python versions falling behind NEP29/SPEC0
- Missing version information in generated Sphinx docs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cookiecutter-cms.