conda / grayskull

Grayskull :skull: - Recipe generator for Conda

Conda is a cross-platform, language-agnostic binary package manager. It is the package manager used by Anaconda installations, but it may be used for other systems as well. Conda makes environments first-class citizens, making it easy to create independent environments even for C libraries. Conda is written entirely in Python, and is BSD licensed open source.

Conda is enhanced by organizations, tools, and repositories created and managed by the amazing members of the conda community. Some of them can be found here.

Installation

Conda is a part of the Anaconda Distribution. Use Miniconda to bootstrap a minimal installation that only includes conda and its dependencies.

Updating conda

To update conda to the newest version, use the following command:

$ conda update -n base conda

Tip

It is possible that conda update does not install the newest version if the existing conda version is far behind the current release. In this case, updating needs to be done in stages.

For example, to update from conda 4.12 to conda 23.10.0, conda 22.11.1 needs to be installed first:

$ conda install -n base conda=22.11.1
$ conda update conda

Getting Started

If you install the Anaconda Distribution, you will already have hundreds of packages installed. You can see what packages are installed by running:

$ conda list

to see all the packages that are available, use:

$ conda search

and to install a package, use

$ conda install <package-name>

The real power of conda comes from its ability to manage environments. In conda, an environment can be thought of as a completely separate installation. Conda installs packages into environments efficiently using hard links by default when it is possible, so environments are space efficient, and take seconds to create.

The default environment, which conda itself is installed into, is called base. To create another environment, use the conda create command. For instance, to create an environment with PyTorch, you would run:

$ conda create --name ml-project pytorch

This creates an environment called ml-project with the latest version of PyTorch, and its dependencies.

We can now activate this environment:

$ conda activate ml-project

This puts the bin directory of the ml-project environment in the front of the PATH, and sets it as the default environment for all subsequent conda commands.

To go back to the base environment, use:

$ conda deactivate

Building Your Own Packages

You can easily build your own packages for conda, and upload them to anaconda.org, a free service for hosting packages for conda, as well as other package managers. To build a package, create a recipe. Package building documentation is available here. See AnacondaRecipes for the recipes that make up the Anaconda Distribution and defaults channel. Conda-forge and Bioconda are community-driven conda-based distributions.

To upload to anaconda.org, create an account. Then, install the anaconda-client and login:

$ conda install anaconda-client
$ anaconda login

Then, after you build your recipe:

$ conda build <recipe-dir>

you will be prompted to upload to anaconda.org.

To add your anaconda.org channel, or other's channels, to conda so that conda install will find and install their packages, run:

$ conda config --add channels https://conda.anaconda.org/username

(replacing username with the username of the person whose channel you want to add).

Getting Help

• Documentation
• Twitter
• Slack
• Bug Reports/Feature Requests
• Installer/Package Issues
• Discourse

Contributing

Contributions to conda are welcome. See the contributing documentation for instructions on setting up a development environment.

grayskull's Issues

Improve CLI

It would be nice to add terminal commands to be able to load and update some sections of the recipe.
Something like
--update all --load PATH
Or
--update requirements --load PATH
And add possibly to choose the version of the package
--version 5.1.2

Create custom Dumper for yaml

Add possibility to disable output

Add possibility to disable output

Add CI to run linters

Track different pkg names on PyPI and conda-forge

Track different pkg names on PyPI and conda-forge

For example:

• python-opencv is just opencv on conda-forge
• tables on PyPI is pytables on conda-forge (and there is a package on pypi with the name pytables which is not the same pytables on conda-forge)

cc: @ocefpaf

Enable pytest-xdist

Add support for setup.cfg

It would be nice to also add support for setup.cfg

Bug - adding tags to the deps

I found this problem when I was reviewing some recipes on staged-recipes for this one in particular zipp.

On the host requirements it added
setuptools_scm [toml] >=3.4.1
instead of just
````setuptools_scm >=3.4.1``

Grayskull should remove those [toml]

Other recipes to take a look

• telegram-send
• perspective-python

enable handling selectors

Per our gitter discussion, duplicate keys with selectors present special challenges.

Don't use deprecated SPDX identifiers

For eg: GPL-2.0 is deprecated, but there's a test

get_short_license_id("GPL 2.0") == "GPL-2.0"

Add support to fortran

Add support to fortran compilers

Improvements on requirements discory for host

It would be interesting to add a way to recover the modules loaded after the execution of the setup.py. So grayskull will be able to analyse all the modules loaded to install the package

Format output colors based on terminal configuration

Add option to drop sdist package in a specific folder

Add option (maybe --sdist-download-folder) to drop sdist package in a specific folder

cc: @ocefpaf

Cover corner cases for pkgs which need compilers

• cython
• pybind11

Add azure pipelines to run the tests

Remove comments from dependencies

Package to test:

• kubernetes_asyncio

cc: @ocefpaf

Packages and feedstocks we should test this

Pretty much any Python package in https://github.com/regro/cf-graph-countyfair/blob/master/status/could_use_help.json, bonus point if this creates a recipe that passes the CI!

• hypothesis (almost daily updates and high change of changing metadata)
• boto/botocore (one is tied to another and with the proper metadata boto's PR should fail until botocore is merged)
• gsw (compiler)
• matplotlib (messy outputs)
• pytest (lots of new syntax for package metadata

disable compilers when dependencies are extra

For the following package tornado_sqlalchemy_login there is a dependency for pybind11 but it is just an extra and in dev mode. However, grayskull is adding the compilers anyway. It should not add it

Generate PyPi recipes

Generate recipe from PyPi.

Recipes which are interesting to test with:

• pytest
• pandas
• matplotlib
• botocore
• boto
• numpy
• scipy
• altair
• vega
• gsw
• requests

Review the home in about

Try to get the home key from pypi, for some reason is always prioritizing the pypi webpage

Issue found when testing with a few recipes

• djangorestframework-xml had the wrong import test and license
• nbdime also got the wrong license

Are you getting the import test from the package name or the modules list?

Speed up generation of recipes for some packages

It needs investigation to understand why grayskull pypi web3 is taking about 1min to finish

Matplotlib-base rather than matplotlib as default

For most packages we can use matplotlib-base rather than matplotlib.

Create docs

• Create Wiki
• Create Docs
• Update README

Use ruamel yaml instead of PyYaml

That would be useful to add, modify or delete eol comments.
It would be interesting to create an interface in the base class to abstract the manipulation regarding delimiters and selectors

exclude modules in the same folder as setup.py from host

Grayskull needs to exclude modules from the same level as setup.py
one of the problems is because everything in the same level as setup.py will be seen by python as a module, any new .py file or any folder with a __init__.py

Recipes to test with this problem:

• bokeh : adding _setup_support to host section
• ipycytoscape : adding setupbase to host section

thanks @ocefpaf for reporting it

Add support for toml

Injection on disutils to be able to recover more metadata

That is important especially when the pypi metadata is missing

Give a better message when executing grayskull without any options

It was raised by @step21_gitlab
When executing just grayskull without any options it is raising an exception.

A proposal fix is to return the help when grayskull is called without any option.

Pragmmatically discover License type

Update license according to package and version. Possibility to search on tarballs, repo, and pypi metadata.

Code already created for conda-forge, it would be interesting to base in this implementation:
https://github.com/regro/cf-scripts/pull/637/files#diff-65ad1c9390fe432dca7ca211091e3287R101

Move the license module to a separated package

Fail if PyPi package does not contain sdist

The new skeleton should fail if the package does not contain sdist on PyPi.

We may try to get the source code from GitHub tarballs if possible.

Example of a package that just has wheels. (PyQt5)
https://pypi.org/project/PyQt5/#files

Generate Package on PyPi and Conda package

Exclude hidden folders when searching for the license

Exclude hidden folders when searching for the license

Add functional tests

Generate recipe from Cran

Generate recipe based on the Cran repository for R packages.
Bioconda already has its own implementation of skeleton. It would be interesting to discuss with them to adapt their code to this new structure.

Wrong recipes generated

as explained by @ocefpaf

• importlib-metadata
  • wrong license
  • added setuptool-scm twice
• keyring
  • added importlib-metadata twice
• clikit
  • wrong indentation
  • check selectors (it probably needs a parenthesis to group them)

Adding incorrect selectors

grayskull generated a recipe for requests which added the following dependency

- win-inet-pton                 # [win and py==27]

which came from pypi metadata

"win-inet-pton ; (sys_platform == \"win32\" and python_version == \"2.7\") and extra == 'socks'"

It needs a few improvements in the parser

Keep license file name from source distribution

I noticed when creating a recipe that the upstream repo named the license file LICENSE.md and grayskull made a copy called LICENSE. It might be useful to match the naming convention in the upstream repo so that when the license file is eventually included in the upstream repo, there is no need to make changes to the meta.yaml in addition to deleting the manually packaged copy.

Taking too long to generate nipy

Grayskull is taking too much time to generate the recipe for nipy
https://nipy.org/
https://pypi.org/project/nipy/

cc: @ocefpaf

Add tests for entry_points - PyPI

as suggested by @ocefpaf

For example in pytest recipe which has the following entry points

    - pytest = pytest:main
    - py.test = pytest:main

it would be nice to test the entry points adding in the test section the following commands:

test:
  commands:
    - pytest --help
    - py.test --help

Add pip check

Add pip check to the tests

Improve logging and user feedback

It is necessary to improve the logs and it would be interesting to add user feedback in a nice output format

add possibility to deal with multiple licenses

license_family is not generated or stubbed out

Many recipes are now being submitted to conda-forge without this field.

[Design] Remove coupling between AbstractRecipeModel and other classes

It would be interesting to use composition instead of inheritance.
For that purpose, it is possible to extract the whole AbstractRecipeModel to a normal class and after that, the PyPI class would use it as an internal object to be able to handle the recipe metadata.
The PyPI class should inherit just from a pure abstract class which will define the methods the interface (PyPI, CRAN, ...) should implement.

Missed mypy entrypoints

Mypy has 3 entry-points and the new recipe generated with grayskull missed those. Not sure why, investigating. (I used version 0.2.2.)

Wrong license discovered

• pytest-django: found BSD-4 and it should be 3
• humanize found the wrong license
• adlfs found BSD-1-Clause and it should be BSD-3-Clause

cc: @ocefpaf

Add entry_points