conda-forge / cctbx-base-feedstock

A conda-smithy repository for cctbx-base.

About cctbx-base-feedstock

Feedstock license: BSD-3-Clause

Home: https://github.com/cctbx/cctbx_project

Package license: BSD-3-Clause-LBNL AND BSD-3-Clause AND BSL-1.0 AND LGPL-2.0-only AND LGPL-2.1-only AND LGPL-3.0-only AND MIT AND LGPL-2.0-or-later WITH WxWindows-exception-3.1

Summary: The Computational Crystallography Toolbox

Development: https://github.com/cctbx/cctbx_project

Documentation: https://cctbx.github.io/

The Computational Crystallography Toolbox (cctbx) is being developed as the open source component of the Phenix system. The goal of the Phenix project is to advance automation of macromolecular structure determination. Phenix depends on the cctbx, but not vice versa. This hierarchical approach enforces a clean design as a reusable library. The cctbx is therefore also useful for small-molecule crystallography and even general scientific applications.

Current build status

Azure

Variant Status linux_64_numpy1.22python3.10.____cpython linux_64_numpy1.22python3.8.____cpython linux_64_numpy1.22python3.9.____cpython linux_64_numpy1.23python3.11.____cpython linux_aarch64_numpy1.22python3.10.____cpython linux_aarch64_numpy1.22python3.8.____cpython linux_aarch64_numpy1.22python3.9.____cpython linux_aarch64_numpy1.23python3.11.____cpython linux_ppc64le_numpy1.22python3.10.____cpython linux_ppc64le_numpy1.22python3.8.____cpython linux_ppc64le_numpy1.22python3.9.____cpython linux_ppc64le_numpy1.23python3.11.____cpython osx_64_numpy1.22python3.10.____cpython osx_64_numpy1.22python3.8.____cpython osx_64_numpy1.22python3.9.____cpython osx_64_numpy1.23python3.11.____cpython osx_arm64_numpy1.22python3.10.____cpython osx_arm64_numpy1.22python3.8.____cpython osx_arm64_numpy1.22python3.9.____cpython osx_arm64_numpy1.23python3.11.____cpython win_64_numpy1.22python3.10.____cpython win_64_numpy1.22python3.8.____cpython win_64_numpy1.22python3.9.____cpython win_64_numpy1.23python3.11.____cpython

Current release info

Name	Downloads	Version	Platforms

Installing cctbx-base

Installing cctbx-base from the conda-forge channel can be achieved by adding conda-forge to your channels with:

conda config --add channels conda-forge
conda config --set channel_priority strict

Once the conda-forge channel has been enabled, cctbx, cctbx-base can be installed with conda:

conda install cctbx cctbx-base

or with mamba:

mamba install cctbx cctbx-base

It is possible to list all of the versions of cctbx available on your platform with conda:

conda search cctbx --channel conda-forge

or with mamba:

mamba search cctbx --channel conda-forge

Alternatively, mamba repoquery may provide more information:

# Search all versions available on your platform:
mamba repoquery search cctbx --channel conda-forge

# List packages depending on `cctbx`:
mamba repoquery whoneeds cctbx --channel conda-forge

# List dependencies of `cctbx`:
mamba repoquery depends cctbx --channel conda-forge

About conda-forge

conda-forge is a community-led conda channel of installable packages. In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.

A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services. Thanks to the awesome service provided by Azure, GitHub, CircleCI, AppVeyor, Drone, and TravisCI it is possible to build and upload installable packages to the conda-forge anaconda.org channel for Linux, Windows and OSX respectively.

To manage the continuous integration and simplify feedstock maintenance conda-smithy has been developed. Using the conda-forge.yml within this repository, it is possible to re-render all of this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender.

For more information please check the conda-forge documentation.

Terminology

feedstock - the conda recipe (raw material), supporting scripts and CI configuration.

conda-smithy - the tool which helps orchestrate the feedstock. Its primary use is in the construction of the CI .yml files and simplify the management of many feedstocks.

conda-forge - the place where the feedstock and smithy live and work to produce the finished article (built conda distributions)

Updating cctbx-base-feedstock

If you would like to improve the cctbx-base recipe or build a new package version, please fork this repository and submit a PR. Upon submission, your changes will be run on the appropriate platforms to give the reviewer an opportunity to confirm that the changes result in a successful build. Once merged, the recipe will be re-built and uploaded automatically to the conda-forge channel, whereupon the built conda packages will be available for everybody to install and use from the conda-forge channel. Note that all branches in the conda-forge/cctbx-base-feedstock are immediately built and any created packages are uploaded, so PRs should be based on branches in forks and branches in the main repository should only be used to build distinct package versions.

In order to produce a uniquely identifiable distribution:

• If the version of a package is not being increased, please add or increase the build/number.
• If the version of a package is being increased, please remember to return the build/number back to 0.

Feedstock Maintainers

• @bkpoon
• @phyy-nx

cctbx-base-feedstock's Issues

flex ABI broken in conda-forge package

In a correctly working environment I expect the following output:

$ python
>>> from scitbx.array_family import flex                                                                                    
>>> assert (flex.complex_double() == None) is False
>>> flex.complex_double([None])
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    flex.complex_double([None])
TypeError: No registered converter was able to produce a C++ rvalue of type std::complex<double> from this Python object of type NoneType

However, with cctbx-base 2021.4 and python 3.8 in a freshly created conda environment on MacOS I get:

$ conda create -p $(pwd)/cctbx-conda-3.8 -c conda-forge --override cctbx-base python=3.8
$ conda activate $(pwd)/cctbx-conda-3.8
$ python
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 06:39:48) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from scitbx.array_family import flex            
>>> assert (flex.complex_double() == None) is False
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError
>>> flex.complex_double() == None
<scitbx_array_family_flex_ext.bool object at 0x1140681b0>
>>> flex.complex_double([None])
<scitbx_array_family_flex_ext.complex_double object at 0x10d940040>

I added these checks to cctbx in cctbx/cctbx_project#616. The pull request builds are currently running, but I suspect that they will pass, which is why I'm reporting this here.

Originally found in dials/dials#1001

Environment (conda list):

$ conda list
# packages in environment at /Users/wra62962/cctbx-conda-3.8:
#
# Name                    Version                   Build  Channel
biopython                 1.78             py38h96a0964_2    conda-forge
boost                     1.74.0           py38h692b87f_3    conda-forge
boost-cpp                 1.74.0               hff03dee_4    conda-forge
brotlipy                  0.7.0           py38h5406a74_1001    conda-forge
bzip2                     1.0.8                hc929b4f_4    conda-forge
ca-certificates           2020.12.5            h033912b_0    conda-forge
cctbx-base                2021.4           py38hfaf3399_0    conda-forge
certifi                   2020.12.5        py38h50d1736_1    conda-forge
cffi                      1.14.5           py38ha97d567_0    conda-forge
chardet                   4.0.0            py38h50d1736_1    conda-forge
conda                     4.10.1           py38h50d1736_0    conda-forge
conda-package-handling    1.7.3            py38h96a0964_0    conda-forge
cryptography              3.4.7            py38h1fa4640_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
freetype                  2.10.4               h4cff582_1    conda-forge
future                    0.18.2           py38h50d1736_3    conda-forge
icu                       68.1                 h74dc148_0    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
jbig                      2.1               h0d85af4_2003    conda-forge
jpeg                      9d                   hbcb3906_0    conda-forge
kiwisolver                1.3.1            py38hd9c93a9_1    conda-forge
lcms2                     2.12                 h577c468_0    conda-forge
lerc                      2.2.1                h046ec9c_0    conda-forge
libblas                   3.9.0                9_openblas    conda-forge
libcblas                  3.9.0                9_openblas    conda-forge
libcxx                    11.1.0               habf9029_0    conda-forge
libdeflate                1.7                  h35c211d_5    conda-forge
libffi                    3.3                  h046ec9c_2    conda-forge
libgfortran               5.0.0           9_3_0_h6c81a4c_22    conda-forge
libgfortran5              9.3.0               h6c81a4c_22    conda-forge
liblapack                 3.9.0                9_openblas    conda-forge
libopenblas               0.3.15          openmp_h5e1b9a4_1    conda-forge
libpng                    1.6.37               hb0a8c7a_2    conda-forge
libsvm                    325                  he49afe7_0    conda-forge
libtiff                   4.3.0                h1167814_1    conda-forge
libwebp-base              1.2.0                h0d85af4_2    conda-forge
llvm-openmp               11.1.0               hda6cdc1_1    conda-forge
lz4-c                     1.9.3                h046ec9c_0    conda-forge
matplotlib-base           3.4.2            py38h6152e83_0    conda-forge
mrcfile                   1.3.0              pyh44b312d_0    conda-forge
ncurses                   6.2                  h2e338ed_4    conda-forge
numpy                     1.20.3           py38had91d27_1    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                h6e7aa92_1    conda-forge
openssl                   1.1.1k               h0d85af4_0    conda-forge
pillow                    8.2.0            py38h83525de_1    conda-forge
pip                       21.1.2             pyhd8ed1ab_0    conda-forge
psutil                    5.8.0            py38h5406a74_1    conda-forge
pycosat                   0.6.3           py38h5406a74_1006    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pysocks                   1.7.1            py38h50d1736_3    conda-forge
python                    3.8.10          h3a889e0_1_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python.app                1.3              py38h5406a74_4    conda-forge
python_abi                3.8                      1_cp38    conda-forge
readline                  8.1                  h05e3726_0    conda-forge
reportlab                 3.5.67           py38hf6ac518_0    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
ruamel_yaml               0.15.80         py38hca655e8_1004    conda-forge
setuptools                49.6.0           py38h50d1736_3    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.35.5               h44b9ce1_0    conda-forge
tk                        8.6.10               hb0a8c7a_1    conda-forge
tornado                   6.1              py38h5406a74_1    conda-forge
tqdm                      4.61.0             pyhd8ed1ab_0    conda-forge
urllib3                   1.26.5             pyhd8ed1ab_0    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
yaml                      0.2.5                haf1e3a3_0    conda-forge
zlib                      1.2.11            h7795811_1010    conda-forge
zstd                      1.5.0                h582d3a0_0    conda-forge

Details about conda and system ( conda info ):

$ conda info
     active environment : base
    active env location : /Users/wra62962/cctbx-conda-3.8
            shell level : 1
       user config file : /Users/wra62962/.condarc
 populated config files : 
          conda version : 4.10.1
    conda-build version : not installed
         python version : 3.8.10.final.0
       virtual packages : __osx=10.14.6=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /Users/wra62962/cctbx-conda-3.8  (writable)
      conda av data dir : /Users/wra62962/cctbx-conda-3.8/etc/conda
  conda av metadata url : https://repo.anaconda.com/pkgs/main
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/wra62962/cctbx-conda-3.8/pkgs
                          /Users/wra62962/.conda/pkgs
       envs directories : /Users/wra62962/cctbx-conda-3.8/envs
                          /Users/wra62962/.conda/envs
               platform : osx-64
             user-agent : conda/4.10.1 requests/2.25.1 CPython/3.8.10 Darwin/18.7.0 OSX/10.14.6
                UID:GID : 503:20
             netrc file : None
           offline mode : False

@conda-forge-admin, rerender

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Environment (conda list):

$ conda list

Details about conda and system ( conda info ):

$ conda info

Can't libtbx configure on Windows

running in an empty build directory next to the usual modules/ checkouts with cctbx-base 2021.6 gives:

python ..\modules\cctbx_project\libtbx\configure.py scitbx iotbx cbflib

scitbx is already installed
Traceback (most recent call last):
  File "..\modules\cctbx_project\libtbx\configure.py", line 34, in <module>
    if not run():
  File "..\modules\cctbx_project\libtbx\configure.py", line 29, in run
    libtbx.env_config.cold_start(sys.argv)
  File "C:\DialsBuild\modules\cctbx_project\libtbx\env_config.py", line 3010, in cold_start
    env.process_args(pre_processed_args=pre_processed_args)
  File "C:\DialsBuild\modules\cctbx_project\libtbx\env_config.py", line 1046, in process_args
    assert self.path_utility.isfile()
AssertionError

a
print(abs(self.path_utility))
yields
d:\bld\cctbx-base_1625205021395\_h_env\Lib\site-packages\command_line\path_utility.py,
so it looks like this comes from the prebuilt package?

@conda-forge-admin, please update version

Additional comment:

No response

ccp4io/ccp4io_adaptbx present but not configured

>>> import libtbx.load_env
>>> libtbx.env.has_module("ccp4io")
False
>>> libtbx.env.has_module("ccp4io_adaptbx")
False
>>> import ccp4io_adaptbx
>>> ccp4io_adaptbx       
<module 'ccp4io_adaptbx' from '(PREFIX)/lib/python3.8/site-packages/ccp4io_adaptbx/__init__.py'>

however

$ ls -d (PREFIX)/lib/python3.8/site-packages/ccp4io*
lib/python3.8/site-packages/ccp4io  lib/python3.8/site-packages/ccp4io_adaptbx

Does it make sense for the packages to be both present and not configured?

Environment (conda list):

$ conda list
...
cctbx-base                2020.10          py38hc79ae55_0    conda-forge
...

remaining extra annlib dependency

I now have a preliminary dxtbx build that is based on the pre-compiled cctbx package:
https://dev.azure.com/azure-dxtbx/dxtbx/_build/results?buildId=382&view=logs&j=027b87a4-6108-54a2-25c3-e49002181cdd

I noted that this still needs an extra annlib/annlib_adaptbx checkout, presumably because the prebuilt cctbx enables rstbx by default. A standard plain dxtbx build from scratch doesn't need rstbx, therefore doesn't configure it, and therefore works just fine without any annlib* stuff present:
https://dev.azure.com/azure-dxtbx/dxtbx/_build/results?buildId=382&view=logs&j=bbb31556-568f-5bdc-5f88-8c2cd67ed7a6&t=398c8d41-a382-5623-6665-088a1dc79e38&l=33

Do you have any suggestions on this?

MacOS .dylib libraries have broken `-install_name`

The .dylib -install_name/Library ID values appear to be broken for cctbx packages on MacOS in a way that makes it hard to successfully link to from downstream packages e.g.

$ mamba create -yp ENV cctbx-base
...
$ otool -D ENV/lib/libscitbx_boost_python.dylib
ENV/lib/libscitbx_boost_python.dylib:
lib/libscitbx_boost_python.dylib

so, has a relative ID, instead of having an RPATH and being RPATH-relative, like the boost libraries:

$ otool -D ENV/lib/libboost_python310.dylib
ENV/lib/libboost_python310.dylib:
@rpath/libboost_python310.dylib

$ otool -l ENV/lib/libboost_python310.dylib
...
          cmd LC_RPATH
      cmdsize 32
         path @loader_path/ (offset 12)

There appears to be no way to actually override this when linking, because the Mac ecosystem seems to rely on a library accurately knowing it's own ID, leaving the only options to either fiddle with install_name_tool (on the built package, because cannot change the upstream library, because conda), or setting a DYLD_LIBRARY_PATH (might be technically possible on a conda-forge package, but is definitely bad practice).

This should either be an @rpath/libscitbx_boost_python.dylib (so the builder is responsible for adding to the RPATH search folders, or an absolute path (in which case I believe conda-build prefix-rewriting will take care of the issue).

Specific Example

As a specific example, setting up with:

$ mamba create -yp ENV cctbx-base
$ echo "int main(){}" > test.c
$ clang test.c -o test ENV/lib/libscitbx_boost_python.dylib

Running test cannot find the linked library:

$ ./test
dyld: Library not loaded: lib/libscitbx_boost_python.dylib
  Referenced from: /private/var/folders/yq/n9kwymws3q317f02dwpl9jrr0000gn/T/tmp.2K5uCIxY/./test
  Reason: image not found
Abort trap: 6

... unless you run in a CWD that has the library in the relative path:

$ cd ENV
$ ../test

(this fails, because minimal test and boost isn't linked, but fails further down the chain, after it's found libscitbx_boost_python at runtime).

Windows distribution includes dxtbx entry_points

$ wget https://anaconda.org/conda-forge/cctbx-base/2022.5/download/win-64/cctbx-base-2022.5-py38h1c842d2_0.tar.bz2
$ tar xf cctbx-base-2022.5-py38h1c842d2_0.tar.bz2
$ rg EigerStream
Lib/site-packages/libtbx.dxtbx.egg-info/entry_points.txt
42:FormatEigerStream:FormatMultiImage,Format = dxtbx.format.FormatEigerStream:FormatEigerStream

This is causing the dxtbx 3.10.0 conda-forge builds to fail on windows, because FormatEigerStream was removed. The libtbx.dxtbx.egg-info should probably not be distributed.

Of the non-windows platform packages I checked, this directory was not present.

misleading libtbx.configure output

When configuring new modules in an otherwise clean conda environment with cctbx-base installed, the sources in /modules/ and running in an empty /build/:

$ libtbx.configure dxtbx dials xia2
dxtbx is already installed
dials is already installed
xia2 is already installed
(...)

Cannot build anything

Starting from an empty environment containing cctbx-base, installing all dials dependencies, cloning sources for dxtbx/dials/xia2 into /modules and running in /build:

$ libtbx.configure dxtbx dials xia2 --compiler=conda --enable_cxx11 --use_conda 
$ make
libtbx.scons -j "`libtbx.show_number_of_processors`"
scons: Reading SConscript files ...
libtbx.scons: using flags from initial environment: 
              CXXFLAGS =  -fvisibility-inlines-hidden -std=c++11 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem (...)/environment/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cctbx-base_1600872004520/work=/usr/local/src/conda/cctbx-base-2020.8 -fdebug-prefix-map=(...)/environment=/usr/local/src/conda-prefix
              CFLAGS =  -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem (...)/environment/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/cctbx-base_1600872004520/work=/usr/local/src/conda/cctbx-base-2020.8 -fdebug-prefix-map=(...)/environment=/usr/local/src/conda-prefix
              CPPFLAGS =  -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem (...)/environment/include
              LDFLAGS =  -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,(...)/environment/lib -Wl,-rpath-link,(...)/environment/lib -L(...)/environment/lib
libtbx.scons: OpenMP is disabled.
BOOST_VERSION: 107200
AttributeError: 'empty' object has no attribute 'dxtbx_include':
  File "(...)/environment/build/SConstruct", line 19:
    SConscript("spotfinder/SConscript")
  File "(...)/environment/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 661:
    return method(*args, **kw)
  File "(...)/environment/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "(...)/environment/lib/python3.8/site-packages/SCons/Script/SConscript.py", line 287:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "(...)/environment/lib/python3.8/site-packages/spotfinder/SConscript", line 22:
    env_etc.dxtbx_include,
usr+sys time: 0.43 seconds
wall clock time: 1.48 seconds
make: *** [default] Error 2

package vendors ANN

cctbx-base 2020.8 vendors ANN (https://github.com/conda-forge/ann-feedstock) in /lib/$PYTHON/site-packages/annlib.

missing scitbx entry point

Just copying over the information from conda-forge/staged-recipes#10021 so it is easier to find:

Entry points aren't registered properly. For example:

$ python -c "import pkg_resources;print(list(pkg_resources.iter_entry_points('pytest_randomly.random_seeder')))" 
[]

The expected output is:

$ python -c "import pkg_resources;print(list(pkg_resources.iter_entry_points('pytest_randomly.random_seeder')))" 
[EntryPoint.parse('scitbx_flex = scitbx.array_family.flex:set_random_seed')]

The entry point is so that flex arrays can be correctly used in randomized tests of any downstream package.

Basically what we do in the development installation is roughly equivalent to

setuptools.setup(
    name='libtbx.scitbx',
    description='libtbx entry point manager for scitbx',
    entry_points={
        "pytest_randomly.random_seeder": [
            "scitbx_flex = scitbx.array_family.flex:set_random_seed"
        ],
    },
)

ANN/ANN.h missing

#include <ANN/ANN.h> is used by lib/python3.10/site-packages/annlib_adaptbx/include/annlib_adaptbx/ann_adaptor.h, currently itself included by DIALS indexing.

The lib/libann.so (built by annlib_adaptbx using the annlib sources) and share/cctbx/annlib_adaptbx/include/ANNSELF_INCLUDE/ANN.h from the build/ folder are included, so it looks like it's mostly a simple case of just being missed. I've never worked out the reason for the ANN/ANNSELF_INCLUDE duplication for headers (duplication except the #define guard, and namespace), but copying them both into the share/cctbx/annlib_adaptbx/include folder might be sufficient?

I don't know what the state of this was when #12, but that implies it was missing and most of it seems to actually be included now.