conda-forge / cramjam-feedstock

A conda-smithy repository for cramjam.

About cramjam

Home: https://github.com/milesgranger/pyrus-cramjam

Package license: MIT

Feedstock license: BSD-3-Clause

Summary: python bindings to rust-implemented compression

Documentation: https://docs.rs/cramjam

Extremely thin Python bindings to de/compression algorithms in Rust. Allows for using algorithms such as Snappy, without any system dependencies. This is handy when being used in environments like AWS Lambda, where installing packages like python-snappy becomes difficult because of system level dependencies.

Current build status

Azure

Variant Status linux_64_python3.10.____cpython linux_64_python3.11.____cpython linux_64_python3.8.____cpython linux_64_python3.9.____cpython linux_aarch64_python3.10.____cpython linux_aarch64_python3.11.____cpython linux_aarch64_python3.8.____cpython linux_aarch64_python3.9.____cpython linux_ppc64le_python3.10.____cpython linux_ppc64le_python3.11.____cpython linux_ppc64le_python3.8.____cpython linux_ppc64le_python3.9.____cpython osx_64_python3.10.____cpython osx_64_python3.11.____cpython osx_64_python3.8.____cpython osx_64_python3.9.____cpython osx_arm64_python3.10.____cpython osx_arm64_python3.11.____cpython osx_arm64_python3.8.____cpython osx_arm64_python3.9.____cpython win_64_python3.10.____cpython win_64_python3.11.____cpython win_64_python3.8.____cpython win_64_python3.9.____cpython

Current release info

Name	Downloads	Version	Platforms

Installing cramjam

Installing cramjam from the conda-forge channel can be achieved by adding conda-forge to your channels with:

conda config --add channels conda-forge
conda config --set channel_priority strict

Once the conda-forge channel has been enabled, cramjam can be installed with conda:

conda install cramjam

or with mamba:

mamba install cramjam

It is possible to list all of the versions of cramjam available on your platform with conda:

conda search cramjam --channel conda-forge

or with mamba:

mamba search cramjam --channel conda-forge

Alternatively, mamba repoquery may provide more information:

# Search all versions available on your platform:
mamba repoquery search cramjam --channel conda-forge

# List packages depending on `cramjam`:
mamba repoquery whoneeds cramjam --channel conda-forge

# List dependencies of `cramjam`:
mamba repoquery depends cramjam --channel conda-forge

About conda-forge

conda-forge is a community-led conda channel of installable packages. In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.

A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services. Thanks to the awesome service provided by Azure, GitHub, CircleCI, AppVeyor, Drone, and TravisCI it is possible to build and upload installable packages to the conda-forge Anaconda-Cloud channel for Linux, Windows and OSX respectively.

To manage the continuous integration and simplify feedstock maintenance conda-smithy has been developed. Using the conda-forge.yml within this repository, it is possible to re-render all of this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender.

For more information please check the conda-forge documentation.

Terminology

feedstock - the conda recipe (raw material), supporting scripts and CI configuration.

conda-smithy - the tool which helps orchestrate the feedstock. Its primary use is in the construction of the CI .yml files and simplify the management of many feedstocks.

conda-forge - the place where the feedstock and smithy live and work to produce the finished article (built conda distributions)

Updating cramjam-feedstock

If you would like to improve the cramjam recipe or build a new package version, please fork this repository and submit a PR. Upon submission, your changes will be run on the appropriate platforms to give the reviewer an opportunity to confirm that the changes result in a successful build. Once merged, the recipe will be re-built and uploaded automatically to the conda-forge channel, whereupon the built conda packages will be available for everybody to install and use from the conda-forge channel. Note that all branches in the conda-forge/cramjam-feedstock are immediately built and any created packages are uploaded, so PRs should be based on branches in forks and branches in the main repository should only be used to build distinct package versions.

In order to produce a uniquely identifiable distribution:

• If the version of a package is not being increased, please add or increase the build/number.
• If the version of a package is being increased, please remember to return the build/number back to 0.

Feedstock Maintainers

• @martindurant
• @milesgranger

cramjam-feedstock's Issues

Cramjam wrong architecture error when installing on M1 with conda (mambaforge)

[X] I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

I'm working on Apple M1 with a mambaforge deployment of conda.
When trying to save a dataframe to parquet using fastparquet I'm getting the following error:

ImportError: dlopen(/Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so, 2): no suitable image found. Did find:
/Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so: mach-o, but wrong architecture
/Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so: mach-o, but wrong architecture
Seems like conda is giving me the wrong version of cramjam?

MRE:

conda create -n cramjam
conda install cramjam ipython
ipython
import cramjam

ImportError: dlopen(/Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so, 2): no suitable image found. Did find:
/Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so: mach-o, but wrong architecture
/Users/rpelgrim/mambaforge/envs/cramjam/lib/python3.9/site-packages/cramjam.cpython-39-darwin.so: mach-o, but wrong architecture

import cramjam does work when doing a pip install cramjam. But fails again if I then do a subsequent conda install fastparquet.

$ conda list
# Name                    Version                   Build  Channel
ca-certificates           2021.5.30            h4653dfc_0    conda-forge
cramjam                   2.3.2                    pypi_0    pypi
fastparquet               0.7.1            py39h20ef057_0    conda-forge
fsspec                    2021.8.1           pyhd8ed1ab_0    conda-forge
libblas                   3.9.0           11_osxarm64_openblas    conda-forge
libcblas                  3.9.0           11_osxarm64_openblas    conda-forge
libcxx                    12.0.1               h168391b_0    conda-forge
libgfortran               5.0.0.dev0      11_0_1_hf114ba7_23    conda-forge
libgfortran5              11.0.1.dev0         hf114ba7_23    conda-forge
liblapack                 3.9.0           11_osxarm64_openblas    conda-forge
libopenblas               0.3.17          openmp_h5dd58f0_1    conda-forge
llvm-openmp               12.0.1               hf3c4609_1    conda-forge
ncurses                   6.2                  h9aa5885_4    conda-forge
numpy                     1.21.2           py39h1f3b974_0    conda-forge
openssl                   1.1.1l               h3422bc3_0    conda-forge
pandas                    1.3.2            py39h7f752ed_0    conda-forge
pip                       21.2.4             pyhd8ed1ab_0    conda-forge
python                    3.9.7           h54d631c_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
readline                  8.1                  hedafd6a_0    conda-forge
setuptools                57.4.0           py39h2804cbe_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.36.0               h72a2b83_0    conda-forge
thrift                    0.13.0           py39h93dc1e2_2    conda-forge
tk                        8.6.11               he1e0b03_1    conda-forge
tzdata                    2021a                he74cb21_1    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
zlib                      1.2.11            h31e879b_1009    conda-forge

$ conda info

    active environment : cramjam
    active env location : /Users/rpelgrim/mambaforge/envs/cramjam
            shell level : 2
       user config file : /Users/rpelgrim/.condarc
 populated config files : /Users/rpelgrim/mambaforge/.condarc
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.9.6.final.0
       virtual packages : __osx=11.3.1=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/rpelgrim/mambaforge  (writable)
      conda av data dir : /Users/rpelgrim/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/rpelgrim/mambaforge/pkgs
                          /Users/rpelgrim/.conda/pkgs
       envs directories : /Users/rpelgrim/mambaforge/envs
                          /Users/rpelgrim/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.10.3 requests/2.26.0 CPython/3.9.6 Darwin/20.4.0 OSX/11.3.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Cramjam buffer is read-only

The result of a cramjam.*.decompress is a cramjam.Buffer. I can take memoryviews of this or use it as the buffer for a numpy array, but I cannot modify it. The only way around this would be to create the buffer up front in python and use decompress_into. This is causing some unnecessary copies in fastparquet.