Coder Social home page Coder Social logo

conda-forge / pyarrow-feedstock Goto Github PK

View Code? Open in Web Editor NEW
6.0 15.0 26.0 189 KB

Now part of https://github.com/conda-forge/arrow-cpp-feedstock / A conda-smithy repository for pyarrow.

License: BSD 3-Clause "New" or "Revised" License

Shell 39.84% Batchfile 60.16%

pyarrow-feedstock's Introduction

About pyarrow

Home: http://github.com/apache/arrow

Package license: Apache-2.0

Feedstock license: BSD 3-Clause

Summary: Python libraries for Apache Arrow

Current build status

Drone linux
Azure
VariantStatus
linux_aarch64_python3.6.____cpython variant
linux_aarch64_python3.7.____cpython variant
linux_aarch64_python3.8.____cpython variant
linux_python3.6.____cpython variant
linux_python3.7.____cpython variant
linux_python3.8.____cpython variant
osx_python3.6.____cpython variant
osx_python3.7.____cpython variant
osx_python3.8.____cpython variant
win_python3.6.____cpython variant
win_python3.7.____cpython variant
win_python3.8.____cpython variant
Linux_ppc64le ppc64le disabled

Current release info

Name Downloads Version Platforms
Conda Recipe Conda Downloads Conda Version Conda Platforms

Installing pyarrow

Installing pyarrow from the conda-forge channel can be achieved by adding conda-forge to your channels with:

conda config --add channels conda-forge

Once the conda-forge channel has been enabled, pyarrow can be installed with:

conda install pyarrow

It is possible to list all of the versions of pyarrow available on your platform with:

conda search pyarrow --channel conda-forge

About conda-forge

Powered by NumFOCUS

conda-forge is a community-led conda channel of installable packages. In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.

A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services. Thanks to the awesome service provided by CircleCI, AppVeyor and TravisCI it is possible to build and upload installable packages to the conda-forge Anaconda-Cloud channel for Linux, Windows and OSX respectively.

To manage the continuous integration and simplify feedstock maintenance conda-smithy has been developed. Using the conda-forge.yml within this repository, it is possible to re-render all of this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender.

For more information please check the conda-forge documentation.

Terminology

feedstock - the conda recipe (raw material), supporting scripts and CI configuration.

conda-smithy - the tool which helps orchestrate the feedstock. Its primary use is in the construction of the CI .yml files and simplify the management of many feedstocks.

conda-forge - the place where the feedstock and smithy live and work to produce the finished article (built conda distributions)

Updating pyarrow-feedstock

If you would like to improve the pyarrow recipe or build a new package version, please fork this repository and submit a PR. Upon submission, your changes will be run on the appropriate platforms to give the reviewer an opportunity to confirm that the changes result in a successful build. Once merged, the recipe will be re-built and uploaded automatically to the conda-forge channel, whereupon the built conda packages will be available for everybody to install and use from the conda-forge channel. Note that all branches in the conda-forge/pyarrow-feedstock are immediately built and any created packages are uploaded, so PRs should be based on branches in forks and branches in the main repository should only be used to build distinct package versions.

In order to produce a uniquely identifiable distribution:

  • If the version of a package is not being increased, please add or increase the build/number.
  • If the version of a package is being increased, please remember to return the build/number back to 0.

Feedstock Maintainers

pyarrow-feedstock's People

Contributors

beckermr avatar cat-yu avatar conda-forge-admin avatar cpcloud avatar github-actions[bot] avatar jakirkham avatar jcrist avatar jreback avatar ksangeek avatar kszucs avatar leifwalsh avatar regro-cf-autotick-bot avatar wesm avatar xhochy avatar

Stargazers

avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

wesm conda-forge-admin xhochy leifwalsh jreback bmabey isuruf jcrist anacondarecipes kszucs regro-cf-autotick-bot conda-forge-linter josiahsams ksangeek mike-wendt jakirkham-feedstocks ocefpaf cat-yu justcherie quansight h-vetinari graingert anaconda-community

pyarrow-feedstock's Issues

Updating pyarrow from conda-forge causing errors on Mac py27 

$ conda install pyarrow -c conda-forge
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /Users/steve/anaconda:

The following packages will be SUPERCEDED by a higher-priority channel:

    conda: 4.3.14-py27_0 --> 4.2.13-py27_0 conda-forge

Proceed ([y]/n)? y

Then checking installed

$ python
Python 2.7.12 |Anaconda custom (x86_64)| (default, Jul  2 2016, 17:43:17) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import pyarrow.parquet as pa
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/__init__.py", line 28, in <module>
    import pyarrow.config
ImportError: No module named config

Then checking folder for config

$ ls .../anaconda/lib/python2.7/site-packages/pyarrow

__init__.py		compat.py		filesystem.pyc		jemalloc.pyx		scalar.pyx		table_api.h
__init__.pyc		compat.pyc		formatting.py		libpyarrow.dylib	scalar.so		tests
_parquet.pxd		config.pyx		formatting.pyc		memory.pxd		schema.pxd		util.py
_parquet.pyx		config.so		io.pxd			memory.pyx		schema.pyx		util.pyc
_parquet.so		error.pxd		io.pyx			memory.so		schema.so
array.pxd		error.pyx		io.so			parquet.py		table.pxd
array.pyx		error.so		ipc.py			parquet.pyc		table.pyx
array.so		filesystem.py		ipc.pyc			scalar.pxd		table.so

If I try to import pyarrow generally get different warning:

$ python

import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/steve/anaconda/lib/python2.7/site-packages/pyarrow/__init__.py", line 20, in <module>
    from pkg_resources import get_distribution, DistributionNotFound
  File "/Users/steve/anaconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 21, in <module>
    try:
ImportError: dlopen(./io.so, 2): Symbol not found: _pyarrow_ARRAY_API
  Referenced from: /Users/steve/anaconda/lib/python2.7/site-packages/pyarrow/libpyarrow.dylib
  Expected in: flat namespace
 in /Users/steve/anaconda/lib/python2.7/site-packages/pyarrow/libpyarrow.dylib

ImportError: DLL load failed: The specified module could not be found

The dreaded DLL load failed error :(

This is on win64 with py36 and the latest pyarrow. I tested that this also happens in a new environment:

λ conda create -n arw python=3.6 defaults::numpy conda-forge::arrow-cpp conda-forge::parquet-cpp conda-forge::pyarrow conda-forge::vc --yes
Solving environment: done

## Package Plan ##

  environment location: C:\Miniconda3\envs\arw

  added / updated specs:
    - conda-forge::arrow-cpp
    - conda-forge::parquet-cpp
    - conda-forge::pyarrow
    - conda-forge::vc
    - defaults::numpy
    - python=3.6


The following NEW packages will be INSTALLED:

    arrow-cpp:       0.10.0-py36_vc14h12fa3ca_0 conda-forge [vc14]
    blas:            1.0-mkl
    boost-cpp:       1.67.0-hfa6e2cd_4
    certifi:         2018.8.13-py36_0
    icc_rt:          2018.0.2-0                 numba
    intel-openmp:    2018.0.3-0
    libboost:        1.67.0-hfd51bdf_4
    mkl:             2018.0.3-1
    mkl_fft:         1.0.6-py36_0               conda-forge
    mkl_random:      1.0.1-py36h77b88f5_1
    numpy:           1.15.0-py36h9fa60d3_0
    numpy-base:      1.15.0-py36h4a99626_0
    pandas:          0.23.4-py36h830ac7b_0
    parquet-cpp:     1.5.0.pre-h037026a_0       conda-forge
    pip:             18.0-py36_1                conda-forge
    pyarrow:         0.10.0-py36h6538335_0      conda-forge
    python:          3.6.6-hea74fb7_0
    python-dateutil: 2.7.3-py36_0
    pytz:            2018.5-py36_0
    setuptools:      40.0.0-py36_0
    six:             1.11.0-py36_1
    snappy:          1.1.7-h777316e_3
    vc:              14-0                       conda-forge
    vs2015_runtime:  15.5.2-3
    wheel:           0.31.1-py36_0
    wincertstore:    0.2-py36h7fe50ca_0
    xz:              5.2.4-h2fa13f4_4
    zlib:            1.2.11-h8395fce_2

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Results in a broken environment:

λ activate arw
(arw) λ  python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Miniconda3\envs\arw\lib\site-packages\pyarrow\__init__.py", line 60, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: DLL load failed: The specified module could not be found.
>>>

setuptools is a run dependency for conda package

Issue
setuptools is listed as a run dependency for conda package

Environment
While working on the pyarrow conda recipe for linux-ppc64le I realized that there is no need to have setuptools as a run-dependency for pyarrow.

Expected behavior
setuptools package should not be run dependency for pyarrow conda package.

pyarrow 0.14 conflicts with blas from conda-forge?

@xhochy @kszucs
Issue:
When I try to update environment with conda update --all it resolves like this:

The following packages will be SUPERSEDED by a higher-priority channel:

  blas                                            pkgs/main --> conda-forge

The following packages will be DOWNGRADED:

  arrow-cpp                           0.14.0-py37h1b0c03e_0 --> 0.13.0-py37h37df3c6_3
  boost-cpp                               1.70.0-h6a4c333_0 --> 1.68.0-h6a4c333_1000
  grpc-cpp                                1.22.0-h4d7d3fa_0 --> 1.21.4-hfa5417f_0
  libprotobuf                              3.8.0-h1a1b453_0 --> 3.7.1-h1a1b453_0
  libtiff                              4.0.10-h6512ee2_1003 --> 4.0.10-h016b793_1002
  pyarrow                             0.14.0-py37h803c963_0 --> 0.13.0-py37h8c67754_1
  thrift-cpp                           0.12.0-hd042d19_1004 --> 0.12.0-h59828bf_1002
  zstd                                     1.4.0-hd8a0e53_0 --> 1.3.3-vc14_1

Environment (conda list): 
# Name                    Version                   Build  Channel
_py-xgboost-mutex         2.0                       cpu_0
arrow-cpp                 0.14.0           py37h1b0c03e_0    conda-forge
asn1crypto                0.24.0                py37_1003    conda-forge
atomicwrites              1.3.0                      py_0    conda-forge
attrs                     19.1.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
blas                      1.0                         mkl
bleach                    3.1.0                      py_0    conda-forge
bokeh                     1.2.0                    py37_0    conda-forge
boost-cpp                 1.70.0               h6a4c333_0    conda-forge
boto                      2.49.0                     py_0    conda-forge
boto3                     1.9.183                    py_0    conda-forge
botocore                  1.12.183                   py_0    conda-forge
brotli                    1.0.7             he025d50_1000    conda-forge
bz2file                   0.98                       py_0    conda-forge
c-ares                    1.15.0            h2fa13f4_1001    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.6.16                py37_0    conda-forge
cffi                      1.12.3           py37hb32ad35_0    conda-forge
chardet                   3.0.4                 py37_1003    conda-forge
click                     7.0                        py_0    conda-forge
cloudpickle               1.2.1                      py_0    conda-forge
colorama                  0.4.1                      py_0    conda-forge
cryptography              2.7              py37hb32ad35_0    conda-forge
cycler                    0.10.0                     py_1    conda-forge
cytoolz                   0.9.0.1         py37hfa6e2cd_1001    conda-forge
dask                      2.0.0                      py_0    conda-forge
dask-core                 2.0.0                      py_0    conda-forge
decorator                 4.4.0                      py_0    conda-forge
defusedxml                0.5.0                      py_1    conda-forge
distributed               2.0.1                      py_0    conda-forge
docutils                  0.14                  py37_1001    conda-forge
double-conversion         3.1.5                h6538335_1    conda-forge
entrypoints               0.3                   py37_1000    conda-forge
fastparquet               0.3.1            py37hfa6e2cd_1    conda-forge
featuretools              0.9.1                      py_0    conda-forge
freetype                  2.10.0               h5db478b_0    conda-forge
funcsigs                  1.0.2                      py_3    conda-forge
future                    0.17.1                py37_1000    conda-forge
gflags                    2.2.2             he025d50_1001    conda-forge
glog                      0.4.0                he025d50_1    conda-forge
graphviz                  2.38.0            h6538335_1011    conda-forge
grpc-cpp                  1.22.0               h4d7d3fa_0    conda-forge
heapdict                  1.0.0                 py37_1000    conda-forge
hyperopt                  0.1.2                      py_0    conda-forge
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.1                     vc14_0    conda-forge
idna                      2.8                   py37_1000    conda-forge
imageio                   2.5.0                    py37_0    conda-forge
importlib_metadata        0.18                     py37_0    conda-forge
intel-openmp              2019.4                      245
ipykernel                 5.1.1            py37h39e3cac_0    conda-forge
ipython                   7.6.1            py37h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.14.0                   py37_0    conda-forge
jinja2                    2.10.1                     py_0    conda-forge
jmespath                  0.9.4                      py_0    conda-forge
joblib                    0.13.2                     py_0    conda-forge
jpeg                      9c                hfa6e2cd_1001    conda-forge
json5                     0.8.4                      py_0    conda-forge
jsonschema                3.0.1                    py37_0    conda-forge
jupyter_client            5.2.4                      py_3    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterlab                1.0.1                    py37_0    conda-forge
jupyterlab_server         1.0.0                      py_1    conda-forge
kiwisolver                1.1.0            py37he980bc4_0    conda-forge
libblas                   3.8.0                     8_mkl    conda-forge
libcblas                  3.8.0                     8_mkl    conda-forge
libpng                    1.6.37               h7602738_0    conda-forge
libprotobuf               3.8.0                h1a1b453_0    conda-forge
libsodium                 1.0.16            h2fa13f4_1001    conda-forge
libtiff                   4.0.10            h6512ee2_1003    conda-forge
libxgboost                0.90                          0
llvmlite                  0.29.0           py37hed17590_1    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.8.3             he025d50_1001    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
markupsafe                1.1.1            py37hfa6e2cd_0    conda-forge
matplotlib                3.1.1                    py37_0    conda-forge
matplotlib-base           3.1.1            py37h2852a4a_0    conda-forge
mistune                   0.8.4           py37hfa6e2cd_1000    conda-forge
mkl                       2019.4                      245
mkl_fft                   1.0.13           py37hfa6e2cd_1    conda-forge
mkl_random                1.0.4            py37h830ac7b_0    conda-forge
more-itertools            7.1.0                      py_0    conda-forge
msgpack-python            0.6.1            py37he980bc4_0    conda-forge
msys2-conda-epoch         20160418                      1
nbconvert                 5.5.0                      py_0    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
networkx                  2.3                        py_0    conda-forge
notebook                  5.7.8                    py37_1    conda-forge
numba                     0.44.1           py37hf9181ef_0
numpy                     1.16.4           py37h19fb1c0_0
numpy-base                1.16.4           py37hc3f5095_0
olefile                   0.46                       py_0    conda-forge
openssl                   1.1.1b               hfa6e2cd_2    conda-forge
packaging                 19.0                       py_0    conda-forge
pandas                    0.24.2           py37he350917_0    conda-forge
pandoc                    2.7.3                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.5.0                      py_0    conda-forge
partd                     1.0.0                      py_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pickleshare               0.7.5                 py37_1000    conda-forge
pillow                    6.1.0            py37h9a613e6_0    conda-forge
pip                       19.1.1                   py37_0    conda-forge
pluggy                    0.12.0                     py_0    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt_toolkit            2.0.9                      py_0    conda-forge
psutil                    5.6.3            py37hfa6e2cd_0    conda-forge
py                        1.8.0                      py_0    conda-forge
py-xgboost                0.90                     py37_0
pyarrow                   0.14.0           py37h803c963_0    conda-forge
pycparser                 2.19                     py37_1    conda-forge
pygments                  2.4.2                      py_0    conda-forge
pymongo                   3.8.0            py37h6538335_0    conda-forge
pyodbc                    4.0.26           py37h6538335_0    conda-forge
pyopenssl                 19.0.0                   py37_0    conda-forge
pyparsing                 2.4.0                      py_0    conda-forge
pyqt                      5.9.2            py37h6538335_0    conda-forge
pyrsistent                0.15.3           py37hfa6e2cd_0    conda-forge
pysocks                   1.7.0                    py37_0    conda-forge
pytest                    5.0.1                    py37_0    conda-forge
python                    3.7.3                h510b542_1    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
python-graphviz           0.11.1                     py_0    conda-forge
python-snappy             0.5.4            py37hd25c944_0    conda-forge
pytz                      2019.1                     py_0    conda-forge
pywavelets                1.0.3            py37h452e1ab_1    conda-forge
pywin32                   224             py37hfa6e2cd_1000    conda-forge
pywinpty                  0.5.5                 py37_1000    conda-forge
pyyaml                    5.1.1            py37hfa6e2cd_0    conda-forge
pyzmq                     18.0.2           py37he7828b0_0    conda-forge
qt                        5.9.7                hc6833c9_1    conda-forge
re2                       2019.07.01       vc14h6538335_0  [vc14]  conda-forge
requests                  2.22.0                   py37_0    conda-forge
s3fs                      0.2.2                      py_0    conda-forge
s3transfer                0.2.1                    py37_0    conda-forge
scikit-image              0.15.0           py37he350917_2    conda-forge
scikit-learn              0.21.2           py37h7208079_1    conda-forge
scipy                     1.2.1            py37h29ff71c_0
seaborn                   0.9.0                      py_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                41.0.1                   py37_0    conda-forge
shap                      0.29.3           py37he350917_0    conda-forge
sip                       4.19.8          py37h6538335_1000    conda-forge
six                       1.12.0                py37_1000    conda-forge
smart_open                1.8.4                      py_0    conda-forge
snappy                    1.1.7             h6538335_1002    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
sqlalchemy                1.3.5            py37hfa6e2cd_0    conda-forge
sqlite                    3.28.0               hfa6e2cd_1    conda-forge
statsmodels               0.10.0           py37hfa6e2cd_0    conda-forge
tblib                     1.4.0                      py_0    conda-forge
terminado                 0.8.2                    py37_0    conda-forge
testpath                  0.4.2                   py_1001    conda-forge
thrift                    0.11.0          py37h6538335_1001    conda-forge
thrift-cpp                0.12.0            hd042d19_1004    conda-forge
tk                        8.6.9             hfa6e2cd_1002    conda-forge
toolz                     0.9.0                      py_1    conda-forge
tornado                   6.0.3            py37hfa6e2cd_0    conda-forge
tqdm                      4.32.2                     py_0    conda-forge
traitlets                 4.3.2                 py37_1000    conda-forge
uriparser                 0.9.3                he025d50_1    conda-forge
urllib3                   1.24.3                   py37_0    conda-forge
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_4
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.33.4                   py37_0    conda-forge
win_inet_pton             1.1.0                    py37_0    conda-forge
wincertstore              0.2                   py37_1002    conda-forge
winpty                    0.4.3                         4    conda-forge
xlrd                      1.2.0                      py_0    conda-forge
xlsxwriter                1.1.8                      py_0    conda-forge
xlwt                      1.3.0                      py_1    conda-forge
xz                        5.2.4             h2fa13f4_1001    conda-forge
yaml                      0.1.7             hfa6e2cd_1001    conda-forge
zeromq                    4.3.1             he025d50_1000    conda-forge
zict                      1.0.0                      py_0    conda-forge
zipp                      0.5.1                      py_0    conda-forge
zlib                      1.2.11            h2fa13f4_1004    conda-forge
zstd                      1.4.0                hd8a0e53_0    conda-forge

Details about conda and system ( conda info ): 
     active environment : cyclone
    active env location : C:\Users\user\AppData\Local\Continuum\miniconda3\envs\cyclone
            shell level : 2
       user config file : C:\Users\user\.condarc
 populated config files : C:\Users\user\.condarc
          conda version : 4.6.14
    conda-build version : not installed
         python version : 3.7.3.final.0
       base environment : C:\Users\user\AppData\Local\Continuum\miniconda3
 (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\user\AppData\Local\Continuum\miniconda3\pkgs
                          C:\Users\user\.conda\pkgs
                          C:\Users\user\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\user\AppData\Local\Continuum\miniconda3\envs
                          C:\Users\user\.conda\envs
                          C:\Users\user\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.14 requests/2.22.0 CPython/3.7.3 Windows/7 Windows/6.1.7601
          administrator : False
             netrc file : None
           offline mode : False

Version 0.15.0 : DLL load failed - works with 0.14.1

Problem: the latest pyarrow version fails to load (ImportError due to DLL error).

Minimum non-working example

conda create -n test1 python=3 pyarrow=0.15.0
conda activate test1
python -c "import platform; print(platform.platform()); import pyarrow"

The output from the last command is

Windows-10-10.0.17763-SP0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Anaconda3\envs\test1\lib\site-packages\pyarrow\__init__.py", line 49, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: DLL load failed: The specified module could not be found.

Output from conda list:

# packages in environment at C:\Anaconda3\envs\test1:
#
# Name                    Version                   Build  Channel
arrow-cpp                 0.15.0           py37h00ef551_1    https://conda.anaconda.org/conda-forge
blas                      1.0                         mkl
boost-cpp                 1.70.0               h6a4c333_2    https://conda.anaconda.org/conda-forge
brotli                    1.0.7                h33f27b4_0
bzip2                     1.0.8                he774522_0
c-ares                    1.15.0            h62dcd97_1001
ca-certificates           2019.8.28                     0
certifi                   2019.9.11                py37_0
double-conversion         3.1.5                ha925a31_1
gflags                    2.2.2                ha925a31_0
glog                      0.4.0                h33f27b4_0
grpc-cpp                  1.23.0               h4d7d3fa_0    https://conda.anaconda.org/conda-forge
icc_rt                    2019.0.0             h0cc432a_1
intel-openmp              2019.4                      245
libprotobuf               3.8.0                h7bd577a_0
lz4-c                     1.8.3             he025d50_1001    https://conda.anaconda.org/conda-forge
mkl                       2019.4                      245
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.0.14           py37h14836fe_0
mkl_random                1.1.0            py37h675688f_0
numpy                     1.16.5           py37h19fb1c0_0
numpy-base                1.16.5           py37hc3f5095_0
openssl                   1.1.1d               he774522_2
pandas                    0.25.1           py37ha925a31_0
parquet-cpp               1.5.1                         2    https://conda.anaconda.org/conda-forge
pip                       19.2.3                   py37_0
pyarrow                   0.15.0           py37h803c963_0    https://conda.anaconda.org/conda-forge
python                    3.7.4                h5263a28_0
python-dateutil           2.8.0                    py37_0
pytz                      2019.3                     py_0
re2                       2019.09.01       vc14h6538335_0  [vc14]  https://conda.anaconda.org/conda-forge
setuptools                41.4.0                   py37_0
six                       1.12.0                   py37_0
snappy                    1.1.7                h777316e_3
sqlite                    3.30.0               he774522_0
thrift-cpp                0.12.0            hd042d19_1004    https://conda.anaconda.org/conda-forge
uriparser                 0.9.3                he025d50_1    https://conda.anaconda.org/conda-forge
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_0
wheel                     0.33.6                   py37_0
wincertstore              0.2                      py37_0
xz                        5.2.4                h2fa13f4_4
zlib                      1.2.11               h62dcd97_3
zstd                      1.4.3                hd8a0e53_0    https://conda.anaconda.org/conda-forge

Minimum working example, using the previous version of pyarrow

conda create -n test2 python=3 pyarrow=0.14.1
conda activate test2
python -c "import platform; print(platform.platform()); import pyarrow"

This outputs Windows-10-10.0.17763-SP0 without any subsequent ImportError

The output from conda list:

# packages in environment at C:\Anaconda3\envs\test2:
#
# Name                    Version                   Build  Channel
arrow-cpp                 0.14.1           py37h5c295ed_4    https://conda.anaconda.org/conda-forge
blas                      1.0                         mkl
boost-cpp                 1.70.0               h6a4c333_2    https://conda.anaconda.org/conda-forge
brotli                    1.0.7                h33f27b4_0
bzip2                     1.0.8                he774522_0
c-ares                    1.15.0            h62dcd97_1001
ca-certificates           2019.8.28                     0
certifi                   2019.9.11                py37_0
double-conversion         3.1.5                ha925a31_1
gflags                    2.2.2                ha925a31_0
glog                      0.4.0                h33f27b4_0
grpc-cpp                  1.23.0               h4d7d3fa_0    https://conda.anaconda.org/conda-forge
icc_rt                    2019.0.0             h0cc432a_1
intel-openmp              2019.4                      245
libprotobuf               3.8.0                h7bd577a_0
lz4-c                     1.8.3             he025d50_1001    https://conda.anaconda.org/conda-forge
mkl                       2019.4                      245
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.0.14           py37h14836fe_0
mkl_random                1.1.0            py37h675688f_0
numpy                     1.16.5           py37h19fb1c0_0
numpy-base                1.16.5           py37hc3f5095_0
openssl                   1.1.1d               he774522_2
pandas                    0.25.1           py37ha925a31_0
parquet-cpp               1.5.1                         2    https://conda.anaconda.org/conda-forge
pip                       19.2.3                   py37_0
pyarrow                   0.14.1           py37h803c963_2    https://conda.anaconda.org/conda-forge
python                    3.7.4                h5263a28_0
python-dateutil           2.8.0                    py37_0
pytz                      2019.3                     py_0
re2                       2019.09.01       vc14h6538335_0  [vc14]  https://conda.anaconda.org/conda-forge
setuptools                41.4.0                   py37_0
six                       1.12.0                   py37_0
snappy                    1.1.7                h777316e_3
sqlite                    3.30.0               he774522_0
thrift-cpp                0.12.0            hd042d19_1004    https://conda.anaconda.org/conda-forge
uriparser                 0.9.3                he025d50_1    https://conda.anaconda.org/conda-forge
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_0
wheel                     0.33.6                   py37_0
wincertstore              0.2                      py37_0
xz                        5.2.4                h2fa13f4_4
zlib                      1.2.11               h62dcd97_3
zstd                      1.4.0                hd8a0e53_0    https://conda.anaconda.org/conda-forge

Illegal instructions in x86_64 pyarrow linux libraries

Issue:

For x86_64 linux some of the pyarrow extension libraries contain instructions from an instruction set greater than nocona (default for the Anaconda toolchain compilers).

Reproducer:

$ conda create -n tmp_pyarrow_bad -c conda-forge python=3 pyarrow -y -q
$ conda activate tmp_pyarrow_bad
$ for x in $(find $(dirname `which python`)/../lib/python3.7/site-packages/pyarrow/*.so); do echo $x; objdump -D $x|grep pinsrq|head -1; done

should yield output like:

<redacted>/envs/tmp_pyarrow_bad/bin/../lib/python3.7/site-packages/pyarrow/_csv.cpython-37m-x86_64-linux-gnu.so                                                                                                                             
    9e83:       66 49 0f 3a 22 c6 01    pinsrq $0x1,%r14,%xmm0                                                                  
<redacted>/envs/tmp_pyarrow_bad/bin/../lib/python3.7/site-packages/pyarrow/_flight.cpython-37m-x86_64-linux-gnu.so                                                                                                                          
   14f64:       66 48 0f 3a 22 c0 01    pinsrq $0x1,%rax,%xmm0                                                                  
<redacted>/envs/tmp_pyarrow_bad/bin/../lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-x86_64-linux-gnu.so                                                                                                                          
    ee8e:       66 48 0f 3a 22 c2 01    pinsrq $0x1,%rdx,%xmm0                                                                  
<redacted>/envs/tmp_pyarrow_bad/bin/../lib/python3.7/site-packages/pyarrow/_json.cpython-37m-x86_64-linux-gnu.so                                                                                                                            
<redacted>/envs/tmp_pyarrow_bad/bin/../lib/python3.7/site-packages/pyarrow/lib.cpython-37m-x86_64-linux-gnu.so
   53c71:       66 48 0f 3a 22 c0 01    pinsrq $0x1,%rax,%xmm0
<redacted>/envs/tmp_pyarrow_bad/bin/../lib/python3.7/site-packages/pyarrow/_orc.cpython-37m-x86_64-linux-gnu.so
<redacted>/envs/tmp_pyarrow_bad/bin/../lib/python3.7/site-packages/pyarrow/_parquet.cpython-37m-x86_64-linux-gnu.so
    e447:       66 48 0f 3a 22 c0 01    pinsrq $0x1,%rax,%xmm0
<redacted>/envs/tmp_pyarrow_bad/bin/../lib/python3.7/site-packages/pyarrow/_plasma.cpython-37m-x86_64-linux-gnu.so
    a570:       66 48 0f 3a 22 c0 01    pinsrq $0x1,%rax,%xmm0

The instruction pinsrq is SSE 4.1+, nocona supports MMX, SSE, SSE2 and SSE3. The effect is a SIGILL on attempted load from a CPU without SSE 4.1.

xref: AnacondaRecipes#1


Environment (conda list): 
# Name                    Version                   Build  Channel                                                              
_libgcc_mutex             0.1                        main  
arrow-cpp                 0.14.1           py37hb2cae1d_2    conda-forge
boost-cpp                 1.70.0               h8e57a91_2    conda-forge
brotli                    1.0.7             he1b5a44_1000    conda-forge
bzip2                     1.0.8                h516909a_1    conda-forge
c-ares                    1.15.0            h516909a_1001    conda-forge
ca-certificates           2019.9.11            hecc5488_0    conda-forge
certifi                   2019.9.11                py37_0    conda-forge
double-conversion         3.1.5                he1b5a44_1    conda-forge
gflags                    2.2.2             he1b5a44_1001    conda-forge
glog                      0.4.0                he1b5a44_1    conda-forge
grpc-cpp                  1.23.0               h18db393_0    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libevent                  2.1.10               h72c5cf5_0    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
liblapack                 3.8.0               12_openblas    conda-forge
libopenblas               0.3.7                h6e990d7_1    conda-forge
libprotobuf               3.8.0                h8b12597_0    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.17.2           py37h95a1406_0    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
pandas                    0.25.1           py37hb3f55d8_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
pip                       19.2.3                   py37_0    conda-forge
pyarrow                   0.14.1           py37h8b68381_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
pytz                      2019.2                     py_0    conda-forge
re2                       2019.09.01           he1b5a44_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
setuptools                41.2.0                   py37_0    conda-forge
six                       1.12.0                py37_1000    conda-forge
snappy                    1.1.7             he1b5a44_1002    conda-forge
sqlite                    3.29.0               hcee41ef_1    conda-forge
thrift-cpp                0.12.0            hf3afdfd_1004    conda-forge
tk                        8.6.9             hed695b0_1003    conda-forge
uriparser                 0.9.3                he1b5a44_1    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.0                h3b9ef0a_0    conda-forge

Details about conda and system ( conda info ): 
N/A

0.12.0 installs as 0.12.0a0

This causes pip install pyarrow==0.12 to do the following:

Installing collected packages: pyarrow
  Found existing installation: pyarrow 0.12.0a0
    Uninstalling pyarrow-0.12.0a0:
      Successfully uninstalled pyarrow-0.12.0a0

Python version not correctly recorded

Issue:

Files for python 3.5 are being installed on my 3.7 environment.

$ conda create -p /tmp/rr python=3.7 pyarrow
$ conda activate /tmp/rr
$ python                                                            (/tmp/rr) 
Python 3.7.0 (default, Jun 28 2018, 13:15:42) 
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyarrow'
$ head anaconda3/pkgs/pyarrow-0.1.post-1/info/files                                                       (/tmp/rr) 
lib/python3.5/site-packages/Cython/Compiler/__pycache__/AnalysedTreeTransforms.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/Annotate.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/AutoDocTransforms.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/Buffer.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/Builtin.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/CmdLine.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/CodeGeneration.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/CythonScope.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/DebugFlags.cpython-35.pyc
lib/python3.5/site-packages/Cython/Compiler/__pycache__/Errors.cpython-35.pyc

Environment (conda list):

Note that the build string does not specify the python version as with the other packages.

$ conda list
 $ conda list                                                                                              (/tmp/rr) 
# packages in environment at /tmp/rr:
#
# Name                    Version                   Build  Channel
arrow-cpp                 0.2.post                      0    conda-forge
blas                      1.0                         mkl  
ca-certificates           2018.03.07                    0  
certifi                   2018.4.16                py37_0  
intel-openmp              2018.0.3                      0  
jemalloc                  5.0.1                hf484d3e_1  
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                hdf63c60_3  
libgfortran-ng            7.2.0                hdf63c60_3  
libstdcxx-ng              7.2.0                hdf63c60_3  
mkl                       2018.0.3                      1  
mkl_fft                   1.0.4            py37h4414c95_1  
mkl_random                1.0.1            py37h4414c95_1  
ncurses                   6.1                  hf484d3e_0  
numpy                     1.15.0           py37h1b885b7_0  
numpy-base                1.15.0           py37h3dfced4_0  
openssl                   1.0.2o               h14c3975_1  
pandas                    0.23.4           py37h04863e7_0  
parquet-cpp               1.1.0                         1    conda-forge
pip                       10.0.1                   py37_0  
pyarrow                   0.1.post                      1    conda-forge
python                    3.7.0                hc3d631a_0  
python-dateutil           2.7.3                    py37_0  
pytz                      2018.5                   py37_0  
readline                  7.0                  ha6073c6_4  
setuptools                39.2.0                   py37_0  
six                       1.11.0                   py37_1  
sqlite                    3.24.0               h84994c4_0  
tk                        8.6.7                hc745277_3  
wheel                     0.31.1                   py37_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               ha838bed_2

Details about conda and system ( conda info ): 
$ conda info                                                                                              (/tmp/rr) 

     active environment : /tmp/rr
    active env location : /tmp/rr
            shell level : 1
       user config file : /home/zah/.condarc
 populated config files : /home/zah/.condarc
          conda version : 4.5.9
    conda-build version : not installed
         python version : 3.7.0.final.0
       base environment : /home/zah/anaconda3  (writable)
           channel URLs : 
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/zah/anaconda3/pkgs
                          /home/zah/.conda/pkgs
       envs directories : /home/zah/anaconda3/envs
                          /home/zah/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.9 requests/2.19.1 CPython/3.7.0 Linux/4.15.0-30-generic linuxmint/19 glibc/2.27
                UID:GID : 1000:1000
             netrc file : /home/zah/.netrc
           offline mode : False

Nightly builds

It would be useful to have nightly (or weekly) dev builds of Arrow. It is used by many projects these days, releases at a 3-month cadence, and is challenging for many of its users to build (at least the Python folks).

What would need to happen to have nightly builds here? cc'ing @jakirkham , and at his suggestion (I think), @conda-forge/core

different errors for pyarrow-0.2.post and pyarrow-0.3.pre on ec2

Conda Environment on EC2

[steven.simpson@ip-10-20-74-154 ~]$ /mnt/var/lib/anaconda2/bin/conda info
Current conda install:

               platform : linux-64
          conda version : 4.2.13
       conda is private : False
      conda-env version : 4.2.13
    conda-build version : 1.21.3
         python version : 2.7.12.final.0
       requests version : 2.10.0
       root environment : /mnt/var/lib/anaconda2  (read only)
    default environment : /mnt/var/lib/anaconda2
       envs directories : /home/steven.simpson/.conda/envs
                          /mnt/var/lib/anaconda2/envs
          package cache : /home/steven.simpson/.conda/envs/.pkgs
                          /mnt/var/lib/anaconda2/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
           offline mode : False


[steven.simpson@ip-10-20-74-154 ~]$ /mnt/var/lib/anaconda2/bin/conda list
# packages in environment at /mnt/var/lib/anaconda2:
#
_nb_ext_conf              0.2.0                    py27_0  
alabaster                 0.7.8                    py27_0  
anaconda                  4.1.1               np111py27_0  
anaconda-client           1.4.0                    py27_0  
anaconda-navigator        1.2.1                    py27_0  
argcomplete               1.0.0                    py27_1  
astropy                   1.2.1               np111py27_0  
babel                     2.3.3                    py27_0  
backports                 1.0                      py27_0  
backports_abc             0.4                      py27_0  
beautifulsoup4            4.4.1                    py27_0  
bitarray                  0.8.1                    py27_0  
blaze                     0.10.1                   py27_0  
bokeh                     0.12.0                   py27_0  
boto                      2.40.0                   py27_0  
bottleneck                1.1.0               np111py27_0  
cairo                     1.12.18                       6  
cdecimal                  2.3                      py27_2  
cffi                      1.6.0                    py27_0  
chest                     0.2.3                    py27_0  
click                     6.6                      py27_0  
cloudpickle               0.2.1                    py27_0  
clyent                    1.2.2                    py27_0  
colorama                  0.3.7                    py27_0  
conda                     4.1.6                    py27_0  
conda-build               1.21.3                   py27_0  
conda-env                 2.5.1                    py27_0  
configobj                 5.0.6                    py27_0  
configparser              3.5.0b2                  py27_1  
contextlib2               0.5.3                    py27_0  
cryptography              1.4                      py27_0  
curl                      7.49.0                        0  
cycler                    0.10.0                   py27_0  
cython                    0.24                     py27_0  
cytoolz                   0.8.0                    py27_0  
dask                      0.10.0                   py27_0  
datashape                 0.5.2                    py27_0  
decorator                 4.0.10                   py27_0  
dill                      0.2.5                    py27_0  
docutils                  0.12                     py27_2  
dynd-python               0.7.2                    py27_0  
entrypoints               0.2.2                    py27_0  
enum34                    1.1.6                    py27_0  
et_xmlfile                1.0.1                    py27_0  
fastcache                 1.0.2                    py27_1  
flask                     0.11.1                   py27_0  
flask-cors                2.1.2                    py27_0  
fontconfig                2.11.1                        6  
freetype                  2.5.5                         1  
funcsigs                  1.0.2                    py27_0  
functools32               3.2.3.2                  py27_0  
futures                   3.0.5                    py27_0  
get_terminal_size         1.0.0                    py27_0  
gevent                    1.1.1                    py27_0  
greenlet                  0.4.10                   py27_0  
grin                      1.2.1                    py27_3  
h5py                      2.6.0               np111py27_1  
hdf5                      1.8.16                        0  
heapdict                  1.0.0                    py27_1  
idna                      2.1                      py27_0  
imagesize                 0.7.1                    py27_0  
ipaddress                 1.0.16                   py27_0  
ipykernel                 4.3.1                    py27_0  
ipython                   4.2.0                    py27_0  
ipython_genutils          0.1.0                    py27_0  
ipywidgets                4.1.1                    py27_0  
itsdangerous              0.24                     py27_0  
jbig                      2.1                           0  
jdcal                     1.2                      py27_1  
jedi                      0.9.0                    py27_1  
jinja2                    2.8                      py27_1  
jpeg                      8d                            1  
jsonschema                2.5.1                    py27_0  
jupyter                   1.0.0                    py27_3  
jupyter_client            4.3.0                    py27_0  
jupyter_console           4.1.1                    py27_0  
jupyter_core              4.1.0                    py27_0  
libdynd                   0.7.2                         0  
libffi                    3.2.1                         0  
libgfortran               3.0.0                         1  
libpng                    1.6.22                        0  
libsodium                 1.0.10                        0  
libtiff                   4.0.6                         2  
libxml2                   2.9.2                         0  
libxslt                   1.1.28                        0  
llvmlite                  0.11.0                   py27_0  
locket                    0.2.0                    py27_1  
lxml                      3.6.0                    py27_0  
markupsafe                0.23                     py27_2  
matplotlib                1.5.1               np111py27_0  
mistune                   0.7.2                    py27_0  
mkl                       11.3.3                        0  
mkl-service               1.1.2                    py27_2  
mpmath                    0.19                     py27_1  
multipledispatch          0.4.8                    py27_0  
nb_anacondacloud          1.1.0                    py27_0  
nb_conda                  1.1.0                    py27_0  
nb_conda_kernels          1.0.3                    py27_0  
nbconvert                 4.2.0                    py27_0  
nbformat                  4.0.1                    py27_0  
nbpresent                 3.0.2                    py27_0  
networkx                  1.11                     py27_0  
nltk                      3.2.1                    py27_0  
nose                      1.3.7                    py27_1  
notebook                  4.2.1                    py27_0  
numba                     0.26.0              np111py27_0  
numexpr                   2.6.0               np111py27_0  
numpy                     1.11.1                   py27_0  
odo                       0.5.0                    py27_1  
openpyxl                  2.3.2                    py27_0  
openssl                   1.0.2h                        1  
pandas                    0.18.1              np111py27_0  
partd                     0.3.4                    py27_0  
patchelf                  0.9                           0  
path.py                   8.2.1                    py27_0  
pathlib2                  2.1.0                    py27_0  
patsy                     0.4.1                    py27_0  
pep8                      1.7.0                    py27_0  
pexpect                   4.0.1                    py27_0  
pickleshare               0.7.2                    py27_0  
pillow                    3.2.0                    py27_1  
pip                       8.1.2                    py27_0  
pixman                    0.32.6                        0  
ply                       3.8                      py27_0  
psutil                    4.3.0                    py27_0  
ptyprocess                0.5.1                    py27_0  
py                        1.4.31                   py27_0  
pyasn1                    0.1.9                    py27_0  
pycairo                   1.10.0                   py27_0  
pycosat                   0.6.1                    py27_1  
pycparser                 2.14                     py27_1  
pycrypto                  2.6.1                    py27_4  
pycurl                    7.43.0                   py27_0  
pyflakes                  1.2.3                    py27_0  
pygments                  2.1.3                    py27_0  
pyopenssl                 0.16.0                   py27_0  
pyparsing                 2.1.4                    py27_0  
pyqt                      4.11.4                   py27_3  
pytables                  3.2.2               np111py27_4  
pytest                    2.9.2                    py27_0  
python                    2.7.12                        1  
python-dateutil           2.5.3                    py27_0  
pytz                      2016.4                   py27_0  
pyyaml                    3.11                     py27_4  
pyzmq                     15.2.0                   py27_1  
qt                        4.8.7                         3  
qtconsole                 4.2.1                    py27_0  
qtpy                      1.0.2                    py27_0  
readline                  6.2                           2  
redis                     3.2.0                         0  
redis-py                  2.10.5                   py27_0  
requests                  2.10.0                   py27_0  
rope                      0.9.4                    py27_1  
ruamel_yaml               0.11.7                   py27_0  
scikit-image              0.12.3              np111py27_1  
scikit-learn              0.17.1              np111py27_2  
scipy                     0.17.1              np111py27_1  
setuptools                23.0.0                   py27_0  
simplegeneric             0.8.1                    py27_1  
singledispatch            3.4.0.3                  py27_0  
sip                       4.16.9                   py27_0  
six                       1.10.0                   py27_0  
snowballstemmer           1.2.1                    py27_0  
sockjs-tornado            1.0.3                    py27_0  
sphinx                    1.4.1                    py27_0  
sphinx_rtd_theme          0.1.9                    py27_0  
spyder                    2.3.9                    py27_0  
sqlalchemy                1.0.13                   py27_0  
sqlite                    3.13.0                        0  
ssl_match_hostname        3.4.0.2                  py27_1  
statsmodels               0.6.1               np111py27_1  
sympy                     1.0                      py27_0  
terminado                 0.6                      py27_0  
tk                        8.5.18                        0  
toolz                     0.8.0                    py27_0  
tornado                   4.3                      py27_1  
traitlets                 4.2.1                    py27_0  
unicodecsv                0.14.1                   py27_0  
werkzeug                  0.11.10                  py27_0  
wheel                     0.29.0                   py27_0  
xlrd                      1.0.0                    py27_0  
xlsxwriter                0.9.2                    py27_0  
xlwt                      1.1.2                    py27_0  
xz                        5.2.2                         0  
yaml                      0.1.6                         0  
zeromq                    4.1.4                         0  
zlib                      1.2.8                         3

I install pyarrow ver 0.2.post because that's what I was able to get working on my machine.

[steven.simpson@ip-10-20-74-154 ~]$ sudo su
[root@ip-10-20-74-154 steven.simpson]# /mnt/var/lib/anaconda2/bin/conda install -c conda-forge pyarrow=0.2.post
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata .........
Solving package specifications: ..........

Package plan for installation in environment /mnt/var/lib/anaconda2:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-env-2.6.0            |                0         1017 B  conda-forge
    jemalloc-4.4.0             |                0         2.3 MB  conda-forge
    parquet-cpp-1.1.0pre       |                1         1.6 MB  conda-forge
    ruamel_yaml-0.11.14        |           py27_0         420 KB  conda-forge
    arrow-cpp-0.3.pre          |      np111py27_1         1.3 MB  conda-forge
    conda-4.2.13               |           py27_0         374 KB  conda-forge
    pyarrow-0.2.post           |      np111py27_0         3.3 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         9.2 MB

The following NEW packages will be INSTALLED:

    arrow-cpp:   0.3.pre-np111py27_1  conda-forge
    jemalloc:    4.4.0-0              conda-forge
    parquet-cpp: 1.1.0pre-1           conda-forge
    pyarrow:     0.2.post-np111py27_0 conda-forge

The following packages will be UPDATED:

    conda:       4.1.6-py27_0                     --> 4.2.13-py27_0  conda-forge
    conda-env:   2.5.1-py27_0                     --> 2.6.0-0        conda-forge
    ruamel_yaml: 0.11.7-py27_0                    --> 0.11.14-py27_0 conda-forge

Proceed ([y]/n)? y  

Fetching packages ...
conda-env-2.6. 100% |######################################################################| Time: 0:00:00   1.40 MB/s
jemalloc-4.4.0 100% |######################################################################| Time: 0:00:01   2.25 MB/s
parquet-cpp-1. 100% |######################################################################| Time: 0:00:00   1.97 MB/s
ruamel_yaml-0. 100% |######################################################################| Time: 0:00:00 911.23 kB/s
arrow-cpp-0.3. 100% |######################################################################| Time: 0:00:00   1.63 MB/s
conda-4.2.13-p 100% |######################################################################| Time: 0:00:00 659.74 kB/s
pyarrow-0.2.po 100% |######################################################################| Time: 0:00:00   3.72 MB/s
Extracting packages ...
[      COMPLETE      ]|#########################################################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|#########################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|#########################################################################################| 100%

When I try to import pyarrow I get a libarrow_io.so error

[steven.simpson@ip-10-20-74-154 ~]$ python
Python 2.7.12 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:42:40) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/var/lib/anaconda2/lib/python2.7/site-packages/pyarrow/__init__.py", line 28, in <module>
    import pyarrow.config
ImportError: libarrow_io.so: cannot open shared object file: No such file or directory
>>> quit()

So I try to update pyarrow to 0.3.pre, which I've had trouble with before

[steven.simpson@ip-10-20-74-154 ~]$ sudo su
[root@ip-10-20-74-154 steven.simpson]# /mnt/var/lib/anaconda2/bin/conda install -c conda-forge pyarrow
Fetching package metadata .........
Solving package specifications: ..........

Package plan for installation in environment /mnt/var/lib/anaconda2:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pyarrow-0.3.pre            |      np111py27_0         2.3 MB  conda-forge

The following packages will be UPDATED:

    pyarrow: 0.2.post-np111py27_0 conda-forge --> 0.3.pre-np111py27_0 conda-forge

Proceed ([y]/n)? yes

Fetching packages ...
pyarrow-0.3.pr 100% |######################################################################| Time: 0:00:01   2.25 MB/s
Extracting packages ...
[      COMPLETE      ]|#########################################################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|#########################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|#########################################################################################| 100%
[root@ip-10-20-74-154 steven.simpson]# exit

Then try to rerun import pyarrow in python interpreter and get a Numpy related error

[steven.simpson@ip-10-20-74-154 ~]$ python
Python 2.7.12 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:42:40) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/var/lib/anaconda2/lib/python2.7/site-packages/pyarrow/__init__.py", line 28, in <module>
    import pyarrow._config
ImportError: /mnt/var/lib/anaconda2/lib/python2.7/site-packages/pyarrow/_config.so: undefined symbol: _ZN5arrow2py9InitNumPyEv

What's weird and frustrating is that I can get pyarrow working on my local conda env:

alabaster                 0.7.9                    py27_0  
anaconda                  custom                   py27_0  
anaconda-client           1.6.0                    py27_0  
anaconda-navigator        1.5.0                    py27_0  
anaconda-project          0.4.1                    py27_0  
appnope                   0.1.0                    py27_0  
appscript                 1.0.1                    py27_0  
argcomplete               1.0.0                    py27_1  
arrow-cpp                 0.2.post                      0    conda-forge
astroid                   1.4.9                    py27_0  
astropy                   1.3                 np111py27_0  
babel                     2.3.4                    py27_0  
backports                 1.0                      py27_0  
backports_abc             0.5                      py27_0  
beautifulsoup4            4.5.3                    py27_0  
bitarray                  0.8.1                    py27_0  
blaze                     0.10.1                   py27_0  
bokeh                     0.12.5                   py27_0    conda-forge
boto                      2.45.0                   py27_0  
boto3                     1.4.4                    py27_0    conda-forge
botocore                  1.5.39                   py27_0    conda-forge
bottleneck                1.2.0               np111py27_0  
cdecimal                  2.3                      py27_2  
cffi                      1.9.1                    py27_0  
chardet                   2.3.0                    py27_0  
chest                     0.2.3                    py27_0  
click                     6.7                      py27_0  
cloudpickle               0.2.2                    py27_0  
clyent                    1.2.2                    py27_0  
colorama                  0.3.7                    py27_0  
conda                     4.3.16                   py27_0  
conda-env                 2.6.0                         0  
configobj                 5.0.6                    py27_0  
configparser              3.5.0                    py27_0  
contextlib2               0.5.4                    py27_0  
cryptography              1.7.1                    py27_0  
curl                      7.52.1                        0  
cycler                    0.10.0                   py27_0  
cython                    0.25.2                   py27_0  
cytoolz                   0.8.2                    py27_0  
dask                      0.14.1                   py27_0    conda-forge
datashape                 0.5.4                    py27_0  
decorator                 4.0.11                   py27_0  
dill                      0.2.5                    py27_0  
distributed               1.16.1                   py27_0    conda-forge
docutils                  0.13.1                   py27_0  
entrypoints               0.2.2                    py27_0  
enum34                    1.1.6                    py27_0  
et_xmlfile                1.0.1                    py27_0  
fastcache                 1.0.2                    py27_1  
feather-format            0.3.1                    py27_1    conda-forge
flask                     0.12                     py27_0  
flask-cors                3.0.2                    py27_0  
freetype                  2.7                           1    conda-forge
funcsigs                  1.0.2                    py27_0  
functools32               3.2.3.2                  py27_0  
futures                   3.0.5                    py27_0  
get_terminal_size         1.0.0                    py27_0  
gevent                    1.2.1                    py27_0  
graphviz                  2.38.0                        4    conda-forge
greenlet                  0.4.11                   py27_0  
grin                      1.2.1                    py27_3  
h5py                      2.6.0               np111py27_2  
hdf5                      1.8.17                        1  
heapdict                  1.0.0                    py27_1  
icu                       54.1                          0  
idna                      2.2                      py27_0  
imagesize                 0.7.1                    py27_0  
ipaddress                 1.0.18                   py27_0  
ipykernel                 4.5.2                    py27_0  
ipython                   5.1.0                    py27_1  
ipython_genutils          0.1.0                    py27_0  
ipywidgets                5.2.2                    py27_1  
isort                     4.2.5                    py27_0  
itsdangerous              0.24                     py27_0  
jbig                      2.1                           0  
jdcal                     1.3                      py27_0  
jedi                      0.9.0                    py27_1  
jemalloc                  4.4.0                         0    conda-forge
jinja2                    2.9.4                    py27_0  
jmespath                  0.9.2                    py27_0    conda-forge
jpeg                      9b                            0  
jsonschema                2.5.1                    py27_0  
jupyter                   1.0.0                    py27_0    conda-forge
jupyter_client            4.4.0                    py27_0  
jupyter_console           5.0.0                    py27_0  
jupyter_core              4.2.1                    py27_0  
lazy-object-proxy         1.2.2                    py27_0  
libiconv                  1.14                          0  
libpng                    1.6.28                        0    conda-forge
libtiff                   4.0.6                         3  
libxml2                   2.9.4                         0  
libxslt                   1.1.29                        0  
llvmlite                  0.15.0                   py27_0  
locket                    0.2.0                    py27_1  
lxml                      3.7.2                    py27_0  
markupsafe                0.23                     py27_2  
matplotlib                2.0.0               np111py27_3    conda-forge
mistune                   0.7.3                    py27_1  
mkl                       11.3.3                        0  
mkl-service               1.1.2                    py27_3  
mpmath                    0.19                     py27_1  
msgpack-python            0.4.8                    py27_0    conda-forge
multipledispatch          0.4.9                    py27_0  
nbconvert                 4.2.0                    py27_0  
nbformat                  4.2.0                    py27_0  
networkx                  1.11                     py27_0  
nltk                      3.2.2                    py27_0  
nose                      1.3.7                    py27_1  
notebook                  4.3.1                    py27_0  
numba                     0.30.1              np111py27_0  
numexpr                   2.6.1               np111py27_1  
numpy                     1.11.2                   py27_0  
numpydoc                  0.6.0                    py27_0  
odo                       0.5.0                    py27_1  
olefile                   0.44                     py27_0    conda-forge
openpyxl                  2.4.1                    py27_0  
openssl                   1.0.2k                        1  
pandas                    0.19.2              np111py27_1  
pandas-datareader         0.3.0.post0               <pip>
parquet-cpp               0.1.pre                      14    conda-forge
partd                     0.3.7                    py27_0  
path.py                   10.0                     py27_0  
pathlib2                  2.2.0                    py27_0  
patsy                     0.4.1                    py27_0  
pep8                      1.7.0                    py27_0  
pexpect                   4.2.1                    py27_0  
pickleshare               0.7.4                    py27_0  
pillow                    4.1.0                    py27_0    conda-forge
pip                       9.0.1                    py27_1  
ply                       3.9                      py27_0  
prompt_toolkit            1.0.9                    py27_0  
psutil                    5.0.1                    py27_0  
ptyprocess                0.5.1                    py27_0  
py                        1.4.32                   py27_0  
pyarrow                   0.2.post            np111py27_0    conda-forge
pyasn1                    0.1.9                    py27_0  
pyaudio                   0.2.7                    py27_0  
pycosat                   0.6.1                    py27_1  
pycparser                 2.17                     py27_0  
pycrypto                  2.6.1                    py27_4  
pycurl                    7.43.0                   py27_2  
pyflakes                  1.5.0                    py27_0  
pygments                  2.1.3                    py27_0  
pylint                    1.6.4                    py27_1  
pyopenssl                 16.2.0                   py27_0  
pyparsing                 2.1.4                    py27_0  
pyqt                      5.6.0                    py27_1  
pytables                  3.3.0               np111py27_0  
pytest                    3.0.5                    py27_0  
python                    2.7.13                        0  
python-dateutil           2.6.0                    py27_0  
python-graphviz           0.6                      py27_1    conda-forge
python.app                1.2                      py27_4  
pytz                      2016.10                  py27_0  
pyyaml                    3.12                     py27_0  
pyzmq                     16.0.2                   py27_0  
qt                        5.6.2                         0  
qtawesome                 0.4.3                    py27_0  
qtconsole                 4.2.1                    py27_1  
qtpy                      1.2.1                    py27_0  
readline                  6.2                           2  
redis                     3.2.0                         0  
redis-py                  2.10.5                   py27_0  
requests                  2.12.4                   py27_0  
requests-file             1.4.1                     <pip>
requests-ftp              0.3.1                     <pip>
rope                      0.9.4                    py27_1  
ruamel_yaml               0.11.14                  py27_1  
s3fs                      0.0.9                    py27_0    conda-forge
s3transfer                0.1.10                   py27_1    conda-forge
scandir                   1.4                      py27_0  
scikit-image              0.12.3              np111py27_1  
scikit-learn              0.17.1              np111py27_2  
scikit-learn              0.17                      <pip>
scipy                     0.18.1              np111py27_0  
seaborn                   0.7.1                    py27_0  
setuptools                27.2.0                   py27_0  
simplegeneric             0.8.1                    py27_1  
singledispatch            3.4.0.3                  py27_0  
sip                       4.18                     py27_0  
six                       1.10.0                   py27_0  
snowballstemmer           1.2.1                    py27_0  
sockjs-tornado            1.0.3                    py27_0  
sortedcontainers          1.5.3                    py27_0    conda-forge
sphinx                    1.5.1                    py27_0  
spyder                    3.1.2                    py27_0  
sqlalchemy                1.1.5                    py27_0  
sqlite                    3.13.0                        0  
ssl_match_hostname        3.4.0.2                  py27_1  
statsmodels               0.6.1               np111py27_1  
subprocess32              3.2.7                    py27_0  
sympy                     1.0                      py27_0  
tblib                     1.3.2                    py27_0    conda-forge
terminado                 0.6                      py27_0  
tk                        8.5.18                        0  
toolz                     0.8.2                    py27_0  
tornado                   4.4.2                    py27_0  
traitlets                 4.3.1                    py27_0  
unicodecsv                0.14.1                   py27_0  
wcwidth                   0.1.7                    py27_0  
werkzeug                  0.11.15                  py27_0  
wheel                     0.29.0                   py27_0  
widgetsnbextension        1.2.6                    py27_0  
wrapt                     1.10.8                   py27_0  
xlrd                      1.0.0                    py27_0  
xlsxwriter                0.9.6                    py27_0  
xlwings                   0.10.2                   py27_0  
xlwt                      1.2.0                    py27_0  
xz                        5.2.2                         1  
yaml                      0.1.6                         0  
zict                      0.1.1                    py27_0    conda-forge
zlib                      1.2.8                         3  




               platform : osx-64
          conda version : 4.2.13
       conda is private : False
      conda-env version : 4.2.13
    conda-build version : not installed
         python version : 2.7.13.final.0
       requests version : 2.12.4
       root environment : /Users/steve/anaconda  (writable)
    default environment : /Users/steve/anaconda
       envs directories : /Users/steve/anaconda/envs
          package cache : /Users/steve/anaconda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/osx-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/pro/osx-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
           offline mode : False

install fails

Issue:

When installing pyarrow there is a verification failed error:

Preparing transaction: done
Verifying transaction: failed

RemoveError: 'requests' is a dependency of conda and cannot be removed from
conda's operating environment.
RemoveError: 'setuptools' is a dependency of conda and cannot be removed from
conda's operating environment.

Environment (conda list): 
$ conda list

# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu    anaconda
absl-py                   0.7.0                 py36_1000    conda-forge
anaconda                  custom           py36hbbc8b67_0  
astor                     0.7.1                      py_0    conda-forge
attrs                     18.2.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
binutils_impl_linux-64    2.31.1               h6176602_1    conda-forge
binutils_linux-64         2.31.1               h6176602_3    conda-forge
blas                      1.0                         mkl  
bleach                    3.1.0                      py_0    conda-forge
c-ares                    1.15.0            h14c3975_1001    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py36_1000    conda-forge
cudatoolkit               9.2                           0    anaconda
cudnn                     7.2.1                 cuda9.2_0    anaconda
cupti                     9.2.148                       0    anaconda
decorator                 4.3.2                      py_0    conda-forge
entrypoints               0.3                   py36_1000    conda-forge
gast                      0.2.1.post0                py_0    conda-forge
gcc_impl_linux-64         7.3.0                habb00fd_1    conda-forge
gcc_linux-64              7.3.0                h553295d_3    conda-forge
grpcio                    1.16.1           py36hf8bcb03_1  
gxx_impl_linux-64         7.3.0                hdf63c60_1    conda-forge
gxx_linux-64              7.3.0                h553295d_3    conda-forge
h5py                      2.9.0           nompi_py36hf008753_1102    conda-forge
hdf5                      1.10.4          nompi_h11e915b_1105    conda-forge
intel-openmp              2019.1                      144  
ipykernel                 5.1.0           py36h24bf2e0_1002    conda-forge
ipython                   7.2.0           py36h24bf2e0_1000    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.13.2                py36_1000    conda-forge
jinja2                    2.10                       py_1    conda-forge
jsonschema                3.0.0a3               py36_1000    conda-forge
jupyter_client            5.2.4                      py_1    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterlab                0.35.4                   py36_0    conda-forge
jupyterlab_server         0.2.0                      py_0    conda-forge
keras                     2.2.4                         0    anaconda
keras-applications        1.0.6                    py36_0    anaconda
keras-base                2.2.4                    py36_0    anaconda
keras-preprocessing       1.0.5                    py36_0    anaconda
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libgpuarray               0.7.6             h14c3975_1003    conda-forge
libprotobuf               3.6.1             hdbcaa40_1000    conda-forge
libsodium                 1.0.16            h14c3975_1001    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1  
mako                      1.0.7                      py_1    conda-forge
markdown                  2.6.11                     py_0    conda-forge
markupsafe                1.1.0           py36h14c3975_1000    conda-forge
mistune                   0.8.4           py36h14c3975_1000    conda-forge
mkl                       2019.1                      144  
mkl_fft                   1.0.10           py36h14c3975_1    conda-forge
mkl_random                1.0.2            py36h637b7d7_2    conda-forge
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1                  he6710b0_1  
notebook                  5.7.4                 py36_1000    conda-forge
numpy                     1.15.4           py36h7e9f1db_0  
numpy-base                1.15.4           py36hde5b4d6_0  
openssl                   1.1.1a            h14c3975_1000    conda-forge
pandoc                    2.6                           1    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.3.4                      py_0    conda-forge
pexpect                   4.6.0                 py36_1000    conda-forge
pickleshare               0.7.5                 py36_1000    conda-forge
pip                       19.0.1                   py36_0  
prometheus_client         0.5.0                      py_0    conda-forge
prompt_toolkit            2.0.8                      py_0    conda-forge
protobuf                  3.6.1           py36hf484d3e_1001    conda-forge
ptyprocess                0.6.0                 py36_1000    conda-forge
pygments                  2.3.1                      py_0    conda-forge
pygpu                     0.7.6           py36h3010b51_1000    conda-forge
pyrsistent                0.14.10          py36h14c3975_0    conda-forge
python                    3.6.8                h0371630_0  
python-dateutil           2.8.0                      py_0    conda-forge
pyyaml                    3.13            py36h14c3975_1001    conda-forge
pyzmq                     17.1.2          py36h6afc9c9_1001    conda-forge
readline                  7.0                  h7b6447c_5  
scipy                     1.2.0            py36h7c811a0_0  
send2trash                1.5.0                      py_0    conda-forge
setuptools                40.7.3                   py36_0  
six                       1.12.0                py36_1000    conda-forge
sqlite                    3.26.0               h7b6447c_0  
tensorboard               1.12.2           py36he6710b0_0    anaconda
tensorflow                1.12.0          gpu_py36he74679b_0    anaconda
tensorflow-base           1.12.0          gpu_py36had579c0_0    anaconda
tensorflow-gpu            1.12.0               h0d30ee6_0    anaconda
termcolor                 1.1.0                      py_2    conda-forge
terminado                 0.8.1                 py36_1001    conda-forge
testpath                  0.4.2                 py36_1000    conda-forge
theano                    1.0.4           py36hf484d3e_1000    conda-forge
tk                        8.6.8                hbc83047_0  
tornado                   5.1.1           py36h14c3975_1000    conda-forge
traitlets                 4.3.2                 py36_1000    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
werkzeug                  0.14.1                     py_0    conda-forge
wheel                     0.32.3                   py36_0  
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7             h14c3975_1001    conda-forge
zeromq                    4.2.5             hf484d3e_1006    conda-forge
zlib                      1.2.11               h7b6447c_3

Details about conda and system ( conda info ):

Not sure why conda info says python 2.7, the python version for my env is 3.6.8

$ conda info

     active environment : ds
    active env location : /data/anaconda/envs/ds
            shell level : 2
       user config file : /home/peterquill/.condarc
 populated config files : 
          conda version : 4.6.3
    conda-build version : 3.10.5
         python version : 2.7.15.final.0
       base environment : /data/anaconda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /data/anaconda/pkgs
                          /home/peterquill/.conda/pkgs
       envs directories : /data/anaconda/envs
                          /home/peterquill/.conda/envs
               platform : linux-64
             user-agent : conda/4.6.3 requests/2.9.1 CPython/2.7.15 Linux/4.15.0-1037-azure ubuntu/16.04.5 glibc/2.23
                UID:GID : 1003:1003
             netrc file : None
           offline mode : False

Use of parquet-cpp 1.5.0.pre

I'm attempting to update Anaconda's arrow-cpp and pyarrow recipes. I'm having some trouble with how things have been changed. Specifically:

  • parquet-cpp 1.5.0 (tagged) relies on an arrow-cpp class that is not defined in arrow-cpp 0.10.0 (AllocateEmptyBitmap).
  • parquet-cpp 1.4.0 is broken by arrow-cpp 0.10.0.

It appears that it is only possible to build pyarrow with an untagged parquet-cpp version. We don't really like to build against arbitrary commits, as it seems you are doing with parquet-cpp right now (labelling the version .pre, which does not correspond to a tag, does not confer what software you're actually running).

The plan appears to be to fold parquet-cpp back into arrow-cpp (apache/parquet-cpp@apache-parquet-cpp-1.5.0...master#diff-af3b638bc2a3e6c650974192a53c7291R21) - can anyone comment on a timescale for that? Can there be an arrow-cpp 0.10.1 release that works with the tagged parquet-cpp 1.5.0 in the meantime?

ImportError: undefined symbol

Issue: ImportError: undefined symbol

Not sure if it's an Arrow issue or a Conda one. Or (more likely) just something broken in my environment.

I had pyarrow 0.11 installed at first. Downgrading to 0.10 and 0.9 didn't fix the problem.

import pyarrow.parquet as pq
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "/<redacted>/projects/<redacted>/conda-env/lib/python3.6/site-packages/pyarrow/__init__.py", line 47, in <module>
#     from pyarrow.lib import cpu_count, set_cpu_count
# ImportError: /<redacted>/projects/<redacted>/conda-env/lib/python3.6/site-packages/pyarrow/../../../libarrow.so.0: undefined symbol: _ZN5boost13match_resultsIN9__gnu_cxx17__normal_iteratorIPKcSsEESaINS_9sub_matchIS5_EEEE12maybe_assignERKS9_

Environment (conda list): 
# packages in environment at /opt/app/data/home_extenstion/gregory-werbin/projects/tts-classification/conda-env:
#
# Name                    Version                   Build  Channel
_py-xgboost-mutex         2.0                       cpu_0  
appdirs                   1.4.3                     <pip>
arrow-cpp                 0.9.0            py36h70250a7_7    conda-forge
asn1crypto                0.24.0                   py36_0  
atomicwrites              1.2.1                      py_0    conda-forge
attrs                     18.2.0                     py_0    conda-forge
backcall                  0.1.0                    py36_0  
bidict                    0.17.3                     py_0    conda-forge
blas                      1.0                         mkl  
boost-cpp                 1.67.0               h14c3975_4  
boto                      2.49.0                   py36_0  
boto3                     1.9.21                   py36_0  
botocore                  1.12.23                  py36_0  
bz2file                   0.98                      <pip>
bzip2                     1.0.6                h14c3975_5  
ca-certificates           2018.10.15           ha4d7672_0    conda-forge
category_encoders         1.3.0                      py_0    conda-forge
certifi                   2018.10.15            py36_1000    conda-forge
cffi                      1.11.5           py36he75722e_1  
chardet                   3.0.4                    py36_1  
ConfigArgParse            0.13.0                    <pip>
cryptography              2.3.1            py36hc365091_0  
cycler                    0.10.0                   py36_0  
cymem                     2.0.2            py36hfd86e86_0  
cytoolz                   0.9.0.1          py36h14c3975_1  
datrie                    0.7.1                     <pip>
dbus                      1.13.2               h714fa37_1  
decorator                 4.3.0                    py36_0  
dill                      0.2.8.2                  py36_0  
docutils                  0.14                     py36_0  
expat                     2.2.6                he6710b0_0  
fastparquet               0.1.6                    py36_0    conda-forge
fasttext                  0.1.0                hfc679d8_0    conda-forge
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_1  
gensim                    3.6.0                     <pip>
gitdb2                    2.0.5                     <pip>
GitPython                 2.1.11                    <pip>
glib                      2.56.2               hd408876_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
icu                       58.2                 h9c2bf20_1  
idna                      2.7                      py36_0  
imbalanced-learn          0.4.1                      py_0    conda-forge
intel-openmp              2019.0                      118  
ipykernel                 5.1.0            py36h39e3cac_0  
ipython                   7.0.1            py36h39e3cac_0  
ipython_genutils          0.2.0                    py36_0  
jedi                      0.13.1                   py36_0  
jmespath                  0.9.3                    py36_0  
jpeg                      9b                   h024ee3a_2  
jsonschema                2.6.0                     <pip>
jupyter_client            5.2.3                    py36_0  
jupyter_core              4.4.0                    py36_0  
kiwisolver                1.0.1            py36hf484d3e_0  
libboost                  1.67.0               h46d08c1_4  
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libpng                    1.6.35               hbc83047_0  
libsodium                 1.0.16               h1bed415_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxgboost                0.80                 hfc679d8_1    conda-forge
libxml2                   2.9.8                h26e45fe_1  
lightgbm                  2.2.1            py36hfc679d8_0    conda-forge
llvmlite                  0.25.0           py36hd28b015_1    conda-forge
matplotlib                3.0.0            py36h5429711_0  
mkl                       2019.0                      118  
mkl_fft                   1.0.6            py36h7dd41cf_0  
mkl_random                1.0.1            py36h4414c95_1  
more-itertools            4.3.0                 py36_1000    conda-forge
msgpack-numpy             0.4.3.2                  py36_0  
msgpack-python            0.5.6            py36h6bb024c_1  
murmurhash                1.0.1            py36he6710b0_0  
ncurses                   6.1                  hf484d3e_0  
nltk                      3.3.0                    py36_0  
numba                     0.40.0           py36hf8a1672_0    conda-forge
numpy                     1.15.2           py36h1d66e8a_1  
numpy-base                1.15.2           py36h81de0dd_1  
openssl                   1.0.2p               h470a237_1    conda-forge
pandas                    0.23.4           py36h04863e7_0  
parquet-cpp               1.4.0                h83d4a3d_2    conda-forge
parso                     0.3.1                    py36_0  
patsy                     0.5.0                    py36_0  
pcre                      8.42                 h439df22_0  
pexpect                   4.6.0                    py36_0  
pickleshare               0.7.5                    py36_0  
pip                       10.0.1                   py36_0  
plac                      0.9.6                    py36_0  
pluggy                    0.8.0                      py_0    conda-forge
preshed                   2.0.1            py36he6710b0_0  
prompt_toolkit            2.0.6                    py36_0  
ptyprocess                0.6.0                    py36_0  
py                        1.7.0                      py_0    conda-forge
py-xgboost                0.80             py36hfc679d8_1    conda-forge
pyarrow                   0.9.0            py36hfc679d8_2    conda-forge
pycparser                 2.19                     py36_0  
pygments                  2.2.0                    py36_0  
pyopenssl                 18.0.0                   py36_0  
pyparsing                 2.2.2                    py36_0  
pyqt                      5.9.2            py36h05f1152_2  
pysocks                   1.6.8                    py36_0  
pytest                    3.9.1                 py36_1000    conda-forge
python                    3.6.6                h6e4f718_2  
python-dateutil           2.7.3                    py36_0  
pytz                      2018.5                   py36_0  
PyYAML                    3.13                      <pip>
pyzmq                     17.1.2           py36h14c3975_0  
qt                        5.9.6                h8703b6f_2  
ratelimiter               1.2.0.post0               <pip>
readline                  7.0                  h7b6447c_5  
regex                     2018.08.29       py36h7b6447c_0  
requests                  2.19.1                   py36_0  
s3transfer                0.1.13                   py36_0  
scikit-learn              0.20.0           py36h4989274_1  
scipy                     1.1.0            py36hfa4b5c9_1  
setuptools                40.4.3                   py36_0  
simplegeneric             0.8.1                    py36_2  
sip                       4.19.8           py36hf484d3e_0  
six                       1.11.0                   py36_1  
smart_open                1.7.1                    py36_0  
smmap2                    2.0.5                     <pip>
snakemake                 5.3.0                     <pip>
spacy                     2.0.16           py36h962f231_0  
sqlite                    3.25.2               h7b6447c_0  
statsmodels               0.9.0            py36h035aef0_0  
thinc                     6.12.0           py36h4989274_0  
thrift                    0.11.0           py36hfc679d8_1    conda-forge
tk                        8.6.8                hbc83047_0  
toolz                     0.9.0                    py36_0  
tornado                   5.1.1            py36h7b6447c_0  
tqdm                      4.27.0             pyha03479a_0    <redacted>
traitlets                 4.3.2                    py36_0  
ujson                     1.35             py36h14c3975_0  
urllib3                   1.23                     py36_0  
wcwidth                   0.1.7                    py36_0  
wheel                     0.32.1                   py36_0  
wrapt                     1.10.11          py36h14c3975_2  
xlrd                      1.1.0                    py36_1  
xz                        5.2.4                h14c3975_4  
zeromq                    4.2.5                hf484d3e_1  
zlib                      1.2.11               ha838bed_2

Details about conda and system ( conda info ): 

     active environment : /<redacted>/projects/<redacted>/conda-env
    active env location : /<redacted>/projects/<redacted>/conda-env
            shell level : 1
       user config file : /<redacted>/.condarc
 populated config files : /<redacted>/.condarc
          conda version : 4.5.0
    conda-build version : 3.0.31
         python version : 3.6.1.final.0
       base environment : /<redacted>/anaconda3  (read only)
           channel URLs : <redacted>/linux-64
                          <redacted>/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /opt/app/anaconda3/pkgs
                          /<redacted>/.conda/pkgs
       envs directories : /<redacted>/.conda/envs
                          /<redacted>/envs
               platform : linux-64
             user-agent : conda/4.5.0 requests/2.18.4 CPython/3.6.1 Linux/3.10.0-514.26.2.el7.x86_64 rhel/7.3 glibc/2.17
                UID:GID : 51757:51714
             netrc file : None
           offline mode : False

Fastparquet vs pyarrow file size

I moved from fastparquet to pyarrow after this post
I'm wondering why saving a dataframe to .parq with snappy compression lead to a bigger file using pyarrow.

I generated a dataframe using the function generate_data from the linked post and saved to files

pq.write_table(pa.Table.from_pandas(df), 'csv/df_pa.parq', compression='SNAPPY')
fastparquet.write("csv/df_fp.parq", df, compression='SNAPPY')

Using pyarrow the wall time is twice as long as using fastparquet (I guess the culprit is pa.Table.from_pandas) and df_pa.parq's size is 513.7 MB vs 490.2 MB of df_fp.parq
Using other df I found an even bigger difference. Is there a way to have a better compression?

ImportError on Ubuntu 16.10 

╭─miki@xubi  arrow (git:integration) 
╰─<conda:arrow>$ conda install -c conda-forge pyarrow            Mon Feb 27, 07:18 
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /opt/miniconda3/envs/arrow:

The following packages will be UPDATED:

    pyarrow: 0.1.post-np111py36_7 conda-forge --> 0.2.post-np111py36_0 conda-forge

Proceed ([y]/n)? 

pyarrow-0.2.po 100% |####################################| Time: 0:00:02   1.65 MB/s
╭─miki@xubi  arrow (git:integration) 
╰─<conda:arrow>$ conda install ipython                           Mon Feb 27, 07:18 
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /opt/miniconda3/envs/arrow:

The following packages will be UPDATED:

    ipython: 5.2.2-py36_0 --> 5.3.0-py36_0

Proceed ([y]/n)? 

╭─miki@xubi  arrow (git:integration) 
╰─<conda:arrow>$ ipy                                             Mon Feb 27, 07:19 
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 12:22:00) 
Type "copyright", "credits" or "license" for more information.

IPython 5.3.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import pyarrow
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-57968f417579> in <module>()
----> 1 import pyarrow

/opt/miniconda3/envs/arrow/lib/python3.6/site-packages/pyarrow/__init__.py in <module>()
     26 
     27 
---> 28 import pyarrow.config
     29 from pyarrow.config import cpu_count, set_cpu_count
     30 

ImportError: /opt/miniconda3/envs/arrow/lib/python3.6/site-packages/pyarrow/./libpyarrow.so: undefined symbol: _ZN5arrow2io9Writeable5FlushEv

In [2]:                                                                             
╭─miki@xubi  arrow (git:integration) 
╰─<conda:arrow>$ source deactivate
╭─miki@xubi  arrow (git:integration) 
╰─$ lsb_release -d                                               Mon Feb 27, 07:41 
Description:	Ubuntu 16.10

build issue, missing boost symbol on macOS 

ImportError                               Traceback (most recent call last)
~/anaconda3/envs/tmpfiona/lib/python3.6/site-packages/pandas/io/feather_format.py in _try_import()
     13     try:
---> 14         import feather
     15     except ImportError:

~/anaconda3/envs/tmpfiona/lib/python3.6/site-packages/feather/__init__.py in <module>()
     16 
---> 17 from feather.api import (read_dataframe, write_dataframe,
     18                          FeatherError, FeatherReader, FeatherWriter)

~/anaconda3/envs/tmpfiona/lib/python3.6/site-packages/feather/api.py in <module>()
     14 
---> 15 from pyarrow.feather import (read_feather as read_dataframe,  # noqa
     16                              write_feather as write_dataframe,

~/anaconda3/envs/tmpfiona/lib/python3.6/site-packages/pyarrow/__init__.py in <module>()
     46 
---> 47 from pyarrow.lib import cpu_count, set_cpu_count
     48 from pyarrow.lib import (null, bool_,

ImportError: dlopen(/Users/rgommers/anaconda3/envs/tmpfiona/lib/python3.6/site-packages/pyarrow/lib.cpython-36m-darwin.so, 2): Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
  Referenced from: /Users/rgommers/anaconda3/envs/tmpfiona/lib/libparquet.1.dylib
  Expected in: /Users/rgommers/anaconda3/envs/tmpfiona/lib/libboost_regex.dylib
 in /Users/rgommers/anaconda3/envs/tmpfiona/lib/libparquet.1.dylib

Just installed:

    arrow-cpp:      0.9.0-py36h1ae9da6_7 conda-forge
    feather-format: 0.4.0-py36_2         conda-forge
    parquet-cpp:    1.4.0-h83d4a3d_1     conda-forge
    pyarrow:        0.9.0-py36_1         conda-forge

Full package list:

# packages in environment at /Users/rgommers/anaconda3/envs/tmpfiona:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.11                    <pip>
altair                    2.0.1                      py_0    conda-forge
appnope                   0.1.0                    py36_0    conda-forge
arrow-cpp                 0.9.0            py36h1ae9da6_7    conda-forge
Babel                     2.6.0                     <pip>
backcall                  0.1.0                      py_0    conda-forge
bleach                    2.1.3                      py_0    conda-forge
bokeh                     0.12.16                  py36_0    conda-forge
boost                     1.66.0                   py36_1    conda-forge
boost-cpp                 1.66.0                        1    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.4.16                     0    conda-forge
cairo                     1.14.10                       0    conda-forge
certifi                   2018.4.16                py36_0    conda-forge
chardet                   3.0.4                     <pip>
clangdev                  6.0.0                 default_0    conda-forge
click                     6.7                        py_1    conda-forge
click-plugins             1.0.3                    py36_0    conda-forge
cligj                     0.4.0                    py36_0    conda-forge
curl                      7.60.0                        0    conda-forge
cycler                    0.10.0                   py36_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
descartes                 1.1.0                      py_1    conda-forge
docutils                  0.14                      <pip>
entrypoints               0.2.3                    py36_1    conda-forge
esda                      1.0.1.dev0                <pip>
expat                     2.2.5                         0    conda-forge
feather-format            0.4.0                    py36_2    conda-forge
fiona                     1.7.11                   py36_3    conda-forge
fontconfig                2.12.6                        0    conda-forge
freetds                   1.00.44                       2    conda-forge
freetype                  2.8.1                         0    conda-forge
freexl                    1.0.5                         0    conda-forge
gdal                      2.2.4                    py36_0    conda-forge
geopandas                 0.3.0                    py36_0    conda-forge
geopandas                 0.3.0+63.g6e0fd9b           <pip>
geos                      3.6.2                         1    conda-forge
geotiff                   1.4.2                         1    conda-forge
gettext                   0.19.8.1                      0    conda-forge
giddy                     1.1.1                     <pip>
giflib                    5.1.4                         0    conda-forge
gitdb2                    2.0.3                     <pip>
GitPython                 2.1.10                    <pip>
glib                      2.55.0                        0    conda-forge
hdf4                      4.2.13                        0    conda-forge
hdf5                      1.10.1                        2    conda-forge
html5lib                  1.0.1                      py_0    conda-forge
icu                       58.2                          0    conda-forge
idna                      2.7                       <pip>
imagesize                 1.0.0                     <pip>
intel-openmp              2018.0.0                      8  
ipykernel                 4.8.2                    py36_0    conda-forge
ipython                   6.4.0                    py36_0    conda-forge
ipython_genutils          0.2.0                    py36_0    conda-forge
ipywidgets                7.2.1                    py36_1    conda-forge
jedi                      0.12.0                   py36_0    conda-forge
jinja2                    2.10                     py36_0    conda-forge
jpeg                      9b                            2    conda-forge
json-c                    0.12.1                        0    conda-forge
jsonschema                2.6.0                    py36_1    conda-forge
jupyter_client            5.2.3                    py36_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterlab                0.32.1                   py36_0    conda-forge
jupyterlab_launcher       0.10.5                   py36_0    conda-forge
kealib                    1.4.7                         4    conda-forge
kiwisolver                1.0.1                    py36_1    conda-forge
krb5                      1.14.6                        0    conda-forge
libcxx                    6.0.0                         0    conda-forge
libdap4                   3.18.3                        2    conda-forge
libedit                   3.1.20170329                  0    conda-forge
libffi                    3.2.1                         3    conda-forge
libgdal                   2.2.4                         2    conda-forge
libgfortran               3.0.1                h93005f0_2  
libiconv                  1.15                          0    conda-forge
libkml                    1.3.0                         6    conda-forge
libnetcdf                 4.6.1                         2    conda-forge
libopenblas               0.2.20               hdc02c5d_4  
libpng                    1.6.34                        0    conda-forge
libpq                     9.6.3                         0    conda-forge
libpysal                  3.0.6                     <pip>
libsodium                 1.0.16                        0    conda-forge
libspatialindex           1.8.5                         1    conda-forge
libspatialite             4.3.0a                       19    conda-forge
libssh2                   1.8.0                         2    conda-forge
libtiff                   4.0.9                         0    conda-forge
libxml2                   2.9.8                         0    conda-forge
llvm-meta                 6.0.0                         0    conda-forge
llvmdev                   6.0.0                 default_2    conda-forge
mapclassify               1.0.1                     <pip>
markupsafe                1.0                      py36_0    conda-forge
matplotlib                2.2.2                    py36_1    conda-forge
mistune                   0.8.3                    py36_1    conda-forge
mkl                       2018.0.2                      1  
munch                     2.3.2                      py_0    conda-forge
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                    py36_0    conda-forge
ncurses                   5.9                          10    conda-forge
nose                      1.3.7                    py36_2    conda-forge
notebook                  5.5.0                    py36_0    conda-forge
numpy                     1.14.3           py36he6379a5_1  
numpy-base                1.14.3           py36h7ef55bc_1  
numpydoc                  0.8.0                     <pip>
openjpeg                  2.3.0                         2    conda-forge
openssl                   1.0.2o                        0    conda-forge
packaging                 17.1                       py_0    conda-forge
palettable                3.1.1                     <pip>
pandas                    0.23.0                   py36_1    conda-forge
pandoc                    2.2.1                hde52d81_0    conda-forge
pandocfilters             1.4.2                    py36_0    conda-forge
parquet-cpp               1.4.0                h83d4a3d_1    conda-forge
parso                     0.2.1                      py_0    conda-forge
patsy                     0.5.0                    py36_0    conda-forge
pcre                      8.41                          1    conda-forge
pexpect                   4.5.0                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0    conda-forge
pip                       9.0.3                    py36_0    conda-forge
pixman                    0.34.0                        2    conda-forge
poppler                   0.61.1                        3    conda-forge
poppler-data              0.4.9                         0    conda-forge
proj4                     4.9.3                         5    conda-forge
prompt_toolkit            1.0.15                   py36_0    conda-forge
psycopg2                  2.7.4                    py36_0    conda-forge
ptyprocess                0.5.2                    py36_0    conda-forge
pyarrow                   0.9.0                    py36_1    conda-forge
pygments                  2.2.0                    py36_0    conda-forge
pymssql                   2.1.3.post16             py36_0    conda-forge
pyparsing                 2.2.0                    py36_0    conda-forge
pyproj                    1.9.5.1                  py36_0    conda-forge
pysal                     1.14.3                   py36_0    conda-forge
python                    3.6.5                         1    conda-forge
python-dateutil           2.7.3                     <pip>
python-dateutil           2.7.3                      py_0    conda-forge
pytz                      2018.4                     py_0    conda-forge
pyyaml                    3.12                     py36_1    conda-forge
pyzmq                     17.0.0                   py36_4    conda-forge
readline                  7.0                           0    conda-forge
requests                  2.19.1                    <pip>
rtree                     0.8.3                    py36_0    conda-forge
scipy                     1.1.0            py36hcaad992_0  
seaborn                   0.8.1                    py36_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                39.2.0                   py36_0    conda-forge
shapely                   1.6.4                    py36_0    conda-forge
simplegeneric             0.8.1                    py36_0    conda-forge
six                       1.11.0                   py36_1    conda-forge
smmap2                    2.0.3                     <pip>
snowballstemmer           1.2.1                     <pip>
Sphinx                    1.7.5                     <pip>
sphinx-rtd-theme          0.4.0                     <pip>
sphinxcontrib-websupport  1.1.0                     <pip>
splot                     1.0.0.dev0                <pip>
sqlalchemy                1.2.7            py36h65ede16_0    conda-forge
sqlite                    3.20.1                        2    conda-forge
statsmodels               0.9.0                    py36_0    conda-forge
terminado                 0.8.1                    py36_0    conda-forge
testpath                  0.3.1                    py36_0    conda-forge
tk                        8.6.7                         0    conda-forge
toolz                     0.9.0                      py_0    conda-forge
tornado                   5.0.2                    py36_0    conda-forge
traitlets                 4.3.2                    py36_0    conda-forge
typing                    3.6.4                    py36_0    conda-forge
unixodbc                  2.3.5                h09ba92c_1    conda-forge
urllib3                   1.23                      <pip>
vega_datasets             0.5.0                      py_0    conda-forge
wcwidth                   0.1.7                    py36_0    conda-forge
webencodings              0.5.1                    py36_0    conda-forge
wheel                     0.31.0                   py36_0    conda-forge
widgetsnbextension        3.2.1                    py36_0    conda-forge
xerces-c                  3.2.0                         0    conda-forge
xlrd                      1.1.0                      py_2    conda-forge
xz                        5.2.3                         0    conda-forge
yaml                      0.1.7                         0    conda-forge
zeromq                    4.2.5                         1    conda-forge
zlib                      1.2.11               h470a237_2    conda-forge

Upgrade pyarrow defaults to 0.16.0

Issue:

When install pyarrow in clean linux conda environment (2020.02):

conda install -c conda-forge pyarrow

The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    arrow-cpp-0.11.1           |py37h0e61e49_1004         6.3 MB  conda-forge
    boost-cpp-1.68.0           |    h11c811c_1000        20.5 MB  conda-forge
    conda-4.8.3                |   py37hc8dfbb8_1         3.0 MB  conda-forge
    libprotobuf-3.6.1          |    hdbcaa40_1001         4.0 MB  conda-forge
    parquet-cpp-1.5.1          |                3           3 KB  conda-forge
    pyarrow-0.11.1             |py37hbbcf98d_1002         2.0 MB  conda-forge
    python_abi-3.7             |          1_cp37m           4 KB  conda-forge
    thrift-cpp-0.12.0          |    h0a07b25_1002         2.4 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        38.2 MB

The default version is pyarrow-0.11.1, while conda repo actually has the latest version 0.16.0 ( https://anaconda.org/conda-forge/pyarrow ).

Specify the version does not help at all:

conda install -c conda-forge pyarrow=0.16.0

Workaround

I have to manually download below packages from conda then install them locally:

arrow-cpp-0.16.0-py37hb0edad2_0.tar.bz2
aws-sdk-cpp-1.7.164-h1f8afcc_0.tar.bz2
boost-cpp-1.70.0-h8e57a91_2.tar.bz2
brotli-1.0.7-he1b5a44_1000.tar.bz2
c-ares-1.15.0-h516909a_1001.tar.bz2
gflags-2.2.2-he1b5a44_1002.tar.bz2
glog-0.4.0-he1b5a44_1.tar.bz2
grpc-cpp-1.25.0-h213be95_2.tar.bz2
libprotobuf-3.11.3-h8b12597_0.tar.bz2
lz4-c-1.8.3-he1b5a44_1001.tar.bz2
parquet-cpp-1.5.1-1.tar.bz2
pyarrow-0.16.0-py37h8b68381_1.tar.bz2
re2-2020.01.01-he1b5a44_0.tar.bz2
snappy-1.1.8-he1b5a44_1.tar.bz2
thrift-cpp-0.12.0-hf3afdfd_1004.tar.bz2
zstd-1.4.4-h3b9ef0a_1.tar.bz2


Details about conda and system ( conda info ): 
$ conda info
     active environment : base
    active env location : /home/dwang/anaconda3
            shell level : 1
       user config file : /home/dwang/.condarc
 populated config files :
          conda version : 4.8.2
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __glibc=2.27
       base environment : /home/dwang/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/dwang/anaconda3/pkgs
                          /home/dwang/.conda/pkgs
       envs directories : /home/dwang/anaconda3/envs
                          /home/dwang/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Linux/5.0.0-1033-gcp ubuntu/18.04.4 glibc/2.27
                UID:GID : 1001:1002
             netrc file : None
           offline mode : False

Import error with pyarrow 0.13 on mac.

Issue:

I've got the following error with pyarrow 0.13 on mac:

>>> import pyarrow as pa
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ueshin/workspace/pytmp/miniconda/envs/pytmp_3.6_pa0.13/lib/python3.6/site-packages/pyarrow/__init__.py", line 47, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: dlopen(/Users/ueshin/workspace/pytmp/miniconda/envs/pytmp_3.6_pa0.13/lib/python3.6/site-packages/pyarrow/lib.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libglog.0.dylib
  Referenced from: /Users/ueshin/workspace/pytmp/miniconda/envs/pytmp_3.6_pa0.13/lib/libarrow.13.dylib
  Reason: Incompatible library version: libarrow.13.dylib requires version 1.0.0 or later, but libglog.0.dylib provides version 0.0.0

With Python 3.5 and 3.7 the same error was raised as well.


Environment (conda list): 
$ conda list

# Name                    Version                   Build  Channel
appnope                   0.1.0                 py36_1000    conda-forge
arrow-cpp                 0.13.0           py36hbfb945d_2    conda-forge
attrs                     19.1.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
boost-cpp                 1.68.0            h6f8c590_1000    conda-forge
brotli                    1.0.7             h0a44026_1000    conda-forge
bzip2                     1.0.6             h1de35cc_1002    conda-forge
ca-certificates           2019.3.9             hecc5488_0    conda-forge
certifi                   2019.3.9                 py36_0    conda-forge
decorator                 4.4.0                      py_0    conda-forge
defusedxml                0.5.0                      py_1    conda-forge
entrypoints               0.3                   py36_1000    conda-forge
gflags                    2.2.2             h0a44026_1001    conda-forge
glog                      0.4.0                h6de7cb9_0    conda-forge
icu                       58.2              h0a44026_1000    conda-forge
ipykernel                 5.1.0           py36h24bf2e0_1002    conda-forge
ipython                   7.5.0            py36h24bf2e0_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.4.2                      py_0    conda-forge
jedi                      0.13.3                   py36_0    conda-forge
jinja2                    2.10.1                     py_0    conda-forge
jpeg                      9c                h1de35cc_1001    conda-forge
jsonschema                3.0.1                    py36_0    conda-forge
jupyter                   1.0.0                      py_2    conda-forge
jupyter_client            5.2.4                      py_3    conda-forge
jupyter_console           6.0.0                      py_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
libblas                   3.8.0                8_openblas    conda-forge
libcblas                  3.8.0                8_openblas    conda-forge
libcxx                    8.0.0                         2    conda-forge
libcxxabi                 8.0.0                         2    conda-forge
libffi                    3.2.1             h6de7cb9_1006    conda-forge
libgfortran               3.0.1                         0    conda-forge
liblapack                 3.8.0                8_openblas    conda-forge
libpng                    1.6.37               h2573ce8_0    conda-forge
libprotobuf               3.7.1                hfbae3c0_0    conda-forge
libsodium                 1.0.16            h1de35cc_1001    conda-forge
lz4-c                     1.8.3             h6de7cb9_1001    conda-forge
markupsafe                1.1.1            py36h1de35cc_0    conda-forge
mistune                   0.8.4           py36h1de35cc_1000    conda-forge
nbconvert                 5.5.0                      py_0    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1               h0a44026_1002    conda-forge
notebook                  5.7.8                    py36_0    conda-forge
numpy                     1.16.3           py36hdf140aa_0    conda-forge
openblas                  0.3.6                hd44dcd8_1    conda-forge
openssl                   1.1.1b               h01d97ff_2    conda-forge
pandas                    0.24.2           py36h0a44026_0    conda-forge
pandoc                    2.7.2                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.4.0                      py_0    conda-forge
pexpect                   4.7.0                    py36_0    conda-forge
pickleshare               0.7.5                 py36_1000    conda-forge
pip                       19.1                     py36_0    conda-forge
prometheus_client         0.6.0                      py_0    conda-forge
prompt_toolkit            2.0.9                      py_0    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pyarrow                   0.13.0           py36hb37e6aa_0    conda-forge
pygments                  2.3.1                      py_0    conda-forge
pyqt                      5.6.0           py36hc26a216_1008    conda-forge
pyrsistent                0.15.1           py36h01d97ff_0    conda-forge
python                    3.6.7             h8dc6b48_1004    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
pytz                      2019.1                     py_0    conda-forge
pyzmq                     18.0.1           py36h2d07e9b_1    conda-forge
qt                        5.6.2             h822fa55_1013    conda-forge
qtconsole                 4.4.3                      py_0    conda-forge
re2                       2019.04.01           h6de7cb9_0    conda-forge
readline                  7.0               hcfe32e1_1001    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                41.0.1                   py36_0    conda-forge
sip                       4.18.1          py36h0a44026_1000    conda-forge
six                       1.12.0                py36_1000    conda-forge
snappy                    1.1.7             h0a44026_1002    conda-forge
sqlite                    3.26.0            h1765d9f_1001    conda-forge
terminado                 0.8.2                    py36_0    conda-forge
testpath                  0.4.2                   py_1001    conda-forge
thrift-cpp                0.12.0            hdedcb34_1002    conda-forge
tk                        8.6.9             ha441bb4_1001    conda-forge
tornado                   6.0.2            py36h01d97ff_0    conda-forge
traitlets                 4.3.2                 py36_1000    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.33.1                   py36_0    conda-forge
widgetsnbextension        3.4.2                 py36_1000    conda-forge
xz                        5.2.4             h1de35cc_1001    conda-forge
zeromq                    4.3.1             h0a44026_1000    conda-forge
zlib                      1.2.11            h1de35cc_1004    conda-forge
zstd                      1.3.3                         1    conda-forge

Details about conda and system ( conda info ): 
$ conda info

     active environment : /Users/ueshin/workspace/pytmp/miniconda/envs/pytmp_3.6_pa0.13
    active env location : /Users/ueshin/workspace/pytmp/miniconda/envs/pytmp_3.6_pa0.13
            shell level : 1
       user config file : /Users/ueshin/.condarc
 populated config files : /Users/ueshin/.condarc
          conda version : 4.6.14
    conda-build version : not installed
         python version : 3.7.1.final.0
       base environment : /usr/local/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/osx-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /usr/local/miniconda3/pkgs
                          /Users/ueshin/.conda/pkgs
       envs directories : /usr/local/miniconda3/envs
                          /Users/ueshin/.conda/envs
               platform : osx-64
             user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.1 Darwin/18.5.0 OSX/10.14.4
                UID:GID : 501:20
             netrc file : None
           offline mode : False

NumPy ABI warning

Got this in a fresh environment installed this morning

In [1]: import pyarrow as pa                                                                   
/home/wesm/miniconda/envs/pyarrow-conda-test/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)

This is concerning, should we rebuild the package using latest dependencies?

Failed to import pyarrow with ModuleNotFoundError (Windows 10)

OS: Windows 10 64
Installed anaconda 5.1.0
installed pyarrow as:

conda config --add channels conda-forge
conda install pyarrow

so far so good

created jupyter notebook
with the code cell:

import pandas as pd
import numpy as np
import pyarrow as pa
df1=pd.read_csv("./myCsv.csv")
table = pa.Table.from_pandas(df1)

Error I got:

ModuleNotFoundError Traceback (most recent call last)
in ()
1 import numpy as np
----> 2 import pyarrow as pa
3 table = pa.Table.from_pandas(df1)
4 table

ModuleNotFoundError: No module named 'pyarrow'

pyarrow==0.15.1 requires Python 3.7?

We're still on Python 3.6

$ conda install conda-forge::pyarrow==0.15.1

produces following conflicts:

Package openssl conflicts for:
python==3.6.6 -> openssl[version='>=1.0.2o,<1.0.3a|>=1.0.2p,<1.0.3a']
conda-forge::pyarrow==0.15.1 -> python[version='>=3.7,<3.8.0a0'] -> openssl[version='>=1.1.1a,<1.1.2a|>=1.1.1b,<1.1.2a|>=1.1.1c,<1.1.2a|>=1.1.1d,<1.1.2a']
Package python-dateutil conflicts for:
conda-forge::pyarrow==0.15.1 -> pandas -> python-dateutil[version='>=2.5.*|>=2.6.1']
Package python conflicts for:
python==3.6.6

Notice conda-forge::pyarrow==0.15.1 -> python[version='>=3.7,<3.8.0a0'] part

pyarrow==0.15.1 requires Python 3.7?

conda-build warnings

Building the feedstock currently issues the following warnings:

Adding .* to spec 'arrow-cpp 0.12.0' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING:conda_build.utils:Adding .* to spec 'arrow-cpp 0.12.0' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
Adding .* to spec 'parquet-cpp 1.5.1' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING:conda_build.utils:Adding .* to spec 'parquet-cpp 1.5.1' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level

I will release 0.12.0 once it passes and fix them in a follow-up PR.

No Builds for 0.6,win,py35

The current 0.6.0 package has not been built for the combination of Windows and Python 3.5, thus packages depending on it will fail for this combination.

OSX Support

Currently we only have Linux builds. Is there a problem with the OSX ones?

Fresh Windows environment crashing on Appveyor

I have an appveyor build that installs a fresh conda environment (with conda-forge added as channel, with strict channel priority etc), but pyarrow is crashing when trying to write/read a parquet file ("Windows fatal exception: access violation").

Example build: https://ci.appveyor.com/project/jorisvandenbossche/geopandas-us6w0/builds/32270758/job/i08hhck39jb53exk

And the env yaml file: https://github.com/geopandas/geopandas/blob/3e23acfa083f2f03aa1711e541e893b265a729a5/ci/travis/37-latest-conda-forge.yaml
It includes a set of packages from the geospatial stack (not sure if that is related)

The conda env is listed here: https://ci.appveyor.com/project/jorisvandenbossche/geopandas-us6w0/builds/32270758/job/i08hhck39jb53exk#L435 (there are some defaults packages installed, like mkl and intel_openmp, not sure if this can be the reason or how to prevent them from being installed)

Environment (conda list):

$ conda list
# packages in environment at C:\Miniconda37-x64\envs\test:
#
# Name                    Version                   Build  Channel
abseil-cpp                20200225.1           he025d50_2    conda-forge
affine                    2.3.0                      py_0    conda-forge
appdirs                   1.4.3                      py_1    conda-forge
arrow-cpp                 0.16.0           py37h8247c2b_3    conda-forge
atomicwrites              1.3.0                      py_0    conda-forge
attrs                     19.3.0                     py_0    conda-forge
aws-requests-auth         0.4.2                      py_0    conda-forge
aws-sdk-cpp               1.7.164          vc14h867dc94_1  [vc14]  conda-forge
backcall                  0.1.0                      py_0    conda-forge
beautifulsoup4            4.9.0            py37hc8dfbb8_0    conda-forge
blas                      1.0                         mkl    conda-forge
bleach                    3.1.4              pyh9f0ad1d_0    conda-forge
blosc                     1.18.1               h6538335_0    conda-forge
boost-cpp                 1.72.0               h0caebb8_0    conda-forge
boto3                     1.12.41            pyh9f0ad1d_0    conda-forge
botocore                  1.15.41            pyh9f0ad1d_0    conda-forge
brotli                    1.0.7             he025d50_1001    conda-forge
bzip2                     1.0.8                hfa6e2cd_2    conda-forge
c-ares                    1.15.0            h2fa13f4_1001    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
certifi                   2020.4.5.1       py37hc8dfbb8_0    conda-forge
cffi                      1.14.0           py37ha419a9e_0    conda-forge
cfitsio                   3.470                hfa6e2cd_2    conda-forge
chardet                   3.0.4           py37hc8dfbb8_1006    conda-forge
click                     7.1.1              pyh8c360ce_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.5.0                      py_0    conda-forge
codecov                   2.0.19             pyh9f0ad1d_0    conda-forge
colorama                  0.4.3                      py_0    conda-forge
coverage                  5.1              py37h4ab8f01_0    conda-forge
cryptography              2.8              py37hb32ad35_1    conda-forge
curl                      7.69.1               h1dcc11c_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
decorator                 4.4.2                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
deprecated                1.2.8              pyh9f0ad1d_0    conda-forge
descartes                 1.1.0                      py_4    conda-forge
dnspython                 1.16.0                     py_1    conda-forge
docutils                  0.15.2                   py37_0    conda-forge
ecdsa                     0.13                       py_0    conda-forge
entrypoints               0.3             py37hc8dfbb8_1001    conda-forge
esda                      2.2.1                      py_0    conda-forge
expat                     2.2.9                he025d50_2    conda-forge
fastcache                 1.1.0            py37h8055547_1    conda-forge
fiona                     1.8.13           py37hb7fdc2d_0    conda-forge
flask                     1.1.2              pyh9f0ad1d_0    conda-forge
flask_cors                3.0.8                      py_0    conda-forge
flask_json                0.3.4                      py_0    conda-forge
freetype                  2.10.1               ha9979f8_0    conda-forge
freexl                    1.0.5             hd288d7e_1002    conda-forge
gdal                      3.0.4            py37hd44be9e_7    conda-forge
geographiclib             1.50                       py_0    conda-forge
geopandas                 0.7.0                      py_1    conda-forge
geopy                     1.21.0                     py_0    conda-forge
geos                      3.8.1                he025d50_0    conda-forge
geotiff                   1.5.1               h3d29ae3_10    conda-forge
gettext                   0.19.8.1          hb01d8f6_1002    conda-forge
gflags                    2.2.2             he025d50_1002    conda-forge
giddy                     2.3.0                      py_0    conda-forge
glib                      2.64.2               he4de6d7_0    conda-forge
glog                      0.4.0                h0174b99_3    conda-forge
grpc-cpp                  1.28.1               hb1a2610_1    conda-forge
hdf4                      4.2.13            hf8e6fe8_1003    conda-forge
hdf5                      1.10.6          nompi_ha405e13_100    conda-forge
icc_rt                    2019.0.0             h0cc432a_1    defaults
icu                       64.2                 he025d50_1    conda-forge
idna                      2.9                        py_1    conda-forge
importlib-metadata        1.6.0            py37hc8dfbb8_0    conda-forge
importlib_metadata        1.6.0                         0    conda-forge
inequality                1.0.0                      py_0    conda-forge
intel-openmp              2019.4                      245    defaults
ipykernel                 5.2.0            py37h5ca1d4c_1    conda-forge
ipython                   7.13.0           py37hc8dfbb8_2    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.5.1                      py_0    conda-forge
itsdangerous              1.1.0                      py_0    conda-forge
jedi                      0.17.0           py37hc8dfbb8_0    conda-forge
jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
jmespath                  0.9.5                      py_0    conda-forge
joblib                    0.14.1                     py_0    conda-forge
jpeg                      9c                hfa6e2cd_1001    conda-forge
jsonlines                 1.2.0                      py_1    conda-forge
jsonschema                3.2.0            py37hc8dfbb8_1    conda-forge
jupyter_client            6.1.3                      py_0    conda-forge
jupyter_core              4.6.3            py37hc8dfbb8_1    conda-forge
kealib                    1.4.13               h3b59ab9_1    conda-forge
kiwisolver                1.2.0            py37heaa310e_0    conda-forge
krb5                      1.17.1               hdd46e55_0    conda-forge
libblas                   3.8.0                    14_mkl    conda-forge
libcblas                  3.8.0                    14_mkl    conda-forge
libclang                  9.0.1           default_hf44288c_0    conda-forge
libcurl                   7.69.1               h1dcc11c_0    conda-forge
libffi                    3.2.1             h6538335_1007    conda-forge
libgdal                   3.0.4                hf164de3_7    conda-forge
libiconv                  1.15              hfa6e2cd_1006    conda-forge
libkml                    1.3.0             h7e985d0_1011    conda-forge
liblapack                 3.8.0                    14_mkl    conda-forge
libnetcdf                 4.7.4           nompi_h256d12c_102    conda-forge
libpng                    1.6.37               hfe6a214_1    conda-forge
libpq                     12.2                 hd9aa61d_1    conda-forge
libprotobuf               3.11.4               h1a1b453_0    conda-forge
libpysal                  4.2.2                      py_0    conda-forge
libsodium                 1.0.17               h2fa13f4_0    conda-forge
libspatialindex           1.9.3                he025d50_3    conda-forge
libspatialite             4.3.0a            h51df0ed_1038    conda-forge
libssh2                   1.8.2                h642c060_2    conda-forge
libtiff                   4.1.0                h885aae3_6    conda-forge
libwebp-base              1.1.0                hfa6e2cd_3    conda-forge
libxml2                   2.9.10               h9ce36c8_0    conda-forge
llvmlite                  0.31.0           py37hc8dfbb8_1    conda-forge
lz4-c                     1.9.2                h33f27b4_0    conda-forge
m2w64-expat               2.1.1                         2    defaults
m2w64-gcc-libgfortran     5.3.0                         6    defaults
m2w64-gcc-libs            5.3.0                         7    defaults
m2w64-gcc-libs-core       5.3.0                         7    defaults
m2w64-gettext             0.19.7                        2    defaults
m2w64-gmp                 6.1.0                         2    defaults
m2w64-libiconv            1.14                          6    defaults
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    defaults
m2w64-xz                  5.2.2                         2    defaults
mapclassify               2.2.0                      py_0    conda-forge
markupsafe                1.1.1            py37h8055547_1    conda-forge
matplotlib                3.2.1                         0    conda-forge
matplotlib-base           3.2.1            py37h911224e_0    conda-forge
mgwr                      2.1.1                      py_0    conda-forge
mistune                   0.8.4           py37h8055547_1001    conda-forge
mkl                       2019.4                      245    defaults
mkl-service               2.3.0            py37hfa6e2cd_0    conda-forge
mock                      3.0.5            py37hc8dfbb8_1    conda-forge
more-itertools            8.2.0                      py_0    conda-forge
mpmath                    1.1.0                      py_0    conda-forge
msys2-conda-epoch         20160418                      1    defaults
munch                     2.5.0                      py_0    conda-forge
nbconvert                 5.6.1            py37hc8dfbb8_1    conda-forge
nbformat                  5.0.4                      py_0    conda-forge
nose                      1.3.7           py37hc8dfbb8_1004    conda-forge
notebook                  6.0.3                    py37_0    conda-forge
numba                     0.48.0           py37he350917_0    conda-forge
numexpr                   2.7.1            py37h63f7a3c_1    conda-forge
numpy                     1.18.1           py37h90d3380_1    conda-forge
openjpeg                  2.3.1                h57dd2e7_3    conda-forge
openssl                   1.1.1f               hfa6e2cd_0    conda-forge
packaging                 20.1                       py_0    conda-forge
pandas                    1.0.3            py37h3bbf574_1    conda-forge
pandoc                    2.9.2.1                       0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.7.0              pyh9f0ad1d_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pcre                      8.44                 h6538335_0    conda-forge
pickleshare               0.7.5           py37hc8dfbb8_1001    conda-forge
pip                       20.0.2                     py_2    conda-forge
pluggy                    0.13.0                   py37_0    conda-forge
pointpats                 2.1.0                      py_1    conda-forge
poppler                   0.67.0               h1707e21_8    conda-forge
poppler-data              0.4.9                         1    conda-forge
postgresql                12.2                 he14cc48_1    conda-forge
proj                      7.0.0                haa36216_3    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt-toolkit            3.0.5                      py_0    conda-forge
psycopg2                  2.8.5            py37h26f1ce3_1    conda-forge
py                        1.8.1                      py_0    conda-forge
pyarrow                   0.16.0           py37h8702343_2    conda-forge
pycparser                 2.20                       py_0    conda-forge
pycryptodome              3.9.7            py37h8055547_1    conda-forge
pygeos                    0.7.1            py37h422e0ae_0    conda-forge
pygments                  2.6.1                      py_0    conda-forge
pyopenssl                 19.1.0                     py_1    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyproj                    2.6.0            py37he833962_1    conda-forge
pyqt                      5.12.3           py37h6538335_1    conda-forge
pyqt5-sip                 4.19.18                  pypi_0    pypi
pyqtwebengine             5.12.1                   pypi_0    pypi
pyrsistent                0.16.0           py37h8055547_0    conda-forge
pysal                     2.2.0                      py_0    conda-forge
pysocks                   1.7.1            py37hc8dfbb8_1    conda-forge
pytables                  3.6.1            py37h2d87964_2    conda-forge
pytest                    5.4.1            py37hc8dfbb8_0    conda-forge
pytest-cov                2.8.1                      py_0    conda-forge
python                    3.7.6           h60c2a47_5_cpython    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2019.3                     py_0    conda-forge
pywin32                   227              py37hfa6e2cd_0    conda-forge
pywinpty                  0.5.7                    py37_0    conda-forge
pyyaml                    5.3.1            py37h8055547_0    conda-forge
pyzmq                     19.0.0           py37h8c16cda_1    conda-forge
qt                        5.12.5               h7ef1ec2_0    conda-forge
quantecon                 0.4.5                    py37_1    conda-forge
quilt3                    3.1.13           py37hc8dfbb8_0    conda-forge
rasterio                  1.1.3            py37h2617b1b_0    conda-forge
rasterstats               0.14.0                     py_0    conda-forge
re2                       2020.04.01       vc14h6538335_0  [vc14]  conda-forge
requests                  2.23.0             pyh8c360ce_2    conda-forge
requests-futures          1.0.0            py37hc8dfbb8_1    conda-forge
rtree                     0.9.4            py37h804a536_1    conda-forge
s3transfer                0.3.3            py37hc8dfbb8_1    conda-forge
scikit-learn              0.22.2.post1     py37h7208079_0    conda-forge
scipy                     1.3.1            py37h29ff71c_0    conda-forge
seaborn                   0.10.0                     py_1    conda-forge
segregation               1.2.0                      py_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                46.1.3           py37hc8dfbb8_0    conda-forge
shapely                   1.7.0            py37he1cf020_3    conda-forge
simplejson                3.17.0           py37hfa6e2cd_0    conda-forge
six                       1.14.0                     py_1    conda-forge
snappy                    1.1.8                he025d50_1    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
soupsieve                 1.9.4            py37hc8dfbb8_1    conda-forge
spaghetti                 1.4.2.post2                py_0    conda-forge
spglm                     1.0.7                      py_0    conda-forge
spint                     1.0.6                      py_0    conda-forge
splot                     1.1.3                      py_0    conda-forge
spreg                     1.1.1                      py_0    conda-forge
spvcm                     0.3.0                      py_0    conda-forge
sqlalchemy                1.3.16           py37h8055547_0    conda-forge
sqlite                    3.30.1               hfa6e2cd_0    conda-forge
statsmodels               0.11.1           py37h8055547_1    conda-forge
sympy                     1.5.1            py37hc8dfbb8_3    conda-forge
tbb                       2018.0.5             he980bc4_0    conda-forge
tenacity                  6.1.0            py37hc8dfbb8_1    conda-forge
terminado                 0.8.3            py37hc8dfbb8_1    conda-forge
testpath                  0.4.4                      py_0    conda-forge
thrift-cpp                0.13.0               h1907cbf_2    conda-forge
tiledb                    1.7.7                h0b90766_1    conda-forge
tk                        8.6.10               hfa6e2cd_0    conda-forge
tobler                    0.2.2                      py_0    conda-forge
tornado                   6.0.4            py37hfa6e2cd_0    conda-forge
tqdm                      4.45.0             pyh9f0ad1d_0    conda-forge
traitlets                 4.3.3            py37hc8dfbb8_1    conda-forge
urllib3                   1.24.3                   py37_0    conda-forge
vc                        14.1                 h869be7e_1    conda-forge
vs2015_runtime            14.16.27012          h30e32a0_2    conda-forge
wcwidth                   0.1.9              pyh9f0ad1d_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
widgetsnbextension        3.5.1                    py37_0    conda-forge
win_inet_pton             1.1.0                    py37_0    conda-forge
wincertstore              0.2                   py37_1003    conda-forge
winpty                    0.4.3                         4    conda-forge
wrapt                     1.12.1           py37h8055547_1    conda-forge
xerces-c                  3.2.2             h6538335_1004    conda-forge
xz                        5.2.5                h2fa13f4_0    conda-forge
yaml                      0.2.3                he774522_0    conda-forge
zeromq                    4.3.2                h6538335_2    conda-forge
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11            h2fa13f4_1006    conda-forge
zstd                      1.4.4                h9f78265_3    conda-forge

Details about conda and system ( conda info ): (this is before the fresh env is created) 
$ conda info
     active environment : base
    active env location : C:\Miniconda37-x64
            shell level : 1
       user config file : C:\Users\appveyor\.condarc
 populated config files : C:\Users\appveyor\.condarc
          conda version : 4.8.3
    conda-build version : not installed
         python version : 3.7.4.final.0
       virtual packages : 
       base environment : C:\Miniconda37-x64  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Miniconda37-x64\pkgs
                          C:\Users\appveyor\.conda\pkgs
                          C:\Users\appveyor\AppData\Local\conda\conda\pkgs
       envs directories : C:\Miniconda37-x64\envs
                          C:\Users\appveyor\.conda\envs
                          C:\Users\appveyor\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.3 requests/2.23.0 CPython/3.7.4 Windows/2012ServerR2 Windows/6.3.9600
          administrator : True
             netrc file : None
           offline mode : False

ImportError: No module named _config (when importing pyarrow.parquet)

I have the same issue as in:

#20 (comment)

import pyarrow.parquet
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/init.py", line 28, in
import pyarrow._config
ImportError: No module named _config

I used instructions at https://arrow.apache.org/docs/python/install.html to build arrow/cpp, parquet-cpp, and then pyarrow, with the following deviations (I view them as possibly bugs in the instructions):

  1. arrow/cpp build:
    export ARROW_HOME=$HOME/local
    I had to specify -DARROW_PYTHON=on and -DPARQUET_ARROW=ON to the cmake command (besides the -DCMAKE_INSTALL_PREFIX=$ARROW_HOME)

  2. parquet-cpp build:

    export ARROW_HOME=$HOME/local

    cmake -DARROW_HOME=$HOME/local -DPARQUET_ARROW_LINKAGE=static -DPARQUET_ARROW=ON .
    make

    sudo make install ----> this installs parquet libs in the std systems location (/usr/local/lib) so that the pyarrow build (see below) can find the parquet libs

  3. pyarrow build:

    export ARROW_HOME=$HOME/local (not a deviation; just repeating here)

    export LD_LIBRARY_PATH=$HOME/local/lib:$HOME/parquet4/parquet-cpp/build/latest

    sudo python setup.py build_ext --with-parquet --with-jemalloc --build-type=release install

    sudo python setup.py install

    (sudo is needed to install in /usr/local/lib/python2.7/dist-packages )

These are the steps and modifications to the instructions needed for me to build the pyarrow.parquet package. However, when I now try to import the package I get the error specified above.

Maybe I did something wrong in my steps which I kind of put together by searching for these issues...but really can't tell what. It took me almost a whole day to get to the point where I can build pyarrow and parquet, and now I can't use what I built.

Any comments, help appreciated! Thanks in advance.

How to install CUDA enabled pyarrow?

Issue:

Running

conda create -n arrow-test arrow-cpp pyarrow "arrow-cpp-proc=*=cuda"

suggests

The following NEW packages will be INSTALLED:
  ...
  arrow-cpp          conda-forge/linux-64::arrow-cpp-0.17.1-py38h217c556_2_cuda
  arrow-cpp-proc     conda-forge/linux-64::arrow-cpp-proc-1.0.0-cuda
  ...
  pyarrow            conda-forge/linux-64::pyarrow-0.17.1-py38h56ab25f_0

which is wrong as there is no CUDA enabled pyarrow 0.17 available yet. Is there a way to prevent such installation suggestion?

The following conda install combinations lead to correct installation suggestions:

conda create -n arrow-test "arrow-cpp=0.16" "pyarrow=0.16" "arrow-cpp-proc=*=cuda"
conda create -n arrow-test arrow-cpp "pyarrow=0.16" "arrow-cpp-proc=*=cuda"
conda create -n arrow-test "pyarrow=0.16" "arrow-cpp-proc=*=cuda"
conda create -n arrow-test "pyarrow=0.16=*cuda"

but the following do not:

conda create -n arrow-test arrow-cpp pyarrow "arrow-cpp-proc=*=cuda"
conda create -n arrow-test "arrow-cpp=0.16" pyarrow "arrow-cpp-proc=*=cuda"

Considering the above, there exist two issues:

  • one is able to install arrow-cpp and pyarrow with different versions and build_ext that will likely lead to conflicts at the usage level.
  • arrow-cpp-proc is not always effective (that is understandable since the current pyarrow=0.17 does not use it) and seems to be not really needed from the user's perspective.

Installing with py2.7 causing errors

Issue:

Unable to install pyarrow==0.13.0 with py27.
The following command fails for me -

/root/miniconda3/bin/conda create --name py27_env python==2.7.5 pyarrow==0.13.0 -c conda-forge

Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - pyarrow==0.13.0 -> arrow-cpp[version='>=0.13.0,<0.14.0a0,>=0.13.0,<1.0a0'] -> python[version='>=3.6,<3.7.0a0'] -> openssl[version='>=1.1.1a,<1.1.2a']
  - pyarrow==0.13.0 -> arrow-cpp[version='>=0.13.0,<0.14.0a0,>=0.13.0,<1.0a0'] -> python[version='>=3.6,<3.7.0a0'] -> tk[version='>=8.6.9,<8.7.0a0']
  - python==2.7.5
Use "conda search <package> --info" to see the dependencies for each package.

If I roll back pyarrow to v0.11.1, I am able to create the environment -

/root/miniconda3/bin/conda create --name py27_env python==2.7.5 pyarrow==0.11.1 -c conda-forge

but I run into this issue -

Python 2.7.5 |Continuum Analytics, Inc.| (default, Nov  4 2013, 15:30:26)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/py27_env/lib/python2.7/site-packages/pyarrow/__init__.py", line 54, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: /root/miniconda3/envs/py27_env/lib/python2.7/site-packages/pyarrow/lib.so: undefined symbol: _ZNK5arrow11RecordBatch11column_nameB5cxx11Ei

## Package Plan ##

  environment location: /root/miniconda3/envs/py27_env

  added / updated specs:
    - pyarrow==0.11.1
    - python==2.7.5


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    arrow-cpp-0.11.1           |   py27h3bd774a_1         6.1 MB  conda-forge
    certifi-2019.3.9           |           py27_0         149 KB  conda-forge
    futures-3.2.0              |        py27_1000          24 KB  conda-forge
    numpy-1.16.4               |   py27h95a1406_0         4.2 MB  conda-forge
    openssl-1.0.1k             |                1         2.6 MB
    pandas-0.24.2              |   py27hb3f55d8_0        10.9 MB  conda-forge
    parquet-cpp-1.5.1          |                3           3 KB  conda-forge
    pip-19.1                   |           py27_0         1.8 MB  conda-forge
    pyarrow-0.11.1             |py27hbbcf98d_1002         1.9 MB  conda-forge
    python-2.7.5               |                3        12.3 MB
    setuptools-41.0.1          |           py27_0         650 KB  conda-forge
    six-1.12.0                 |        py27_1000          21 KB  conda-forge
    sqlite-3.19.3              |                1         1.3 MB  conda-forge
    system-5.8                 |                2         170 KB
    thrift-cpp-0.11.0          |                2         2.6 MB  conda-forge
    wheel-0.33.4               |           py27_0          34 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        44.9 MB

The following NEW packages will be INSTALLED:

  arrow-cpp          conda-forge/linux-64::arrow-cpp-0.11.1-py27h3bd774a_1
  boost-cpp          conda-forge/linux-64::boost-cpp-1.68.0-h11c811c_1000
  bzip2              conda-forge/linux-64::bzip2-1.0.6-h14c3975_1002
  certifi            conda-forge/linux-64::certifi-2019.3.9-py27_0
  futures            conda-forge/linux-64::futures-3.2.0-py27_1000
  icu                conda-forge/linux-64::icu-58.2-hf484d3e_1000
  libblas            conda-forge/linux-64::libblas-3.8.0-7_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-7_openblas
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1
  libgfortran        conda-forge/linux-64::libgfortran-3.0.0-1
  liblapack          conda-forge/linux-64::liblapack-3.8.0-7_openblas
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1
  ncurses            conda-forge/linux-64::ncurses-5.9-10
  numpy              conda-forge/linux-64::numpy-1.16.4-py27h95a1406_0
  openblas           conda-forge/linux-64::openblas-0.3.5-ha44fe06_0
  openssl            pkgs/free/linux-64::openssl-1.0.1k-1
  pandas             conda-forge/linux-64::pandas-0.24.2-py27hb3f55d8_0
  parquet-cpp        conda-forge/noarch::parquet-cpp-1.5.1-3
  pip                conda-forge/linux-64::pip-19.1-py27_0
  pyarrow            conda-forge/linux-64::pyarrow-0.11.1-py27hbbcf98d_1002
  python             pkgs/free/linux-64::python-2.7.5-3
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.0-py_0
  pytz               conda-forge/noarch::pytz-2019.1-py_0
  readline           conda-forge/linux-64::readline-6.2-0
  setuptools         conda-forge/linux-64::setuptools-41.0.1-py27_0
  six                conda-forge/linux-64::six-1.12.0-py27_1000
  sqlite             conda-forge/linux-64::sqlite-3.19.3-1
  system             pkgs/free/linux-64::system-5.8-2
  thrift-cpp         conda-forge/linux-64::thrift-cpp-0.11.0-2
  tk                 conda-forge/linux-64::tk-8.5.19-2
  wheel              conda-forge/linux-64::wheel-0.33.4-py27_0
  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001
  zlib               conda-forge/linux-64::zlib-1.2.11-h14c3975_1004
Any pointers around what might be going wrong?
Environment (conda list): 
$ conda list

# packages in environment at /root/miniconda3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0
ca-certificates           2019.1.23                     0
certifi                   2019.3.9                 py37_0
cffi                      1.12.2           py37h2e261b9_1
chardet                   3.0.4                    py37_1
conda                     4.6.14                   py37_0
cryptography              2.6.1            py37h1ba5d50_0
idna                      2.8                      py37_0
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libstdcxx-ng              8.2.0                hdf63c60_1
ncurses                   6.1                  he6710b0_1
openssl                   1.1.1b               h7b6447c_1
pip                       19.0.3                   py37_0
pycosat                   0.6.3            py37h14c3975_0
pycparser                 2.19                     py37_0
pyopenssl                 19.0.0                   py37_0
pysocks                   1.6.8                    py37_0
python                    3.7.3                h0371630_0
readline                  7.0                  h7b6447c_5
requests                  2.21.0                   py37_0
ruamel_yaml               0.15.46          py37h14c3975_0
setuptools                41.0.0                   py37_0
six                       1.12.0                   py37_0
sqlite                    3.27.2               h7b6447c_0
tk                        8.6.8                hbc83047_0
urllib3                   1.24.1                   py37_0
wheel                     0.33.1                   py37_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zlib                      1.2.11               h7b6447c_3

Details about conda and system ( conda info ): 
$ conda info

     active environment : None
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.6.14
    conda-build version : not installed
         python version : 3.7.3.final.0
       base environment : /root/miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /root/miniconda3/pkgs
                          /root/.conda/pkgs
       envs directories : /root/miniconda3/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.3 Linux/4.15.0-45-generic ubuntu/16.04.6 glibc/2.23
                UID:GID : 0:0
             netrc file : None
           offline mode : False

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.