conda-forge / scrapy-feedstock

A conda-smithy repository for scrapy.

About scrapy-feedstock

Feedstock license: BSD-3-Clause

Home: https://scrapy.org/

Package license: BSD-3-Clause-Clear

Summary: A high-level Python Screen Scraping framework

Development: https://github.com/scrapy/scrapy

Documentation: https://docs.scrapy.org

Scrapy is an open source and collaborative framework for extracting the data you need from websites in a fast, simple, yet extensible way.

Current build status

Azure

Variant Status linux_64_python3.10.____cpython linux_64_python3.11.____cpython linux_64_python3.12.____cpython linux_64_python3.8.____cpython linux_64_python3.9.____cpython linux_aarch64_python3.10.____cpython linux_aarch64_python3.11.____cpython linux_aarch64_python3.12.____cpython linux_aarch64_python3.8.____cpython linux_aarch64_python3.9.____cpython osx_64_python3.10.____cpython osx_64_python3.11.____cpython osx_64_python3.12.____cpython osx_64_python3.8.____cpython osx_64_python3.9.____cpython osx_arm64_python3.10.____cpython osx_arm64_python3.11.____cpython osx_arm64_python3.12.____cpython osx_arm64_python3.8.____cpython osx_arm64_python3.9.____cpython win_64_python3.10.____cpython win_64_python3.11.____cpython win_64_python3.12.____cpython win_64_python3.8.____cpython win_64_python3.9.____cpython

Current release info

Name	Downloads	Version	Platforms

Installing scrapy

Installing scrapy from the conda-forge channel can be achieved by adding conda-forge to your channels with:

conda config --add channels conda-forge
conda config --set channel_priority strict

Once the conda-forge channel has been enabled, scrapy can be installed with conda:

conda install scrapy

or with mamba:

mamba install scrapy

It is possible to list all of the versions of scrapy available on your platform with conda:

conda search scrapy --channel conda-forge

or with mamba:

mamba search scrapy --channel conda-forge

Alternatively, mamba repoquery may provide more information:

# Search all versions available on your platform:
mamba repoquery search scrapy --channel conda-forge

# List packages depending on `scrapy`:
mamba repoquery whoneeds scrapy --channel conda-forge

# List dependencies of `scrapy`:
mamba repoquery depends scrapy --channel conda-forge

About conda-forge

conda-forge is a community-led conda channel of installable packages. In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.

A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services. Thanks to the awesome service provided by Azure, GitHub, CircleCI, AppVeyor, Drone, and TravisCI it is possible to build and upload installable packages to the conda-forge anaconda.org channel for Linux, Windows and OSX respectively.

To manage the continuous integration and simplify feedstock maintenance conda-smithy has been developed. Using the conda-forge.yml within this repository, it is possible to re-render all of this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender.

For more information please check the conda-forge documentation.

Terminology

feedstock - the conda recipe (raw material), supporting scripts and CI configuration.

conda-smithy - the tool which helps orchestrate the feedstock. Its primary use is in the construction of the CI .yml files and simplify the management of many feedstocks.

conda-forge - the place where the feedstock and smithy live and work to produce the finished article (built conda distributions)

Updating scrapy-feedstock

If you would like to improve the scrapy recipe or build a new package version, please fork this repository and submit a PR. Upon submission, your changes will be run on the appropriate platforms to give the reviewer an opportunity to confirm that the changes result in a successful build. Once merged, the recipe will be re-built and uploaded automatically to the conda-forge channel, whereupon the built conda packages will be available for everybody to install and use from the conda-forge channel. Note that all branches in the conda-forge/scrapy-feedstock are immediately built and any created packages are uploaded, so PRs should be based on branches in forks and branches in the main repository should only be used to build distinct package versions.

In order to produce a uniquely identifiable distribution:

• If the version of a package is not being increased, please add or increase the build/number.
• If the version of a package is being increased, please remember to return the build/number back to 0.

Feedstock Maintainers

• @Gallaecio
• @dangra
• @kmike
• @rmax
• @wRAR

scrapy-feedstock's Issues

Missing package on v2.7.0

Packages mentioned in #75 seems could not be ignored, otherwise it will fail with missing module

Packages found by source code inspection but not in the meta.yaml:

• h2
• hpack
• packaging

Packages found in the meta.yaml but not found by source code inspection:

• pywin32

❯ scrapy crawl  statement
Traceback (most recent call last):
  File "/usr/local/conda/envs/mas/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/local/conda/envs/mas/lib/python3.10/site-packages/scrapy/cmdline.py", line 133, in execute
    cmds = _get_commands_dict(settings, inproject)
  File "/usr/local/conda/envs/mas/lib/python3.10/site-packages/scrapy/cmdline.py", line 61, in _get_commands_dict
    cmds = _get_commands_from_module('scrapy.commands', inproject)
  File "/usr/local/conda/envs/mas/lib/python3.10/site-packages/scrapy/cmdline.py", line 42, in _get_commands_from_module
    for cmd in _iter_command_classes(module):
  File "/usr/local/conda/envs/mas/lib/python3.10/site-packages/scrapy/cmdline.py", line 29, in _iter_command_classes
    for module in walk_modules(module_name):
  File "/usr/local/conda/envs/mas/lib/python3.10/site-packages/scrapy/utils/misc.py", line 88, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/conda/envs/mas/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/conda/envs/mas/lib/python3.10/site-packages/scrapy/commands/parse.py", line 12, in <module>
    from scrapy.utils import display
  File "/usr/local/conda/envs/mas/lib/python3.10/site-packages/scrapy/utils/display.py", line 8, in <module>
    from packaging.version import Version as parse_version
ModuleNotFoundError: No module named 'packaging'

Issue with conda-forge scrapy>1.6.0 on Windows

Issue: Already reported in the scrapy repo: scrapy/scrapy#4289

When installing scrapy from conda-forge, no error appears during the process but when trying to run scrapy the following error appears:

Fatal error in launcher: Unable to create process using '"d:\bld\scrapy_1584555945006\_h_env\python.exe" "C:\Users\benoi\anaconda3\envs\scrapy\Scripts\scrapy.exe" ': The system cannot find the file specified.

I have no D: drive on my machine.
As reported in the bug on the scrapy repo, installing withthe previous 1.6.0 version with conda install -c conda-forge scrapy=1.6.0 works.

Environment (conda list):

# packages in environment at C:\Users\benoi\anaconda3\envs\scrapy:
#
# Name                    Version                   Build  Channel
astroid                   2.3.3                    py38_0
atomicwrites              1.3.0                      py_0    conda-forge
attrs                     19.3.0                     py_0    conda-forge
automat                   20.2.0                     py_0    conda-forge
bcrypt                    3.1.7            py38h9de7a3e_1    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
certifi                   2020.4.5.1       py38h32f6830_0    conda-forge
cffi                      1.14.0           py38ha419a9e_0    conda-forge
colorama                  0.4.3                      py_0    conda-forge
constantly                15.1.0                     py_0    conda-forge
cryptography              2.8              py38hb32ad35_1    conda-forge
cssselect                 1.1.0                      py_0    conda-forge
hyperlink                 17.3.1                     py_0    conda-forge
idna                      2.9                        py_1    conda-forge
importlib-metadata        1.6.0            py38h32f6830_0    conda-forge
importlib_metadata        1.6.0                         0    conda-forge
incremental               17.5.0                     py_0    conda-forge
isort                     4.3.21                   py38_0
lazy-object-proxy         1.4.3            py38he774522_0
libiconv                  1.15              hfa6e2cd_1006    conda-forge
libxml2                   2.9.10               h9ce36c8_0    conda-forge
libxslt                   1.1.33               heafd4d3_0    conda-forge
lxml                      4.5.0            py38h7b22a32_1    conda-forge
mccabe                    0.6.1                    py38_1
more-itertools            8.2.0                      py_0    conda-forge
openssl                   1.1.1f               hfa6e2cd_0    conda-forge
packaging                 20.1                       py_0    conda-forge
parsel                    1.5.2                    py38_0
pip                       20.0.2                     py_2    conda-forge
pluggy                    0.13.1                   py38_0    conda-forge
protego                   0.1.16                     py_0    conda-forge
py                        1.8.1                      py_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.20                       py_0    conda-forge
pydispatcher              2.0.5                      py_1    conda-forge
pyhamcrest                2.0.2                      py_0    conda-forge
pylint                    2.4.4                    py38_0
pyopenssl                 19.1.0                     py_1    conda-forge
pyparsing                 2.4.6                      py_0    conda-forge
pytest                    5.4.1            py38h32f6830_0    conda-forge
pytest-runner             5.2                        py_0    conda-forge
python                    3.8.2           h5fd99cc_5_cpython    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pywin32                   227              py38hfa6e2cd_0    conda-forge
queuelib                  1.5.0                    py38_0
scrapy                    1.6.0                    py38_0
service_identity          18.1.0                     py_0    conda-forge
setuptools                46.1.3           py38h32f6830_0    conda-forge
six                       1.14.0                     py_1    conda-forge
sqlite                    3.30.1               hfa6e2cd_0    conda-forge
twisted                   20.3.0           py38h9de7a3e_0    conda-forge
vc                        14.1                 h869be7e_1    conda-forge
vs2015_runtime            14.16.27012          h30e32a0_1    conda-forge
w3lib                     1.20.0                     py_0    conda-forge
wcwidth                   0.1.9              pyh9f0ad1d_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
wincertstore              0.2                   py38_1003    conda-forge
wrapt                     1.12.1           py38he774522_1
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11            h2fa13f4_1006    conda-forge
zope.interface            5.0.2            py38h9de7a3e_0    conda-forg

Details about conda and system ( conda info ):

     active environment : scrapy
    active env location : C:\Users\benoi\anaconda3\envs\scrapy
            shell level : 2
       user config file : C:\Users\benoi\.condarc
 populated config files : C:\Users\benoi\.condarc
          conda version : 4.8.2
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages :
       base environment : C:\Users\benoi\anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\benoi\anaconda3\pkgs
                          C:\Users\benoi\.conda\pkgs
                          C:\Users\benoi\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\benoi\anaconda3\envs
                          C:\Users\benoi\.conda\envs
                          C:\Users\benoi\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Windows/10 Windows/10.0.18362
          administrator : False
             netrc file : None
           offline mode : False

Support python 3.9 and 3.10

Scrapy 2.6.0 has offical Python 3.10 support, but in conda we only have 3.8 . @Gallaecio

Check if we can build an usable package for win+python3

With latest changes in twisted it may be possible to run scrapy on win+py3.

I think the basic check we have to do on a windows host is:

1. Install twisted either latest or dev version.
2. Install scrapy via pip or from source.
3. Run a spider and tests.