pip install in editable mode not working about clickhouse-sqlalchemy

[xzkostyan]

Hi,

Thanks for reporting this issue. Are there another alternatives or good examples of setup.py?

This is the most confusing part of setup.py for me.

[AchilleAsh]

As a good example you can follow this guide about python packaging: https://python-packaging.readthedocs.io/en/latest/index.html
It may not be perfect but does the job.

Following this guide packages setting in setup.py should be dead simple when src/ is renamed to clickhouse_sqlalchemy/:

    packages=['clickhouse_sqlalchemy'],
   # and remove package_dir setting

(also don't forget to edit the path in the read_version function in setup.py)

I let you do the refacto so you can check if everything is okay

Timothé

[xzkostyan]

Hi, @AchilleAsh

I can't reproduce this issue:

kostya@kostya-laptop:/tmp$ git clone https://github.com/xzkostyan/clickhouse-sqlalchemy
Cloning into 'clickhouse-sqlalchemy'...
remote: Counting objects: 401, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 401 (delta 2), reused 3 (delta 0), pack-reused 389
Receiving objects: 100% (401/401), 81.04 KiB, done.
Resolving deltas: 100% (241/241), done.
kostya@kostya-laptop:/tmp$ cd clickhouse-sqlalchemy/
kostya@kostya-laptop:/tmp/clickhouse-sqlalchemy$ mkvirtualenv test
New python executable in /home/kostya/.virtualenvs/test/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/preactivate
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/postactivate
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/get_env_details
(test) kostya@kostya-laptop:/tmp/clickhouse-sqlalchemy$ pip install -e .
Obtaining file:///tmp/clickhouse-sqlalchemy
Collecting sqlalchemy (from clickhouse-sqlalchemy==0.0.5)
/home/kostya/.virtualenvs/test/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/home/kostya/.virtualenvs/test/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
/home/kostya/.virtualenvs/test/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/b4/9c/411a9bac1a471bed54ec447dc183aeed12a75c1b648307e18b56e3829363/SQLAlchemy-1.2.8.tar.gz (5.6MB)
    100% |████████████████████████████████| 5.6MB 1.3MB/s 
Collecting requests (from clickhouse-sqlalchemy==0.0.5)
  Downloading https://files.pythonhosted.org/packages/cc/15/e1c318dbc20032ffbe5628837ca0de2d5b116ffd1b849c699634010f6a5d/requests-2.19.0-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 3.9MB/s 
Collecting clickhouse_driver>=0.0.8 (from clickhouse-sqlalchemy==0.0.5)
  Downloading https://files.pythonhosted.org/packages/58/f5/f58b258e61eafe59430bd100f5c43c148cf8d8ce1bf4152544bd6e21feac/clickhouse-driver-0.0.11.tar.gz
Collecting idna<2.8,>=2.5 (from requests->clickhouse-sqlalchemy==0.0.5)
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 9.1MB/s 
Collecting certifi>=2017.4.17 (from requests->clickhouse-sqlalchemy==0.0.5)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 4.2MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests->clickhouse-sqlalchemy==0.0.5)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 5.2MB/s 
Collecting urllib3<1.24,>=1.21.1 (from requests->clickhouse-sqlalchemy==0.0.5)
  Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 4.0MB/s 
Collecting pytz (from clickhouse_driver>=0.0.8->clickhouse-sqlalchemy==0.0.5)
  Downloading https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl (510kB)
    100% |████████████████████████████████| 512kB 3.2MB/s 
Collecting enum34 (from clickhouse_driver>=0.0.8->clickhouse-sqlalchemy==0.0.5)
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Building wheels for collected packages: sqlalchemy, clickhouse-driver
  Running setup.py bdist_wheel for sqlalchemy ... done
  Stored in directory: /home/kostya/.cache/pip/wheels/df/fc/61/df2f43ec3f11f864554bdc006a866a3ffffa59740bcf3674ef
  Running setup.py bdist_wheel for clickhouse-driver ... done
  Stored in directory: /home/kostya/.cache/pip/wheels/8e/20/5a/22c4db3c3cfe7d75b2d2446d08a16062cbc3d2f31e44d57f2a
Successfully built sqlalchemy clickhouse-driver
Installing collected packages: sqlalchemy, idna, certifi, chardet, urllib3, requests, pytz, enum34, clickhouse-driver, clickhouse-sqlalchemy
  Running setup.py develop for clickhouse-sqlalchemy
Successfully installed certifi-2018.4.16 chardet-3.0.4 clickhouse-driver-0.0.11 clickhouse-sqlalchemy enum34-1.1.6 idna-2.7 pytz-2018.4 requests-2.19.0 sqlalchemy-1.2.8 urllib3-1.23
(test) kostya@kostya-laptop:/tmp/clickhouse-sqlalchemy$ 

Creating test package with simple setup.py is fine too:

from setuptools import setup

setup(
    name='test',
    version='0.0.1',

    install_requires=[
        'clickhouse-sqlalchemy',
    ],
)

kostya@kostya-laptop:/tmp/test$ mkvirtualenv test
New python executable in /home/kostya/.virtualenvs/test/bin/python
Installing setuptools, pip, wheel... done.
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/preactivate
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/postactivate
virtualenvwrapper.user_scripts creating /home/kostya/.virtualenvs/test/bin/get_env_details
(test) kostya@kostya-laptop:/tmp/test$ pip install -e .
Obtaining file:///tmp/test
Collecting clickhouse-sqlalchemy (from test==0.0.1)
/home/kostya/.virtualenvs/test/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/home/kostya/.virtualenvs/test/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
/home/kostya/.virtualenvs/test/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/d5/04/13f4d0a82d1029279aeb52065da5eb4c059128b4aa52ddc5ff66eda06250/clickhouse-sqlalchemy-0.0.5.tar.gz
Collecting sqlalchemy (from clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/b4/9c/411a9bac1a471bed54ec447dc183aeed12a75c1b648307e18b56e3829363/SQLAlchemy-1.2.8.tar.gz (5.6MB)
    100% |████████████████████████████████| 5.6MB 1.7MB/s 
Collecting requests (from clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/cc/15/e1c318dbc20032ffbe5628837ca0de2d5b116ffd1b849c699634010f6a5d/requests-2.19.0-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 4.4MB/s 
Collecting clickhouse_driver>=0.0.8 (from clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/58/f5/f58b258e61eafe59430bd100f5c43c148cf8d8ce1bf4152544bd6e21feac/clickhouse-driver-0.0.11.tar.gz
Collecting idna<2.8,>=2.5 (from requests->clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 8.7MB/s 
Collecting certifi>=2017.4.17 (from requests->clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 4.7MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests->clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 5.7MB/s 
Collecting urllib3<1.24,>=1.21.1 (from requests->clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 7.1MB/s 
Collecting pytz (from clickhouse_driver>=0.0.8->clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl (510kB)
    100% |████████████████████████████████| 512kB 6.6MB/s 
Collecting enum34 (from clickhouse_driver>=0.0.8->clickhouse-sqlalchemy->test==0.0.1)
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Building wheels for collected packages: clickhouse-sqlalchemy, sqlalchemy, clickhouse-driver
  Running setup.py bdist_wheel for clickhouse-sqlalchemy ... done
  Stored in directory: /home/kostya/.cache/pip/wheels/33/21/a2/88c5af6ed85cfa0dfca0a2585e0ba5354b4cd14687bf24b9c5
  Running setup.py bdist_wheel for sqlalchemy ... done
  Stored in directory: /home/kostya/.cache/pip/wheels/df/fc/61/df2f43ec3f11f864554bdc006a866a3ffffa59740bcf3674ef
  Running setup.py bdist_wheel for clickhouse-driver ... done
  Stored in directory: /home/kostya/.cache/pip/wheels/8e/20/5a/22c4db3c3cfe7d75b2d2446d08a16062cbc3d2f31e44d57f2a
Successfully built clickhouse-sqlalchemy sqlalchemy clickhouse-driver
Installing collected packages: sqlalchemy, idna, certifi, chardet, urllib3, requests, pytz, enum34, clickhouse-driver, clickhouse-sqlalchemy, test
  Running setup.py develop for test
Successfully installed certifi-2018.4.16 chardet-3.0.4 clickhouse-driver-0.0.11 clickhouse-sqlalchemy-0.0.5 enum34-1.1.6 idna-2.7 pytz-2018.4 requests-2.19.0 sqlalchemy-1.2.8 test urllib3-1.23
(test) kostya@kostya-laptop:/tmp/test$ 

Can you provide some example to reproduce your issue?

[AchilleAsh]

Hi,

Indeed I forgot to detail the issue, my apologies.
Let's start just as you did:

achille@darth-vader ~/prog                                            
> $ pew new tmp
Using base prefix '/usr'
New python executable in /home/achille/.local/share/virtualenvs/tmp/bin/python3
Also creating executable in /home/achille/.local/share/virtualenvs/tmp/bin/python
Installing setuptools, pip, wheel...done.
Launching subshell in virtual environment. Type 'exit' or 'Ctrl+D' to return.

achille@darth-vader ~/prog      
(tmp) > $  git clone https://github.com/xzkostyan/clickhouse-sqlalchemy  
Clonage dans 'clickhouse-sqlalchemy'...
remote: Counting objects: 401, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 401 (delta 2), reused 3 (delta 0), pack-reused 389
Réception d'objets: 100% (401/401), 81.04 KiB | 838.00 KiB/s, fait.
Résolution des deltas: 100% (241/241), fait.

achille@darth-vader ~/prog        
(tmp) > $ cd clickhouse-sqlalchemy 

achille@darth-vader ~/prog/clickhouse-sqlalchemy                                
(tmp) > $ pip install -e .   
[...]
  Running setup.py develop for clickhouse-sqlalchemy
Successfully installed certifi-2018.4.16 chardet-3.0.4 clickhouse-driver-0.0.11 clickhouse-sqlalchemy idna-2.7 pytz-2018.4 requests-2.19.0 sqlalchemy-1.2.8 urllib3-1.23

Now it seems that it worked, however:

achille@darth-vader ~/prog/clickhouse-sqlalchemy 
(tmp) > $ python 
Python 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from clickhouse_driver import Client  # Works
>>> from clickhouse_sqlalchemy import Table # Boum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'clickhouse_sqlalchemy'

Because the setup.py file is using replace on the package files, it messes with the symlink pip uses when using develop mode.
Using simple packages value in the setup file makes the symlink works : the package can be imported.

--
Timothé

[xzkostyan]

Fixed in last master commit: 7fe8386

[AchilleAsh]

Hi @xzkostyan ,
I can confirm that it's working now.
Could you release a new version to PyPI to include all changes made since the last release?

Thank you for your work!

[AchilleAsh]

Oops, I've noticed that your tests folder is at the same level as clickhouse_sqlalchemy/, and is being installed by the setup.py in the global "tests" namespace...

To fix, simply replace packages=find_packages('.') by packages=["clickhouse_sqlalchemy"]

[xzkostyan]

Fixed.