edwardgeorge / virtualenv-clone

A script for cloning a non-relocatable virtualenv. originated here: https://gist.github.com/860822

virtualenv cloning script.

A script for cloning a non-relocatable virtualenv.

Virtualenv provides a way to make virtualenv's relocatable which could then be copied as we wanted. However making a virtualenv relocatable this way breaks the no-site-packages isolation of the virtualenv as well as other aspects that come with relative paths and /usr/bin/env shebangs that may be undesirable.

Also, the .pth and .egg-link rewriting doesn't seem to work as intended. This attempts to overcome these issues and provide a way to easily clone an existing virtualenv.

It performs the following:

• copies sys.argv[1] dir to sys.argv[2]

• updates the hardcoded VIRTUAL_ENV variable in the activate script to the new repo location. (--relocatable doesn't touch this)

• updates the shebangs of the various scripts in bin to the new Python if they pointed to the old Python. (version numbering is retained.)

  it can also change /usr/bin/env python shebangs to be absolute too, though this functionality is not exposed at present.

• checks sys.path of the cloned virtualenv and if any of the paths are from the old environment it finds any .pth or .egg link files within sys.path located in the new environment and makes sure any absolute paths to the old environment are updated to the new environment.

• finally it double checks sys.path again and will fail if there are still paths from the old environment present.

NOTE: This script requires Python 2.7 or 3.4+

virtualenv-clone's Issues

Can't build package with python 3.9

Can't build python3-module-virtualenv-clone with new python version.

tests/test_virtualenv_sys.py::TestVirtualenvSys::test_clone_syspath FAILED [ 83%]
tests/test_virtualenv_sys.py::TestVirtualenvSys::test_clone_version FAILED [ 88%]
tests/test_virtualenv_sys.py::TestVirtualenvSys::test_virtualenv_syspath FAILED [ 94%]
tests/test_virtualenv_sys.py::TestVirtualenvSys::test_virtualenv_versions FAILED [100%]

=================================== FAILURES ===================================`
_____________________ TestVirtualenvSys.test_clone_syspath _____________________

self = <tests.test_virtualenv_sys.TestVirtualenvSys testMethod=test_clone_syspath>

    def test_clone_syspath(self):
        """
        Verify syspath for cloned virtualenvs`
    
        This really is a test for fixup_syspath as well
        """
       `for version in start_version_test():`

tests/test_virtualenv_sys.py:72: 

    def start_version_test():
        ran_once = False
        for version in versions:
            # create a virtualenv
            if subprocess.call(['virtualenv3', '-p', 'python' + version,
                                venv_path]) != 0:
                continue
    
            ran_once = True
            yield version
    
       assert ran_once, "All versions were skipped."
       AssertionError: All versions were skipped.
       assert False

tests/test_virtualenv_sys.py:20: AssertionError

----------------------------- Captured stdout call -----------------------------
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python2.7'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.4'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.5'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.6'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.7'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8'
_____________________ TestVirtualenvSys.test_clone_version _____________________


self = <tests.test_virtualenv_sys.TestVirtualenvSys testMethod=test_clone_version>

    def test_clone_version(self):
        """Verify version for cloned virtualenvs"""
       for version in start_version_test():

tests/test_virtualenv_sys.py:55: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def start_version_test():
        ran_once = False
        for version in versions:
            # create a virtualenv
            if subprocess.call(['virtualenv3', '-p', 'python' + version,
                                venv_path]) != 0:
                continue
    
            ran_once = True
            yield version
    
       assert ran_once, "All versions were skipped."
       AssertionError: All versions were skipped.
       assert False

tests/test_virtualenv_sys.py:20: AssertionError
----------------------------- Captured stdout call -----------------------------
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python2.7'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.4'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.5'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.6'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.7'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8'
__________________ TestVirtualenvSys.test_virtualenv_syspath ___________________

self = <tests.test_virtualenv_sys.TestVirtualenvSys testMethod=test_virtualenv_syspath>

    def test_virtualenv_syspath(self):
        """Verify syspath for created virtualenvs"""
>       for version in start_version_test():

tests/test_virtualenv_sys.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def start_version_test():
        ran_once = False
        for version in versions:
            # create a virtualenv
            if subprocess.call(['virtualenv3', '-p', 'python' + version,
                                venv_path]) != 0:
                continue
    
            ran_once = True
            yield version
    
       assert ran_once, "All versions were skipped."
       AssertionError: All versions were skipped.
       assert False

tests/test_virtualenv_sys.py:20: AssertionError
----------------------------- Captured stdout call -----------------------------
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python2.7'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.4'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.5'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.6'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.7'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8'
__________________ TestVirtualenvSys.test_virtualenv_versions __________________

self = <tests.test_virtualenv_sys.TestVirtualenvSys testMethod=test_virtualenv_versions>

    def test_virtualenv_versions(self):
        """Verify version for created virtualenvs"""
       for version in start_version_test():

tests/test_virtualenv_sys.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def start_version_test():
        ran_once = False
        for version in versions:
            # create a virtualenv
            if subprocess.call(['virtualenv3', '-p', 'python' + version,
                                venv_path]) != 0:
                continue
    
            ran_once = True
            yield version
    
       assert ran_once, "All versions were skipped."
       AssertionError: All versions were skipped.
       assert False

tests/test_virtualenv_sys.py:20: AssertionError
----------------------------- Captured stdout call -----------------------------
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python2.7'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.4'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.5'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.6'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.7'
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8'
=========================== short test summary info ============================
FAILED tests/test_virtualenv_sys.py::TestVirtualenvSys::test_clone_syspath - ...
FAILED tests/test_virtualenv_sys.py::TestVirtualenvSys::test_clone_version - ...
FAILED tests/test_virtualenv_sys.py::TestVirtualenvSys::test_virtualenv_syspath
FAILED tests/test_virtualenv_sys.py::TestVirtualenvSys::test_virtualenv_versions
======================== 4 failed, 14 passed in 22.94s =========================
ERROR: InvocationError for command /usr/src/RPM/BUILD/python3-module-virtualenv-clone-0.5.4/.tox/py39/bin/py.test -v (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py39: commands failed

python version: 3.9.2
virtualenv-clone version: 0.5.3

Cannot uninstall packages using pip in cloned venvs

Example below uses virtualenv-clone HEAD, virtualenvwrapper 3.4-2 (Ubuntu 13.10 package) and virtualenv 1.10.1-1:

will@banana ~ $ mkvirtualenv testvenv1
New python executable in testvenv1/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/preactivate
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/postactivate
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/get_env_details
(testvenv1)will@banana ~ $ pip install nose
Downloading/unpacking nose
  Downloading nose-1.3.0.tar.gz (404kB): 404kB downloaded
  Running setup.py egg_info for package nose

    no previously-included directories found matching 'doc/.build'
Installing collected packages: nose
  Running setup.py install for nose

    no previously-included directories found matching 'doc/.build'
    Installing nosetests script to /home/will/.virtualenvs/testvenv1/bin
    Installing nosetests-2.7 script to /home/will/.virtualenvs/testvenv1/bin
Successfully installed nose
Cleaning up...
(testvenv1)will@banana ~ $ deactivate 
will@banana ~ $ python dev/virtualenv-clone/clonevirtualenv.py .virtualenvs/testvenv1 .virtualenvs/testvenv2
will@banana ~ $ workon testvenv2
(testvenv2)will@banana ~ $ pip uninstall nose
Not uninstalling nose at /home/will/.virtualenvs/testvenv1/lib/python2.7/site-packages, outside environment /home/will/.virtualenvs/testvenv2

yet

(testvenv2)will@banana ~ $ python -c 'import nose; print nose.__file__'
/home/will/.virtualenvs/testvenv2/local/lib/python2.7/site-packages/nose/__init__.pyc

This needs a new release on pypi as the current version breaks PEW

see berdario/invewrapper/#10

fixup_script_() is run on python2 executable

Trying to clone a Python 2 virtual environment on Arch Linux results in the following error:

Traceback (most recent call last):
File "/usr/bin/virtualenv-clone", line 9, in
load_entry_point('virtualenv-clone==0.2.3', 'console_scripts', 'virtualenv-clone')()
File "/usr/lib/python2.7/site-packages/clonevirtualenv.py", line 247, in main
clone_virtualenv(old_dir, new_dir)
File "/usr/lib/python2.7/site-packages/clonevirtualenv.py", line 70, in clone_virtualenv
fixup_scripts(src_dir, dst_dir, version)
File "/usr/lib/python2.7/site-packages/clonevirtualenv.py", line 98, in fixup_scripts
rewrite_env_python=rewrite_env_python)
File "/usr/lib/python2.7/site-packages/clonevirtualenv.py", line 126, in fixup_script_
bang = lines[0].decode('utf-8').strip()
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd8 in position 24: invalid continuation byte

Looking into why this happens, I found that virtualenv-clone runs fixup_script_() on $HOME/.virtualenvs/test2/bin/python2 — 'test2' being the target virtualenv and 'python2' being the Python 2 interpreter executable.

I'm not sure how this should be fixed; maybe ignore binary files or add 'python2' (and 'python3'?) to the ignored filenames in fixup_scripts().

Pull request migrating to argparse welcome?

Hi!

I'm surprised to see use of optparse here. I'd make a pull request migrating to argparse if it has a chance to be accepted. What do you think?

Best, Sebastian

Get Travis CI working

This repo is signed up for Travis CI, but doesn't have a working .travis.yml file.

Source tarball of 0.4.0 release

I was unable to find a source tarball for the 0.4.0 release. It would be convenient for Linux distro packaging to have it available. (Similar to virtualenv-clone-0.3.0.tar.gz on PyPI.)

Please add tags for releases

The pypi tarballs are useless for testing. Please provide a tagged releases on GitHub so that we could get the whole package.

Can we adopt this package?

Hey @edwardgeorge, it seems this repository isn't being updated, I imagine because you don't have time anymore. We use this project in Zulip (github.com/zulip/zulip/), and would really like to have another release come out with some bugs fixes. Can we take over managing this package here and on pypi?

Thanks for your consideration!

(📚) Update readme to not mention virtualenv relocatable

relocatable was experimental and dropped quite a while ago. I think it would be a good idea to remove it's reference from the readme.

Python >= 2.5 required

Thanks for this script! You might want to point out the Python 2.5 requirement in the README. (I'm running CentOS, which uses Python 2.4.3 as its built-in Python.)

Note: Your script works fine if installed in an existing virtualenv running Python 2.5+. So I was able to successfully clone virtualenv A from within virtualenv A, which is pretty neat.

sys_path fixer not working in Python 3

Issue is in virtualenv-clone/0.5.0 and was discovered while using Python 3.6.8.

If there are elements to be fixed in the sys.path, code is triggered that doesn't work in Python 3. Specifically, _virtualenv_sys calls the filter method to return the sys.path from the VE python binary. In Python 2, this is returned as a list, but in Python 3 it is returned as a consumable iterable. The sys_path iterable is consumed in line 83:

if has_old(sys_path)

So when it is used again it is equal to an empty set in line 238:

for path in syspath:

The simple fix should be to convert the iterable to a list, which will make the logic work in Python 3 and should have no impact in Python 2. I will submit a pull request for this change.

The cloned env is linked to the original

I tried the package that comes with Ubuntu 16.04, i.e. version 0.2.5, and I'm seeing an unexpected behavior. Whatever changes I make in a cloned environment reflects in the original too.

Here's an example:

$ mkvirtualenv venv_a
(venv_a)$ pip install django==1.11.7
(venv_a)$ pip freeze
	Django==1.11.7
	pytz==2017.3

$ virtualenv-clone /path/to/venv_a /path/to/venv_b
$ workwon venv_b
(venv_b)$ pip freeze
	Django==1.11.7
	pytz==2017.3
(venv_b)$ pip install django==1.8.7
(venv_b)$ pip freeze
	Django==1.8.7
	pytz==2017.3

At this point I expect vevn_a to be on Django 1.11.7 and venv_b on Django 1.8.7. But when I switch to venv_a, I see it's changed too:

$ workon venv_a
(venv_a)$ pip freeze
	Django==1.8.7
	pytz==2017.3

No such file or directory: 'README.md' in new version 0.5.2

Seems to be some sort of packaging issue with the new virtualenv-clone-0.5.2.tar.gz that is on PyPi currently.

My python version:

$ python --version
Python 3.4.9

Steps to reproduce:

$ python3 -m venv /tmp/myvenv
$ source /tmp/myvenv/bin/activate
(my-venv) $ wget https://files.pythonhosted.org/packages/9c/79/c5250612d1ef85d0051ebcc50ac57ef222767db938f1151fc2c904c7fcc6/virtualenv-clone-0.5.2.tar.gz
(my-venv) $ python3 -m pip install ./virtualenv-clone-0.5.2.tar.gz 
DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).
Processing ./virtualenv-clone-0.5.2.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-brnab2gf/setup.py", line 9, in <module>
        with open('README.md') as f:
    FileNotFoundError: [Errno 2] No such file or directory: 'README.md'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-brnab2gf/

I didn't hit this directly, I ran into it by way of ./virtualenv-burrito.sh

Confusion on pypi (2.4 vs bugfix-2.5)

There are 2 versions on pypi. 2.4 and bugfix-2.5. Could this be cleaned up? Perhaps release 2.6 with tox.ini included.

Thanks!

Problem when LD_LIBRARY_PATH is required to run python

I'm in a managed environment using my own version of python. As a result, I need to set LD_LIBRARY_PATH to get python to work. In clonevirtualenv.py, around line 63, a python subprocess is opened, but the environment it uses is set to {}. To get it to work in my environment, I changed that to:

     env={'LD_LIBRARY_PATH': os.environ.get('LD_LIBRARY_PATH')},            

I'm adding this issue mostly just to document it in case others search for it rather than suggesting it as a general solution.

Add support for in-place relocation

Virtualenv-clone could provide a command-line option to relocate a virtualenv's pathing (/bin shebangs and symlinks) without copying the virtualenv. This would be useful when preparing a virtualenv in a fake root directory for .rpm/.deb generation. A system package build process would create the virtualenv, install all dependencies into it, relocate the paths to exclude the fake root's prefix, and package up the fake root directory.

Creating a virtualenv in a chroot or container under a fake root directory works, but often isn't an option due to build pipelines' limited user accounts.

filenotfounderror : [Errno 2] No such file or directory /bin/python

Ubuntu 20.04
installed virtualenv-clone in the virtual environment
command: virtualenv-clone ~/Rasa/rasa_kritik_check/ ~/Rasa/playground_rasa_kritik

traceback:

~/Rasa/rasa_kritik_check$ virtualenv-clone ~/Rasa/rasa_kritik_check/ ~/Rasa/playground_rasa_kritik
Traceback (most recent call last):
  File "/home/randy/Rasa/rasa_kritik_check/rasa_kritik_check/bin/virtualenv-clone", line 8, in <module>
    sys.exit(main())
  File "/home/randy/Rasa/rasa_kritik_check/rasa_kritik_check/lib/python3.7/site-packages/clonevirtualenv.py", line 311, in main
    clone_virtualenv(old_dir, new_dir)
  File "/home/randy/Rasa/rasa_kritik_check/rasa_kritik_check/lib/python3.7/site-packages/clonevirtualenv.py", line 77, in clone_virtualenv
    version, sys_path = _virtualenv_sys(dst_dir)
  File "/home/randy/Rasa/rasa_kritik_check/rasa_kritik_check/lib/python3.7/site-packages/clonevirtualenv.py", line 60, in _virtualenv_sys
    stdout=subprocess.PIPE)
  File "/home/randy/.pyenv/versions/3.7.3/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/home/randy/.pyenv/versions/3.7.3/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/randy/Rasa/playground_rasa_kritik/bin/python': '/home/randy/Rasa/playground_rasa_kritik/bin/python'

In this case /bin/python is a symbolic link

lrwxrwxrwx 1 randy randy   44 Feb 21 23:22 python -> /home/randy/.pyenv/versions/3.7.3/bin/python
lrwxrwxrwx 1 randy randy    6 Feb 21 23:22 python3 -> python

virtualenv-clone doesn't work correctly on windows

When I call clonevirtualenv.clone_virtualenv on Windows, I got this error:

In [8]: clonevirtualenv.clone_virtualenv("D:\\anaconda3\\envs\\kedro-environment\\", str(tmpdir / "ked-install" ))
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-8-fb15ea24257a> in <module>
----> 1 clonevirtualenv.clone_virtualenv("D:\\anaconda3\\envs\\kedro-environment\\", str(tmpdir / "ked-install" ))

d:\anaconda3\envs\kedro-environment\lib\site-packages\clonevirtualenv.py in clone_virtualenv(src_dir, dst_dir)
     75     shutil.copytree(src_dir, dst_dir, symlinks=True,
     76             ignore=shutil.ignore_patterns('*.pyc'))
---> 77     version, sys_path = _virtualenv_sys(dst_dir)
     78     logger.info('fixing scripts in bin...')
     79     fixup_scripts(src_dir, dst_dir, version)

d:\anaconda3\envs\kedro-environment\lib\site-packages\clonevirtualenv.py in _virtualenv_sys(venv_path)
     58               'print ("\\n".join(sys.path));'],
     59         env={},
---> 60         stdout=subprocess.PIPE)
     61     stdout, err = p.communicate()
     62     assert not p.returncode and stdout

d:\anaconda3\envs\kedro-environment\lib\subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
    798                                 c2pread, c2pwrite,
    799                                 errread, errwrite,
--> 800                                 restore_signals, start_new_session)
    801         except:
    802             # Cleanup if the child failed starting.

d:\anaconda3\envs\kedro-environment\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
   1205                                          env,
   1206                                          os.fspath(cwd) if cwd is not None else None,
-> 1207                                          startupinfo)
   1208             finally:
   1209                 # Child is launched. Close the parent's copy of those pipe

FileNotFoundError: [WinError 2] The system cannot find the file specified

I have narrowed down one bug to be this line: https://github.com/edwardgeorge/virtualenv-clone/blob/master/clonevirtualenv.py#L52. There might be more. I'm still trying to confirm.

On Windows, at least on my Windows (Windows 10, python 3.7), there is no python executable in Scripts. There is python.exe in its parent directory though.

assertion error on clone

I have a fairly heavy VE with several dozen packages installed. When I attempt to clone, the following assertion failure is shown, where remaining==[u'2.7']. Is this a show-stopper or can it be safely ignored?

kjacobs@0-compute:~/projects/locus-ve/0.0.1$ time bin/virtualenv-clone . ../clone3
Traceback (most recent call last):
File "bin/virtualenv-clone", line 9, in
load_entry_point('virtualenv-clone==0.2.4', 'console_scripts', 'virtualenv-clone')()
File "/locus/home/kjacobs/projects/locus-ve/0.0.1/local/lib/python2.7/site-packages/clonevirtualenv.py", line 268, in main
clone_virtualenv(old_dir, new_dir)
File "/locus/home/kjacobs/projects/locus-ve/0.0.1/local/lib/python2.7/site-packages/clonevirtualenv.py", line 89, in clone_virtualenv
assert not remaining, _virtualenv_sys(dst_dir)
AssertionError: (u'2.7', [u'/locus/home/kjacobs/projects/locus-ve/0.0.1/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/local/lib/python2.7/site-packages/locus_ve-0.0.1-py2.7.egg', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7/site-packages/locus_ve-0.0.1-py2.7.egg', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7/plat-linux2', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7/lib-tk', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7/lib-old', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7/lib-dynload', u'/usr/lib/python2.7', u'/usr/lib/python2.7/plat-linux2', u'/usr/lib/python2.7/lib-tk', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/local/lib/python2.7/site-packages', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7/site-packages', u'/locus/home/kjacobs/projects/locus-ve/0.0.1/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info'])

Is this tool support copy-on-write?

That can reduce the disk space usage.

Please ship tox.ini

Hi there,

I'm building an RPM and "setup.py test" depends on tox.ini. However this is not shipped in 2.4 or 2.5. Could you add it to MANIFEST.in?

Thanks!

it's support between two hosts

virtualenv-clone A:/path B:/Path
A:192.168.2.10
B:192.168.2.24

Virtual environment activation doesn't update prompt properly

So far, virtualenv-clone seems to work for our purposes - we want to install some PyTest "smoke" tests, but not have the development libraries in production.

Environment:
ubuntu 20.04
Python 3.8.10
virtualenv-clone 0.5.7

However, activating the environment makes it seem like you're still in the original environment.

I checked the activate script and it look like the old hard coded environment name is still in there.

Seems like this could be an additional search and replace?

I checked under Powershell, and the prompt DOES update :)

== John ==

Not removing or modifying (outside of prefix)

Environment: HEAD(currently 79a2667)

Process:

1. install django through wheel
2. clone venv(v1 -> v2)
3. uninstall django from v2
   =>it says:

Not removing or modifying (outside of prefix)
/run/shm/v1/bin/django-admin

When install a wheel with executable script,
(eg. package django has the script django-admin installed to v1/bin/ registered by entry_points.txt)
there's will be a record/line in Django-1.8.2.dist-info/RECORD saying
/run/shm/v1/bin/django-admin,sha256=OhnVC0qu9vUaGIQ8FdZ2det2wZFRNckxaQVkGcSWGRA,272.

After cpvenv from v1 to v2, that line is still v1.

This is the source of this bug.

However, the django-admin and django-admin.py are removed from v2 correctly.
Therefore, currently this is just a annoying message causing nothing wrong

Make a Git tag for 0.5.7 release

Could you please make a Git tag for the 0.5.7 release?

pypi is missing 0.3.0 release

Dear @timabbott
Is it possible to publish 0.3.0 to pypi?

https://pypi.python.org/pypi/virtualenv-clone

Strange error with multiple tracebacks

I'm trying to clone python2 virtual env.

$ /usr/bin/python2 -m clonevirtualenv -v ~/.virtualenvs/project ~/.virtualenvs/project-django15
cloning virtualenv '/home/nuklea/.virtualenvs/project' => '/home/nuklea/.virtualenvs/project-django15'...
fixing scripts in bin...
fixing paths in sys.path...

Traceback (most recent call last):
  File "/home/nuklea/.virtualenvs/project-django15/lib/python2.7/site.py", line 703, in <module>
    main()
  File "/home/nuklea/.virtualenvs/project-django15/lib/python2.7/site.py", line 683, in main
    paths_in_sys = addsitepackages(paths_in_sys)
  File "/home/nuklea/.virtualenvs/project-django15/lib/python2.7/site.py", line 282, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/home/nuklea/.virtualenvs/project-django15/lib/python2.7/site.py", line 204, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/home/nuklea/.virtualenvs/project-django15/lib/python2.7/site.py", line 177, in addpackage
    if not dircase in known_paths and os.path.exists(dir):
  File "/home/nuklea/.virtualenvs/project-django15/lib/python2.7/genericpath.py", line 26, in exists
    os.stat(path)
TypeError: must be encoded string without NULL bytes, not str

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/clonevirtualenv.py", line 307, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/clonevirtualenv.py", line 300, in main
    clone_virtualenv(old_dir, new_dir)
  File "/usr/lib/python2.7/site-packages/clonevirtualenv.py", line 92, in clone_virtualenv
    v_sys = _virtualenv_sys(dst_dir)
  File "/usr/lib/python2.7/site-packages/clonevirtualenv.py", line 66, in _virtualenv_sys
    assert not p.returncode and stdout
AssertionError

$ LC_ALL=C pacman -Qi python2-virtualenv-clone 
Name           : python2-virtualenv-clone
Version        : 0.2.6-1

Can't build package with python 3.8

Can't build python3-module-virtualenv-clone with new python version.
Build log:
Cannot build this package.
Please investigate.
Excerpt from build log:

ImportError: No module named ast
_____________________ TestVirtualenvSys.test_clone_version _____________________

self = <tests.test_virtualenv_sys.TestVirtualenvSys testMethod=test_clone_version>

def test_clone_version(self):
    """Verify version for cloned virtualenvs"""
   for version in start_version_test():

tests/test_virtualenv_sys.py:55:

---

def start_version_test():
    ran_once = False
    for version in versions:
        # create a virtualenv
        if subprocess.call(['virtualenv3', '-p', 'python' + version,
                            venv_path]) != 0:
            continue

        ran_once = True
        yield version

   assert ran_once, "All versions were skipped."

E AssertionError: All versions were skipped.
E assert False

tests/test_virtualenv_sys.py:20: AssertionError
----------------------------- Captured stdout call -----------------------------
Running virtualenv with interpreter /usr/bin/python2.7
The path python3.4 (from --python=python3.4) does not exist
The path python3.5 (from --python=python3.5) does not exist
The path python3.6 (from --python=python3.6) does not exist
The path python3.7 (from --python=python3.7) does not exist
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
File "/usr/lib/python3/site-packages/virtualenv.py", line 20, in
import ast
ImportError: No module named ast
__________________ TestVirtualenvSys.test_virtualenv_syspath ___________________

self = <tests.test_virtualenv_sys.TestVirtualenvSys testMethod=test_virtualenv_syspath>

def test_virtualenv_syspath(self):
    """Verify syspath for created virtualenvs"""
  for version in start_version_test():

tests/test_virtualenv_sys.py:44:

---

def start_version_test():
    ran_once = False
    for version in versions:
        # create a virtualenv
        if subprocess.call(['virtualenv3', '-p', 'python' + version,
                            venv_path]) != 0:
            continue

        ran_once = True
        yield version

   assert ran_once, "All versions were skipped."

E AssertionError: All versions were skipped.
E assert False

tests/test_virtualenv_sys.py:20: AssertionError
----------------------------- Captured stdout call -----------------------------
Running virtualenv with interpreter /usr/bin/python2.7
The path python3.4 (from --python=python3.4) does not exist
The path python3.5 (from --python=python3.5) does not exist
The path python3.6 (from --python=python3.6) does not exist
The path python3.7 (from --python=python3.7) does not exist
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
File "/usr/lib/python3/site-packages/virtualenv.py", line 20, in
import ast
ImportError: No module named ast
__________________ TestVirtualenvSys.test_virtualenv_versions __________________

self = <tests.test_virtualenv_sys.TestVirtualenvSys testMethod=test_virtualenv_versions>

def test_virtualenv_versions(self):
    """Verify version for created virtualenvs"""
   for version in start_version_test():

tests/test_virtualenv_sys.py:35:

---

def start_version_test():
    ran_once = False
    for version in versions:
        # create a virtualenv
        if subprocess.call(['virtualenv3', '-p', 'python' + version,
                            venv_path]) != 0:
            continue

        ran_once = True
        yield version

   assert ran_once, "All versions were skipped."

E AssertionError: All versions were skipped.
E assert False

tests/test_virtualenv_sys.py:20: AssertionError
----------------------------- Captured stdout call -----------------------------
Running virtualenv with interpreter /usr/bin/python2.7
The path python3.4 (from --python=python3.4) does not exist
The path python3.5 (from --python=python3.5) does not exist
The path python3.6 (from --python=python3.6) does not exist
The path python3.7 (from --python=python3.7) does not exist
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
File "/usr/lib/python3/site-packages/virtualenv.py", line 20, in
import ast
ImportError: No module named ast
=================== 4 failed, 14 passed in 139.67s (0:02:19) ===================
ERROR: InvocationError for command /usr/src/RPM/BUILD/python3-module-virtualenv-clone-0.5.3/.tox/py38/bin/py.test -v (exited with code 1)
___________________________________ summary ____________________________________
ERROR: py38: commands failed
error: Bad exit status from /usr/src/tmp/rpm-tmp.27630 (%check)

RPM build errors:
Bad exit status from /usr/src/tmp/rpm-tmp.27630 (%check)
Command exited with non-zero status 1
138.16user 10.69system 2:42.36elapsed 91%CPU (0avgtext+0avgdata 65796maxresident)k
0inputs+0outputs (0major+2013679minor)pagefaults 0swaps
hsh-rebuild: rebuild of `python3-module-virtualenv-clone-0.5.3-alt1.src.rpm' failed.
Command exited with non-zero status 1

python version: 3.8.1
virtualenv-clone version: 0.5.3

Problem in fixup_pth_file()

There is a problem when the method fixup_pth_file() has to rewrite a file - every line is decoded from utf-8 using unicode internally, but the file is written as a unicode file at the end - no encode('utf-8') when writing.

version numbers must be collected into one place

Hello, right now package version info and py version info is not same and all version numbers are needed to change when a bump needed.
So my suggestion is to collect all these info to a seperate file lets name it version.py and in that file a variable for example version , and in whole other places ( in setup.py , in main app py , ... ) Ve use that variable that will let us fix all version related problems

Global name 'filename' is not defined

Hi, great script, it's saved me loads of time. But I'm getting the following error trying to clone a pretty robust virtualenv, could you take a look?

Traceback (most recent call last):
File "/home/mae/bin/virtualenv-clone", line 8, in
load_entry_point('virtualenv-clone==0.1.2', 'console_scripts', 'virtualenv-clone')()
File "/home/mae/lib/python2.6/clonevirtualenv.py", line 214, in main
clone_virtualenv(old_dir, new_dir)
File "/home/mae/lib/python2.6/clonevirtualenv.py", line 65, in clone_virtualenv
fixup_scripts(src_dir, dst_dir, version)
File "/home/mae/lib/python2.6/clonevirtualenv.py", line 86, in fixup_scripts
target = os.readlink(filename)
NameError: global name 'filename' is not defined

Pypi repo tarball incorrect for version 0.5.6

The tarball for version 0.5.6 differs in pypi vs github. Pypi's tarball is particularly missing tox.ini which is necessary to run tests when packaging this library for GNU Guix.

The difference between pypi can be confirmed by comparing the checksums for the two. In pypi, the checksum is 997c7d225eabc4d09e77672461f4bdf9f3a8ea9dc9e4a847b0e83dc8bad9573a. However, github's checksum for tarball is 949777a53fd8284819110661abb8f6a97420919bda34ee01353c8e3793486a28. Both checksums use sha256 and are base16 formatted.
I used the following commands to determine this.
guix download -H sha256 -f base16 https://files.pythonhosted.org/packages/a9/bb/ab59bdb1dc8e14add54c70e9bccb8d54379bdf73ec8cebbbf638ab76590f/virtualenv-clone-0.5.6.tar.gz
guix download -H sha256 -f base16 https://github.com/edwardgeorge/virtualenv-clone/archive/refs/tags/0.5.6.tar.gz

My current workaround is to grab the tarball from github rather than pypi.

Can't build package with python 3.8

Virtualenv-clone tests do not work with Python 3.8.

(🐞) Windows executables are not updated

I think that windows executables (.venv\Scripts\*.exe) are not updated and are hardcoded to the old venv:

> py -m venv .venv
...
> .venv\scripts\Activate.ps1
...
> pip install virtualenv-clone
...
>  virtualenv-clone .venv .venv2
...
> deactivate
...
> .venv2\scripts\Activate.ps1
...
> pip.exe -V
.\.venv\Lib\site-packages\pip

packages installed as editable from dvc repositories using pip -e aren't cloned

Any packages installed using the pip -e git+.... or pip -e hg+.... syntax which are installed to the src folder are not cloned.

bin/pip script shebang doesn't get fixed after clone (python3 only)

Using virtualenv-clone 0.2.6

I cloned old_env to new_env, and can't install anything in the new_env using pip, because its shebang line still points to old_env:

(new_env)u1@pc1:~> head -n1 `which pip`
#!/home/u1/.virtualenvs/old_env/bin/python3

The reason of this is because shebang line ends with 'python3', but not with 'python3.4'.
clonevirtualenv._virtualenv_sys('/home/u1/.virtualenvs/old_env/') returns '3.4'

Steps to reproduce:

u1@pc1:~> mkvirtualenv -p /usr/bin/python3 old
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in old/bin/python3
Also creating executable in old/bin/python
Installing setuptools, pip...done.
virtualenvwrapper.user_scripts creating /home/u1/.virtualenvs/old/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/u1/.virtualenvs/old/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/u1/.virtualenvs/old/bin/preactivate
virtualenvwrapper.user_scripts creating /home/u1/.virtualenvs/old/bin/postactivate
virtualenvwrapper.user_scripts creating /home/u1/.virtualenvs/old/bin/get_env_details
(old)u1@pc1:~> head -n1 `which pip`
#!/home/u1/.virtualenvs/old/bin/python3
(old)u1@pc1:~> virtualenv-clone ~/.virtualenvs/old ~/.virtualenvs/new
(old)u1@pc1:~> workon new                           
(new)u1@pc1:~> head -n1 `which pip`
#!/home/u1/.virtualenvs/old/bin/python3

This does not affect python2-based environments, because with python2 - pip shebang will look like this:

(new_py2)u1@pc1:~> head -n1 `which pip`
#!/home/u1/.virtualenvs/new_py2/bin/python

And it will get properly fixed after clone.