Make Python module recognize and auto-discover virtualenv environments, including those made by virtualenvwrapper. about python

[bitemyapp]

virtualenvwrapper uses ~/.virtualenv

[berdario]

There's already a discussion going on in this LT issue

[bitemyapp]

@berdario you don't need to necessarily specify it because there's a common location that 99% of virtualenvs made by virtualenvwrapper are put into.

[ibdknox]

That discussion should move here :)

[berdario]

@bitemyapp you need to specify it because you don't know the name of the virtualenv that the project is using (in theory, there could even be multiple environments for the same project, but this use case shouldn't concern LT since it's something that's handled by testing tools like Tox)

[bitemyapp]

@berdario hum, that's a good point but you could make an educated guess based on the name of the project you're inside of.

Given that, I'd lean towards just letting the user specify the virtualenv.

[berdario]

Guessing is surely not the most robust solution, but I agree: it would help to get 90% there...
how should the user specify the virtualenv?

workspace behavior seems the way to go, but they're really underdocumented... also: every time that I change workspace, everything that I wrote in workspace.behavior is lost

[jeffwidman]

I'm a bit of a n00b python guy, so still figuring things out, but it seems the ideal for me would be:

1. Within user.behavior, set default path for folder containing my virtualenvs so that I can later reference virtualenvs by just their name. Have this default path default to ~/.virtualenvs but let it be overridden if the user, like me, stores them elsewhere.

2. Within workspace.behavior, set a default virtualenv for that workspace:
   a) using just the virtualenv name if it resides within my virtualenv container folder, eg 'projectone'
   b) using the full path if the virtualenv lives outside the container folder, eg '~/code/projectone/virtualenv'

3. Optional: Occasionally I want to switch back and forth between different virtualenvs for a project, so somehow make it possible to choose a different virtualenv when launching the interpreter... maybe within 'Connections' > 'Python' > 'Enter for default, otherwise use arrow keys to select a different virtualenv'

[iluminite]

virtualenvwrapper uses $WORKON_HOME, so LT could also use this. but this path, ~.virtualenv/ is the HOME for many virtualenv. EG, if you open two projects in different virtualenvs, you open two separate python executables. LT needs to be smart about this.

[waynew]

I, for one, am particularly interested in this issue. I'm just getting started with lighttable, but I certainly like following the virtualenvwrapper conventions for convenience/portability. And alternatively specifying the interpreter to use.

[foobarbecue]

People like me are excited about using LightTable with Django, but LT is pretty useless for Django development until we can do this. Would be wonderful if I could tell the connection to run "manage.py shell" with a virtualenv enabled.

[saulshanabrook]

I actually think that trying to find an existing virtuelenv that matches the project might not be the best way (or at least not the only way) to run code with certain dependencies.

I just started learning Clojure on LightTable and the way it loads dependencies seems to be very straightforward. If you are evaluating code in a project, then it finds the project.clj file (the equivalent of a setup.py or requirements.txt) and uses those to find dependencies, download, install, and then evaluate code. If you are not working in a project, it does not install any dependencies.

I think setting up Python dependencies the same way as Clojure has some advantages over looking for a virtualenv to run code in.

1. Always ensures that code is being run against the dependencies specified in your project. This ensures that you list all your dependencies in either the requirements.txt or setup.py and that it never gets out of sync with the environemnt you are working in.
2. Don't have to worry about trying to find virtualenv for project.
3. Simpler to set up for new projects (no set up).

It could use virtualenv under the surface, creating a new virtualenv whenever you run code from a project. The main problem is that you don't want to download and install all your dependencies whenever want to run any code. It wouldn't be hard to use the same environment every time you started LightTable, but even reinstalling requirements on every open is unrealistic.

Setting up the Pip cache eliminates the download problem, but we are still stuck installing things on every run. One way to solve this would be to create a virtualenv for each project path and reinstall overtop, when connecting to the python client. This is what Heroku does when deploying new Python code. Another option would be to detect changes between the environment and requirements, and only install or uninstall to match. pip-tools could be of use if we went this route.

Any thoughts on this approach? If anything I said didn't make sense, please let me know.

[waynew]

@saulshanabrook I think you're right that automatically trying to find the right env is probably not the right way. I don't know that automagically overriding the typical workflow is the best way either. It may be the expected behavior for Clojure, but I think it would be a bit surprising for most Pythonistas.

I think illumin touched on it - Python already has conventions for working with virtualenvs. It makes sense to at least first consider the existing conventions before deciding to embark on another route.

Because many Python devs are going to have a $WORKONHOME defined, LightTable would do well do follow that convention. I think it should just follow this out to its logical conclusion and effectively just imitate virtualenvwrapper. Provide a Workon: ... and Make Virtualenv: commands. Perhaps automatically keep the env in sync with the requirements.txt file like you suggest, though that raises the question of how to let the user specify some of the options (e.g. --allow-external)

I think it probably just makes sense at least initially to clone the virtualenvwrapper api, and then expose the underlying pip api through a Pip: command that's just a pass-thru to the underlying pip install.

[foobarbecue]

All I really want to do is specify what command LT runs when it starts a python interpreter (I want it to run ./manage.py shell ). This virtualenv stuff would be icing on the cake and I worry it will delay implementation of the essential feature here. Then people can write their own scripts that start the interpreter and load whatever environment they want.

[ashwoods]

Python doesn't really have established conventions regarding packaging nor interpreters and it's a landscape that continually changes: you have buildout, pip, wheels, virtualenv with/without virtualenvwrapper, venv (python3), custom builds (with debian packages for example), pyenv (uses shims) + remote interpreters (i.e. inside a vagrant vm or docker container) and a mix of several of the above. The one thing that works for all the above is specifiying a path to the interpreter (although the remote interpreters over ssh require a bit more work). If you decide to add some plugins for virtualenvwrapper, pyenv, or what-else, should maybe be considered as extra plugin.

[ergo]

I don't think there is any established convention for environment handling and I agree that best option would be to provide an easy way for us to just specify interpreter path - that would probably solve most use cases.

[ibdknox]

There's a behavior for doing exactly that: Python: Set the path to the python executable for clients :)

[ergo]

Hmm, I did that but I was hoping for some intellisense like code completion where i can do:

from package import foo

and have the autocomplete suggest actual packages and symbols from them as I type this, this kind of functionality is currently not implemented?

[lyndsysimon]

I'm new to LT, but I've been doing Python for a few years now. Here's my suggestion:

Instead of thinking about this only in terms of activating the right virtualenv, I propose that there be a mechanism to set environment variables on a per-client basis. This would then let LT look for an environment variable - say, LT_CLIENT_BIN - to choose the Python executable to use.

There should probably also be some UI sugar for letting the user choose the Python binary to use, but the real meat of the functionality would be the per-client environment variable management. This would allow other functionality like storing your app configuration in the environment a'la 12 Factor, and having multiple client connections set up in your workspace - so testing a webapp with sqlite might be the default (for speed), but running the same app against PostgreSQL would just be a matter of connecting to a different client.

I'm basically doing this by brute force now - I open a terminal, activate my virtualenv, then invoke LT. Since the virtualenv bin directory is prepended to PATH, the default behavior gets me the right environment.

Running tests, however, is a bit of a pain still :)

[lufeihaidao]

Hi,

I use pyenv to manage my python, and I found that it seems different between my terminal python and lt python.

lt use /bin/python, is it right? Could I customize the python path?

[radix]

It's unclear to me what the conclusion to this was -- is there any way at all to specify the python executable or virtualenv for evaluating code in LT?

[thewhitetulip]

Will this issue be resolved by enabling access to the default shell?
That way all we have to do is provide an actual console in the console window and let the user run anything he/she wants.

[lyndsysimon]

This was closed - but what was the resolution?

[lyndsysimon]

Still no conclusion to this - at least, not one of which I'm aware.

@ibdknox said:

There's a behavior for doing exactly that: Python: Set the path to the python executable for clients :)

Admittedly I'm not expert with LT (I can't use it yet, because I've not been able ot get it to work well with Python) - but when I open the commands pane I can't get anything beginning with "Python" to come up.

[kenny-evitt]

@lyndsysimon Does #26 help?