nanobox-io / nanobox-engine-python

Engine for running Python apps on Nanobox

Python

This is a Python engine for developing python apps on Nanobox.

Usage

To use the Python engine, specify python as your engine in your boxfile.yml

run.config:
  engine: python

Configuration Options

This engine exposes configuration options through the boxfile.yml, a yaml config file used to provision and configure your app's infrastructure when using Nanobox.

Python Settings

The following setting allows you to define your Python runtime environment.

runtime

Specifies which Python runtime and version to use. The following runtimes are available:

• python-2.7
• python-3.4
• python-3.5
• python-3.6

run.config:
  engine.config:
    runtime: python-3.5

pip_install

Allows for a custom user-defined pip install command.

note: The custom pip install command should always include -I to ensure the binaries get copied on each build.

run.config:
  engine.config:
    pip_install: 'pip install --index-url https://my.custom.index -r requirements/private.txt -I'

Help & Support

This is a Python engine provided by Nanobox. If you need help with this engine, you can reach out to us in the #nanobox IRC channel. If you are running into an issue with the engine, feel free to create a new issue on this project.

nanobox-engine-python's Issues

UTF-16

Grep doesn't seem to like UTF-16 formatted files.
When searching the requirements.txt file, grep fails to find anything if it's UTF-16

Provide dependencies for Pillow by default

Pillow is the go-to Python package for image manipulation. For example, Django requires it when using model fields for images. Currently it is kind of a hassle to get working due to the fact that it requires quite some packages to install properly (probably to compile the bindings). Therefore it would be nice if these dependencies would be installed automagically.

The documentation has a very comprehensive list of dependencies: https://pillow.readthedocs.io/en/4.1.x/installation.html#building-from-source

As can be seen, not all packages are strictly required, however, I would recommend just installing them all so there are no nasty surprises when people think all features are supported.

After some experimentation I can verify that Pillow compiles when I add the following extra packages to my boxfile (although this list is probably not complete for full support, also, it might install more than strictly necessary):

run.config:
  engine: python
  engine.config:
    runtime: python-3.6
  extra_packages:
    - libjpeg-turbo
    - tiff
    - zlib
    - freetype2
    - lcms2
    - libwebp
    - tcl
    - tk

Another option of course is just providing a precompiled version of Pillow, but I am not sure how that would work in combination with the requirements file and pip.

Boxfile exec field isn't used at the end of nanobox run

What I do :

$ git clone https://github.com/harobed/nanobox-python-sample.git
$ cd nanobox-python-sample
$ nanobox destroy
$ nanobox  update
$ nanobox update-images
$ nanobox run

I got :

...
PIP INSTALL --------------------------->
   You are using pip version 6.0.8, however version 7.1.2 is available.
   You should consider upgrading via the 'pip install --upgrade pip' command.
   Requirement already satisfied (use --upgrade to upgrade): flask in ./env/lib/python2.7/site-packages (from -r /opt/code/requirements.txt (line 1))
   Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.7 in ./env/lib/python2.7/site-packages (from flask->-r /opt/code/requirements.txt (line 1))
   Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.4 in ./env/lib/python2.7/site-packages (from flask->-r /opt/code/requirements.txt (line 1))
   Requirement already satisfied (use --upgrade to upgrade): itsdangerous>=0.21 in ./env/lib/python2.7/site-packages (from flask->-r /opt/code/requirements.txt (line 1))
   Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in ./env/lib/python2.7/site-packages (from Jinja2>=2.4->flask->-r /opt/code/requirements.txt (line 1))
   [√] SUCCESS
+> Copying release into live code directory...
+> Moving final build into release environment...

--------------------------------------------------------------------------------
[√] APP SUCCESSFULLY BUILT   ///   DEV URL : test_nanobox.dev
--------------------------------------------------------------------------------

++> STREAMING LOGS (ctrl-c to exit) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

My execution command specify here : https://github.com/harobed/nanobox-python-sample/blob/master/Boxfile#L3 isn't executed.

Other note : the nanobox output say nothing about line executed.

/opt/nos/lib/payload.sh: line 9: PL_error:: command not found

When I execute

$ nanobox run 

in my Python project folder, I've this error :

...
+> Copying code into build container...
/opt/nos/lib/payload.sh: line 9: PL_error:: command not found
...

python-3.6 and psycopg2 (Error: pg_config executable not found.)

I'm trying to setup an environment with nanobox python engine. If I use python-2.7 and then install a psycopg2 package (pip command within a container or with a requirements.txt) it works ok. But when I switch to python-3.6 I got an error "Error: pg_config executable not found.". This is a well known issue but I can't solve it with a nanobox container. I can't install any dependencies (python3-dev or libpq-dev) since there is none of these in Nanobox pkgsrc and even can't find a path to pg_conf.

Is there any way to use python3 with psycopg2 with nanobox?

My starting config:

run.config:
engine: python
engine.config:
runtime: python-3.6
extra_packages:
- nodejs
cache_dirs:
- client/node_modules
extra_path_dirs:
- client/node_modules/.bin
build_triggers:
- server/requirements.txt
- client/package.json
- client/package-lock.json
extra_steps:
- cd server && pip install -r requirements.txt
- cd client && npm install
fs_watch: true

data.postgresql:
image: nanobox/postgresql
config:
version: 9.6
`

Python engine cmd error

"FAILED TO CLONING" should be singular "CLONE"

ON failure: nanobox run
- Setting up directories and permissions... - Cloning engine :

! FAILED TO CLONING HTTPS://GITHUB.COM/NANOBOX-IO/NANOBOX-ENGINE-PYTHONG.GIT !

Need help understanding Python

Nanobox Desktop automates the creation of isolated, virtual development environments. It scans your code and builds a suitable environment within a VM. In order to best automate the build of Python environments, we would greatly appreciate it if you would be willing to answer some or all of the following questions, in as much detail as you can. There are no wrong answers:

Detection

• What is the most accurate way you can think of to detect a Python app?

Environment

• Which python runtimes need to be available to your app?
• Are any other runtimes (nodejs, ruby, etc) needed?
• Are any utilities, webservers, or other binaries (apache, nginx, etc) needed?
• How do you manage/install dependencies?
• Where are your dependencies stored? (which directories)

Service detection

• Does your app depend on services like postgres, redis, etc?
• How could we detect that your app depends on them?
• Does your app require any special configuration for these services?

Application configuration

• Does your app require special configuration via config files or environment variables?
• Where and how could we configure your app to connect to the services that were previously detected?

Compilation/Build

• Does your app need to be compiled?
• Is there a build process required to generate a release?
• Ultimately, what should be copied into the live environment? (all source code, specific directories, etc)

Finally, running your app

• Is your app strictly a web (http) app, or does it provide a service on tcp/udp as well?
• How do you run your app?

Thanks in advance for sharing your time and expertise!

bad interpreter: No such file or directory

Getting the following error message when trying to run pip.

bash: /code/env/bin/pip: /opt/code/env/bin/python2.7: bad interpreter: No such file or directory

My Boxfile looks like the following

env:
  ENVIRONMENT: dev

build:
  engine: python
  python_runtime: python27
  nodejs_runtime: nodejs-4.2

I took a look in /code/env/bin/pip and it looks like this

#!/opt/code/env/bin/python2.7

should be this?

#!/code/env/bin/python2.7

requirements packages are installed but not accessible

What I do :

$ git clone https://github.com/harobed/nanobox-python-sample.git
$ cd nanobox-python-sample
$ nanobox destroy
$ nanobox  update
$ nanobox update-images
$ nanobox dev

requirements are installed with success https://gist.github.com/harobed/67a3351fc4d880634d8f but :

[gonano@test_nanobox /code]$ ls
Boxfile  README.md  app.py  requirements.txt
[gonano@test_nanobox /code]$ python app.py
Traceback (most recent call last):
  File "app.py", line 1, in <module>
    from flask import Flask
ImportError: No module named flask
[gonano@test_nanobox /code]$ cat requirements.txt
flask
[gonano@test_nanobox /code]$ pip
bash: pip: command not found

Problems :

• flask package isn't accessible
• where is virtualenv ?
• where is pip ?

Best regards,
Stephane

Pip installs packages to different site-packages directory

Note: This is without using virtualenv.

Installing a package with the following

[gonano@api /code]$ pip install falcon
<...>
Successfully installed Cython-0.23.4 falcon-0.3.0 python-mimeparse-0.1.4 six-1.10.0

and attempting to import the module leads to an error.

[gonano@api /code]$ python
Python 2.7.10 (default, Oct 26 2015, 17:54:35)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import falcon
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named falcon
>>>

Looking at sys.path, I noticed the following

[gonano@api /code]$ python -c 'import sys; print "\n".join(sys.path)'

<...>
/data/lib/python2.7/site-packages

which does not include the path where the package was actually installed (which is /code/env/lib/python2.7/site-packages).

[gonano@api /code]$ pip install falcon
<...>
Requirement already satisfied (use --upgrade to upgrade): falcon in /code/env/lib/python2.7/site-packages
<...>

and looking in the directory did indeed reveal the package.

[gonano@api /code]$ ls -al /code/env/lib/python2.7/site-packages
total 156
<...>
drwxr-xr-x  7 gonano gonano  4096 Dec 11 23:15 falcon
drwxr-xr-x  2 gonano gonano  4096 Dec 11 23:15 falcon-0.3.0-py2.7.egg-info
<...>
[gonano@api /code]$

Missing sqlite in python3.6

Creating a django project with python3.6 causes:

django.core.exceptions.ImproperlyConfigured: Error loading either pysqlite2 or sqlite3 modules (tried in that order): No module named '_sqlite3'

Refactor workflow

Update boxfile script

• Move code.build to run.config

Python from pkgin is compiled with old flags

When I was using this Nanobox engine, I was surprised to find that some of Python's popular packages such as tensorflow and torch would not install with pip:

$ pip install tensorflow
Could not find a version that satisfies the requirement tensorflow (from versions: )

It turned out that these packages only distribute wheels for Linux with the cp35m (or different Python version equivalent) ABI tag, but the version of Python that pkgin distributes can only run wheels with a cp35 or none ABI tag.

Sure enough, looking in the Makefile for building Python on pkgin, it comes with the --without-pymalloc flag, and so cannot support the above libraries out the box.

Nowadays, I believe most people compile their packages with pymalloc enabled - and indeed it is the default compiler option on recent versions of Python. Notably, the Python distributed by apt-get (and probably other package repos) is compiled with --with-pymalloc.

I can see a few suggestions to make building a wider breadth of packages work:

• Ask package maintainers (torch, tensorflow etc) to make wheels that can run on cp35 ABI (not exactly scalable to all packages)
• Change the Python installed on Nanobox to come from apt-get instead of pkgin (my personal favourite!)
• Submit request to change --without-pymalloc to --with-pymalloc on pkgin
• Something else...!

Thanks in advance - it would be great to have Nanobox able to run these out the box.

ignoring Boxfile when specifying python34

I've specified

build: 
  runtime: python34

in my Boxfile, but nanobox doesn't seem to recognize this and I get a box with python27.

Ensure pip modules are caching between builds

There is some suspicion that pip modules aren't being reserved between runtime builds. This needs to be investigated.

Could not open requirements file: [Errno 2] No such file or directory: '/requirements.txt' => my requirements.txt is in /code/

I've "requirements.txt" in my working folder and I've this error :

Could not open requirements file: [Errno 2] No such file or directory: '/requirements.txt'

More log lines :

INSTALLING PACKAGES ------------------------------------------------->
+> installing python27...
+> installing py27-virtualenv...
+> installing nodejs-0.12...
   $ virtualenv env

VIRTUALENV ENV ------------------------>
   New python executable in env/bin/python2.7
   Also creating executable in env/bin/python
   Installing setuptools, pip...done.
   [√] SUCCESS
   $ env/bin/pip install -r /requirements.txt

PIP INSTALL --------------------------->
   You are using pip version 6.0.8, however version 7.1.2 is available.
   You should consider upgrading via the 'pip install --upgrade pip' command.
   Could not open requirements file: [Errno 2] No such file or directory: '/requirements.txt'
   [!] FAILED
+> Opening a nanobox console:


                                     **
                                  ********
                               ***************
                            *********************
                              *****************
                            ::    *********    ::
                               ::    ***    ::
                             ++   :::   :::   ++
                                ++   :::   ++
                                   ++   ++
                                      +

                      _  _ ____ _  _ ____ ___  ____ _  _
                      |\ | |__| |\ | |  | |__) |  |  \/
                      | \| |  | | \| |__| |__) |__| _/\_

--------------------------------------------------------------------------------
+ You are in a virtual machine (vm)
+ Your local source code has been mounted into the vm, and changes in either
the vm or local will be mirrored.
+ If you run a server, access it at >> test_nanobox.dev
--------------------------------------------------------------------------------
[gonano@test_nanobox /code]$ ls
Boxfile  app.py  requirements.txt
[gonano@test_nanobox /code]$ ls ..
bin  boot  code  data  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[gonano@test_nanobox /code]$ ls
Boxfile  app.py  requirements.txt

Proposal for automatically installing OS packages based on Python requirements

At the moment the requirements.txt file is checked for certain package names. This is not ideal as sometimes packages are implicitly installed as a dependency of another package. Not everyone likes to always pip freeze > requirements.txt, instead, people specify their direct dependencies including bounding versions and let the other packages resolve themselves.

I have thought of a way to do this better, allowing for both scenarios outlined above. There are basically three steps to this:

1. Let pip figure out the dependencies. Part of this process is to download them (this is the only way of really knowing the dependencies of a package).
2. Analyze the packages that were downloaded and install the required OS packages. Whether this should be hard coded or provided some other way has to be decided.
3. Install the downloaded Python packages from the cache.

1. Downloading the Python packages

For this step, pip download -r requirements.txt -d /tmp/python-packages/ should come in handy. The output of the command can be processed as for every package the output seems consistent in some format, which is listed below. The destination directory has to be specified via command line arguments, otherwise the packages are placed in the current directory.

Collecting {{package_name}}
  Downloading {{file_name}}
    (progress bar)
  Saved {{destination_directory}}/{{file_name}}
Collecting {{package_name}} (from {{dependent_package_name}})
  Downloading {{file_name}}
    (progress bar)
  Saved {{destination_directory}}/{{file_name}}

2. Analyzing the downloaded packages

From the format above the exact package names can be extracted. Given these package names the OS dependencies can be looked up in some system (to be specified). These packages can be installed.

3. Installing the Python packages

Now we should have all the dependencies (at least of the packages that are known), so it is safe to install the packages with pip install -r requirements.txt -f /tmp/python-packages/. This makes sure that the already downloaded packages are used instead of downloading them again.