nanobox-io / nanobox-engine-python Goto Github PK
View Code? Open in Web Editor NEWEngine for running Python apps on Nanobox
Home Page: https://nanobox.io
License: MIT License
Engine for running Python apps on Nanobox
Home Page: https://nanobox.io
License: MIT License
When I was using this Nanobox engine, I was surprised to find that some of Python's popular packages such as tensorflow
and torch
would not install with pip
:
$ pip install tensorflow
Could not find a version that satisfies the requirement tensorflow (from versions: )
It turned out that these packages only distribute wheels for Linux with the cp35m
(or different Python version equivalent) ABI tag, but the version of Python that pkgin
distributes can only run wheels with a cp35
or none
ABI tag.
Sure enough, looking in the Makefile for building Python on pkgin
, it comes with the --without-pymalloc
flag, and so cannot support the above libraries out the box.
Nowadays, I believe most people compile their packages with pymalloc enabled - and indeed it is the default compiler option on recent versions of Python. Notably, the Python distributed by apt-get
(and probably other package repos) is compiled with --with-pymalloc
.
I can see a few suggestions to make building a wider breadth of packages work:
cp35
ABI (not exactly scalable to all packages)apt-get
instead of pkgin
(my personal favourite!)--without-pymalloc
to --with-pymalloc
on pkgin
Thanks in advance - it would be great to have Nanobox able to run these out the box.
When I execute
$ nanobox run
in my Python project folder, I've this error :
...
+> Copying code into build container...
/opt/nos/lib/payload.sh: line 9: PL_error:: command not found
...
I've "requirements.txt" in my working folder and I've this error :
Could not open requirements file: [Errno 2] No such file or directory: '/requirements.txt'
More log lines :
INSTALLING PACKAGES ------------------------------------------------->
+> installing python27...
+> installing py27-virtualenv...
+> installing nodejs-0.12...
$ virtualenv env
VIRTUALENV ENV ------------------------>
New python executable in env/bin/python2.7
Also creating executable in env/bin/python
Installing setuptools, pip...done.
[√] SUCCESS
$ env/bin/pip install -r /requirements.txt
PIP INSTALL --------------------------->
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Could not open requirements file: [Errno 2] No such file or directory: '/requirements.txt'
[!] FAILED
+> Opening a nanobox console:
**
********
***************
*********************
*****************
:: ********* ::
:: *** ::
++ ::: ::: ++
++ ::: ++
++ ++
+
_ _ ____ _ _ ____ ___ ____ _ _
|\ | |__| |\ | | | |__) | | \/
| \| | | | \| |__| |__) |__| _/\_
--------------------------------------------------------------------------------
+ You are in a virtual machine (vm)
+ Your local source code has been mounted into the vm, and changes in either
the vm or local will be mirrored.
+ If you run a server, access it at >> test_nanobox.dev
--------------------------------------------------------------------------------
[gonano@test_nanobox /code]$ ls
Boxfile app.py requirements.txt
[gonano@test_nanobox /code]$ ls ..
bin boot code data dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
[gonano@test_nanobox /code]$ ls
Boxfile app.py requirements.txt
I'm trying to setup an environment with nanobox python engine. If I use python-2.7 and then install a psycopg2 package (pip command within a container or with a requirements.txt) it works ok. But when I switch to python-3.6 I got an error "Error: pg_config executable not found.". This is a well known issue but I can't solve it with a nanobox container. I can't install any dependencies (python3-dev or libpq-dev) since there is none of these in Nanobox pkgsrc and even can't find a path to pg_conf.
Is there any way to use python3 with psycopg2 with nanobox?
My starting config:
run.config:
engine: python
engine.config:
runtime: python-3.6
extra_packages:
- nodejs
cache_dirs:
- client/node_modules
extra_path_dirs:
- client/node_modules/.bin
build_triggers:
- server/requirements.txt
- client/package.json
- client/package-lock.json
extra_steps:
- cd server && pip install -r requirements.txt
- cd client && npm install
fs_watch: true
data.postgresql:
image: nanobox/postgresql
config:
version: 9.6
`
At the moment the requirements.txt
file is checked for certain package names. This is not ideal as sometimes packages are implicitly installed as a dependency of another package. Not everyone likes to always pip freeze > requirements.txt
, instead, people specify their direct dependencies including bounding versions and let the other packages resolve themselves.
I have thought of a way to do this better, allowing for both scenarios outlined above. There are basically three steps to this:
pip
figure out the dependencies. Part of this process is to download them (this is the only way of really knowing the dependencies of a package).For this step, pip download -r requirements.txt -d /tmp/python-packages/
should come in handy. The output of the command can be processed as for every package the output seems consistent in some format, which is listed below. The destination directory has to be specified via command line arguments, otherwise the packages are placed in the current directory.
Collecting {{package_name}}
Downloading {{file_name}}
(progress bar)
Saved {{destination_directory}}/{{file_name}}
Collecting {{package_name}} (from {{dependent_package_name}})
Downloading {{file_name}}
(progress bar)
Saved {{destination_directory}}/{{file_name}}
From the format above the exact package names can be extracted. Given these package names the OS dependencies can be looked up in some system (to be specified). These packages can be installed.
Now we should have all the dependencies (at least of the packages that are known), so it is safe to install the packages with pip install -r requirements.txt -f /tmp/python-packages/
. This makes sure that the already downloaded packages are used instead of downloading them again.
What I do :
$ git clone https://github.com/harobed/nanobox-python-sample.git
$ cd nanobox-python-sample
$ nanobox destroy
$ nanobox update
$ nanobox update-images
$ nanobox run
I got :
...
PIP INSTALL --------------------------->
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied (use --upgrade to upgrade): flask in ./env/lib/python2.7/site-packages (from -r /opt/code/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.7 in ./env/lib/python2.7/site-packages (from flask->-r /opt/code/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.4 in ./env/lib/python2.7/site-packages (from flask->-r /opt/code/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): itsdangerous>=0.21 in ./env/lib/python2.7/site-packages (from flask->-r /opt/code/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in ./env/lib/python2.7/site-packages (from Jinja2>=2.4->flask->-r /opt/code/requirements.txt (line 1))
[√] SUCCESS
+> Copying release into live code directory...
+> Moving final build into release environment...
--------------------------------------------------------------------------------
[√] APP SUCCESSFULLY BUILT /// DEV URL : test_nanobox.dev
--------------------------------------------------------------------------------
++> STREAMING LOGS (ctrl-c to exit) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
My execution command specify here : https://github.com/harobed/nanobox-python-sample/blob/master/Boxfile#L3 isn't executed.
Other note : the nanobox output say nothing about line executed.
There is some suspicion that pip modules aren't being reserved between runtime builds. This needs to be investigated.
"FAILED TO CLONING" should be singular "CLONE"
ON failure: nanobox run
- Setting up directories and permissions... - Cloning engine :
! FAILED TO CLONING HTTPS://GITHUB.COM/NANOBOX-IO/NANOBOX-ENGINE-PYTHONG.GIT !
What I do :
$ git clone https://github.com/harobed/nanobox-python-sample.git
$ cd nanobox-python-sample
$ nanobox destroy
$ nanobox update
$ nanobox update-images
$ nanobox dev
requirements are installed with success https://gist.github.com/harobed/67a3351fc4d880634d8f but :
[gonano@test_nanobox /code]$ ls
Boxfile README.md app.py requirements.txt
[gonano@test_nanobox /code]$ python app.py
Traceback (most recent call last):
File "app.py", line 1, in <module>
from flask import Flask
ImportError: No module named flask
[gonano@test_nanobox /code]$ cat requirements.txt
flask
[gonano@test_nanobox /code]$ pip
bash: pip: command not found
Problems :
Best regards,
Stephane
Update boxfile script
Grep doesn't seem to like UTF-16 formatted files.
When searching the requirements.txt file, grep fails to find anything if it's UTF-16
Creating a django project with python3.6 causes:
django.core.exceptions.ImproperlyConfigured: Error loading either pysqlite2 or sqlite3 modules (tried in that order): No module named '_sqlite3'
I've specified
build:
runtime: python34
in my Boxfile, but nanobox doesn't seem to recognize this and I get a box with python27.
Getting the following error message when trying to run pip
.
bash: /code/env/bin/pip: /opt/code/env/bin/python2.7: bad interpreter: No such file or directory
My Boxfile looks like the following
env:
ENVIRONMENT: dev
build:
engine: python
python_runtime: python27
nodejs_runtime: nodejs-4.2
I took a look in /code/env/bin/pip
and it looks like this
#!/opt/code/env/bin/python2.7
should be this?
#!/code/env/bin/python2.7
Nanobox Desktop automates the creation of isolated, virtual development environments. It scans your code and builds a suitable environment within a VM. In order to best automate the build of Python environments, we would greatly appreciate it if you would be willing to answer some or all of the following questions, in as much detail as you can. There are no wrong answers:
Thanks in advance for sharing your time and expertise!
Pillow is the go-to Python package for image manipulation. For example, Django requires it when using model fields for images. Currently it is kind of a hassle to get working due to the fact that it requires quite some packages to install properly (probably to compile the bindings). Therefore it would be nice if these dependencies would be installed automagically.
The documentation has a very comprehensive list of dependencies: https://pillow.readthedocs.io/en/4.1.x/installation.html#building-from-source
As can be seen, not all packages are strictly required, however, I would recommend just installing them all so there are no nasty surprises when people think all features are supported.
After some experimentation I can verify that Pillow compiles when I add the following extra packages to my boxfile (although this list is probably not complete for full support, also, it might install more than strictly necessary):
run.config:
engine: python
engine.config:
runtime: python-3.6
extra_packages:
- libjpeg-turbo
- tiff
- zlib
- freetype2
- lcms2
- libwebp
- tcl
- tk
Another option of course is just providing a precompiled version of Pillow, but I am not sure how that would work in combination with the requirements file and pip
.
virtualenv
.Installing a package with the following
[gonano@api /code]$ pip install falcon
<...>
Successfully installed Cython-0.23.4 falcon-0.3.0 python-mimeparse-0.1.4 six-1.10.0
and attempting to import the module leads to an error.
[gonano@api /code]$ python
Python 2.7.10 (default, Oct 26 2015, 17:54:35)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import falcon
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named falcon
>>>
Looking at sys.path
, I noticed the following
[gonano@api /code]$ python -c 'import sys; print "\n".join(sys.path)'
<...>
/data/lib/python2.7/site-packages
which does not include the path where the package was actually installed (which is /code/env/lib/python2.7/site-packages
).
[gonano@api /code]$ pip install falcon
<...>
Requirement already satisfied (use --upgrade to upgrade): falcon in /code/env/lib/python2.7/site-packages
<...>
and looking in the directory did indeed reveal the package.
[gonano@api /code]$ ls -al /code/env/lib/python2.7/site-packages
total 156
<...>
drwxr-xr-x 7 gonano gonano 4096 Dec 11 23:15 falcon
drwxr-xr-x 2 gonano gonano 4096 Dec 11 23:15 falcon-0.3.0-py2.7.egg-info
<...>
[gonano@api /code]$
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.