About dependencies about installscript

I have not investigated all dependencies listed above, but gdebi was added by me as a replacement for dpkg here: 3d76467 and commit message explains briefly the reason of replacement... as you can see in the commit, the line:

sudo dpkg -i wkhtmltox-0.12.1_linux-trusty-amd64.deb

is replaced with:

sudo gdebi --n `basename $_url`

which in further version becomes: gdebi --n basename $_url`` with removed sudo.
dpkg is a low level installation tool, often used by higher level installation tools internally... for comparison apt-get is a higher level tool which deals with package dependencies automatically, so if you install package "a" with apt-get and "a" depends on "b", then apt-get installs "b" for you as well. But we use apt-get for packages available through repositories... dpkg as a low level tool dos not automate dependency installation for installed package, by using dpkg we leave some room for missing dependencies of installed package if we force to install .deb package, or installation failure because of unmet dependencies. whereas gdebi is exactly same level tool as apt-get, but for offline installation of .deb packages. So, gdebi is offline equivalent of apt-get, but dpkg is not (In general if you tend to install offline .deb packages downloaded from internet or otherwise obtained, then it's handy tool). of course we can use dpkg, but it'll require us to maintain dependency list for wkthmltox as well, and not only for Ubuntu 14, for now other platforms are supported by simply replacing download links here, including other versions of Ubuntu and Debian as well.

about python-pip, here is pip command used, and pip command is installed from python-pip package, so we need it if we're not going to remove mentioned pip command... if it's already installed in a particular distribution, then it'll skipped anyway.

from remove/add candidate list by @sebastian-code, from my side I leave under question following (I excluded gdebi-core and python-pip):

• Remove:
  • subversion
  • bzr
  • bzrtools
• Replacement:
  • python-pil vs Pillow
• Python libraries to REMOVE:
  • python-webdav
  • python-zsi
• Python libraries to ADD:
  • python-geoip
  • python-unicodecsv
  • python-serial

it's interesting more details about need for change regarding listed items

[Yenthe666]

@tvibliani and @sebastian-code I've cleaned it up a bit for now. See this one: 815257c
New status:

• Remove:
  • subversion
  • bzr
  • bzrtools
• Replacement:
  • python-pil vs Pillow
• Python libraries to REMOVE:
  • python-webdav
  • python-zsi
• Python libraries to ADD:
  • python-geoip
  • python-unicodecsv
  • python-serial

Gave this script a test on a new VM without any libraries or any Odoo and everything seems to work fine! Only thing to do it the python-pil vs Pillow, need to find the correct library for this one. If you guys know it, free to throw it in here 👍

Edit: Also added a little more info in the admin notification (fa9148e)

[sebastian-code]

Hi @Yenthe666 the correct library is python-pillow

[Yenthe666]

Thanks @sebastian-code I'll add it in and test it ASAP 👍

[qupro]

Hi, what about solve dependencies installation like this:

cd /tmp && wget https://raw.githubusercontent.com/odoo/odoo/9.0/requirements.txt && sudo pip install -r requirements.txt

as its said in https://www.odoo.com/es_ES/forum/help-1/question/how-can-i-install-odoo-9-88323, so this way, InstallScript will be always automatically update.

[Yenthe666]

Hmm interesting concept @qupro! I tried this once but I did see some errors showing up.
Execute script:

administrator@odoo8:~$ ./odoo_test.sh

Output:

--2015-11-05 09:28:06--  https://raw.githubusercontent.com/odoo/odoo/9.0/requirements.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 23.235.43.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|23.235.43.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 619 [text/plain]
Saving to: ‘requirements.txt’

100%[======================================>] 619         --.-K/s   in 0s      

2015-11-05 09:28:06 (94.2 MB/s) - ‘requirements.txt’ saved [619/619]

Requirement already satisfied (use --upgrade to upgrade): Babel==1.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 1))
Downloading/unpacking Jinja2==2.7.3 (from -r requirements.txt (line 2))
  Downloading Jinja2-2.7.3.tar.gz (378kB): 378kB downloaded
  Running setup.py (path:/tmp/pip_build_root/Jinja2/setup.py) egg_info for package Jinja2

    warning: no files found matching '*' under directory 'custom_fixers'
    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
Downloading/unpacking Mako==1.0.1 (from -r requirements.txt (line 3))
  Downloading Mako-1.0.1.tar.gz (473kB): 473kB downloaded
  Running setup.py (path:/tmp/pip_build_root/Mako/setup.py) egg_info for package Mako

    warning: no files found matching '*.xml' under directory 'examples'
    warning: no files found matching '*.mako' under directory 'examples'
    warning: no files found matching 'ez_setup.py'
    no previously-included directories found matching 'doc/build/output'
Downloading/unpacking MarkupSafe==0.23 (from -r requirements.txt (line 4))
  Downloading MarkupSafe-0.23.tar.gz
  Running setup.py (path:/tmp/pip_build_root/MarkupSafe/setup.py) egg_info for package MarkupSafe

Downloading/unpacking Pillow==2.7.0 (from -r requirements.txt (line 5))
  Downloading Pillow-2.7.0.tar.gz (7.4MB): 7.4MB downloaded
  Running setup.py (path:/tmp/pip_build_root/Pillow/setup.py) egg_info for package Pillow

Downloading/unpacking Python-Chart==1.39 (from -r requirements.txt (line 6))
  Downloading Python-Chart-1.39.tar.gz (76kB): 76kB downloaded
  Running setup.py (path:/tmp/pip_build_root/Python-Chart/setup.py) egg_info for package Python-Chart

Downloading/unpacking PyYAML==3.11 (from -r requirements.txt (line 7))
  Downloading PyYAML-3.11.tar.gz (248kB): 248kB downloaded
  Running setup.py (path:/tmp/pip_build_root/PyYAML/setup.py) egg_info for package PyYAML

Downloading/unpacking Werkzeug==0.9.6 (from -r requirements.txt (line 8))
  Downloading Werkzeug-0.9.6.tar.gz (1.1MB): 1.1MB downloaded
  Running setup.py (path:/tmp/pip_build_root/Werkzeug/setup.py) egg_info for package Werkzeug

    warning: no files found matching '*' under directory 'werkzeug/debug/templates'
    warning: no files found matching '*' under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'examples'
    warning: no previously-included files matching '*.pyo' found under directory 'examples'
    no previously-included directories found matching 'docs/_build'
Requirement already satisfied (use --upgrade to upgrade): argparse==1.2.1 in /usr/lib/python2.7 (from -r requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): decorator==3.4.0 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 10))
Downloading/unpacking docutils==0.12 (from -r requirements.txt (line 11))
  Downloading docutils-0.12.tar.gz (1.6MB): 1.6MB downloaded
  Running setup.py (path:/tmp/pip_build_root/docutils/setup.py) egg_info for package docutils

    warning: no files found matching 'MANIFEST'
    warning: no files found matching '*' under directory 'extras'
    warning: no previously-included files matching '.cvsignore' found under directory '*'
    warning: no previously-included files matching '*.pyc' found under directory '*'
    warning: no previously-included files matching '*~' found under directory '*'
    warning: no previously-included files matching '.DS_Store' found under directory '*'
Requirement already satisfied (use --upgrade to upgrade): feedparser==5.1.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): gdata==2.0.18 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 13))
Downloading/unpacking gevent==1.0.2 (from -r requirements.txt (line 14))
  Downloading gevent-1.0.2.tar.gz (1.7MB): 1.7MB downloaded
  Running setup.py (path:/tmp/pip_build_root/gevent/setup.py) egg_info for package gevent

Downloading/unpacking greenlet==0.4.7 (from -r requirements.txt (line 15))
  Downloading greenlet-0.4.7.zip (77kB): 77kB downloaded
  Running setup.py (path:/tmp/pip_build_root/greenlet/setup.py) egg_info for package greenlet

Downloading/unpacking jcconv==0.2.3 (from -r requirements.txt (line 16))
  Downloading jcconv-0.2.3.tar.gz
  Running setup.py (path:/tmp/pip_build_root/jcconv/setup.py) egg_info for package jcconv

    file jcconv.py (for module jcconv) not found
    file jcconv_test.py (for module jcconv_test) not found
Downloading/unpacking lxml==3.4.1 (from -r requirements.txt (line 17))
  Downloading lxml-3.4.1.tar.gz (3.5MB): 3.5MB downloaded
  Running setup.py (path:/tmp/pip_build_root/lxml/setup.py) egg_info for package lxml
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    Building lxml version 3.4.1.
    Building without Cython.
    ERROR: /bin/sh: 1: xslt-config: not found

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt

    warning: no previously-included files found matching '*.py'
Requirement already satisfied (use --upgrade to upgrade): mock==1.0.1 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 18))
Downloading/unpacking ofxparse==0.14 (from -r requirements.txt (line 19))
  Downloading ofxparse-0.14.tar.gz (42kB): 42kB downloaded
  Running setup.py (path:/tmp/pip_build_root/ofxparse/setup.py) egg_info for package ofxparse

Downloading/unpacking passlib==1.6.2 (from -r requirements.txt (line 20))
  Downloading passlib-1.6.2.tar.gz (408kB): 408kB downloaded
  Running setup.py (path:/tmp/pip_build_root/passlib/setup.py) egg_info for package passlib

Downloading/unpacking psutil==2.2.0 (from -r requirements.txt (line 21))
  Downloading psutil-2.2.0.tar.gz (223kB): 223kB downloaded
  Running setup.py (path:/tmp/pip_build_root/psutil/setup.py) egg_info for package psutil

    warning: no previously-included files matching '*' found under directory 'docs/_build'
Downloading/unpacking psycogreen==1.0 (from -r requirements.txt (line 22))
  Downloading psycogreen-1.0.tar.gz
  Running setup.py (path:/tmp/pip_build_root/psycogreen/setup.py) egg_info for package psycogreen

Downloading/unpacking psycopg2==2.5.4 (from -r requirements.txt (line 23))
  Downloading psycopg2-2.5.4.tar.gz (682kB): 682kB downloaded
  Running setup.py (path:/tmp/pip_build_root/psycopg2/setup.py) egg_info for package psycopg2

    Error: You need to install postgresql-server-dev-X.Y for building a server-side extension or libpq-dev for building a client-side application.

    Complete output from command python setup.py egg_info:
    running egg_info

creating pip-egg-info/psycopg2.egg-info

writing pip-egg-info/psycopg2.egg-info/PKG-INFO

writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt

writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

warning: manifest_maker: standard file '-c' not found



Error: You need to install postgresql-server-dev-X.Y for building a server-side extension or libpq-dev for building a client-side application.



----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/psycopg2
Storing debug log for failure in /home/administrator/.pip/pip.log

[qupro]

According http://stackoverflow.com/questions/13014984/warnings-and-errors-after-trying-to-install-flask-0-9 and http://stackoverflow.com/questions/18204181/warning-in-django-installation-via-pip seems that warnings could be ignored.

For solving last error: sudo apt-get install postgresql-server-dev-X.Y

[Yenthe666]

@qupro yes the problems do not seem to give specific problems or errors but feel unnatural for the average user I'd say.
As for the problem about the last error with postgresql-server-dev-X.Y.. this would mean we would still need to add one apt-get to the installation and I honestly do not get why we'd need this either way. It is a development package anyways..
@sebastian-code and @tvibliani what do you guys think about this?

[sebastian-code]

Hi guys.

I was tempted to answer to the idea from @qupro when I saw it, but I wanted to wait until someone else answer. But enough prologue, and to the meat and bones of the idea.

Using pip is mainly designed to be used with Python virtual enviroments, and is aimed to work in heavy development enviroments, you can use it, but actually there would be need to include some changes into the script to be sure than the Odoo installation seeks it's dependencies in the right places.

Additionally I disagree with ignoring warnings. If those are there, then you should fix them, no ignore them, even if they can be safely ignored. From my point of view, things shouldn't be accepted that way.

Also, some of the Python libraries need to be compiled because they have C dependencies, and that's done in different ways if you go with pip or if you go with apt-get, and actually those processes has different requirements, in that point, pip has higher requirements with memory, development libraries and additional dependencies.

And last, but not least, a regular installation with pip is harder to maintain than it's counterpart with apt-get, you have to be too careful because you can broke dependencies now than they are not automatically managed.

[qupro]

Ok, it seems logical, so without using pip to read and install requirements from requirements.txt directly looks difficult to me:

After @sebastian-code comment I thought in something like sudo apt-get -s install $(cat requirements.txt) using some regular expressions to avoid == characters etc, but after solve that there would be a problem because generally there are differences in package names between Canonical and Pyhton Foundation repositories (http://askubuntu.com/questions/431780/apt-get-install-vs-pip-install/431792#431792)

[sebastian-code]

Obviously than my position has it's downsides, an example would be the need to keep record of the dependencies for Odoo using apt-get or any other command line install tool, because you have to review always the python dependencies and then look their equivalents into the distro repositories, when compared against pip, where the same people from Odoo will keep the dependencies updated inside the requirements.txt file.

But that's the thing about software development, the amount of work is done, is because the developer wants to reduce the amount of work the user, system administrator and the long list of "others" has to do.

Also keeping the pip list would mean than you need to know and maintain the different dependencies to build the python libraries, when you can do it automaticly just using apt-get or the command line installation tool of choice (I mean aptitude in Debian based distros, but there are other distros and other command line tools).

I agree with @sebastian-code, installations with pip are harder to maintain then apt-get. There are two kind of dependencies, explicit dependencies or direct dependencies, that are listed in requirements.txt or in odoo_install.sh, and there are implicit dependencies, or dependencies of dependencies... including tools needed to installation process itself, as you already noted above pip installation sometimes requires some software build stuff (compilers, libraries) to be available at installation time, this stuff also goes into the implicit dependencies. With apt-get and gdebi (at the current state) we have completely automated installation of implicit dependencies if any (it was main reason for replacing dpkg with gdebi at a time), but we have to carry about maintaining explicit dependency list in the odoo_install.sh. If we go with pip + requirements.txt version, then we'll have almost inverted situation, we automate explicit dependency installation for odoo, but we'll have to carry about implicit dependencies of pip install, which may change unexpectedly for example because of internal implementation changes in some explicit dependency package, that requires new libraries to compile it. etc. However, it's an interesting idea to automate dependency installation, it should be great if we come across a solution which automate installation of both kind of dependencies. And for pip + requirements.txt version, maybe it's possible to overcome it's downsides, without compromising reliability of script? I have no time currently to investigate it deeper or test it, but I'll check it later.

[Yenthe666]

@qupro after the detailed information from both @sebastian-code and @tvibliani I'm afraid we won't add this in.. This has far more downsides than upsides.
As Temur said though, if you'd manage to find a way to have the plus sides of both the requirements.txt and the apt-get then I would love to hear more! I'm afraid I do not have enough time to investigate this thoroughly either..

[sebastian-code]

I do agree with @tvibliani about the idea from @qupro perhaps there is a way to have the good sides from both worlds, it would be really nice.

[Yenthe666]

@sebastian-code perhaps there is but I personally really don't have the time to dig through all the docs & I'm not very familair with the Linux packaging systems and way of working either.
So it would boil down to somebody else researching I guess.

[sebastian-code]

I do agree with you @Yenthe666 I was mainly stating that, I'm also really outside of even my uncomfortable zone with that kind of topics, and beyond of my complains and arguments against it, I have not much to offer.

Would it be nice? yes, is it necessary? not really. Right now it's a solved situation, even for Odoo is clear, for them using the native package manager from the distro is the preferred way, even to install Odoo, because they maintain a lot of automatic repositories for the mayor brands out there.

Also I don't want to discard the @qupro 's idea, but I see too many challenges in that way, and I'm busy, not young enough and even lazy enough to not want to have that kind of unnecessary overhead 😜

[gustavovalverde]

I know this is a kind-of-closed subject. I've actually been using, changing and testing this script; but I recently found it was hosted here, so I didn't forked it.

I'm actually working in making the script work with virtualenv as I don't like to mess with the OS python environment, and pip also helps there. Right now I'm using this approach in my own production environment (no errors so far), as is the best way to test it.

I'd like you guys take a look at it and maybe give some advice https://raw.githubusercontent.com/gustavovalverde/odoo-install-scripts/master/odoo_install.sh

P.D.: I'm not a bash expert (not even near) so I might have done some mistakes there.

[sebastian-code]

Hi @gustavovalverde giving it a look, I would add this to everything we said, using pip is cleaner to the distro package manager, and it would help a lot with portability issues because using pip you make the script platform agnostic (besides the PostgreSQL installation), but it has a lot of downsides, as @tvibliani said in his comments, but as an additional:

• If you implement virtualenv you have make sure you're loading it every time you reboot the system, you make some changes, update a library... or everything will fail.
• You need a lot of the -dev libraries of the distro of choice (and the dependency hell it carries: a lot of C python headers, building libraries, headless development dependencies and a really long and boring list of things any SysAdmin would prefer not to have to handle).).
• If you're deploying Odoo on a small VM is almost sure you are going to have issues because you're going to need some minimum from the system resources (to compile Pillow you need more than 1GB of RAM), which rules out small machines for small porpoises, small business and a lot of "and else".

Obviously I'm with you about testing and going along with the idea, but for me (and I guess for the rest of the people here) those are enough reasons to decline on that choice.

[gustavovalverde]

Hi @sebastian-code, thanx for having a look at the script. I don't know if you actually saw the master script or the 16.04 one, here: https://raw.githubusercontent.com/gustavovalverde/odoo-install-scripts/16.04/odoo_install.sh

The last one has been completely reworked considering things like the virtualenv loading everytime. Even though you're right about you'll be needing 1GB just to compile lxml, I didn't saw it as a "killing feature", because for a good production environment where PostgreSQL is on the same host you'll be needing at least 2GB to distribute between PostgreSQL and Odoo and at the very least, two CPU/cores (for Workers).

Something that, as you said, might be a reason to decline this approach is the -dev libraries overhead. Even though you can remove some of these libraries after installing.