canonical / layer-basic

Base layer for building charms using the reactive pattern

Overview

This is the base layer for all reactive Charms. It provides all of the standard Juju hooks and starts the reactive framework when these hooks get executed. It also bootstraps the charm-helpers and charms.reactive libraries, and all of their dependencies for use by the Charm.

Usage

Go read the layer-basic documentation for more info on how to use this layer. It is now hosted together with the charms.reactive documentation in order to reduce the amount of places a charmer needs to search for info.

Python Versions, Built charms and what they can run on.

Due to major backwards incompatibilities between Python 3.10 and previous versions, there is a compatibility break between Python 3.8 (focal) and earlier versions of Python. Why Python 3.8 rather than 3.10? Mostly due to all of the incompatibilities being deprecated and available in Python 3.8, so the ability of authors to test at that version.

As the charmhub.io now offers a place to build reactive charms on a per series ('base') and architecture basis, layer-basic supports at least Python 3.5 to Python 3.10. However, a charm built on Python 3.5 won't work on Python 3.8 and vice-versa. The objective going forwards is to build the charm for the base that the charm will run on.

layer-basic's Issues

config states use hookenv.Config.changed, which can fail

The config states detect if a configuration item has changed using hookenv.config() and its changed method. This was designed assuming the only thing running in a hook context is a hook, which is now untrue. It is possible for an action to be invoked, which can check charm configuration using hookenv.config() and will flag changed config items as seen. If this action happens to be called between the config being changed and an actual hook being called (such as config-changed), then the configuration changed states will be lost.

The solution is for the base layer to cache the config itself in unitdata.kv(), so the cache only gets updated in charms.layer.basic.init_config_states()

Layer options need a CLI

There's no way to use layer options in bash layers currently, because the parsing code is Python only. We need to expose a CLI for layer options. This can either be a script in the base layer, or maybe we want to split layer options out into a separate library pulled in via the wheelhouse (or into reactive?) so that it can use the setup.py tools for registering a CLI.

Merge options from jujubigdata.utils.DistConfig

jujubigdata.utils.DistConfig contains general configuration that would be valuable to have available through layer-basic. I feel like many layers could take advantage of operations like add users, groups, and dirs if they were available through layer basic.

Mixed use of apt_install and check_call

Code uses mixed apt_install and check_call. The former is a mapped function based on host OS; the latter is a straight CLI call.

Propose: consolidate check_call, nearly all of them are "pip install....", and replace them with apt_ version of mapped functions.

This makes it possible to support multiple host OS using the same code base.

pip installation conflict on Xenial charms built from layer-basic

Any charm being deployed on Xenial which was built from layer-basic is failing due to resource conflict. It seems that installing 7.1.2 when 8.1.1 is on the system.

pkg_resources.VersionConflict: (pip 8.1.1 (/usr/lib/python3/dist-packages), Requirement.parse('pip===7.1.2'))

This is failing for all charms built with this layer on Xenial. It appears something /today/ was updated in Xenial to cause this. #70 attempts to address this, but the repercussion of this is pip 8.1.1 will be installed on trusty machines. This does not appear to be a problem at the moment.

Add upgrade-charm hook

Original issue: juju/charm-tools#10

We need a default upgrade-charm hook so that it can be handled in reactive layers.

charm-env needs option to clarify ambiguous cases

If you need to run a charm tool outside of a Juju hook context, charm-env has to guess which charm directory to use, and it can only work properly if the charm is the one and only principle unit on the machine (i.e., not a subordinate, and no hulk-smashes). It should support a parameter to allow the charm author to disambiguate those cases by indicating the charm name (NB: not the unit name). Something like:

#!/usr/local/sbin/charm-env --charm foo python

basic layer doesn't explicitly install python3-dev

basic-layer ends up pip installing pyaml, which requires python3-dev in order to build, but doesn't explicitly install it. If apt is setup with no-recommends on a system, then python3-dev will be missing and pip installing pyaml will fail.

Change venv option defaults

It tends to be surprising for charm authors (even experienced ones) when Python dependency conflicts between charms co-located on a machine (usually subordinates and the principal) happen due to the charms not changing the default value for use_venv. We should make the default for that be true to improve isolation by default. To reduce surprise from apt installed Python packages, we should also change the default of include_system_packages to true; there is still a chance of conflict there, but it is much smaller and the benefit of doing more what the charm author expects out of the box with regard to system packages is worth it.

One caveat is that we will need to create wrappers for the chlp, charms.reactive, and charms.reactive.sh command-line interfaces (from charmhelpers and charms.reactive, respectively) for bash charms and manual debugging which use the Juju env vars to ensure that the correct venv copy of those tools are run.

[bionic] install hook fails on missing package python3-pip

When using layer-basic on Bionic, we see install hook failures on missing package python3-pip. A manual post-deployment sudo apt-get update resolves this, indicating that perhaps the package is trying to be installed before the index is updated.

Reference:

https://bugs.launchpad.net/charm-aodh/+bug/1765212

Add mechanism for enforcing ordering of wheelhouse install

The dependencies in wheelhouse/ are installed in an "undefined" order (it comes from a glob, so I'm pretty sure it's alphabetical). Most of the time, pip will resolve the dependencies properly so that's fine. However, it occasionally fails for things that are not strictly dependencies but still need to be installed in a particular order.

My current thought is to have a wheelhouse_order layer option that is a list of packages from the wheelhouse (just the package names, not full requirement specifiers nor file names). These will be installed from the wheelhouse, in order. Once they are all processed, any remaining items in the wheelhouse are installed in the existing order. That way, only "need to be installed first" items must be listed, and multiple items can be given a defined order.

Base Layer has an aggressive license

GPL-3 on the base layer is a pretty aggressive license. It was suggested we look into LGPL or a slightly less aggressive license for users building on top of the base layer.

Thread originated in layer-logstash, where i looked at the base layer and copied its license, figuring derived works should be of similar license.

juju-solutions/layer-logstash#11

cfg = layer.options('basic') - module object has no attribute 'options'

unit-hue-3[4047]: 2016-01-12 15:40:05 INFO unit.hue/3.install logger.go:40 File "lib/charms/bootstrap.py", line 31, in install_charm_deps
unit-hue-3[4047]: 2016-01-12 15:40:05 INFO unit.hue/3.install logger.go:40 cfg = layer.options('basic')
unit-hue-3[4047]: 2016-01-12 15:40:05 INFO unit.hue/3.install logger.go:40 AttributeError: 'module' object has no attribute 'options'

Apt pkg deps from packages option not installed

Deps not installed when the packages option is used.

Install hook fails due to pkg deps not being installed: http://paste.ubuntu.com/14480678/

http://bazaar.launchpad.net/~jamesbeedy/charms/trusty/fiche/trunk/files

http://bazaar.launchpad.net/~jamesbeedy/charms/trusty/fiche/trunk/view/head:/layer.yaml

Apt in bootstrap should be retried

I'm occasionally seeing a transient error during bootstrap:

unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached Traceback (most recent call last):
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached   File "/var/lib/juju/agents/unit-ghost-0/charm/hooks/app-storage-attached", line 8, in <module>
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached     basic.bootstrap_charm_deps()
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached   File "lib/charms/layer/basic.py", line 54, in bootstrap_charm_deps
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached     'python3-dev',
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached   File "lib/charms/layer/basic.py", line 160, in apt_install
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached     check_call(cmd + packages, env=env)
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached     raise CalledProcessError(retcode, cmd)
unit-ghost-0: 15:22:31 INFO unit.ghost/0.app-storage-attached subprocess.CalledProcessError: Command '['apt-get', '--option=Dpkg::Options::=--force-confold', '--assume-yes', 'install', 'python3-pip', 'python3-setuptools', 'python3-yaml', 'python3-dev']' returned non-zero exi
t status 100

Python2 from charms.layer.execd, not found

Running layer-basic on Python2 environment will break at importing "charms.layer".

Solution: touch an empty init.py in layer-basic/lib/charms/

Improved package support/modularization

The way the layer currently handles packages is inconsistent. Packages to be installed via apt are placed into layer.yaml, but python packages are in wheelhouse.txt, and there's no support for other packaging formats, such as snaps or npm.

A more modular approach would allow developers to write support for a particular packaging format, with a consistent way to install them. Two possible formats of layer.yaml to support this idea:

options:
  basic:
    packages:
      - apt:traceroute
      - pip:paramiko==2.0.1
      - npm:grunt
      - npm:grunt-cli
      - snap:hello-world

options:
basic:
  packages:
    apt:
        - traceroute
    pip:
        - paramiko==2.0.1
    npm:
        - grunt
        - grunt-cli
    snap:
        - hello-world

Multiple calls to unitdata.kv().flush()

charms.layer.basic.clear_config_states() calls unitdata.kv().flush(), which will be the second time the storage is flushed in the hook. In the perverse case where a failure happens between these two calls, we can end up with invalid state.

I think this is a use case for an actual @cleanup phase in charms.reactive, per juju-solutions/charms.reactive#20

NameError: name 'basestring' is not defined

Placing the apt_install in the bootstrap throws the error below.

unit-gbuilder-0[994]: 2016-01-12 13:24:10 INFO unit.gbuilder/0.install logger.go:40 Traceback (most recent call last):
unit-gbuilder-0[994]: 2016-01-12 13:24:10 INFO unit.gbuilder/0.install logger.go:40   File "/var/lib/juju/agents/unit-gbuilder-0/charm/hooks/install", line 8, in <module>
unit-gbuilder-0[994]: 2016-01-12 13:24:10 INFO unit.gbuilder/0.install logger.go:40     bootstrap_charm_deps()
unit-gbuilder-0[994]: 2016-01-12 13:24:10 INFO unit.gbuilder/0.install logger.go:40   File "lib/charms/bootstrap.py", line 12, in bootstrap_charm_deps
unit-gbuilder-0[994]: 2016-01-12 13:24:10 INFO unit.gbuilder/0.install logger.go:40     apt_install(['python3-pip', 'python3-yaml'])
unit-gbuilder-0[994]: 2016-01-12 13:24:10 INFO unit.gbuilder/0.install logger.go:40   File "lib/charms/bootstrap.py", line 36, in apt_install
unit-gbuilder-0[994]: 2016-01-12 13:24:10 INFO unit.gbuilder/0.install logger.go:40     if isinstance(packages, basestring):
unit-gbuilder-0[994]: 2016-01-12 13:24:10 INFO unit.gbuilder/0.install logger.go:40 NameError: name 'basestring' is not defined
unit-gbuilder-0[994]: 2016-01-12 13:24:10 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1

Quick fix that worked for me is to comment out the
#if isinstance(packages, basestring):
# packages = [packages]

Invalid URL in default hook

The default hooks cite https://jujucharms.com/docs/stable/getting-started-with-charms-reactive/ , but this URL does not exist. Perhaps it will exist?

Lacking comprehensive readme

Base layer is the jumping-off point for most charm authors atm, lack of readme detracts from adoption

feature request: expose test actions to inherited layers

Was thinking maybe having the ability to run simple tox tests on layers that inherit the base class without having to go through the whole amulet song and dance for simple isolated charm tests.

Something like

juju do service/0 tox or juju do service/0 tox -e checkservices, etc.

Bootstrap moved without deprecation, all existing layers with hooks broken

The bootstrap code was moved to lib/charms/layer and the hook template altered. A link was not left to the original path though, so all existing layers with their (now out of date) hooks should fail. A second bug in charm-tools hides this, where lib/charms/bootstrap.py is not removed on 'charm build' so broken layers may remain oblivious for some time.

Attempt to preserve system pip doesn't work

Currently, when not using a virtualenv, bootstrap_charm_deps() function saves a copy of /usr/bin/pip3, pip upgrades the system pip, and then restores the copy of /usr/bin/pip

This is presumably trying to undo the effect of pip-upgrading pip. But it does not achieve this, as the /usr/bin/pip3 file is just an entrypoint, that uses "from pip import main", and the pip upgrade is still in
/usr/local/lib/python3.5/dist-packages, and is used by the restored /usr/bin/pip3 just fine

pip3 needs pip uninstalling, as well as restoring /usr/bin/pip3, I think

Install hook fails due to python-yaml already existing in system

Did a rebuild of my layer this morning and now my charm can't be installed as it errors in the base-layer install hook trying to install the pyyaml wheel:

unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40 Installing collected packages: MarkupSafe, PyYAML, pip, netaddr, pyaml, Tempita, six, Jinja2, charmhelpers, charms.reactive
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40   Found existing installation: PyYAML 3.10
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40 Detected a distutils installed project ('PyYAML') which we cannot uninstall. The metadata provided by distutils does not contain a list of files which have been installed, so pip does not know which files to uninstall.
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40 Traceback (most recent call last):
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40   File "/var/lib/juju/agents/unit-errbot-1/charm/hooks/install", line 8, in <module>
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40     bootstrap_charm_deps()
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40   File "lib/charms/bootstrap.py", line 18, in bootstrap_charm_deps
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40     check_call(['pip3', 'install', '-U', '--no-index', '-f', 'wheelhouse'] + glob('wheelhouse/*'))
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40   File "/usr/lib/python3.4/subprocess.py", line 557, in check_call
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40     if retcode:
unit-errbot-1[2474]: 2016-01-21 11:05:48 INFO unit.errbot/1.install logger.go:40 subprocess.CalledProcessError: Command '['pip3', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'wheelhouse/MarkupSafe-0.23.tar.gz', 'wheelhouse/PyYAML-3.11.tar.gz', 'wheelhouse/pip-8.0.0.tar.gz', 'wheelhouse/netaddr-0.7.18.tar.gz', 'wheelhouse/pyaml-15.8.2.tar.gz', 'wheelhouse/Tempita-0.5.2.tar.gz', 'wheelhouse/charms.reactive-0.3.8.tar.gz', 'wheelhouse/charmhelpers-0.6.1.tar.gz', 'wheelhouse/six-1.10.0.tar.gz', 'wheelhouse/Jinja2-2.8.tar.gz']' returned non-zero exit status 1
unit-errbot-1[2474]: 2016-01-21 11:05:48 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1

make lint has redundant behavior

Bundletester explicitly calls charm-proof, so we really don't need to call it in the makefile during lint.

background info:

having this added to the makefile is redundant, and necessitates me to fork the makefile in my upper layer if I wish to remove this behavior. Charm-tools hasn't been bumped in quite a while and charms leveraging new features will consistently fail when using keywords like:

• terms
• resources

I feel like having lint fail consistently, when proof has already failed, raises red herrings.

Can we remove the proof step from the lint target to stop the redundancy?

Hooks are Python2

'#!/bin/env python' will be Python2, and remain so until Python2 disappears from the default install.

We really, really should be encouraging Python3 work rather than encouraging tech-debt.

This issue also exists in charm-tools, as perhaps 'charm build' should correct shebang lines and make them consistent across layers and I think it is somehow responsible for installing the correct charm-helpers and charms.reactive dependencies.

Reactive charms cannot handle a series upgrade

When upgrading from Trusty to Xenial, the ubuntu charm fails with:

2018-03-14 14:12:07 DEBUG update-status Traceback (most recent call last):
2018-03-14 14:12:07 DEBUG update-status   File "/var/lib/juju/agents/unit-ubuntu-0/charm/hooks/update-status", line 9, in <module>
2018-03-14 14:12:07 DEBUG update-status     basic.init_config_states()
2018-03-14 14:12:07 DEBUG update-status   File "lib/charms/layer/basic.py", line 174, in init_config_states
2018-03-14 14:12:07 DEBUG update-status     from charmhelpers.core import hookenv
2018-03-14 14:12:07 DEBUG update-status ImportError: No module named 'charmhelpers'
2018-03-14 14:12:07 ERROR juju.worker.uniter.operation runhook.go:113 hook "update-status" failed: exit status 1

name 'version' is not defined

I seem to be hitting this on some trusty machines, and not on others -> http://paste.ubuntu.com/23666856/

enable travis for this repo

I'm not sure if the inclusion of a .travis.yml in this layer would adversely affect upper layers, but it sure would be nice to at least have travis lint commits to this repo.

Include Git

Git seems to be heavily used in layers consuming this one (at least most of mine use it). Would be nice to just go ahead and apt-install git in this layer and make it available.

Enable support for CentOS7

Many lines in layer-basic are coded to have Python3 and Ubuntu in mind.

I'm working on a project in which we want to build charms for CentOS7. Through my tests there are four areas to change to make layer-basic Python27 compatible:

1. Using python2 modules, eg python-setuptools.
2. Set the correct binary path for your pip.
3. Use platform.linux_distribution()[0] to determine host platform.
4. apt uses --assume-yes, where yum uses --assumeyes.

For Python3 related lines, we can improve it further by detecting the Python version being used and installing packages accordingly.

install hook failure on Ubuntu Xenial

From @javacruft on March 10, 2016 11:20

Xenial appears to not have python3-setuptools installed by default; as a result, layered charms fail to install:

2016-03-10 11:11:07 INFO install 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2016-03-10 11:11:09 INFO install Ignoring indexes: https://pypi.python.org/simple
2016-03-10 11:11:09 INFO install Requirement already up-to-date: pip in /usr/lib/python3/dist-packages
2016-03-10 11:11:10 INFO install Ignoring indexes: https://pypi.python.org/simple
2016-03-10 11:11:10 INFO install Processing ./wheelhouse/MarkupSafe-0.23.tar.gz
2016-03-10 11:11:10 INFO install     Complete output from command python setup.py egg_info:
2016-03-10 11:11:10 INFO install     Traceback (most recent call last):
2016-03-10 11:11:10 INFO install       File "<string>", line 1, in <module>
2016-03-10 11:11:10 INFO install     ImportError: No module named 'setuptools'
2016-03-10 11:11:10 INFO install     
2016-03-10 11:11:10 INFO install     ----------------------------------------
2016-03-10 11:11:10 INFO install Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-tisofq7o-build/
2016-03-10 11:11:11 INFO install Traceback (most recent call last):
2016-03-10 11:11:11 INFO install   File "/var/lib/juju/agents/unit-aodh-1/charm/hooks/install", line 8, in <module>
2016-03-10 11:11:11 INFO install     basic.bootstrap_charm_deps()
2016-03-10 11:11:11 INFO install   File "lib/charms/layer/basic.py", line 53, in bootstrap_charm_deps
2016-03-10 11:11:11 INFO install     glob('wheelhouse/*'))
2016-03-10 11:11:11 INFO install   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
2016-03-10 11:11:11 INFO install     raise CalledProcessError(retcode, cmd)
2016-03-10 11:11:11 INFO install subprocess.CalledProcessError: Command '['pip3', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'wheelhouse/MarkupSafe-0.23.tar.gz', 'wheelhouse/charmhelpers-0.7.0.tar.gz', 'wheelhouse/pip-7.1.2.tar.gz', 'wheelhouse/netifaces-0.10.4.tar.gz', 'wheelhouse/netaddr-0.7.18.tar.gz', 'wheelhouse/PyYAML-3.11.tar.gz', 'wheelhouse/charms.reactive-0.4.0.tar.gz', 'wheelhouse/Tempita-0.5.2.tar.gz', 'wheelhouse/pyaml-15.8.2.tar.gz', 'wheelhouse/six-1.10.0.tar.gz', 'wheelhouse/Jinja2-2.8.tar.gz']' returned non-zero exit status 1
2016-03-10 11:11:11 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1

Copied from original issue: juju-solutions/charms.reactive#59

Magic imports added to lib/charms/layer

Magic import logic has been added to lib/charms/layer, which stops this namespace working like a standard Python package. It seems to be there as a convenience. It is technically a backwards incompatible change as it changes import order, causes imports of files that should not necessarily be imports (eg. conditional imports, such as only importing a module after its dependencies are installed), and seems an unnecessary support burden as we will need to keep this behavior indefinitely.

Perhaps we shouldn't be recommending lib/charms/layer as a shared area for layers to use, as it requires everyone to play by the same rules and remain backwards compatible (which is hard, since you don't know what behavior others are relying on). Layers can use their own distinct namespace like normal Python packages do.

Sunset the documentation in the README

The documentation for this layer should be merged with the Juju and charms.reactive docs. User shouldn't have to search for info in three places. This will also ease the maintenance burden.

• Put Charm Dependencies in charms.reactive docs and provide single place for "how to install what dependency".
• Merge generic layer namespace with both charms.reactive and Juju docs
• Merge generic layer options explanation with https://jujucharms.com/docs/stable/reference-layer-yaml
• Add hooks part to charms.reactive docs
• Add Layer Configuration part to charms.reactive docs
• Remove "Reactive States" docs since these are now present in both the Juju and the charms.reactive docs.
• Put Exec.d support in charms.reactive docs.
• Add link from layer-basic readme to the relevant sections of Juju and charms.reactive docs.

What I'm not certain about is whether we want duplication between charms.reactive docs and the Juju docs.

Backwards compatibility fix for options fails on trusty

The backwards compatibility fix in #116 doesn't work on Python 3.4 which is what trusty uses.

See canonical/charms.reactive#185

Automatic upgrading

Juju Charms should promote best-practices and have sane and production-ready defaults. Turning on automatic updates seems to a good way to ensure security. Do you think this is something we should be doing in the base layer, so all Charms have this automatically?

What do you think?

to be clear: I'm talking about upgrades system packages using apt.

Document exec.d feature

#29 added support for exec.d preinstall hooks, but it's not documented in the README

basic.py does not support centos7.

required to check OS and support yum on centos7

--- error when deploying on centos7 ---
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected Traceback (most recent call last):
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected File "/var/lib/juju/agents/unit-rac-master-0/charm/hooks/leader-elected", line 8, in
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected basic.bootstrap_charm_deps()
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected File "lib/charms/layer/basic.py", line 55, in bootstrap_charm_deps
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected 'python3-dev',
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected File "lib/charms/layer/basic.py", line 163, in apt_install
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected check_call(cmd + packages, env=env)
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected File "/usr/lib64/python3.4/subprocess.py", line 553, in check_call
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected retcode = call(*popenargs, **kwargs)
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected File "/usr/lib64/python3.4/subprocess.py", line 534, in call
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected with Popen(*popenargs, **kwargs) as p:
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected File "/usr/lib64/python3.4/subprocess.py", line 856, in init
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected restore_signals, start_new_session)
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected File "/usr/lib64/python3.4/subprocess.py", line 1460, in _execute_child
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected raise child_exception_type(errno_num, err_msg)
unit-rac-master-0: 23:01:14 INFO unit.rac-master/0.leader-elected FileNotFoundError: [Errno 2] No such file or directory: 'apt-get'

----- fix should be at least here - check for centos and calll yum
def apt_install(packages):
"""
Install apt packages.

This ensures a consistent set of options that are often missed but
should really be set.
"""
if isinstance(packages, (str, bytes)):
    packages = [packages]

env = os.environ.copy()

if 'DEBIAN_FRONTEND' not in env:
    env['DEBIAN_FRONTEND'] = 'noninteractive'

cmd = ['apt-get',
       '--option=Dpkg::Options::=--force-confold',
       '--assume-yes',
       'install']
for attempt in range(3):
    try:
        check_call(cmd + packages, env=env)
    except CalledProcessError:
        if attempt == 2:  # third attempt
            raise
        sleep(5)
    else:
        break

use_venv breaks on trusty

use_venv breaks on trusty because platform doesn't seem to return the series, see below

2016-12-22 01:53:49 INFO install E: Unable to locate package virtualenv
2016-12-22 01:53:49 INFO install Traceback (most recent call last):
2016-12-22 01:53:49 INFO install File "/var/lib/juju/agents/unit-newrelic-sysmond-trusty-8/charm/hooks/install", line 8, in
2016-12-22 01:53:49 INFO install basic.bootstrap_charm_deps()
2016-12-22 01:53:49 INFO install File "lib/charms/layer/basic.py", line 53, in bootstrap_charm_deps
2016-12-22 01:53:49 INFO install apt_install(['virtualenv'])
2016-12-22 01:53:49 INFO install File "lib/charms/layer/basic.py", line 146, in apt_install
2016-12-22 01:53:49 INFO install check_call(cmd + packages, env=env)
2016-12-22 01:53:49 INFO install File "/usr/local/lib/python3.5/subprocess.py", line 584, in check_call
2016-12-22 01:53:49 INFO install raise CalledProcessError(retcode, cmd)
2016-12-22 01:53:49 INFO install subprocess.CalledProcessError: Command '['apt-get', '--option=Dpkg::Options::=--force-confold', '--assume-yes', 'install', 'virtualenv']' returned non-zero exit status 100
2016-12-22 01:53:49 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1

TypeError: __init__() got an unexpected keyword argument 'namespace'

When trying to destroy a service, all the hooks started failing with this error in reactive:

https://paste.ubuntu.com/17319012/

Any idea what this could be? I did have several charms co-located on a maas unit with Juju 1.25...

Missing base hooks: leadership, update-status, storage

New hooks exist since 1.23 of Juju which are not modeled in the base layer:

• leader-elected
• leader-settings-changed
• {storage}-storage-attached
• {storage}-storage-detaching
• update-status

where {storage} is defined under the top level storage key in metadata.yaml

It'd be great to have them there by default

basic layer should pull in apt layer

I think the apt layer should be pulled into the basic layer and reduce the number of ways to install packages and react on them. This would effectively get rid of an additional layer that tends to be included in all charm anyway

centos support issue

Hi All,

I am trying to deploy charms(the charm is using the reactive python and included layer-basic) on Centos machine, the juju status showing error

I ssh to the juju-machine in which I was deploying, I found some errors installing python3. then I installed python34 manually and restarted the jujud. Then I found the apt-get failure. so I checked the basic.py in this project. there was apt-get in this project.

I did build the charm using charm build -s centos7. but the problem is still there. Can anyone tell me whether I should try another way to build the charm or wait for a updated version to fix this issue?

Processing ./wheelhouse/python-dateutil-2.7.0.tar.gz fails in charm software installation

Hitting this https://paste.ubuntu.com/p/Jc7DTKSjbg/ on all charms that pull in python-dateutil as a dependency of something in wheelhouse.txt.

Config options without defaults never get removed from config.changed.x

I think the problem originates from init_config_states() grabbing all of the config options from the yaml file while clear_config_states() only looks at hookenv.config(). Since hookenv.config() does not include the --all flag when executing the config-get command the options without a default are not included. That, and also due to the fact that config options are always changed when they have no previous value.

Makefile does not define tox lint tests correctly

When I run bundletester, I consistently get an error about flake8 not being found. I believe that this is because:

1. I evilly do not have flake8 installed in my global Python environment.
2. The Makefile in this layer incorrectly added the py34 and py35 tox virtual environments to its path.

Instead, the Makefile should add the "lint" virtual environment to its path, as below:

.PHONY: lint
lint: apt_prereqs
	@tox --notest
	@PATH=.tox/lint/bin flake8 $(wildcard hooks reactive lib unit_tests tests)
	@charm proof

I am planning on submitting a patch for this soon -- wanted to record in case I get sidetracked, though.

Null values in config.yaml raises KeyError exception

A null value in the config.yaml cause charm install to fail with:

2016-04-29 21:34:29 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1
2016-04-29 21:35:52 INFO install Traceback (most recent call last):
2016-04-29 21:35:52 INFO install File "/var/lib/juju/agents/unit-congress-3/charm/hooks/install", line 9, in
2016-04-29 21:35:52 INFO install basic.init_config_states()
2016-04-29 21:35:52 INFO install File "lib/charms/layer/basic.py", line 139, in init_config_states
2016-04-29 21:35:52 INFO install toggle_state('config.set.{}'.format(opt), config[opt])
2016-04-29 21:35:52 INFO install KeyError: 'os-public-hostname'

Add CLI interface for layer options

No way to use layer options in bash layers currently.

config states should be evaluated from a whitelist of keys in config.yaml

otherwise, anyone adding keys to config() will incur error