komlenic / drubs Goto Github PK

Presently init only works if pwd = the project config directory. With -f option pwd should be able to be anything and init work as expected.

Prior to running install or update tasks (or more generically drush make), we should check availability of update.drupal.org and/or other parts of critical drupal infrastructure somehow, and abort prior to making any changes. This will prevent leaving a site in a broken state if resources are not available.

I'm not sure how to best handle instances where specific actions (such as disable) will require additional input. The idea of all commands following the 'drubs action nodes' pattern could probably be maintained fine if the command for drubs disable looked like this:

drubs disable [nodes] --message="My message here"

--message, -m would be an optional argument only to the disable action/command.

This is probably how this needs to be handled.

drubs install fails on the second and repeated builds due to the command 'chmod -R u+w ...sites/default': operation not permitted

On remote hosts where this was observed, the ownership of subdirectories of sites/default/files:

drwxrwxr-x. 3 apache  apache     4096 Apr  1 09:44 bootstrap
-rw-r--r--. 1 jenkins www-users  3434 Apr  1 09:44 failed.csv
drwxrwxr-x. 2 jenkins www-users  4096 Apr  1 09:43 notices
drwxrwxr-x. 3 jenkins www-users  4096 Apr  1 09:42 private
drwxrwxr-x. 2 jenkins www-users  4096 Apr  1 09:41 styles
-rw-r--r--. 1 jenkins www-users 49138 Apr  1 09:44 success.csv

The locations which could not be chmod'ed look like 'bootstrap' above - which makes sense. In this case the jenkins user is a member of the apache group, however that does not give jenkins (the user which drubs is logging in as) the permission to chmod the offending directories (such as 'bootstrap').

In drubs, removing the "-R" from the offending command, near or at here: https://github.com/komlenic/drubs/blob/master/drubs/node.py#L233 ...seems to correct this issue, and the directories, regardless of ownership by apache or jenkins are able to be removed on subsequent builds.

This seems likely to be the only fix required - especially considering that predecessor code to drubs did not include the '-R' flag.

sql queries have been problematic forever, largely due to the combined escape sequences necessary to accommodate ssh, bash, fabric, and sql.

Queries imported from legacy projects fail. Although ideally an easier solution should eventually be recommended (such as using a php script and called via 'drush scr'), it should be determined why legacy queries, unmodified, are failing, and a suitable fix applied.

Trivial but useful: removes node from project.yml file and make and py files. One thing to consider would be the usage of shared make/py files: multiple nodes can reference the same make/py files. When removing one of the nodes, do not remove make/py files if they are shared.

Even though d6 is EOL... should probably still include templates for legacy usage. However, if/when the drush version required/recommended by drubs does not support d6, d6 templates should be abandoned.

Due to inertia and how the predecessor to Drubs worked, additional configuration (beyond what can be accomplished in a drush make file) is currently accomplished in a single python file per node. These files are generally just called "dot P.Y." files in conversation. For the rest of this description, when referring to .py files, we're referring not to generic python scripts, but to the node specific configuration files that Drubs uses.

The types of tasks that are commonly needed in these files are:

• drush commands (particularly module enables/disables, drush scr, drush sql etc)
• linux/shell commands (move files, extract archives, create symlinks, etc.)

These tasks seem quite suited for inclusion in a simple shell script (or scripts). This could accomplish several things:

• remove the need to use/write python to use Drubs (a few simple commands are basically all that is used presently, but it would be better to stick to shell/php)
• remove the need for the current boilerplate code that exists in .py files (using shell scripts would mean that every line in the files was written by the user)
• would solve/simplify #12

These limitations may all be able to be resolved:

• current usage of .py files keeps everything in python/fabric scope
• using shell/drush scripts may not allow for precise failure handling that is available in fabric/python
• currently several environment variables are referenced (mostly for pathing issues) within .py files (can these be passed somehow from fabric to shell scripts?)

This is purely for consistency with other longform versions of options, as well as common command line convention.

Both '--fab_debug' and '--fab-debug' are parsed by argparser as args.fab_debug, so changing the option in the argparser argument settings should be all that is required.

Backup functionality for sites can fairly easily be handled by drush archive-dump commands, but logic as to when to employ it will need to be figured out.

Consider a flag/option --no_backup, -n that could also disable this functionality (useful for when simply rebuilding over and over locally during development to test install scripts/routines, where the backup is simply a waste of time)

Empty directories are not staged by git. Adding a .gitignore file to the files directory should resolve this issue.

Currently, running drubs status without arguments produces an error "too few arguments". drubs status without arguments should return the same result as drubs status all.

This is really only for projects using the predecessor to drubs (not publicly available). As such it makes sense for these instructions to NOT be included permanently with the drubs project, however they still should be documented. Where that should exist is unknown.

• Load average: uptime | cut -d' ' -f14-
• Uptime
• Memory usage
• Ping

An action to restore a backup of a site/project on a node should be added. When called without any arguments such as drubs restore prod a prompt should present a numbered list of available backups from which to choose to restore, and automatically request verification to install the most recent backup.

If Y is entered, the most recent backup will be restored (and this can be automated using the '-y' flag like: drubs restore prod -y.

If an integer is entered (corresponding to the numbered list of available backups that is presented) then that specific backup should be restored.

Otherwise, if N is entered (or any other input), exit without doing anything.

Presently, drubs has been constructed to allow actions to be executed on multiple nodes, sequentially.

drubs install staging production

... will run the install action first on a node named 'staging', then on a node named 'production'. This seems to be (or was originally envisioned to be) a useful feature of drubs, but in practice this is very likely unnecessary. I cannot envision a scenario where the above couldn't just as easily be replaced with:

drubs install staging
drubs install production

Considering the complexity that multi-node actions add to the code/architecture, and the difficulty of sanely adding features like #26 #9 we should refactor drubs to be single-action-per-single-node.

In particular, see node.drush_sql() which is only necessary to accommodate difficult sql commands that must be able to execute correctly locally and remotely (over ssh).

The present drush_sql command applies different levels of escaping to a query, and does appear to work, but this should be better understood and accommodated if possible.

Complicating this issue is a chain of escaping that goes through: python, fabric, shell, and drush.

Backups are presently purged before install and update actions which creates an issue when 'backup_minimum_count' and 'backup_lifetime_days' are both set to 0. (Valid settings, if it is desirable to preserve no backups). Since backups are presently purged immediately after a new one is made (at the beginning of install/update tasks), this creates a situation where no backups exist to restore if the build fails.

The desired behavior when 'backup_minimum_count' and 'backup_lifetime_days' are both set to 0, is to create a backup during the build, and remove it after the build is successful, OR after the build fails and the backup is used to restore. The backup settings in project.yml should be thought of as rules that govern how backups are preserved.

Trivial, but useful: the ability to add another node to project.yml (as well as accomanied make and py files).

During initial development, this likely means including instructions for running 'pip install -e .' in the project repository clone.

It seems that running drubs update on a site that isn't fully bootstrapped ... is attempted, but will fail. To replicate this result:

1. Install a site
2. manually delete the site directory using rm -rf etc
3. run drubs update on the node

A simple check for a bootstrapped site prior to executing drubs update would remedy this issue.

Similar to 'git remote': https://git-scm.com/docs/git-remote

• drubs node (lists all nodes)
• drubs node add (adds a new node to project.py, stubs out new make and py files)
• drubs node remove (deletes a node from project.py)

Example:

[vagrant@localdev chem_ebook]$ drubs install localdev
[vagrant@localdev:22] Executing task 'install'
Temporarily disabling access to site...
[localhost] local: cp /home/vagrant/src/drubs/drubs/data/templates/.htaccess.drubs /var/www/html/chem_ebook
Copying project files...
[localhost] local: mkdir -p /tmp/chem_ebook/files
[localhost] local: cp -R /vagrant/chem_ebook/files/* /tmp/chem_ebook/files/
Creating database...
[localhost] local: mysql -hlocalhost -uroot -proot -e "DROP DATABASE IF EXISTS chem_ebook;CREATE DATABASE chem_ebook;"
Warning: Using a password on the command line interface can be insecure.
Creating site root location...
[localhost] local: mkdir -p /var/www/html/chem_ebook
mkdir: cannot create directory ‘/var/www/html/chem_ebook’: File exists

Fatal error: local() encountered an error (return code 1) while executing 'mkdir -p /var/www/html/chem_ebook'

Aborting.

Some sort of reordering of these steps may be needed, as well as a check that the directory exists before copying .htaccess.drubs.

Elapsed time is printed at the end of each action, per node. Include an aggregate elapsed time if more than one node was acted upon. Or something like:

Total elapsed time (n nodes): h m s

This issue/task should likely include two parts:

1. A documentation markdown file explaining and defining all settings in project.yml.
2. Automatically generated comments for each setting in project.yml.

Presently, the init action simply copies make files from the templates directory to the project directory. This either includes a default list of projects that are unpinned, or projects that are pinned (and a burden to keep updated).

The best solution appears to be to parse the make file in the templates directory and if a project has a version specified in the make file, use that version. If a project does not have a specific version specified, then check updates.drupal.org to determine the latest stable version to pin the project to in the makefile that is created in the project directory. See https://updates.drupal.org/release-history/devel/7.x for an example.

This would allow users to specify their desired version (for example consider projects such as 'zen' which may have several stable branches), as well as by default including the latest versions of all other unspecified projects.

Reinstalling a project, using "drubs install " is a likely destructive action if any content has been created on the site. In order to provide protection against this, there are several options:

1. Simple warning and y/n response. This seems not protective enough (especially when you consider that users become accustomed to automatically responding to such messages, when reinstalling a project over and over during development).
2. Add a boolean setting for each node in the project.yml file named "prohibit reinstall". When set to true/1 attempting to run "drubs install foo" would result in an error and no build.
3. Detect if first install, and allow, but prohibit (by default) later reinstal.
4. Some combination of the above ideas...

... including documentation of the most commonly encountered tasks.

Occasionally a build will hang at a particular step waiting on a response and never actually fail (which means a restore from the pre-build backup cannot be triggered). If possible, a method for detecting a hung build should be utilized to trigger a failure and restoration.

This could potentially be in the form of a per node setting in project.yml such as "timeout_minutes".

Consider two nodes in a project that share/use the same directory for backups (in one example this directory is an NFS mount on multiple servers): When remove_old_backups() is run, currently the node name is not considered, only the dates of the backups.

Additionally, the most recently dated backup is used for a restore - and this could theoretically not be the correct backup from the correct node, in a situation where nodes share a backup directory.

Fixing this issue involves segmenting backups that are considered for purge or use in a restore, by node name.

At this time pip install without the '-e' editable option will fail due to:

Requirement already satisfied (use --upgrade to upgrade): fabric in /usr/lib/python2.7/site-packages (from Drubs==0.1.0)
Requirement already satisfied (use --upgrade to upgrade): pyyaml in /usr/lib64/python2.7/site-packages (from Drubs==0.1.0)
Requirement already satisfied (use --upgrade to upgrade): prettytable in /usr/lib/python2.7/site-packages (from Drubs==0.1.0)
Requirement already satisfied (use --upgrade to upgrade): paramiko>=1.10 in /usr/lib/python2.7/site-packages (from fabric->Drubs==0.1.0)
Requirement already satisfied (use --upgrade to upgrade): ecdsa>=0.11 in /usr/lib/python2.7/site-packages (from paramiko>=1.10->fabric->Drubs==0.1.0)
Requirement already satisfied (use --upgrade to upgrade): pycrypto!=2.4,>=2.1 in /usr/lib64/python2.7/site-packages (from paramiko>=1.10->fabric->Drubs==0.1.0)
Installing collected packages: Drubs
  Running setup.py install for Drubs
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-PH4gN1-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-oijjc3-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/drubs
    copying drubs/__init__.py -> build/lib/drubs
    copying drubs/drubs.py -> build/lib/drubs
    copying drubs/node.py -> build/lib/drubs
    copying drubs/tasks.py -> build/lib/drubs
    creating build/lib/drubs/data
    error: can't copy 'drubs/data/templates': doesn't exist or not a regular file

Additional info:
http://stackoverflow.com/questions/13307408/python-packaging-data-files-are-put-properly-in-tar-gz-file-but-are-not-install/
http://stackoverflow.com/questions/779495/python-access-data-in-package-subdirectory

By default, drush make merges existing project directories - that is, on a site that has been previously installed, drush make will merge filesystem changes between the actual installed existing directory and the downloaded project during the make operation. This can, and does, leave files that previously existed but have been deleted from a project on the site - which can cause issues in some cases (and is not the desired behaviour).

It appears that altering drush make calls to include '--overwrite' solves this issue.

In the predecessor to drubs there was a confirmation during the 'install' action. There should, at minimum, be a similar confirmation in drubs. Additionally, with the "new" ability to trigger builds on remote hosts, and the possibility of typing the wrong environment etc by mistake, confirmations should probably exist for 'update', 'destroy', and potentially other actions as well.

For automation and integration with CI tools, adding a '-y' to optional arguments may be used to automatically confirm any... confirmations.

It would be positive to include a list of similar or related tools to drubs, somewhere in the documentation. A preliminary google search reveals:

• https://github.com/acquia/blt
• https://github.com/docksal/docksal
• https://github.com/aquifer/aquifer

Presently all database queries are executed with passwords via the command line, and generate the warning: "Warning: Using a password on the command line interface can be insecure."

In addition to security concerns, this warning makes reliable reading of query results in python more difficult (and is presently being worked around by stripping the warning string above from the string returned from any queries). See site_database_exists() in node.py.

Documentation (specifically the system configuration docs) should be updated to reflect that the drubs user must also be in the apache group.

'drubs help' should become an action.
'drubs help should become an action with specialized help for each action.

It has been observed previously that it is possible for a user to visit a site during the course of a build and see a partially built site, or (in the case of the users table and auto-creation of users) even interfere with the successful building of the site.

We don't necessarily need to require Apache in order to achieve this, however the easiest method seems to be placing a .htaccess file into the site_root. This has the disadvantage of not really disabling the site for the whole build, indeed, after drush make is run, the .htaccess file is overwritten with the new one.

A second thought would be changing the permissions or ownership temporarily of the site_root location such that the build can complete, but such that apache refuses to serve the site. This may also even have the advantage of working under different webservers... nginx etc.

Neither of these solutions are ideal (though if the second is possible, it may suffice).

Several other project-level types of data/files will need to be included in the config repo... such as csv files for potential data imports, or php scripts to be executed by drush php-eval during postconfigure(), behat tests, etc.

In keeping with the goal of not needing a full copy of the config repo on each node, at present the likely scenario would involve temporarily copying the files directory (whatever that may be called... 'files'?) to the node being executed upon, and removing when finished with operations that would potentially need the 'files'. (That's probably only install and update actions).