cloudant / cirrus

python library build, test and devop like things assistant

⚠️ cirrus has been ported to Python 3 and newer versions (>=3.0.0) are not backwards compatible. The last Python 2 release was 2.0.2. ⚠️

cirrus

python library build, test and devop like things assistant

Read Me First!

All new development work on this fork should be based from the new-development branch.

This repo was forked from evansde77/cirrus (a former Cloudant employee), but the forked version broke some Cloudant-specific tools. Our solution was to check out the most recently working version, 0.1.7 and create a new release from that.

• The branch new-development was also created from 0.1.7, and this should be considerd the new develop branch.
• Release 1.0.0 was created from new-development.

Installation Prerequisites

• Cirrus requires python 3.5.
• Git tools are heavily used, git is a requirement as cirrus is accessed via git command aliases.

Installation as a user:

curl -O https://raw.githubusercontent.com/cloudant/cirrus/new-development/installer.sh
bash installer.sh

The installer script will set up an install of cirrus for you in your home directory and prompt for some info so that it can set up some parameters in your .gitconfig The installer will create a virtualenv and install cirrus from pip via the cirrus-cli package, installing the latest available version. If a specific version is required, it may be set via environment variable. For example, if you require version 3.0.0 you can use export CIRRUS_INSTALL_VERSION='==3.0.0'

Installation for Development:

Note: This package uses GitFlow, any development work should be done off the develop branches and pull requests made against develop, not master.

git clone https://github.com/cloudant/cirrus.git
cd cirrus
git cirrus build

User Configuration File:

As cirrus works as a git extension, it will use your gitconfig file. The installer will create a cirrus section in this file and create the following settings:

1. github-user - Your Github Username
2. github-token - A github access token
3. pypi-user - Username of your pypi server
4. pypi-token - Token/password to access your pypi server
5. pypi-ssh-key - SSH key used for scp-like uploads to your pypi server (if HTTP upload isnt supported)

Protip: If you require a different username for ssh access to your pypi server, you can add an optional pypi-ssh-user setting.

Package Configuration Files:

The per package controls used by cirrus live in a cirrus.conf file in the top level of the repo you use with cirrus. This file, coupled with the cirrus setup.py template and command line tools dictate the behaviour of the cirrus commands within the package. Details for the cirrus config are in the CONFIGURATION.md file

Cirrus Commands:

cirrus hello

A simple test command that says hello, verifies that things are working and prints out some info about your cirrus install

Usage:

git cirrus hello

cirrus package

Beta: new in Release 1.6 and above

The cirrus package command provides support for setting up cirrus to work in a new package. Assuming a starting point of a git repo with at least one commit on the master branch, you can run the git cirrus package init command to add the various configuration and packaging files used by cirrus. This is intended to provide a minimal working template, so some configuration post init will be needed to take advantage of other features.

The init command will do the following:

1. set up the develop/master branches for gitflow style work
2. generate a MANIFEST.in file containing the necessary include statements for packaging with setuptools
3. adds the default cirrus setup.py file
4. Generates a basic cirrus.conf file for the package
5. sets up the version number management and history file generation

For example:

# create a minimal repo 
mkdir 
cd test_repo/
git init 
git checkout -b develop 
mkdir src
mkdir throwaway
mkdir src/throwaway
echo "__version__ = '0.0.0'\n" > src/throwaway/__init__.py
echo "requests\n" > requirements.txt
git add src/throwaway/__init__.py requirements.txt
git commit -m "make a package"
git checkout master
git merge develop

# run the package init command
git cirrus package init -p test_repo --no-remote -v 0.0.0 -s src --version-file=src/throwaway/__init__.py 
# can now use cirrus in the package, eg build:  
git cirrus build

Options available for the package init command:

• --repo, -r - Path to repo, defaults to pwd
• --source-dir -s Additional directory structure within package for source code eg if you put your source code in a src subdir in the repo. Assumes repo top level directory if not set
• --package, -p cirrus package name (Required)
• --version, -v initial package version
• --organization, -o Organization name to include in package (Eg, Github org or username)
• --description, -d Package description
• --templates, additional template rules to include in MANIFEST, eg include src/templates/*.json
• --version-file, Version file, defaults to package init.py, where to update the __version__ attribute
• --history-file, changelog history filename, defaults to HISTORY.md
• --requirements, requirements file for pip defaults to requirements.txt
• --master-branch, GitFlow master branch, defaults to master
• --develop-branch, GitFlow develop branch, defaults to develop
• --no-remote, disable pushing changes to remote, commit locally only

cirrus build

Builds a new virtualenv and installs the requirements for the package, setting up a development/testing/deployment environment for the package.

Usage:

git cirrus build

The virtualenv is created in ./venv. Optional parameters for the build command are read from the cirrus.conf, they are;

1. build Section
2. virtualenv_name - sets the name of the virtualenv directory, defaults to venv
3. requirements_file - name of the requirements.txt file, defaults to requirements.txt
4. pypi Section
5. pypi_url - If present, will use the pypi server to install requirements, also requires the pypi username and token to be set in the cirrus section of your gitconfig
6. Other options
7. -c, --clean removes the existing ./venv before building.
8. -d, --docs generate documentation using Sphinx and its generated Makefile. Any optional make commands can be passed along (e.g., --docs clean singlehtml) 1. Requires a sphinx_makefile_dir value set in the docs section of cirrus.conf. 2. sphinx_makefile_dir should point to the directory that contains Sphinx's Makefile.

cirrus feature

Commands related to the creatation and management of a git-flow style feature branch.

1. new - Creates a new feature branch, optionally pushing the new branch upstream following a git-flow style workflow
2. pull-request - Creates a new Pull Request in github requesting to merge the current feature branch with the develop branch, specifying the title, body and list of people to tag in the PR.
3. pr - shorthand for pull-request
4. list - lists all open unmerged feature branches in the repo

Usage:

git cirrus feature new BRANCH_NAME --push
git cirrus feature pull-request --title TITLE --body BODY --notify @AGITHUBUSER,@ANOTHERGITHUBUSER

cirrus review

The cirrus review command provides some utilities for dealing with GitHub pull requests from the cirrus command line. Available commands are:

• git cirrus review list - list all open PRs for the repo, accepts -u or --user to filter for requests from a specific user
• git cirrus review details - Get details for a specific PR, specified by --id
• git cirrus review plusone - Set a Github context for the PR to indicate that the PR has been approved
• git cirrus review review - Add a review comment to a PR, optionally adding the plusone flag to it as well

Examples:

git cirrus review list --user evansde77 # list open PRs by user evansde77
git cirrus review list                  # list all open PRs
git cirrus review plusone --id 500 -c "+1"      # adds the +1 context to the feature via a status update and sets it to success
git cirrus review reviee --id 500 -m "great work, LGTM"  --plus-one -c "+1" # adds a comment to the PR and sets the +1 context status to success

cirrus release

Commands related to creation of a new git-flow style release branch, building the release and uploading it to a pypi server. There are three subcommands:

1. new - creates a new release branch, increments the package version, builds the release notes if configured.
2. build_and_upload - builds a release artifact and uploads it to artifactory. A --dev option is provided to create test releases which can be used for testing (via sapitest002 for example).
3. merge - Runs git-flow style branch merges back to master and develop, optionally waiting on CI or setting flags for GH build contexts if needed

Usage:

git cirrus test # Stop if things are broken
git cirrus release new --micro
git cirrus release build
git cirrus release merge --cleanup
git cirrus release upload --plugin pypi

Options:

1. release new requires one of --micro, --minor or --macro to indicate which semantic version field to increment
2. --bump adds or updates a package==version pair in requirements.txt, e.g. --bump foo==0.0.9 bar==1.2.3.
3. release merge supports the following options:

• --cleanup - removes the remote and local release branch on successful merge
• --context-string - Update the github context string provided when pushed
• --wait-on-ci - Wait for GitHub CI status to be success before uploading

4. upload will push the new release and upload the build artifact to pypi, but may take several non-required options:

• --plugin - Name of the upload plugin module. Options are found in https://github.com/evansde77/cirrus/tree/develop/src/cirrus/plugins/uploaders and can be used to customise the upload process. The pypi plugin does a standard sdist upload to the pypi server configured in your pypirc. The fabric plugin uses fabric to scp the artifact to a custom pypi server.
• --test do not push new release or upload build artifact to pypi
• --pypi-sudo, --no-pypi-sudo use or do not use sudo to move the build artifact to the correct location in the pypi server, defaults to using sudo
• --pypi-url URL override the pypi url from cirrus.conf with URL, equivalent to the -r option for pypi.python.org uploads, can specify a url or a shorthand name from your pypirc.

Release Options in cirrus.conf

You can customise the release merge workflow for each package via the cirrus config with the following settings:

• wait_on_ci - Set true to enable waiting on CI builds for the release branch, defaults to False, this will make the merge command poll the GH status API for the commit and wait for it to become success
• wait_on_ci_develop - Set true to enable waiting on CI builds for the develop branch, defaults to False
• wait_on_ci_master - Set true to enable waiting on CI builds for the master branch, defaults to False
• wait_on_ci_timeout - Timeout in seconds to give up on waiting on CI, defaults to 600s (10 mins)
• wait_on_ci_interval - Interval to poll status in seconds, defaults to 2 seconds
• github_context_string - The github context to update status for if update_github_context is True Eg: continuous-integration/travis-ci
• update_github_context - An alternative to waiting on CI, you can simply flip the status for a context to success if eg you have protected branches without a CI build to wait for. Requires a context to be provided via the github_context_string setting
• push_retry_attempts - Optional number of attempts to try to push during merge
• push_retry_cooloff - Optional time to wait between retries in seconds

Example:

[release]
wait_on_ci = False
wait_on_ci_develop = False
wait_on_ci_master = False
wait_on_ci_timeout = 600
wait_on_ci_interval = 2
github_context_string = continuous-integration/travis-ci
update_github_context = True
push_retry_attempts = 10
push_retry_cooloff = 2

Working with GitHub protected branches

The release command works with two possible mode of [https://github.com/blog/2051-protected-branches-and-required-status-checks](GitHub branch protection) one in which you wait for the CI tests to run and update the status, and one in which there are no checks, so that you have to set status to merge branches in a git flow style.

For the waiting mode, simply flip the boolean wait_on_ci flag to True to wait on CI to run on the release branch. Likewise, the wait_on_ci_develop and wait_on_ci_master params will wait on CI status for the develop and master branches you have configured before merging. You can adjust the total timeout in seconds and the poll interval via the cirrus config file.

For the non-waiting mode, you provide the github context string you want to set and then when the merges back to develop and master occur, that state value is set to success. This mode of operation is useful when your CI system runs release tests in a way not connected to github, but you still have protected branches.

release tips

Protip: If you don't make releases regularly, you'll want to make sure your local repo copy is up to date (cirrus should do these eventually).

git checkout master # get master, if you only have develop
git fetch
git fetch --tags
git checkout develop # you're ready to release!

Protip: If something goes wrong during release building you may end up on a release/A.B.C branch that didn't work out. If you haven't pushed out the tag for the new version you can git checkout develop and git branch -d release/A.B.C. If you've pushed a bad version/tag the best thing to do is resolve the problem and create a new micro version -- don't modify a tag that's already remote (consult your own release workflow).

cirrus test

Command for running tests in a package.

Usage:

git cirrus test

Options and config: --suite SUITE_LOCATION

Must define name of virtualenv in [package] virtualenv Must define [test-default] where (default location for tests, optional if you choose to always include --suite) May define [test-SUITE_LOCATION] where

cirrus qc

Command for running quality control checks via pylint, pyflakes, pep8.

Usage:

git cirrus qc --files --only-changes --pylint --pyflakes --pep8

Options and config: Running with no arguments will run all checks on everything. To run only a specific checker (pylint, pyflakes, or pep8) use the corosponding argument or a combonation of them. Specific files may be ran using '--files' OR check only files that have not yet been commited to the repo by using the '--only-changes' argument. For pylint, a score threshold must be set in cirrus.conf [quality] threshold. The path to an optional rcfile (pylint configuration) may be set at [quality] rcfile.

cirrus deploy

The deploy command provides a plugin driven way to hook into deployment systems like chef and puppet. Since deployment is heavily customisable based on what system is in use, the command supports selecting a plugin and then delegates all CLI options to that plugin.

Usage:

git cirrus deploy --plugin=<plugin_name>  <options for plugin>

Deployer plugins live in cirrus.plugins.deployers and individual docs for each plugin can be found there.

Plugins:

• chef - docs

cirrus docs

Command for publishing Sphinx documentation

Usage:

git cirrus docs build
git cirrus docs pack
git cirrus docs publish

Options and config:

1. git cirrus docs build: --make <options>
   1. When run without --make, the default options clean html are used
   2. Requires a sphinx_makefile_dir value set in the doc section of cirrus.conf.
   3. sphinx_makefile_dir should point to the directory that contains Sphinx's Makefile.
2. git cirrus docs pack requires the following options in cirrus.conf [doc] section:
   • sphinx_doc_dir - should point to the directory where the documentation to be packaged is. E.g. /docs/_build/html
   • artifact_dir - should point to the directory where the documentation artifact should be saved.
3. git cirrus docs publish requires the following options in cirrus.conf [doc] section:
   • publisher - the publisher plugin to use
   1. The publisher selected should have a section in cirrus.conf which contains the publisher options. Available publisher plugins can be found in /cirrus/plugins/publishers
      1. If using the doc_file_server plugin:
         1. in cirrus.conf:
            • doc_file_server_url - the URL of the server the documentation is uploaded to
            • doc_file_server_upload_path - the path to the location on the server the documentation should be uploaded to
            • doc_file_server_sudo - a value of True or False for if sudo should be used when issuing the Fabric put command Note: Optional if doc_file_server_sudo is False
         2. in the [cirrus] section of your .gitconfig:
            • file-server-username - the username used for the documentation file server
            • file-server-keyfile - the path the ssh keyfile to use when uploading the documentation
      2. If using the jenkins plugin:
         1. in cirrus.conf:
            • url - the URL of the Jenkins server
            • doc_job - the name of the Jenkins job for the documentation build
            • doc_var - the variable name which the uploaded documentation tarball will be accessed by (Jenkins File Parameter)
            • arc_var - the variable that will be used to name the file/folder the archive should be unpacked to as determined by the name of the archive filename. I.e. package-0.0.0.tar.gz => package-0.0.0 (Jenkins String Parameter)
            • extra_vars - boolean value indicating if there are move variables to send to Jenkins which should be defined in the section [jenkins_docs_extra_vars]
         2. in the [cirrus] section of your .gitconfig:
            • buildserver-user - Jenkins username for authorization
            • buildserver-token - token or password for authorization

Example cirrus.conf:

[doc]
sphinx_makefile_dir = docs/
sphinx_doc_dir = docs/_build/html
artifact_dir = docs/artifacts
publisher = doc_file_server

[doc_file_server]
doc_file_server_url = http://localhost:8080
doc_file_server_upload_path = /docs/package/archive
doc_file_server_sudo = True

If using publisher = jenkins:

[jenkins]
url = https://localhost:8080
doc_job = doc_build
doc_var = artifact
arc_var = ARCHIVE
extra_vars = True

[jenkins_docs_extra_vars]
var = value
var1 = value1

cirrus's Issues

Look at removing the `release new --rc` feature

Look at removing the release new --rc feature once tagged builds prove to be sufficient.

Remove HISTORY.md file updating

The HISTORY.md file is quite hard to read, insanely large, and it duplicates information we already have in git log.

The file was well intentioned but we should consider removing it.

build_and_upload should use local virtualenv

I ran git cirrus release build_and_upload without the local carburetor virtualenv activated and it failed with the message below:

[shudgston@SeansMBP carburetor (release/14.1.0)]$ git cirrus release build_and_upload
2.7.10 (default, Aug 17 2018, 17:41:52)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]
/Users/shudgston/.cirrus/venv/bin/python
2018-11-28;INFO;Building and uploading release...
[localhost] local: python setup.py egg_info  bdist_wheel upload -r local

Fatal error: local() encountered an error (return code 1) while executing 'python setup.py egg_info  bdist_wheel upload -r local'

Aborting.

The build_and_upload command should use the local virtualenv python bin.