madcore-ai / cli Goto Github PK

View Code? Open in Web Editor NEW

9.0 7.0 4.0 421 KB

Madcore Core CLI - Kubernetes Clusters with a Single Point of Truth

License: MIT License

Python 99.24% Shell 0.76%

kubernetes kops minikube elk efk kafka neo4j madcore

cli's People

Contributors

Stargazers

Watchers

Forkers

doginal aland-zhang christinataft worldie-com

cli's Issues

describe stack issue

An error occurred (SignatureDoesNotMatch) when calling the DescribeStacks operation: Signature expired: 20170205T115251Z is now earlier than 20170205T115553Z (20170205T120053Z - 5 min.)

setup.py missing jenkins

polfilm@petersmacbook2:~/builds$ madcore
Traceback (most recent call last):
  File "/usr/local/bin/madcore", line 9, in <module>
    load_entry_point('madcore==0.3', 'console_scripts', 'madcore')()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 558, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2682, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2355, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2361, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/site-packages/madcore-0.3-py2.7.egg/madcore/madcore.py", line 10, in <module>
    import configure
  File "/usr/local/lib/python2.7/site-packages/madcore-0.3-py2.7.egg/madcore/configure.py", line 14, in <module>
    from base import MadcoreBase
  File "/usr/local/lib/python2.7/site-packages/madcore-0.3-py2.7.egg/madcore/base.py", line 10, in <module>
    from jenkins import Jenkins

configure - parameters from bash

Same we have for plugins parameters here: #83.

plugin remove k8s failing

When running 'plugin remove k8s' fail
When running 'destroy' fail
minor (no piglets)

stack create - is failing if roll back present

If previous stack create has rolled back then Stack is indeed created but it has been rolled back. Therefore stack resources are not present.

We need to auto-remove rolled back stack and install again.

no .madcore/cloudformation folder must trigger configure

currently that doesn't work fix required for 0.2 release

Followme does not need local cloudformation script

I rebuild (which removes repos)
I try to configure (to clone repos) so i can use plugins because they're not loaded
I hit followme (but no plugins folder)

Madcore Core CLI - Deep Learning & Machine Intelligence Infrastructure Controller
v0.5.5.6+ge20c961 Licensed under MIT (c) 2015-2017 Madcore Ltd - https://madcore.ai

2017-03-29 09:46:51,575 [DEBUG] madcore: initialize_app
2017-03-29 09:46:51,579 [DEBUG] madcore: prepare_to_run_command Followme
2017-03-29 09:46:52,667 [INFO] madcore.cmds.followme: Core Followme: Your public IP detected as: 81.145.238.131
2017-03-29 09:46:52,668 [INFO] madcore.cmds.followme: Updating 'MADCORE-FollowMe' Stack...
2017-03-29 09:46:52,669 [ERROR] madcore: [Errno 2] No such file or directory: u'/Users/polfilm/.madcore/cloudformation/sgfm.json'
2017-03-29 09:46:52,669 [DEBUG] madcore: Got an error: [Errno 2] No such file or directory: u'/Users/polfilm/.madcore/cloudformation/sgfm.json'

We know exactly the name of the stack and region in which it is, Do we need the CF file to be on the disk? Can you look up --use-previous-template

http://docs.aws.amazon.com/cli/latest/reference/cloudformation/update-stack.html

missing settings.json

Settings should be kept in .yaml file

core endpoints
core selftest

cannot produce exception

(madcore) core endpoints
[Errno 2] No such file or directory: '/Users/polfilm/.madcore/settings.json'
(madcore) core selftest
[Errno 2] No such file or directory: '/Users/polfilm/.madcore/settings.json'

they should rather tell you what to do to avoid this or what has not been done

ROLLBACK_COMPLETE - not aborting listening sequence in 'stack create'

... as previously stated

AWS::CloudFormation::Stack               ROLLBACK_COMPLETE
Resetting dropped connection: cloudformation.eu-west-2.amazonaws.com
Resetting dropped connection: cloudformation.eu-west-2.amazonaws.com
Resetting dropped connection: cloudformation.eu-west-2.amazonaws.com
Resetting dropped connection: cloudformation.eu-west-2.amazonaws.com

Add plugins scenarios with input params

Provide examples of scenarios with input params for plugins commands.

followme exception when stack does not exist

(madcore) followme
2017-01-15 13:28:52,061 [DEBUG] madcore: prepare_to_run_command Followme
2017-01-15 13:28:52,061 [INFO] madcore: Already configured.
2017-01-15 13:28:52,427 [INFO] madcore.cmds.followme: Core Followme: Your public IP detected as: 90.203.250.106
2017-01-15 13:28:52,724 [ERROR] madcore.cmds.followme: An error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id MADCORE-FollowMe does not exist
2017-01-15 13:28:52,724 [ERROR] madcore: 'NoneType' object has no attribute '__getitem__'
2017-01-15 13:28:52,724 [DEBUG] madcore: Got an error: 'NoneType' object has no attribute '__getitem__'
2017-01-15 13:28:52,725 [DEBUG] madcore: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/display.py", line 112, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/madcore-0.1.60+gdf6e139-py2.7.egg/madcore/cmds/followme.py", line 40, in take_action
    previous_parameters = stack['Parameters']
TypeError: 'NoneType' object has no attribute '__getitem__'

Cognito link must be static

This makes no sense. Region got switched to client region and it should have stayed the same.

2017-01-12 21:56:22,976 [ERROR] madcore: Could not connect to the endpoint URL: "https://cognito-identity.eu-west-2.amazonaws.com/"

allow use of named profile of AWS cli

AWS CLI allows to use named profiles this will be handy if madcore cli would support it.

run backup job after all services is up

Git latest version issue

There is an issue when we get the latest version of the repo using --force option.

git describe version number in CLI header

second line before "Licensed"

this has to work with way we push to pypy

python setup.py register -r pypitest
python setup.py sdist upload -r pypitest

followme row has incorrect number of values

Last step of FollowMe command should be a table showing OLD and NEW address. Something must have changed because that table is now showing an exception.

AWS::CloudFormation::Stack               UPDATE_COMPLETE
2017-01-16 18:12:57,774 [ERROR] madcore: Row has incorrect number of values, (actual) 13!=3 (expected)
2017-01-16 18:12:57,774 [DEBUG] madcore: Got an error: Row has incorrect number of values, (actual) 13!=3 (expected)
2017-01-16 18:12:57,811 [DEBUG] madcore: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cliff-2.4.0-py2.7.egg/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/site-packages/cliff-2.4.0-py2.7.egg/cliff/display.py", line 113, in run
    self.produce_output(parsed_args, column_names, data)
  File "/usr/local/lib/python2.7/site-packages/madcore-0.1.70+gca0c294-py2.7.egg/madcore/libs/cloudformation.py", line 25, in produce_output
    self.formatter.emit_list(column_names, data, Stdout(), parsed_args)
  File "/usr/local/lib/python2.7/site-packages/cliff-2.4.0-py2.7.egg/cliff/formatters/table.py", line 92, in emit_list
    self.add_rows(x, column_names, data)
  File "/usr/local/lib/python2.7/site-packages/cliff-2.4.0-py2.7.egg/cliff/formatters/table.py", line 79, in add_rows
    table.add_row(_format_row(first_row))
  File "/usr/local/lib/python2.7/site-packages/prettytable.py", line 818, in add_row
    raise Exception("Row has incorrect number of values, (actual) %d!=%d (expected)" %(len(row),len(self._field_names)))
Exception: Row has incorrect number of values, (actual) 13!=3 (expected)

Plugin level parameters not present in cmd line

Currently when we have a parameter that is defined on plugin level(so, all jobs will inherit it) parameters can't be set via cmd line.

ASG activities when we install plugin

Currently activities are not read properly when we install a plugin that has initial start nodes defined.

Plugin commands are not visible after install in interactive mode

If we are in interactive more of the cli and install plugin we do no see the new commands, we need to exit it and enter again to see. Investigate and see if this is possible.

Team Nomenclature Fix

We need to adapt the team name filtering so it's equal to the RFC 1034 standard where the only special characters allowed is hyphen, removing the others special characters and creating a plaintext.

Like team grin_creative get filtered to grincreative.

Fix Help. Delete causing exception

Delete Causing Exceptions
Remove Double Space in Commands

Madcore Core CLI - Deep Learning & Machine Intelligence Infrastructure Controller
0.1.60+gdf6e139
Licensed under MIT (c) 2015-2017 Madcore Ltd - https://madcore.ai

2017-01-14 18:23:06,914 [DEBUG] madcore: initialize_app
2017-01-14 18:23:06,914 [DEBUG] madcore: prepare_to_run_command HelpCommand
2017-01-14 18:23:06,914 [INFO] madcore: Already configured.
usage: madcore [--version] [-v | -q] [--log-file LOG_FILE] [-h] [--debug]

Madcore Core CLI - Deep Learning & Machine Intelligence Infrastructure
ControllerLicensed under MIT (c) 2015-2017 Madcore Ltd - https://madcore.ai

optional arguments:
  --version            show program's version number and exit
  -v, --verbose        Increase verbosity of output. Can be repeated.
  -q, --quiet          Suppress output except warnings and errors.
  --log-file LOG_FILE  Specify a file to log output. Disabled by default.
  -h, --help           Show help message and exit.
  --debug              Show tracebacks on errors.


Commands:

  complete       print bash completion command

  configure      Configure madcore

  create         Create madcore

Could not instantiate <cliff.commandmanager.EntryPointWrapper object at 0x7f64a5df8210>: __init__() takes at least 3 arguments (2 given)

  endpoints      

  followme       Update Followme stack with current IP

  help           print detailed help for another command

  registration   Register madcore

  selftest       

  stacks         List stacks

~/.ssh/id_rsa key not found

Can't make assumptions about key. Prompt confirming 1) location 2) file exists is not in the config anymore.

2017-01-14 14:57:59,213 [INFO] madcore.configure: Map local private key file
2017-01-14 14:57:59,213 [ERROR] madcore.configure: No private ssh key found at: '/home/polfilm/.ssh/id_rsa'
2017-01-14 14:57:59,213 [INFO] madcore.configure: Can't continue configuration. EXIT.

Delete repo from config after destroy

Currently when we destroy repo branch/commit remain in config file. We need to remove that so that when we create next time it gets the latest version(or dependent of the input env).

Expose madcore variables for plugins

Allow more madcore variables to be accessed via plugins, we need also:

domain
subdomain

Check plugin installed properly

Currently if plugin is installed we change the client config and mark as installed. Next time this config is checked and if already installed skip it. We need to properly check the status with the madcore and see if it's up and installed without looking at local flag.

Just a note: in case that you want to reinstall again you can use --force command:

madcore plugin install slack --force - this will not check if already installed.

madcore initialisation fails

Steps to reproduce

Remove all .madcore directories
pip uninstall madcore
pip install madcore (Installs version madcore-0.3.6)
madcore

Error ->

2017-01-18 21:33:01,712 [INFO] madcore.configure: User Registration
2017-01-18 21:33:01,713 [INFO] madcore.configure: Start user registration.
2017-01-18 21:33:03,484 [ERROR] madcore: An error occurred (NotAuthorizedException) when calling the GetId operation: Access to IdentityPool 'eu-west-1:964ea940-a8e6-44a5-a88b-c510cfe487d7' is forbidden.
2017-01-18 21:33:03,484 [DEBUG] madcore: Got an error: An error occurred (NotAuthorizedException) when calling the GetId operation: Access to IdentityPool 'eu-west-1:964ea940-a8e6-44a5-a88b-c510cfe487d7' is forbidden.
2017-01-18 21:33:03,486 [DEBUG] madcore: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/local/lib/python2.7/site-packages/cliff/display.py", line 112, in run
column_names, data = self.take_action(parsed_args)
File "/usr/local/lib/python2.7/site-packages/madcore/cmds/configure.py", line 71, in take_action
config_results = configure.take_action(parsed_args)
File "/usr/local/lib/python2.7/site-packages/madcore/configure.py", line 409, in take_action
self.configure_user_registration()
File "/usr/local/lib/python2.7/site-packages/madcore/configure.py", line 306, in configure_user_registration
aws_lambda = AwsLambda()
File "/usr/local/lib/python2.7/site-packages/madcore/libs/aws.py", line 46, in init
self.lambda_client_no_auth = self.create_aws_lambda_client()
File "/usr/local/lib/python2.7/site-packages/madcore/libs/aws.py", line 68, in create_aws_lambda_client
identity_id = self.get_identity_id()
File "/usr/local/lib/python2.7/site-packages/madcore/libs/aws.py", line 60, in get_identity_id
IdentityPoolId=self.identity_pool_id,
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 251, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 537, in _make_api_call
raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (NotAuthorizedException) when calling the GetId operation: Access to IdentityPool 'eu-west-1:964ea940-a8e6-44a5-a88b-c510cfe487d7' is forbidden.

stopwatch wait for jenkins

when waiting for jenkins we should see a message

[date time sec] - Waiting until Jenkins is up...

new message should show up every 10 seconds.

dns delegation check gives up to quickly

during reinstallation dns stack is reused. dns change must propagate. currently TTL is set to 300 seconds. timeout for wait in CLI must be at least 360 seconds to avoid premature error below

|  \ |\ | [__     |  \ |___ |    |___ | __ |__|  |  | |  | |\ | 
|__/ | \| ___]    |__/ |___ |___ |___ |__] |  |  |  | |__| | \| 
                                                                

2017-01-27 11:22:27,891 [INFO] madcore.libs.cloudformation: DNS delegation
2017-01-27 11:22:27,891 [INFO] madcore.libs.cloudformation: DNS delegation start
2017-01-27 11:22:30,976 [INFO] madcore.libs.cloudformation: DNS delegation verified.
2017-01-27 11:22:30,977 [INFO] madcore.libs.cloudformation: DNS delegation end.
2017-01-27 11:22:30,977 [INFO] madcore.libs.cloudformation: Wait until DNS for domain 'digisign.devopshosted.com' is resolved...
2017-01-27 11:22:31,011 [ERROR] madcore.libs.cloudformation: Domain 'digisign.devopshosted.com' points to '34.248.251.122' but should point to '34.249.214.81'
2017-01-27 11:22:31,011 [INFO] madcore.libs.cloudformation: EXIT

madcore status

new "status" command

endpoints - must be dynamic based on what is installed only

status - old idea of status header in cli that shows status of Core (online/offline) and FollowMe (Current/Outdated) + Detected Public IP - all of this in one line

repos - table names + version local + version on core

configure must detect correct key presend by thumbprint match

during configuration after user key is selected we should make sure we can identify matching key in .ssh folder (scan public keys only obviously) and ask user to confirm (set default) of corresponding private key. (this will be required for "core ssh" command)

Post merge fixes in PR#2

Add colored console output

Would be good to have change in color for things like: ERROR, WARNING, that way we can have a better visual on what is happening.

Issue when calling aws lambda


2017-02-02 16:29:03,153 [DEBUG] madcore.configure: Authenticate to bitbucket...
2017-02-02 16:29:03,884 [DEBUG] madcore.configure: Connect to bitbucket and get information...
2017-02-02 16:29:04,510 [INFO] madcore.configure: Check if madcore user already exists: '[email protected]'
Input madcore password:
2017-02-02 16:29:17,945 [INFO] madcore.configure: Login user(automatically)
2017-02-02 16:29:18,667 [ERROR] madcore: An error occurred (InvalidSignatureException) when calling the Invoke operation: Signature expired: 20170202T162917Z is now earlier than 20170202T163114Z (20170202T163614Z - 5 min.)
2017-02-02 16:29:18,668 [DEBUG] madcore: Got an error: An error occurred (InvalidSignatureException) when calling the Invoke operation: Signature expired: 20170202T162917Z is now earlier than 20170202T163114Z (20170202T163614Z - 5 min.)
2017-02-02 16:29:18,672 [DEBUG] madcore: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/display.py", line 112, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/madcore-0.4.2.4+g10bc093-py2.7.egg/madcore/cmds/configure.py", line 94, in take_action
    configure.take_action(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/madcore-0.4.2.4+g10bc093-py2.7.egg/madcore/configure.py", line 397, in take_action
    self.configure_user_registration()
  File "/usr/local/lib/python2.7/dist-packages/madcore-0.4.2.4+g10bc093-py2.7.egg/madcore/configure.py", line 319, in configure_user_registration
    if self.user_login(aws_lambda):
  File "/usr/local/lib/python2.7/dist-packages/madcore-0.4.2.4+g10bc093-py2.7.egg/madcore/configure.py", line 233, in user_login
    login_response = aws_lambda.auth_login(email, password)
  File "/usr/local/lib/python2.7/dist-packages/madcore-0.4.2.4+g10bc093-py2.7.egg/madcore/libs/aws.py", line 159, in auth_login
    Payload=json.dumps(payload)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 543, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidSignatureException) when calling the Invoke operation: Signature expired: 20170202T162917Z is now earlier than 20170202T163114Z (20170202T163614Z - 5 min.)

plugin install INGRESS extension

Example component: flasker-hub

When we install plugin flasker-hub we specify app name that becomes subdomain for example:

plugin install flasker-hub --app_name=hubber

this brings up a service on the PRIVATE access (through follow me security group)

We now have extension (different route) through ingress server. All we need to do is run a single job to create another secure SSL route through Ingress server.

Jobname: madcore.plugin.ingress.add.service
Parameters

SERVICE_NAME - flasker-hub-service
SERVICE_PORT - 9019
SERVICE_NAMESPACE - flasker-hub-plugin
APP_NAME - hubber

This new job should be run at the end using --ingress optional parameter (we need to figure out where to take parameres from hubber is the app name will be same

resulting new service will be available all wide internet access through https://hubber.ext..<domain.ext>

Issue when job fails

More details here: madcore-ai/core#58

Validate index plugin json properly

Currently plugin index json file is growing a lots of things are added. We need to make a validator using http://docs.python-cerberus.org/en/stable/ so that we can run it every time run a cmd related to plugins and make sure we display a proper error that smth is wrong in json.

Also using this validation schema we may understand better how to add new plugin.

destroy missing --force for unattended

bash script test requires destroy to be part of the test. in order to automated --force flag is required to allow confirmation of destruction to be passed as parameter

Refactor (Isolate) madcore.yaml file (no more plugins.json)

In plugins repo:

no more directories representing types of plugins. every plugin file is in root of repo. folder name is name of plugin
no more plugins.json file for ALL the plugins. every plugin will have its own madcore.yaml (not json) file representing configuration for that plugin. file will always have the same name. for example "flasker" plugin will be in folder /flasker/ and plugin file will have path of /flasker/madcore.yaml

configure set repos branch/commitid from command line

does not apply to cli (it should not reset itself) however controls set and propagation of branch/commitid throughout the build. this is extension of prod/dev environments and reset/force flags or can be used independently.

for now it would be good enough to be able to do this example:

madcore configure \
     --branch_core=issue-106-refactor-plugins \
     --commit_core=latest \
     --branch_plugins=development \
     --commit_plugins=709c49e09707aa5034242a2d21619165805f2c08 \
     --branch_cloudformation=master \
     --commit_cloudformation=0.5 \
     --branch_containers=master \
     --commit_containers=latest

commit param should also work with tags since they use same

git reset --hard tag-name

or

git reset --hard commit-id

if commit is set to reserved word "latest" then no reset is required. currently the CLI should automatically delete repo folders (similar to what we have in CI job) and not do PULL. only way to ensure no conflicts and disconnected heads is to re-clone ALL repos on configure.

jenkins pipeline show progress of sub-jobs

As user executing from CLI i want to be able to see output of Pipeline sub-jobs

there might be a plugin that does that on jenkins level
if not a temporary switch to monitor a new job is required so figlet header and output can be displayed as part of displayed sequence. this will be required for ALL jobs executed from CLI

DNS and S3 cannot be present when launching new domain under new account in same region

Changing Domain
Same Region
Different Madcore Account

cannot use same s3 cert backup (will not work cert is for different domain)
cannot use same domain delegation (will never resolve because delegation is for different subdomain)

Add ElasticSearch endpoint

dns delegation attempting for 1h

During Rebuild DNS delegation was attempting every second for 1h before final fail.

This should not be happening at all. No delegation is required since DNS stack is surviving destruction. We could easily verify with dig or other tool that dns to master subdomain is resolving to core.

In case of DNS DELEGATION for existing working delegation process should be replaced with Restore WITHOUT calling Let's Encrypt for certificate regeneration.

Described here:
madcore-ai/core#17

`madcore configure` fails if bitbucket user has no teams

2017-01-18 23:19:54,199 [INFO] madcore.configure: Check if user already exists.
Input madcore password:
2017-01-18 23:20:05,151 [INFO] madcore.configure: Login user(automatically)
2017-01-18 23:20:06,259 [ERROR] madcore.configure: User not found: [email protected]
2017-01-18 23:20:06,259 [INFO] madcore.configure: User does not exists, create: '[email protected]'
2017-01-18 23:20:07,215 [ERROR] madcore: options should not be an empty list
Traceback (most recent call last):
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/cliff/display.py", line 112, in run
    column_names, data = self.take_action(parsed_args)
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/madcore/cmds/configure.py", line 71, in take_action
    config_results = configure.take_action(parsed_args)
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/madcore/configure.py", line 409, in take_action
    self.configure_user_registration()
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/madcore/configure.py", line 348, in configure_user_registration
    selected_team = self.single_prompt('team', options=teams, prompt='Select bitbucket team')
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/madcore/configure.py", line 34, in single_prompt
    return questionnaire.run()
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/questionnaire/__init__.py", line 124, in run
    if not self.ask_questions():
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/questionnaire/__init__.py", line 135, in ask_questions
    if not self.ask_question(question):
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/questionnaire/__init__.py", line 146, in ask_question
    self.answers[q.key], back = q.prompter(prompt, **q.prompter_args)
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/questionnaire/prompters.py", line 35, in single
    picker = Picker(options, title=prompt, indicator='=>')
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/pick/__init__.py", line 26, in __init__
    raise ValueError('options should not be an empty list')
ValueError: options should not be an empty list
2017-01-18 23:20:07,217 [DEBUG] madcore: Got an error: options should not be an empty list
2017-01-18 23:20:07,218 [DEBUG] madcore: Traceback (most recent call last):
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/cliff/display.py", line 112, in run
    column_names, data = self.take_action(parsed_args)
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/madcore/cmds/configure.py", line 71, in take_action
    config_results = configure.take_action(parsed_args)
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/madcore/configure.py", line 409, in take_action
    self.configure_user_registration()
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/madcore/configure.py", line 348, in configure_user_registration
    selected_team = self.single_prompt('team', options=teams, prompt='Select bitbucket team')
  File "/home/me/tmp/madcore/env/local/lib/python2.7/site-packages/madcore/configure.py", line 34, in single_prompt

Read domain when repo not cloned

2017-02-06 14:26:31,110 [ERROR] madcore: object of type 'NoneType' has no len()
2017-02-06 14:26:31,111 [DEBUG] madcore: Got an error: object of type 'NoneType' has no len()
2017-02-06 14:26:31,118 [DEBUG] madcore: Traceback (most recent call last):
  File "/Users/polfilm/builds/venv/lib/python2.7/site-packages/cliff-2.4.0-py2.7.egg/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/Users/polfilm/builds/venv/lib/python2.7/site-packages/cliff-2.4.0-py2.7.egg/cliff/display.py", line 112, in run
    column_names, data = self.take_action(parsed_args)
  File "/Users/polfilm/builds/venv/lib/python2.7/site-packages/madcore-0.4.2.17+gb2a7534-py2.7.egg/madcore/cmds/configure.py", line 114, in take_action
    configure.take_action(parsed_args)
  File "/Users/polfilm/builds/venv/lib/python2.7/site-packages/madcore-0.4.2.17+gb2a7534-py2.7.egg/madcore/configure.py", line 452, in take_action
    self.configure_user_registration()
  File "/Users/polfilm/builds/venv/lib/python2.7/site-packages/madcore-0.4.2.17+gb2a7534-py2.7.egg/madcore/configure.py", line 402, in configure_user_registration
    prompt='Select madcore domain')
  File "/Users/polfilm/builds/venv/lib/python2.7/site-packages/madcore-0.4.2.17+gb2a7534-py2.7.egg/madcore/base.py", line 115, in single_prompt
    return questionnaire.run()
  File "build/bdist.macosx-10.10-x86_64/egg/questionnaire/__init__.py", line 124, in run
    if not self.ask_questions():
  File "build/bdist.macosx-10.10-x86_64/egg/questionnaire/__init__.py", line 135, in ask_questions
    if not self.ask_question(question):
  File "/Users/polfilm/builds/venv/lib/python2.7/site-packages/madcore-0.4.2.17+gb2a7534-py2.7.egg/madcore/libs/input_questions.py", line 80, in ask_question
    answer, back = q.prompter(prompt, jinja_params=self.jinja_params, **q.prompter_args)
  File "build/bdist.macosx-10.10-x86_64/egg/questionnaire/prompters.py", line 35, in single
    picker = Picker(options, title=prompt, indicator='=>')
  File "build/bdist.macosx-10.10-x86_64/egg/pick/__init__.py", line 25, in __init__
    if len(options) == 0:
TypeError: object of type 'NoneType' has no len()

Add <env> name when configure

default from master, parameter specifies ANY branch, that branch will be used for all the repos local and remote, so this has to carry into cloud formation (env parameter) and all subsequent clones.

Reimplement jenkins stream console output

Currently jenkins stream output is implemented wrong.

From jenkins REST API:

"""
Accessing Progressive Console Output

You can retrieve in-progress console output by making repeated GET requests with a parameter. You'll basically send GET request to this URL (or this URL if you want HTML that can be put into <pre> tag.) The start parameter controls the byte offset of where you start.

The response will contain a chunk of the console output, as well as the X-Text-Size header that represents the bytes offset (of the raw log file). This is the number you want to use as the start parameter for the next call.

If the response also contains the X-More-Data: true header, the server is indicating that the build is in progress, and you need to repeat the request after some delay. The Jenkins UI waits 5 seconds before making the next call. When this header is not present, you know that you've retrieved all the data and the build is complete.
"""

Example url is:
https://jenkins.madgeo.devopshosted.com/job/madcore.plugin.k8s.status/1/logText/progressiveText?start=0

plugin parameters from bash

we have requirement to launch CI test on development branch we need to be able to script plugin installs and plugin custom commands parameters from a bash file. this means that both static and dynamically detected parameters when plugin is installed must be added at runtime (or based on config) to the argparse parameters when inspected. obviously no parameters when plugin not installed only available after installation but a sequence end to end must be possible as specified in milestone 0.5 description