tmobile / jazz-installer

Installer for Jazz Serverless Developer Platform!

Jazz Serverless Platform

Seamlessly build, deploy & manage cloud-native applications.

Jazz addresses gaps and pain points with serverless, particularly for production applications. It is not another FaaS implementation. Rather, it enhances the usability of existing FaaS systems. Jazz has a beautiful UI designed to let developers quickly self-start and focus on code. Its modular design makes it easy to add new integrations:

• Services - Today devs can build functions, APIs and static websites. The template-based system makes it easy to define new ones.

• Deployment Targets - Currently we deploy to AWS (Lambda, API gateway and S3). We plan to support Azure Functions.

• Features - Services seamlessly integrate features like monitoring (CloudWatch), logging (ElasticSearch), authentication (Cognito) and secret management (KMS, Vault coming soon).

• Deployment & CI/CD - We leverage Serverless Framework and Git/Bitbucket/Jenkins.

Jazz is open-sourced and under active development by T-Mobile's Cloud Center of Excellence.

Watch the video preview here.

Install

You can install Jazz in your account using the automated installer.

Try Jazz!

You can try out public preview version of Jazz by registering with your email address here. You will need a registration code which can be requested by joining slack. Once in slack, go to #jazz-serverless channel to get a working registration code.

User Guide

Quick version:

Ensure you have a standard install of Python 3.6 or greater, with pip and setuptools

Run:

git clone [email protected]:tmobile/jazz-installer.git
cd jazz-installer
python3 -m virtualenv env && source env/bin/activate
pip install -r requirements.txt
python Installer.py install --stackprefix myteststack --adminemail [email protected] --region us-east-1 scenario3

Uninstall:

cd jazz-installer
source env/bin/activate
python Installer.py uninstall --mode=[all|frameworkonly]

Options:

python Installer.py

For more details, see the Wiki.

Development

If you're interested in submitting a PR, it would be a good idea to set up your editor/IDE to use the following checkers:

• editorconfig so your editor follows the same whitespace/line-ending/indent rules as everyone else.
• flake8 for Python linting
• tflint for Terraform script linting
• foodcritic for Chef script linting

Tooling

New contributions should consist entirely of Python(2, soon to be 3) code or Terraform scripts. No new shell script code will be accepted, we have too much of it and it's not particularly maintainable. If you want to add a new optional feature (rather than simply bugfix) please chat with the maintainers in Slack before starting, and take a look at the feature-extensions subdirectory for an example of how we currently structure such things.

Branching/release flow

1. Breaking/nontrivial features first go into named feature branches cut from develop
2. When/if a feature branch is chosen to be included in the next release, it is merged into develop
3. Release testing happens in develop
4. When confirmed/vetted, develop is merged into master, and master becomes the current release.
5. Small fixes explicitly intended for the next release can be PRed directly into develop without first needing a feature branch.

tl;dr master is always the current release, develop is always the current state of the next release. If you want to contribute a PR, recommend you fork and work in a branch off of develop, then PR against develop. Project owners will move you into a feature branch if they deem it necessary.

License

Jazz is released under the Apache 2.0 License.

jazz-installer's Issues

Keep Terraform file updates by-file

We touch several config files with updates during the Terraform install process.

It's currently really hard to figure out what Terraform scripts touch which config files, because the updates are scatted willy-nilly.

Suggest we create individual/separate <jazz-config-file-name>-update.tf (for instance jazz-installer-vars-update.tf) Terraform files for each file that we need to update during the install, and move every step that touches a file into it's respective Terraform file.

This will also encourage us to avoid updating several different files in one step.

Use Chef only for BYOJ scenario

Right now we're still using Chef to provision both Docker containers and real servers, which is not optimal. Figure out a way to fix that.

Console warnings and errors during Jazz installation

Installed Jazz on the community AMI chef-highperf-centos7-201711290007 on a t2.large EC2 instance.

Command used to install is sudo curl -L https://raw.githubusercontent.com/tmobile/jazz-installer/master/centos7-provision.sh | sh -s -- -ib master && cd jazz-installer && sh Installer.sh -b master

Installation went success and everything works fine. But during installation noticed some errors and warning on console. Just listing those below.

curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
Warning: apt-key output should not be parsed (stdout is not a terminal)

Get:1 https://deb.nodesource.com/node_8.x stretch/main amd64 nodejs amd64 8.12.0-1nodesource1 [13.5 MB]
debconf: delaying package configuration, since apt-utils is not installed

Setting up nodejs (8.12.0-1nodesource1) ...
npm WARN deprecated [email protected]: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.
npm WARN deprecated [email protected]: This version is no longer maintained. Please upgrade to the latest version.
npm WARN deprecated [email protected]: This version is no longer maintained. Please upgrade to the latest version.
npm WARN deprecated [email protected]: This version is no longer maintained. Please upgrade to the latest version

Removing /usr/lib/node_modules/jshint/node_modules/phantomjs-prebuilt/lib/phantom
Copying extracted folder /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1542801954487/phantomjs-2.1.1-linux-x86_64 -> /usr/lib/node_modules/jshint/node_modules/phantomjs-prebuilt/lib/phantom
Phantom installation failed { Error: EACCES: permission denied, link '/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1542801954487/phantomjs-2.1.1-linux-x86_64' -> '/usr/lib/node_modules/jshint/node_modules/phantomjs-prebuilt/lib/phantom'
errno: -13,
code: 'EACCES',
syscall: 'link',
path: '/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1542801954487/phantomjs-2.1.1-linux-x86_64',
dest: '/usr/lib/node_modules/jshint/node_modules/phantomjs-prebuilt/lib/phantom' } Error: EACCES: permission denied, link '/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1542801954487/phantomjs-2.1.1-linux-x86_64' -> '/usr/lib/node_modules/jshint/node_modules/phantomjs-prebuilt/lib/phantom'

[email protected] install /usr/lib/node_modules/@angular/cli/node_modules/node-sass
node scripts/install.js
Unable to save binary /usr/lib/node_modules/@angular/cli/node_modules/node-sass/vendor/linux-x64-57 : { Error: EACCES: permission denied, mkdir '/usr/lib/node_modules/@angular/cli/node_modules/node-sass/vendor'
at Object.fs.mkdirSync (fs.js:885:18)
at sync (/usr/lib/node_modules/@angular/cli/node_modules/mkdirp/index.js:71:13)
at Function.sync (/usr/lib/node_modules/@angular/cli/node_modules/mkdirp/index.js:77:24)
at checkAndDownloadBinary (/usr/lib/node_modules/@angular/cli/node_modules/node-sass/scripts/install.js:114:11)
at Object. (/usr/lib/node_modules/@angular/cli/node_modules/node-sass/scripts/install.js:157:1)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
errno: -13,
code: 'EACCES',
syscall: 'mkdir',
path: '/usr/lib/node_modules/@angular/cli/node_modules/node-sass/vendor' }

node scripts/build.js
Building: /usr/bin/node /usr/lib/node_modules/@angular/cli/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
gyp info it worked if it ends with ok
gyp verb cli [ '/usr/bin/node',
gyp verb cli '/usr/lib/node_modules/@angular/cli/node_modules/node-gyp/bin/node-gyp.js',
gyp verb cli 'rebuild',
gyp verb cli '--verbose',
gyp verb cli '--libsass_ext=',
gyp verb cli '--libsass_cflags=',
gyp verb cli '--libsass_ldflags=',
gyp verb cli '--libsass_library=' ]
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp verb command rebuild []
gyp verb command clean []
gyp verb clean removing "build" directory
gyp verb command configure []
gyp verb check python checking for Python executable "python2" in the PATH
gyp verb which succeeded python2 /usr/bin/python2
gyp verb check python version /usr/bin/python2 -c "import sys; print "2.7.13 gyp verb check python version .%s.%s" % sys.version_info[:3];" returned: %j
gyp verb get node dir no --target version specified, falling back to host node version: 8.12.0
gyp verb command install [ '8.12.0' ]
gyp verb install input version string "8.12.0"
gyp verb install installing version: 8.12.0
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp WARN EACCES user "undefined" does not have permission to access the dev dir "/root/.node-gyp/8.12.0"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp"
gyp verb tmpdir == cwd automatically will remove dev files after to save disk space
gyp verb command install [ '--node_gyp_internal_noretry', '8.12.0' ]
gyp verb install input version string "8.12.0"
gyp verb install installing version: 8.12.0
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version not already installed, continuing with install 8.12.0
gyp verb ensuring nodedir is created /usr/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp/8.12.0
gyp WARN install got an error, rolling back install
gyp verb command remove [ '8.12.0' ]
gyp verb remove using node-gyp dir: /usr/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp
gyp verb remove removing target version: 8.12.0
gyp verb remove removing development files for version: 8.12.0
gyp WARN install got an error, rolling back install
gyp verb command remove [ '8.12.0' ]
gyp verb remove using node-gyp dir: /usr/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp
gyp verb remove removing target version: 8.12.0
gyp verb remove removing development files for version: 8.12.0
gyp ERR! configure error
gyp ERR! stack Error: EACCES: permission denied, mkdir '/usr/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp'
gyp ERR! System Linux 3.10.0-693.5.2.el7.x86_64
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/@angular/cli/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
gyp ERR! cwd /usr/lib/node_modules/@angular/cli/node_modules/node-sass
gyp ERR! node -v v8.12.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
Build failed with error code: 1

node ./scripts/postinstall.js
┌───────────────────────────────────────────────────┐
│ serverless update check failed │
│ Try running with sudo or get access │
│ to the local update config store via │
│ sudo chown -R $USER:$(id -gn $USER) /root/.config │
└───────────────────────────────────────────────────┘
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/@angular/cli/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/jshint/node_modules/phantomjs-prebuilt):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] install: node install.js
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/@angular/cli/node_modules/node-sass):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] postinstall: node scripts/build.js
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1

Our .git history is 1.3 GB because we keep checking in binaries

Our .git history is 1.3 GB because we keep checking in binaries, need to think about ways to fix this.

Merely removing binaries from the current repo won't help, since Git maintains a complete history the fact that those large binaries were in the repo at one point will still bloat the history file size.

We can easily shrink our repo history from 1.3GB to ~130MB with BFG, but note the caveat:

It's best to delete all old clones, as they'll have dirty history that you don't want to risk pushing back into your newly cleaned repo.

That essentially means we'd have to ask all contributors to delete their local branches and re-clone.

Use S3 backend for Terraform state

https://www.terraform.io/docs/backends/types/s3.html

We should look into storing the installer's Terraform state in an S3 bucket rather than in the on-disk terraform.tfstate file. This will make it harder for people to accidentally delete their state file and have to manually remove a Jazz install bit by bit.

S3 buckets created as part of platform install process get public s3 R/W access by default

Description:
S3 buckets that are created as part of platform install process get public S3 R/W permissions by default.

Steps to Reproduce:

• Complete the install process.
• Login in AWS console.
• S3 Buckets are marked as public.

Expected behavior:

• One bucket should have might read access (api-docs)
• Other S3 buckets shouldn't have public access.

Actual behavior:
Unless absolutely required, public access should disabled on S3 buckets.

Reproduces how often:
100%

Additional Information:

• Except for S3 bucket to store swagger docs, public read is not required for others.
• Public write is not required for any bucket.

Install Maven (and other similar Jenkins-required tooling) via Groovy scripts

Right now we install maven on the Jenkins host box ourselves, but this has the downside of

• requiring a Chef cookbook
• because Jenkins is a fussy all-encompassing blob, the Jenkins UI doesn't think maven is installed (even though it's in the PATH) because we didn't install it thru Jenkins itself, as shown here: https://gist.github.com/tszpinda/53e958cf80b0f532c787

Propose we change our manually-installed Jenkins add-ons like Maven to be installed via this method, as it will simplify our installer and make Jenkins happy.

Installer rollback process should be clean and accurate.

The uninstall or rollback process should be clean and accurate. This is applicable when an existing functioning stack is being deleted or when the installer itself rollbacks when an error occurs while installing Jazz. This is also important from a security point of view as well (Example: leaving out credentials in Jenkins without clean up)

To Reproduce
Install Jazz with a user with insufficient privileges, the installer will fail in the middle of the process sometimes after provisioning resources (which should not ideally). Now running the delete script would also fail with permissions errors

Expected behavior
Clean and rollback should be done gracefully in an all scenarios and intimate user if anything left out

New feature: Choose a folder to create platform jobs

Add a new feature in the installer to provide Jenkins folder name to create the jobs. If not provided, apply a default value (from configuration file)

Suggestions: {stack_key}_jazz_xyz

Jazz UI - Admin page is not loading

Installed Jazz on the community AMI chef-highperf-centos7-201711290007 on a t2.large EC2 instance.

Command used to install is sudo curl -L https://raw.githubusercontent.com/tmobile/jazz-installer/master/centos7-provision.sh | sh -s -- -ib master && cd jazz-installer && sh Installer.sh -b master

Used scenario 3

Installation went success and everything works fine. But when i open admin page page gets stuck. Saw a console error

polyfills.a7203ed2c3792ed19849.bundle.js:1 
TypeError: Cannot convert undefined or null to object
    at Function.keys (<anonymous>)
    at n.ngOnInit (2.c15a49c36c77216b6bda.chunk.js:1)
    at Wrapper_n.ngDoCheck (VM1467 wrapper.ngfactory.js:45)
    at View_n3.detectChangesInternal (VM1515 component.ngfactory.js:79)
    at View_n3.t.detectChanges (vendor.6b8406aa1cb0716e8d83.bundle.js:1)
    at t.detectChangesInNestedViews (vendor.6b8406aa1cb0716e8d83.bundle.js:1)
    at proxyViewClass.View_n0.detectChangesInternal (VM1515 component.ngfactory.js:376)
    at proxyViewClass.t.detectChanges (vendor.6b8406aa1cb0716e8d83.bundle.js:1)
    at proxyViewClass.t.internalDetectChanges (vendor.6b8406aa1cb0716e8d83.bundle.js:1)
    at proxyViewClass.View_n_Host0.detectChangesInternal (VM1556 host.ngfactory.js:27)
    at proxyViewClass.t.detectChanges (vendor.6b8406aa1cb0716e8d83.bundle.js:1)
    at t.detectChangesInNestedViews (vendor.6b8406aa1cb0716e8d83.bundle.js:1)
    at proxyViewClass.View_n0.detectChangesInternal (VM1515 component.ngfactory.js:81)
    at proxyViewClass.t.detectChanges (vendor.6b8406aa1cb0716e8d83.bundle.js:1)
    at proxyViewClass.t.internalDetectChanges (vendor.6b8406aa1cb0716e8d83.bundle.js:1)

Rename jazz_installer_vars.json to jazz_config.json

Rename jazz_installer_vars.json to jazz_config.json. jazz_installer_vars.json started as an installer vars file and is now primarily used to drive the behaviour of Jazz at runtime.

This would involve:

1. Renaming the file during install process
2. Updating build packs to load the updated config file (loadBuildModules method)
3. Update admin microservice to reference new file name.

Jenkins configuration is not idempotent

The jenkins-cli commands in our Chef cookbook will fail if you re-run them against a previously-configured Jenkins instance, because Jenkins reports that it has already been configured, and the jenkins-cli JAR file returns a nonzero exit code.

This isn't a new bug, this has been the case for some time. Suggest we decide if users care about this, since

1. Users shouldn't be doing this on their installs anyway
2. We clearly need to focus on clean e2e runs anyway.
3. We probably can't fix it without doing something ill-advised like ignoring exit codes.

Example error here: Chef-error.txt

New feature: Improve BYOJ (Bring Your Own Jenkins) scenario

Split out from #110

We assume you are running a Jenkins server on a RHEL/CentOS instance. This is a little shortsighted, and we should try to realistically support every platform the Chef provisioner understands.

Now that the Chef cookbooks have been tidied it should be easier to get them working on e.g. Ubuntu or Debian.

Note that "improve" might also mean documenting how to manually prep your Jenkins server for use by Jazz, with a list of the prerequisites we expect to be installed on the box, for people that would rather install the prerequisites manually rather than running our Chef cookbook against their server.

Hard coded port number for bitbucket hostname

We have hardcoded port number for bitbucket hostname: https://github.com/tmobile/jazz-installer/blob/master/installscripts/wizard/run.py#L54

Please remove this hard coded value everywhere and by default expect it to be mapped to port 80. If it isn't let the installer have the user input this value.

v1.4: Python based services fail to get activated

When using v1.4 with scenario (3), python based services fail to get activated.

v1.4: Installer doesn't seem to work with personal AWS Account

Description

v1.4: Installer doesn't seem to work with personal AWS Account

Steps to Reproduce

1. Follow install steps as suggested (here)[https://github.com/tmobile/jazz-installer/wiki#install
2. Provide AWS Key details from your personal AWS Account

Expected behavior:
Installer finishes successfully

Actual behavior:
Installer fails with the error below:

provider.aws: No valid credential sources found for AWS Provider.
 Please see https://terraform.io/docs/providers/aws/index.html for more information on
 providing credentials for the AWS Provider^[[0m^[[0m

Reproduces how often:
Multiple attempts to install on a new image stopped at the same place.

Homebrew recipe

Would be great to have a Homebrew for macOS user.

Use Terraform to manage Gitlab provisioning

Looks like we can create and manage Gitlab groups and users and projects via Terraform, which would be a better way to handle it than the custom Python logic we have now.

https://www.terraform.io/docs/providers/gitlab/index.html

v1.4: Installer fails when choosing us-west-2 region

Description

v1.4: Installer doesn't seem to work for certain regions

Steps to Reproduce

1. Follow install steps as suggested (here)[https://github.com/tmobile/jazz-installer/wiki#install
2. Provide AWS Key details from your personal AWS Account
3. Choose region as us-west-2

Expected behavior:
Installer finishes successfully

Actual behavior:
Installer fails with the error below:

Pulling the Jenkins docker image....Failed
Traceback (most recent call last):
 File "./run.py", line 33, in <module>
   main()
 File "./run.py", line 24, in main
   scenarios.execute(key, git_branch_name)
 File "/home/centos/jazz-installer/installscripts/wizard/jazz_scenarios.py", line 25, in execute
   OPTIONS[key][1].start(parameter_list)
 File "/home/centos/jazz-installer/installscripts/wizard/scenarios/stack_with_dockerized_gitlab_jenkins.py", line 37, in start
   get_and_add_docker_jenkins_config(JENKINS_DOCKER_PATH)
 File "/home/centos/jazz-installer/installscripts/wizard/jazz_jenkins.py", line 143, in get_and_add_docker_jenkins_config
   with open("docker_jenkins_vars") as f:
IOError: [Errno 2] No such file or directory: 'docker_jenkins_vars'

Reproduces how often:
Multiple attempts to install on a new image stopped at the same place.

Putting the Installer to sleep makes Jazz unstable

Description

When the EC2 Instance that was used to create a Jazz Instance is put to sleep and is then subsequently made active, the Jazz instance is unstable.

Steps to Reproduce

1. Follow the instructions for "Scenario 3" and create a Jazz Instance
2. Put the EC2 instance created in Step (1) to sleep.
3. Wait for a day and make the EC2 instance from Step (2) active.
4. Try creating new service by logging into your Jazz Instance

Expected behavior:
Service is created successfully

Actual behavior:
Service creation fails

Reproduces how often:
100%

New feature: Figure out how to stand up a Jazz stack from any machine

Split out from #110

Right now we require/assume you are running the installer from a CentOS 7 box in EC2.

We should support running the installer from any kind of OS, from your local laptop, etc.

One way to do this might be to create a "script container" Docker image to replace the current CentOS box. Vagrant is also an option.

Jazz installation issue

tried installing Jazz on us-west-2a , made several attempts but the installation did not succeed
Method1: Installing using existing Jenkins and existing bitbucket, , all the time the installation fails with the below error
"null_resource.configureExistingJenkinsServer (remote-exec): sudo: chef-client: command not found
Error applying plan:

1 error(s) occurred:

• null_resource.configureExistingJenkinsServer: 1 error(s) occurred:"
  Infact there is no way to start the installation from this step and the destroy is not cleaning up the aws resources that were created

Method2: tried installing with existing bitbucket and Jenkins container, the jenkin container creates properly but the installation is not proceeding further with the below error
"

• null_resource.configureExistingJenkinsServer: file: open ../sshkeys/dockerkeys/jenkinskey.pem: no such file or directory in:

${file("${lookup(var.jenkinsservermap, "jenkins_ssh_key")}")}^[[0m^[[0m
Fri Mar 16 09:32:26 UTC 2018"
"
as per my observation container creation process the pem file is getting copied to ../sshekys folder and the subsequent steps are checking for pem file availability within sshkeys/dockerkeys/
here is the ls command that shows the details on pem file location

[centos@ip-172-31-31-201 installscripts]$ cd sshkeys
[centos@ip-172-31-31-201 sshkeys]$ ls -l
total 4
drwxrwxr-x. 2 centos centos 6 Mar 16 09:30 dockerkeys
-r--------. 1 centos centos 1675 Mar 16 09:32 jenkinskey.pem
[centos@ip-172-31-31-201 sshkeys]$ cd dockerkeys
[centos@ip-172-31-31-201 dockerkeys]$ ls -l
total 0
[centos@ip-172-31-31-201 dockerkeys]$

is there a solution to complete the setup either going through method1 or method2

CLI wizard does not fail immediately if `terraform apply` fails

In create.sh we aren't checking to see if terraform apply returns a non-zero exit code (failure).

We should do this, and inform the user that something went wrong, rather than stating that the installation was successful.

Hardcoded Git repo URLs make it impossible to test forks

The T-Mobile Jazz git repo URL is hardcoded in multiple places in the installer, which means it is not possible to stand up a Jazz instance using code from a fork without manually searching and replacing all git URLs.

Also, the installer scripts implicitly assume that a branch with the same name as the jazz-installer branch you specify also exists in the jazz repo, which is not at all a guaranteed scenario since they are completely separate Git repos.

These things ought to be configurable, to ease testing and contributions from Github users.

Naming convention for platform jenkins jobs

Follow a consistent naming convention for all platform jenkins jobs. Currently it is being created as,

bitbucketteam_newService
build-pack-lambda
build-pack-website
build_pack_api
cleanup_cloudfront_distributions
create-service
delete-service
Platform_API_Services

Use all lower case and prefix with jazz

Installer fails when AWS account alias is not configured

Describe the bug
After ~10mins of terraform run the installer failed because the config script assumes there will be one alias set up.

null_resource.pushconfig: Creating...
null_resource.pushconfig: Provisioning with 'local-exec'...
null_resource.pushconfig (local-exec): Executing: ["/bin/sh" "-c" "python ./scripts/config.py jazzteststack_JazzConfig INSTANCE_PREFIX jazzteststack ./provisioners/cookbooks/jenkins/files/default/jazz-installer-vars.json us-west-2 123456789012 arn:aws:iam::123456789012:role/jazzteststack_platform_services arn:aws:iam::123456789012:role/jazzteststack_basic_execution xxxxxx xxxxxx xxxxxxx jazzteststack-apis-deployment-dev-XXXXXXXXXX jazzteststack-apis-deployment-prod-XXXXXXXXXXX jazzteststack-apis-deployment-stg-XXXXXXXXXX origin-access-identity/cloudfront/XXXXXXXXXXXXX sg-XXXXXXXXXX subnet-XXXXXXXXX,subnet-XXXXXXXXX"]
null_resource.pushconfig (local-exec): Traceback (most recent call last):
null_resource.pushconfig (local-exec):   File "./scripts/config.py", line 27, in <module>
null_resource.pushconfig (local-exec):     accountname = boto3.client('iam').list_account_aliases()['AccountAliases'][0]
null_resource.pushconfig (local-exec): IndexError: list index out of range

To Reproduce
Remove the account alias configuration from IAM and run the installer

Expected behavior
I would expect the installer to pick up a default account name if an alias is not configured. Such as the envprefix

Error logs and files
Log in the bug description

Jazz installer arguments

$ python Installer.py install --stackprefix JazzTestStack --adminemail [email protected] --region us-west-2 scenario3

Host (please complete the following information):

• OS: Ubuntu 18.04.3 LTS
• Terraform version Terraform v0.11.14

Improve installer

This is just a tracking issue.

• Make Terraform the entry point, and drop having to stand up a CentOS box as a manual requirement. You should be able to run Terraform directly from your laptop with the appropriate AWS creds and let it do the rest.
• Create a "script container" Docker image to replace the current CentOS box. Note that due to networking constraints it may be necessary to create a dedicated Docker host in AWS to host this (at least temporarily).
• Expand installer docs
• Make it easier to use setup environments that aren't CentOS (Target RHEL and Ubuntu at a minimum) Obsoleted by first bullet
• Improve BYOJ (Bring Your Own Jenkins) scenario. Note that realistically "improve" might mean "document how to do the steps manually on your own Jenkins server and drop the automation" rather than "automate it and account for every possible server configuration in the world"

Installation with Sonar module enabled created a corrupted jazz version

Describe the bug
Jazz Installation with Sonar module enabled – apparently installed a Jazz version which is not fully functional. On-boarding flow works as expected but the deployment flows are failing (not even being triggered)

To Reproduce
Install Jazz with Sonar module enabled.

Remove hardcoded runtimes

As new runtimes get released by AWS (and other cloud platforms) we cannot have hardcoded values in our codebase. We need to come up with a better way(config for eg.) to support the runtimes.

Evaluate/lock down IAM policies and roles for Jazz core services

Description

Some of the AWS IAM roles and policies Jazz creates for its core services are overly broad/permissive, e.g. they use wildcards in the ARN rather than being restricted to the specific resources they actually need.

We can/should self-audit all of the IAM roles we create/expect and make sure they specify the minimum required access.

Master : Unable to login to Jazz

Issue 1:
Unable to login to Jazz after Successful Jazz Installation.

Steps:
Using master branch installed Jazz.

Log Analysis:
From the logs Installation, no failure reported.
All the 56 AWS resources are added.
Jenkins: 'build-deploy-platform-service' all the 13 jobs failed.

Reason:
In jenkins 'build-deploy-platform-service' all the 13 jobs failed.
Logs showed aws command not found

_[build-deploy-platform-service] Running shell script

• aws configure set profile.cloud-api.region us-east-1
  /var/lib/jenkins/workspace/build-deploy-platform-service@tmp/durable-e304fd96/script.sh: line 2: aws: command not found_

Manual Fix: When Manually Installed 'aws cli' in jenkins box and did a rebuild . All the platform services build was successful and was able to login to Jazz.

Issue 2:
API is not getting created.

  Step:
       Create an API from Jazz and in UI it shows no link and API creation fail message.

 Reason:
         Build API failed with log

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.kohsuke.groovy.sandbox.impl.Checker checkedCast java.lang.Class java.lang.Object boolean boolean boolean
at org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhitelist.rejectStaticMethod(StaticWhitelist.java:192)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onStaticCall(SandboxInterceptor.java:142)
at org.kohsuke.groovy.sandbox.impl.Checker$2.call(Checker.java:186)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedStaticCall(Checker.java:190)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:97)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.validateDeploymentConfigurations(WorkflowScript:625)
at WorkflowScript.run(WorkflowScript:202)
at cps.transform(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at

Issue 3:

Dead code : In modifyCodebase.h

�[0m�[0mnull_resource.configureExistingJenkinsServer (local-exec): Executing: /bin/sh -c "./scripts/modifyCodebase.sh sg- subnet-
arn:aws:iam::475452188481:role/infinity_lambda2_basic_execution_1 us-east-1 xxxxx"
�[0m�[0mnull_resource.configureExistingJenkinsServer (local-exec): sed: can't read /home//cookbooks/jenkins/files/node/jazz-installer-vars.json: No such file or directory
�[0m�[0mnull_resource.configureExistingJenkinsServer (local-exec): sed: can't read /home//cookbooks/jenkins/files/node/jazz-installer-vars.json: No such file or directory
�[0m�[0mnull_resource.configureExistingJenkinsServer (local-exec): sed: can't read /home//cookbooks/jenkins/files/node/jazz-installer-vars.json: No such file or directory

SSH-ing into Jenkins container fails

When I pick scenario 3 (Dockerized jenkins and Gitlab), install fails repeatedly at this point:

Granting permissions to other users to pip install....Completed
['18.222.73.235:8081', 'admin', '8dabee45bb234a96bcd2a7c594acf15f', '18.222.73.235', 'root', '2200', 'sg-852143ee', 'subnet-aa5d53d2']
Unable to SSH into the Jenkins instance! Is the jenkinskey key or the username valid? Jenkins IP: 18.222.73.235 Username: root Keypath: /home/centos/jazz-installer/installscripts/dockerfiles/jenkins//jenkinskey.pem Port: 2200

Originally this just logged the failure and did not stop script execution, I added the sys.exit since there's no point in continuing if this fails. I also added some logging to see why the SSH call (copied below)using paramiko fails

def check_jenkins_sshuser_valid(parameter_list, port_number, keypath):
    """
        Check if the ssh login name is a user
    """
    jenkins_server_public_ip = parameter_list[3]
    jenkins_server_ssh_login = parameter_list[4]
    keyfile = keypath + "/jenkinskey.pem"
    try:
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(jenkins_server_public_ip, username=jenkins_server_ssh_login, key_filename=keyfile, port=port_number)
    except:
        sys.exit("Unable to SSH into the Jenkins instance! Is the jenkinskey key or the username valid? Jenkins IP: %s Username: %s Keypath: %s Port: %s" %(jenkins_server_public_ip, jenkins_server_ssh_login, keyfile, port_number,))

The ssh key is in the correct place, but I can verify that the paramikio ssh failed. Looking in to this.

When dealing with containers we shouldn't even need to SSH in in most cases, it's a bit roundabout, we can just docker exec and skip the part where we have to copy ssh keys around.

Hardcoded port number for Jenkins

We have hardcoded port number for Jenkins: https://github.com/tmobile/jazz-installer/blob/master/installscripts/wizard/run.py#L36

Please remove this hard coded value everywhere and by default expect it to be mapped to port 80. If it isn't let the installer have the user input this value.

Remove Jenkins plugin binaries from repo

Right now, Scenario 1 installs Jenkins plugins by extracting the (rather large) plugins archive under installscripts/jenkinsplugins/, and those files are checked into git.

Since the other scenarios that use Docker images rely on a plugins.txt and download the Jenkins plugins on-demand, we should do that for scenario 1 as well and delete these checked in plugin archives.