fidelity / awsrun

CLI and library to execute commands over one or more AWS or Azure accounts concurrently.

awsrun & azurerun

CLI tool and library to concurrently execute user-defined commands across AWS accounts or Azure subscriptions.

Note As of release 3.0.0, installation has changed with the introduction of optional dependencies. Previously, pip install awsrun would install all dependencies needed for awsrun, azurerun, and all of the bundled commands. You must now specify the optional dependencies when installing: pip install "awsrun[aws,azure]". See Installation for more details.

Overview

Awsrun/azurerun is both a CLI tool and Python package that can be used to execute commands concurrently over one or more AWS accounts or Azure subscriptions. Commands are user-defined Python modules that implement a simple interface to abstract away the complications of obtaining credentials for Boto3 and Azure SDK sessions—especially when using SAML authentication and/or cross-account access in AWS. The key features of awsrun/azurerun include the following:

Concurrent Account Processing: Run a command concurrently across subset or all of your accounts/subscriptions. A worker pool manages the execution to ensure accounts/subscriptions are processed quickly, so you don't have to wait for them to be processed one at a time. Process hundreds of accounts in a few minutes.

SAML and Cross-Account Access: Tired of dealing with AWS temporary STS credentials with SAML and cross-account authentication? Use any of the included credential plug-ins based on your needs, or build your own plug-in to provide credentials for your command authors. Don't use SAML? Define profiles in your AWS credentials file instead. With Azure, the default credentials are obtained via the Azure CLI or interactively via the browser.

Built-in Command for AWS CLI & Azure CLI: Ever wish you could run the standard AWS CLI tool or Azure CLI tool across multiple accounts/subscriptions? Now you can using the included aws or az commands. These commands are simple wrappers for AWS's and Azure's CLI tools respectively, but with the added benefits of using metadata to select multiple accounts as well as simplified credential handling.

User-Defined Commands: Build your own commands using the powerful Boto3 SDK or Azure SDK without the hassle of obtaining sessions and credentials. Thanks to a simple interface, commands are easy to build and can be integrated directly into the CLI with custom arguments and help messages.

Metadata Enriched Accounts: Accounts/subscriptions can be enriched with metadata from external sources, such as a corporate CMBD, via the account loader plug-in mechanism. This enables you to use metadata to select accounts to process rather than explicitly listing each account/subscription on the command line. In addition, command authors have access to this metadata, so it can used while processing an account if needed.

Screenshots

These examples demonstrate the wide-range of possibilities when building your own awsrun & azurerun commands. While these commands are included in awsrun, they use the same command library that you would if building your own. Nothing is special about these commands other than they are included in the base install of awsrun. You could have built these yourself.

awsrun aws command

This screencast illustrates the power of awsrun once it has been configured to your environment using appropriate credential and account loader plug-ins. In this demo, we use awsrun to gather VPC information. While we could do the same using only AWS's native CLI, we would be limited to processing one account at a time. Instead, we'll use awsrun and the built-in aws command to execute an AWS CLI command across multiple accounts concurrently. We'll also make use of the awsrun's metadata explorer to select accounts for command execution.

Note: The output has been obfuscated with random account numbers and identifiers.

awsrun last command

This screenshot demonstrates the use of the last command to interactively explore CloudTrail events. Don't have a simple means to view CloudTrail logs? Tired of using the AWS Console? The last command provides a simple way of viewing events in one or more accounts.

awsrun dx_status command

If you manage AWS Direct Connects to provide connectivity to your on-premise corporate networks, you might find the dx_status command helpful. It provides an overview of Direct Connects contained within an account. This includes pulling CloudWatch metrics and generating terminal-based graphs using sparklines and ASCII-based charts.

Installation

Python 3.7 or higher is required.

To install only awsrun:

$ pip install "awsrun[aws]"

To install only azurerun:

$ pip install "awsrun[azure]"

To install both awsrun and azurerun:

$ pip install "awsrun[aws,azure]"

In all cases, the console scripts awsrun and azurerun are installed, but only the dependencies for the specified CSPs are installed.

Some of the bundled commands have additional dependencies. You will be prompted to install those if you use one of them. Alternatively, you can install all of those ahead of time:

$ pip install "awsrun[cmds]"

Finally, to install from source with the development dependencies:

$ git clone https://github.com/fidelity/awsrun.git
$ cd awsrun
$ pip install -e ".[aws,azure,cmds,dev]"

AWS Quick Start

Out of the box, the utility of awsrun is limited as most of its power comes from the configuration of an account loader plug-in (to simplify the selection of multiple accounts) and a credential loader plug-in (to simplify access to those accounts). With that said, however, you can still use it, as it will default to loading credentials from your $HOME/.aws/credentials file. While not convenient when managing hundreds of accounts, it will suffice to get you started.

Assume you wanted to list the EC2 instances in two accounts: 100200300400 and 200300400100. We can use the built-in aws command to execute any AWS CLI command across one or more accounts concurrently. Be sure you have followed the installation instructions in the previous section. Then, create two profiles, [100200300400] and [200300400100], in your local AWS credentials file $HOME/.aws/credentials. If awsrun cannot find a profile for named for the specific account, it will fallback to the [default] profile.

Note: The AWS credentials file is not part of awsrun, but it is used as the default mechanism to obtain credentials if more advanced options have not been configured. For help on the configuration of the AWS credential file, refer to AWS CLI Named Profiles.

With the profiles defined, you can now run the following to list the EC2 instances in both accounts across multiple regions concurrently:

$ awsrun --account 100200300400 --account 200300400100 aws ec2 describe-instances --region us-east-1 --region us-west-2
2 accounts selected:

100200300400, 200300400100

Proceed (y/n)? y
...

If, instead, you want to list lambda functions in those accounts:

$ awsrun --account 100200300400 --account 200300400100 aws lambda list-functions --region us-east-1 --region us-west-2
2 accounts selected:

100200300400, 200300400100

Proceed (y/n)? y
...

There are several other built-in commands that have been included in awsrun. The more interesting ones include the last command to inspect CloudTrail events as well as the dx_status command to check the status of Direct Connect circuits. Remember, you are encouraged to build your own custom commands. These have been provided to show you how to do so.

Azure Quick Start

Let's list all the Azure VMs in two subscriptions: 00000000-0000-0000-0000-000000000000 and 11111111-1111-1111-1111-111111111111. We can use the built-in az command to execute an Azure CLI command across one or more subscriptions concurrently. Assuming you have already followed the installation instructions:

$ az login   # Use the Azure CLI to obtain credentials
$ azurerun --account 00000000-0000-0000-0000-000000000000 --account 11111111-1111-1111-1111-111111111111 az vm list --output table
2 accounts selected:

00000000-0000-0000-0000-000000000000, 11111111-1111-1111-1111-111111111111

Proceed (y/n)? y
...
Name   ResourceGroup      Location    Zones
-----  -----------------  ----------  -----
vm1    rg1                centralus   1
vm2    rg1                eastus2     1

Name   ResourceGroup      Location    Zones
-----  -----------------  ----------  -----
vm1    rg2                centralus   1
vm2    rg2                eastus1     1
vm3    rg2                eastus2     1

Out of the box, the utility of azurerun is limited as most of its power comes from the configuration of an account loader plug-in (to simplify selection of multiple accounts). Using the included AzureCLI plug-in, azurerun will use the Azure CLI to obtain a list of subscriptions and metadata associated with those. Furthermore, assuming you use a naming convention for your subscriptions, we can parse the name to pull out additional metadata. For example, if your subscriptions are named "azure-retail-prod" and "azure-retail-nonprod", then we can use this regexp to add the "bu" and "env" metadata attributes:

azure-(?P<bu>[^-]+)-(?P<env>.+)

Create the following azurerun configuration file called .azurerun.yaml in your home directory:

Accounts:
  plugin: awsrun.plugins.accts.azure.AzureCLI
  options:
    name_regexp: 'azure-(?P<bu>[^-]+)-(?P<env>.+)'

Now, you can use the metadata filters to specify subscriptions instead of enumerating them on the azurerun command line. Here are a few examples:

# Let's see what metadata we can use
$ azurerun --metadata
bu
cloudName
env
homeTenantId
id
isDefault
name
state
tenantId

# Run the command over all subscriptions.
$ azurerun az vm list --output table
...

# Run the command over all prod subscriptions.
$ azurerun --include env=prod az vm list --output table
...

# Run the command over all enabled, nonprod subscriptions
$ azurerun --include state=Enabled --include Env=nonprod az vm list --output table
...

There are several other built-in commands that have been included in azurerun. Remember, you are encouraged to build your own custom commands. These have been provided to show you how to do so.

Documentation

awsrun includes extensive documentation, which includes the following:

• The CLI User Guide includes basic usage, configuration of awsrun, and how to use the account loader and credential loader plug-ins to enhance the user experience on the CLI.

• The Library User Guide includes pointers to the key documentation required to use awsrun programmatically instead of via the CLI. All of the awsrun modules are also extensively documented.

• The How-to Write Your Own Commands guide provides everything you need to write your own custom awsrun commands. This is where you'll spend a lot of time once you become familiar with the capabilities of awsrun.

• The How-to Write Your Own Plug-ins section includes pointers to the documentation required to build your own account loader plug-in and credential plug-in if the included ones are not suitable to your environment.

Change Log

v3.1.0

• Add URLAccountLoader plug-in which supports HTTP authentication (basic, digest, oauth2). This plug-in should be preferred over the existing JSONAccountLoader, YAMLAccountLoader, and CSVAccountLoader plug-ins, which will eventually be deprecated.

• Add HTTP POST support to the SAML credential provider plug-in. Thanks to @RobertShan2000 for the contribution.

v3.0.0

• BREAKING CHANGE: Installation via pip install awsurn no longer installs all of the dependencies for awsrun, azurerun, and the bundled commands. Instead, users must now specify which optional dependendencies to install. See Installation section above for details. Note: while the major version was bumped, there are no API changes to awsrun. This was bumped from "2" to "3" solely to bring awareness to the different installation instructions.

• The terminal-based UI for the last command, a CloudTrail event viewer, was rewritten using the amazing Textual TUI framework. With this change, several new features were added to the command: dark/light modes, mouse support, filtering events, exporting events, OS clipboard integration, highlighting of events with errors, ability to toggle layout, as well as a brand new look and feel. See screenshot above.

• Modernized the python packaging to use pyproject.toml instead of the setup.py style of packgaing.

• Thanks to Gábor Lipták for the GitHub actions contributions. This included adding the new check for PR titles to ensure they adhere to the Conventional Commits standard.

• Added a CONTRIBUTING document thanks to Brian Warner, a member of our open-source office at Fidelity.

v2.5.2

• Resolve all pylint warnings.

• Update dx_status for asciichart API changes and bump version.

• Allow reuse of argparse option flags for command authors that use the same flag across multiple argparse subcommands. Previously, reuse of option flags was prohibited across different instances of argparse.ArgumentParser.

v2.5.1

• Fix tag and redeploy pypi artifacts.

v2.5.0

• last command now appends each CloudTrail event line in non-interactive mode with "ERROR: " and the errorCode from the event if present. This allows users to quickly identify errors (grep ERROR). The CloudTrail API does not provide a means to filter on errors, so it can only be done after retrieving events.

• last command TUI now allows users to filter CloudTrail events interactively via a popup (mapped to 'f' key). Users can specify one or more terms to match events. Terms are logically OR'd together. A term may be prefixed with - to exclude events matching it. For example, "errorCode -s3" will show only events that had errors excluding S3 events.

v2.4.2

• Update kubectl wrapper to use latest v1beta1 Kubernetes API instead of v1alpha1 to fix compatibility issues with the latest version of Kubernetes kubectl and latest AWS CLI tool.

v2.4.1

• Require Python 3.7 or greater in setup.py.

v2.4.0

• Remove Python 3.6 classifier from setup.py.

• Update authentication method in kubectl wrapper from aws-iam-authenticator to use the AWS CLI command aws eks get-token. Users will no longer need to install the separate aws-iam-authenticator helper moving forward.

• Use KUBECONFIG environment variable instead of --kubeconfig command line flag when the kubectl wrapper command invokes the real kubectl command. This change should be transparent to users of the kubectl command.

v2.3.1

• Add new sample awsrun command, dx_maint, that queries the AWS Health API to display recent and upcoming maintenance events (technically any open events) on Direct Connects.

• Add new sample azurerun command, list_udrs, that displays all User Defined Routes (UDRs) in an Azure VNET.

v2.3.0

• Add support for Azure. By default, installation now installs both awsrun as well as azurerun. See the quick start for Azure above.

v2.2.2

• Add a decorator awsrun.runner.max_thread_limit that can be used by command authors to limit the number of concurrent executions. There are some scenarios where a command author may never want their command run concurrently across multiple accounts. By default, awsrun uses a thread pool of ten workers, and users can override, so this gives command authors ability to limit if needed.

v2.2.1

• Reorganized the documentation. The CLI user guide and reference are now part of the awsrun.cli module documentation. The user guide on writing commands has been moved to the awsrun.commands module. Lots of other minor edits were made as part of this reorganization. Hopefully, things are easier to find with the new layout.

v2.2.0

• Add three new built-in commands: console, dx_status, and last. Console generates sign-in URLs for the AWS Console using credentials from awsrun. Dx_status shows the status of Direct Connect circuits (terminal graphs too!). Last provides an easier way to review CloudTrail events in both an interactive and non-interactive manner.

• Add the cloudwatch module to simplify the retrieval of CloudWatch metrics using bulk retrieval for efficiency. This module is used be the new dx_status command.

• Update the included built-in commands that iterate over VPCs to filter out VPCs that have been shared with an account, as opposed to being owned by the account, as generally that is the behavior one is expecting.

• Remove AWS CLI as a python dependency in setup.py. AWS has released v2 of the AWS CLI, so we should not presume to install v1 via pip installation. In addition, AWS has stated that the only supported installation of the AWS CLI v2 is via their own bundled package installers. What does this mean for awsrun users? Install the AWS CLI on your own if you plan on using the built-in aws command.

• Fonts used in documentation have been updated to use Charter and Fira Mono.

v2.1.0

• Add a YAML account loader plug-in to complement the CSV and JSON account loader plug-ins.

• Minor clarification in user guide about the interaction between --accounts and metadata filters (--include/--exclude).

v2.0.0

• Initial open source release of awsrun from Fidelity's CloudX Network team. This version abstracts the Fidelity specific integrations into plug-ins, so others can take advantage of this tool, which has proven to be valuable for our teams.

awsrun's Issues

azurerun too?

While it may be handy to have azurerun mixed in with awsrun, not everyone that uses aws also uses azure. So, building in a private environment, I have to pull in all the azure dependencies down as well, when I have no need for them. Perhaps this should be broken up into multiple packages with the proper dependencies for awsrun vs. azurerun? Just a thought.

No tag for v2.3.1?

Not really a big deal I suppose, since it's not published to pypi, but found this project recently and when I installed it today I noticed that I got v2.3.1, but that the repo is only tagged with v2.3.0. Figured someone missed the tag in the release process?

Update ExecCredential to work for v1beta1

awsrun kubectl get pods -A --region us-east-1 | grep -v Running leads to the below error:
Unable to connect to the server: getting credentials: decoding stdout: no kind "ExecCredential" is registered for version "client.authentication.k8s.io/v1alpha1" in scheme "pkg/client/auth/exec/exec.go:62"

Credentials plugin awsrun.plugins.creds.aws.SAML doesn't support IDP's post forms authentication

In some case, the SAML auth IDP only supports post forms authentication. And our SAML credentials plugin only support get method so far. we had better make some code enhancement to support this use case. Please refer to the code https://github.com/fidelity/awsrun/blob/master/src/awsrun/session/aws.py#L417

Using .aws/credentials provider doesn't seem to work out of the box

I've come across issues attempting to use awsrun with the default credentials provider. I originally had trouble on a Mac and ran into the same issues in multiple Linux distros, so it doesn't seem to be environment-specific. Please see steps to reproduce below from scratch and debug logs/attempts to fix.

Steps to Reproduce

1. Spin up a fresh Ubuntu 20.04 instance (I tried on DigitalOcean separately but using an AWS instance here)
2. sudo apt update; sudo apt install -y python3-pip unzip; curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"; unzip awscliv2.zip; sudo aws/install
3. Place AWS access key ID and secret access key in ~/.aws/credentials under [default] profile (manually or with aws configure command)
4. Confirm aws s3 ls (or any command these credentials have access to) work
5. Install awsrun git clone https://github.com/fidelity/awsrun.git; cd awsrun; sudo pip3 install .
6. Run awsrun --account default aws s3 ls --region us-east-1 and receive the error default/us-east-1: error: expected str, bytes or os.PathLike object, not NoneType (see below for output with --log-level DEBUG)

2021-01-16 02:30:36,316 awsrun.plugmgr INFO [MainThread] loading plug-in: awsrun.plugins.accts.Identity
2021-01-16 02:30:36,317 awsrun.plugmgr INFO [MainThread] parsed args=Namespace(account_file=None, accounts=['default'], cmd_path=['awsrun.commands.aws'], exclude={}, force=False, include={}, log_level='DEBUG', metadata=None, threads=10) remaining args=['aws', 's3', 'ls', '--region', 'us-east-1']
2021-01-16 02:30:36,317 awsrun.plugmgr INFO [MainThread] loading plug-in: awsrun.plugins.creds.aws.Profile
2021-01-16 02:30:36,321 awsrun.plugmgr INFO [MainThread] parsed args=Namespace(account_file=None, accounts=['default'], cmd_path=['awsrun.commands.aws'], exclude={}, force=False, include={}, log_level='DEBUG', metadata=None, threads=10) remaining args=['aws', 's3', 'ls', '--region', 'us-east-1']
2021-01-16 02:30:36,322 awsrun.cmdmgr INFO [MainThread] loading command at 'awsrun.commands.aws.aws'
... [snip irrelevant boto logs] ...
2021-01-16 02:30:36,334 botocore.session DEBUG [ThreadPoolExecutor-0_0] Setting config variable for profile to 'default'
2021-01-16 02:30:36,338 awsrun.commands.aws.aws INFO [ThreadPoolExecutor-0_0] default-us-east-1: AWS CLI command: ['/usr/local/bin/aws', '--region', 'us-east-1', 's3', 'ls']
2021-01-16 02:30:36,338 botocore.utils DEBUG [ThreadPoolExecutor-0_0] IMDS ENDPOINT: http://169.254.169.254/
2021-01-16 02:30:36,340 botocore.credentials DEBUG [ThreadPoolExecutor-0_0] Skipping environment variable credential check because profile name was explicitly set.
2021-01-16 02:30:36,340 botocore.credentials DEBUG [ThreadPoolExecutor-0_0] Looking for credentials via: assume-role
2021-01-16 02:30:36,340 botocore.credentials DEBUG [ThreadPoolExecutor-0_0] Looking for credentials via: assume-role-with-web-identity
2021-01-16 02:30:36,340 botocore.credentials DEBUG [ThreadPoolExecutor-0_0] Looking for credentials via: sso
2021-01-16 02:30:36,340 botocore.credentials DEBUG [ThreadPoolExecutor-0_0] Looking for credentials via: shared-credentials-file
2021-01-16 02:30:36,341 botocore.credentials INFO [ThreadPoolExecutor-0_0] Found credentials in shared credentials file: ~/.aws/credentials
2021-01-16 02:30:36,341 awsrun.commands.aws.aws INFO [MainThread] default/us-east-1: error: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/awsrun/commands/aws/aws.py", line 538, in regional_collect_results
    result = get_result()
  File "/usr/local/lib/python3.8/dist-packages/awsrun/runner.py", line 946, in fn
    raise exception
  File "/usr/local/lib/python3.8/dist-packages/awsrun/runner.py", line 936, in _wrap_result
    result = fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/awsrun/commands/aws/aws.py", line 500, in regional_execute
    result = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1624, in _execute_child
    env_list.append(k + b'=' + os.fsencode(v))
  File "/usr/lib/python3.8/os.py", line 806, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType
default/us-east-1: error: expected str, bytes or os.PathLike object, not NoneType

Processed 1 account in 0:00:00.018664

Debugging

By making the below change in src/awsrun/commands/aws/aws.py, the command above will work (and the expected s3 output is printed) - but only with the default profile. It's a hack that seems to only work in this specific case because the aws cli tool will find the default creds on its own without the help of awsrun (I think). Note that the debug output log looks exactly the same for the successful run, up to the point of the error (last line of Found credentials in shared credentials file: ~/.aws/credentials).

--- a/src/awsrun/commands/aws/aws.py
+++ b/src/awsrun/commands/aws/aws.py
@@ -484,8 +484,8 @@ class CLICommand(RegionalCommand):
             "AWS_SESSION_TOKEN": creds.token,
         }

-        env = ChainMap(new_vars, os.environ)
-
+        #env = ChainMap(new_vars, os.environ)
+        env = os.environ

I could speculate further from here what the exact issue is but I'm not familiar enough with the codebase to say. It seems as though there's some sort of issue setting the environment variables with the default credentials provider. Please let me know if there's anything else I can do to help with this issue.