wagoodman / bridgy Goto Github PK

View Code? Open in Web Editor NEW

407.0 14.0 53.0 781 KB

cloud inventory + ssh + tmux + sshfs

License: MIT License

Python 99.20% Makefile 0.80%

aws ec2 ssh tmux sshfs inventory

bridgy's Issues

Support Bastion Host Per Source

It would ideal to have the ability to assign a bastion host to a source or sources to a bastion.

In my case, I have multiple AWS profiles, i.e. Dev, QA, Prod, Ops, but different bastion hosts for each.

Provide option to search instance within ONE source

I have multiple servers within multiple AWS account therefore I will have multiple AWS source for servers.

It will be much faster and better if there is an option to narrow down instance searching within ONE source.

Support a Kubernetes backend

It would ideal to allow for tracking of assets in Kubernetes. This would further extend the exec command to get a shell into one or more containers (or all containers in a pod with tmux).

This is a good starting point for how the mechanisms of streaming to/from the container (seems more kosher than wrapping kubectl): https://github.com/kubernetes-client/python/blob/master/examples/exec.py

Attach to existing remote sessions

It would be nice to have something that acted like ssh ... -t tmux attach-session or ssh ... -t -- bash -c "tmux attach || tmux new" to attach or create a session. Currently bridgy does:

(TMUX on my local machine)
 |
 + session 1: someotherhost
    \_ window1: ssh user@someotherhost
    \_ window2: ssh user@someotherhost
    \_ ...

but attaching or creating would allow:

(TMUX on my local machine)
 |
 + session 1 linked to an existing session on someotherhost
    \_ window1: shell on someotherhost
    \_ window2: shell on someotherhost
    \_ ...

Note: this needs to be resilient to tmux not being installed on the remote host.

Updating AWS inventory doesn't show changes to instances

When you change the Name tag (or and instance gets a new IP, etc) of an AWS instance and run bridgy update, subsequent commands do not see the updated information. Removing the cache files in ~/.bridgy/inventory/aws/profile/* and running an update causes the correct information to be used.

bridgy init got erro: sample yaml file not found

'''
Traceback (most recent call last):
File "/home/hhhhh/.local/bin/bridgy", line 11, in
sys.exit(main())
File "/home/hhhhh/.local/lib/python3.6/site-packages/bridgy/main.py", line 340, in main
init_handler(args, config)
File "/home/hhhhh/.local/lib/python3.6/site-packages/bridgy/utils.py", line 32, in wrapper
func(*args,**kwargs)
File "/home/hhhhh/.local/lib/python3.6/site-packages/bridgy/main.py", line 312, in init_handler
if config.create():
File "/home/hhhhh/.local/lib/python3.6/site-packages/bridgy/config/base.py", line 89, in create
fh.write(self.config_template_contents)
File "/home/hhhhh/.local/lib/python3.6/site-packages/bridgy/config/base.py", line 64, in config_template_contents
return pkgutil.get_data('bridgy', 'config/samples/' + self.config_template_path)
File "/usr/lib/python3.6/pkgutil.py", line 634, in get_data
return loader.get_data(resource_name)
File "", line 832, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/hhhhh/.local/lib/python3.6/site-packages/bridgy/config/samples/sample_config_2.yml'
'''

Add SSH tunneling

This would at least enable remote debugging.

concept: ssh -L 8080:web-server:80 -L 8443:web-server:443 bastion-host -N
then: curl https://localhost:8443/secure.txt
source: https://solitum.net/an-illustrated-guide-to-ssh-tunnels/

Add ECS integration

Allow the ability to log into the ec2 instance matched by a running task name and actively attach to the docker instance (via exec)

Add option to sync tmux panes

-s to equivalently to :setw synchronize-panes on

Support for clusterssh

Feature Request
Ability to use/enable clusterssh when SSH'ing to multiple servers so you can run the same command on all servers at once.

Remove placebo package

This will enable easier testing for the class

Extend CSV inventory to support per-instance configurations

As hinted to in #9 (comment) , it would be great to be able to map arbitrary CSV columns to instance parameters that override the global or inventory ssh configuration. For example, a CSV inventory with extra columns:

name, address, user, ssh-key
server-1, 123.87.123.223, admin, id_rsa.ss1
awesome-server-2, 22.54.21.123, frank, id_dsa.key42

Bridgy could take advantage of the extra information like so:

inventory:
  source:
    - type: csv
      name: on-site servers
      file: somefile.csv
      fields: name, address, user, ssh_key
      ssh:
        user: fields['user']
        options: -i fields['ssh_key']

The yaml format needs more thought, but that's the general idea (I'm open to suggestions).

Enhancements

Git integration?
Branch deploy or sync.
Mount local sync? Remote mount changes are synced to a local git repo.
Git bare set working tree to remote mount.
Rsync?

show turned off instances with a different color

Currently, bridgy display all available instances, even those that are turned off. It would be neat to display those with a distinctive color to show that connection to them just wont work.

Listing them, even though they are not reachable, is fine imho, because it shows that the listing actually works properly.

Support a ECS backend

It would be ideal to be able to exec into an ECS task (container) and keep and inventory that is searchable by ECS service name.

Invert tmux logic

--no-tmux should really be --tmux and the default behavior should be to not use tmux unless configured to do so with:

...
ssh:
    tmux: true

Support multiple sources

Would be very useful to combine multiple sources of inventory. For example we have multiple AWS accounts and an on-premise environment.

Copy tools from a staging dir to remote server

Add a configuration item that allows one to specify a one-way rsync to the remote guest to sync supports scripts to.

Cannot list-inventory if any are shutdown

It looks like bridgy relies on PrivateIpAddress being set, which is not always the case for every instance:

https://github.com/wagoodman/bridgy/blob/master/bridgy/inventory/aws.py#L60

Some of my instances return a state of:

"State": {
"Code": 48,
"Name": "terminated"
}

And instances of this type do not have a field of PrivateIpAddress, causing the following error when trying to list the inventory:

Traceback (most recent call last):
  File "/home/torme/.local/bin/bridgy", line 11, in <module>
    sys.exit(main())
  File "/home/torme/.local/lib/python2.7/site-packages/bridgy/__main__.py", line 356, in main
    handler(args, config)
  File "/home/torme/.local/lib/python2.7/site-packages/bridgy/utils.py", line 31, in wrapper
    func(*args,**kwargs)
  File "/home/torme/.local/lib/python2.7/site-packages/bridgy/__main__.py", line 258, in list_inventory_handler
    for ip, name, aliases, source in inventory.instances(config):
  File "/home/torme/.local/lib/python2.7/site-packages/bridgy/utils.py", line 45, in __call__
    return self[args]
  File "/home/torme/.local/lib/python2.7/site-packages/bridgy/utils.py", line 47, in __missing__
    ret = self[key] = self.f(*key)
  File "/home/torme/.local/lib/python2.7/site-packages/bridgy/inventory/__init__.py", line 105, in instances
    all_instances = inventory(config).instances()
  File "/home/torme/.local/lib/python2.7/site-packages/bridgy/inventory/source.py", line 88, in instances
    instances.extend(inventory.instances())
  File "/home/torme/.local/lib/python2.7/site-packages/bridgy/inventory/aws.py", line 60, in instances
    elif instance['PrivateIpAddress']:
KeyError: 'PrivateIpAddress'

mac -bash: bridgy: command not found

bridgy init

-bash: bridgy: command not found

Show more information about matched instances to help differentiate.

Maybe showing differentiating tag information on each option could help. Needs some thought still.

Add filter to only allow some systems to be shown

Sometimes there are a set of systems in an inventory that you don't want to be able to log into (e.g. production). It would be nice to have a pattern-like configuration item that would allow for system inventory inclusion and exclusion rules.

Tmux error is not well handled which throws TypeError: a bytes-like object is required, not 'str'

Tmux error is not well handled which cause bridgy tmux session not open.

env:
macOS 10.13 + python3.6.5

bridgy config:

 tmux:
   layout:
       logger:
           - cmd: split-window -h
           - cmd: split-window -h
           - cmd: split-window -h
             run: tail -f /var/log/syslog
           - cmd: set-window-option synchronize-panes on

bridgy command:

bridgy ssh -tl logger core -I mysource

error:

Traceback (most recent call last):
  File "/Users/qinfeng/.local/share/virtualenvs/bridgy-5NqTP_ZY/bin/bridgy", line 11, in <module>
    load_entry_point('bridgy', 'console_scripts', 'bridgy')()
  File "/Users/qinfeng/Projects/github/bridgy/bridgy/__main__.py", line 425, in main
    handler(args, config)
  File "/Users/qinfeng/Projects/github/bridgy/bridgy/utils.py", line 34, in wrapper
    func(*args,**kwargs)
  File "/Users/qinfeng/Projects/github/bridgy/bridgy/__main__.py", line 218, in ssh_handler
    tmux.run(config, commands, args['-w'], layout, args['-d'], args['-s'])
  File "/Users/qinfeng/Projects/github/bridgy/bridgy/tmux.py", line 20, in run
    with TmuxSession(commands=commands, in_windows=in_windows, layout_cmds=layout_cmds, dry_run=dry_run, sync=sync) as tmux:
  File "/Users/qinfeng/Projects/github/bridgy/bridgy/tmux.py", line 58, in __enter__
    self.tmux(*cmd)
  File "/Users/qinfeng/Projects/github/bridgy/bridgy/tmux.py", line 118, in tmux
    repr(std_err.strip("\n").replace("\n",', ')),
TypeError: a bytes-like object is required, not 'str'

It looks like std_err returned by python subprocess pipe is in bytes type which needs to be decoded to str obj

Get only private IPs from AWS

Is it possible to get only Private IPs for AWS Instances?

➜ bridgy ssh aws-env

‣ host01 (ec2-5-14-36-63.compute-1.amazonaws.com)
host02 (ec2-3-17-38-216.compute-1.amazonaws.com)
host03 (10.40.12.34)
host04 (10.32.11.112)

SSH Could be restricted on public IPs

Support a Google Cloud backend

This issue should get broken down into smaller issues as it gets picked up. However, it would be great to get basic inventory support for GCP Compute Engine, followed by GKE when the #31 has been implemented.

Import inventory from ~/.ssh/config

After isntallation I do $ bridgy ssh my-server-alias
And getting error No inventory source specified (~/.bridgy/config.yml):

I already have all my servers aliased in ~/.ssh/config:

Host  my-server-alias
    HostName my-server.my-doamin.com     #ip could be here
    IdentityFile ~/.ssh/my_key_rsa
    User ubuntu
    ForwardAgent yes

That's basiuc config taht let's me use ssh my-server-alias to login correct IP with correct RSA key.

Would be nice to be able to reuse ssh settings for brigy features

Override the ssh user via cli

Though there is a configured user, it would be nice to override the user on cli via the <user>@<host> approach that all ssh-like tools support

botocore.exceptions.NoRegionError: You must specify a region.

I've just installed bridgy on a new machine (having it working on another), and with the exact same configuration file, I've hit this error upon updating: botocore.exceptions.NoRegionError: You must specify a region.. However, I do specify the region in the config file.

The full output of the bridgy update -v command:

Updating inventory...
Loading variable profile from defaults.
Loading variable config_file from defaults.
Loading variable credentials_file from defaults.
Loading variable data_path from defaults.
Loading variable profile from defaults.
Loading variable region from defaults.
attaching to session: Session(region_name=None)
datapath: /Users/pascaldevink/.bridgy/inventory/aws/production
Loading variable profile from defaults.
Loading variable region from defaults.
Loading variable profile from defaults.
Loading variable ca_bundle from defaults.
Loading variable profile from defaults.
Loading variable api_versions from defaults.
Loading variable profile from defaults.
Loading variable credentials_file from defaults.
Loading variable config_file from defaults.
Loading variable profile from defaults.
Loading variable metadata_service_timeout from defaults.
Loading variable profile from defaults.
Loading variable metadata_service_num_attempts from defaults.
Loading variable profile from defaults.
Looking for credentials via: env
Looking for credentials via: assume-role
Looking for credentials via: shared-credentials-file
Found credentials in shared credentials file: ~/.aws/credentials
Loading JSON file: /Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/data/endpoints.json
Loading variable profile from defaults.
Event choose-service-name: calling handler <function handle_service_name_alias at 0x10cf509d8>
Loading JSON file: /Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/data/ec2/2016-11-15/service-2.json
Event creating-client-class.ec2: calling handler <function add_generate_presigned_url at 0x10cf2a488>
Event creating-client-class.ec2: calling handler <bound method Pill._create_client of <placebo.pill.Pill object at 0x10d3c1d68>>
_create_client
Traceback (most recent call last):
  File "/Users/pascaldevink/.local/bin/bridgy", line 11, in <module>
    sys.exit(main())
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/bridgy/__main__.py", line 373, in main
    handler(args, config)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/bridgy/utils.py", line 32, in wrapper
    func(*args,**kwargs)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/bridgy/__main__.py", line 274, in update_handler
    inventory_obj = inventory.inventory(config)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/bridgy/utils.py", line 49, in __call__
    return self[args]
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/bridgy/utils.py", line 51, in __missing__
    ret = self[key] = self.f(*key)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/bridgy/inventory/__init__.py", line 43, in inventory
    inv = AwsInventory(cache_dir, **srcCfg)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/bridgy/inventory/aws.py", line 41, in __init__
    self.client = session.client('ec2')
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/session.py", line 861, in create_client
    client_config=config, api_version=api_version)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/client.py", line 76, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/client.py", line 288, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/args.py", line 45, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/args.py", line 111, in compute_client_args
    service_name, region_name, endpoint_url, is_secure)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/client.py", line 361, in resolve
    service_name, region_name)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/regions.py", line 122, in construct_endpoint
    partition, service_name, region_name)
  File "/Users/pascaldevink/Library/Python/3.6/lib/python/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

The relevant part of the config file:

config-schema: 2
inventory:
  update_at_start: true
  fuzzy_search: true
  source:
    - type: aws
      name: production
      region: eu-west-1

and of course I have a ~/.aws/credentials file with the correct access key and secret.

Any clue what might cause this? Could it be a version thing of a dependency maybe?

wagoodman / bridgy Goto Github PK

bridgy's Issues

Recommend Projects

Recommend Topics

Recommend Org