nir0s / ghost Goto Github PK

A user can put keys in a stash before initializing it. This results in a created but uninitialized stash which also can't be used because its passphrase is unknown.

We should verify that a stash is initialized before allowing to perform any action on it.

When providing a passphrase via the CLI during init, type checking takes place. In Python 2, the passphrase is passed as unicode but the type check is for string and bytes and so it fails. utf-8 encoding should solve this.

Currently, only the TinyDB storage backend is supported in the CLI.

To solve this, we can get all storage class objects and their names. The names could be used by click via type=click.Choice(storage_names) after which if the name the user provides aligns with any of the available implementations, globals()[class.__name__](stash_path), it will be chosen.

clsmembers = inspect.getmembers(sys.modules[__name__], inspect.isclass) can return the desires class objects.

When running ghost list on an empty vault stash it will fail to list. The reason is that it tries to access dictionary items that are not there if no keys were inserted into the stash.

I'm at /opt and trying to run ghost init ~/stash.json.

This will fail since I don't have sudo privileges. The problem is, that the stash will be created and initialized but the passphrase will not be available since it could not be written into a file in the cwd.

The correct behavior should probably be to remove the stash.

In the ES storage, you can run stash.init() as many times as you want. In The TinyDBStorage and (maybe) SQLAlchemyStorage, it'll tell you that the file/db is already initialized and raise an error.

All init functions should be idempotent.

currently, there's no option to do that unless you export and then import to the same stash using a different passphrase.

When exporting a stash, the values stay encrypted. Loading them into a new stash then means that that stash's passphrase must be the same as the one exported to be able to read the values.

A possible solution could be to create an encrypted version of the entire source stash's data based on the destination stash's passphrase and then passing that passphrase to load, where it will be decrypted and each value will be encrypted according to the destination stash's passphrase.

We should probably take a look at the migrate function to see how to implement this nicely.

Currently, it is done via a proprietary function.

I believe it already is, but the appveyor tests don't pass because of testing problems.

Right now, retrieving a key will retrieve all of its values assuming it has more than one.

For instance:

ghost get key a=b b=c
...

ghost get key -j
...
value: a=value1,b=value2
...

It would be great if we could cleanly retrieve a single value for automation purposes:

ghost get key a
...

value1

It would be helpful if every transaction was logged properly to a file in both a machine readable and human readable format for proper auditing.

This is due to decrypting the key's dictionary inside the instance which then makes the second get decrypt an unencrypted value.

As it is now, ghost can only manage a single stash per storage backend. This is due to the fact that it doesn't allow to declare a "path" for the stash. This also means that only a single passphrase can be applied per backend.

By allowing multiple "paths" (i.e. path in vault, index, in ES, table in tinydb/sqlalchemy, etc..), multiple stashes will be accessible.

VaultStorage stashes secrets to Vault with the client's OS path separator instead of the actual commonly used Vault path separator '/'. This could cause issues when trying to modify the same stash from different operating systems.

Allowing to run ghost migrate which will actually perform stash.export() and stash.load() would go along way with allowing easy migratation between stashes.

The CLI could expose it like so:

ghost migrate ~/.my_stash.json http://127.0.0.1:8200 --source-passphrase xxx --destination-passphrase yyy --source-backend tinydb --destination-backend vault

In Python it could be pretty much the same

Like, creation date, passphrase file location, number of stored keys, etc..

Right now, each storage backend test (for put, get, list, delete, etc..) is tested separately. This means that there could be inconsistencies in how the data structured returned by each of the backends' methods are built.

We should have generic storage backend tests which are run on the returned value of each base method to validate that they are all the same.

flow:
ghost export
ghost init (on new machine)
mv passphrase.ghost to /home/centos/.ghost/passphrases/haviv
ghost load new-stash -p new-stash

error:
ghost get -s /home/centos/.ghost/cloudify.json[haviv] -p $(cat /home/centos/.ghost/passphrases/haviv) aws
STDOUT: Stash: tinydb at /home/centos/.ghost/cloudify.json[haviv]

STDERR: Traceback (most recent call last):
File "/usr/bin/ghost", line 11, in
load_entry_point('ghost==0.5.0', 'console_scripts', 'ghost')()
File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ghost.py", line 1237, in get_key
record = stash.get(key_name=key_name, decrypt=not no_decrypt)
File "/usr/lib/python2.7/site-packages/ghost.py", line 262, in get
key['value'] = self._decrypt(key['value'])
File "/usr/lib/python2.7/site-packages/ghost.py", line 449, in _decrypt
encrypted_value).decode('ascii')
File "/usr/lib64/python2.7/site-packages/cryptography/fernet.py", line 103, in decrypt
raise InvalidToken
cryptography.fernet.InvalidToken

Currently, if we init a stash in a certain path and then set the GHOST_STASH_PATH env var to another path, everything will work on the other path. The reason for having an init phase in the first place is to make a stash a reserved ghost database. Ghost stores a passphrase object within the DB to identify the stash with but doesn't actually perform the identification process at any point. This should be changed so that a stash path that wasn't formally initialized could not be used.

Providing a relative path for sqlalchemy results in an error

Initializing stash...
Traceback (most recent call last):
  File "/home/nir0s/.virtualenvs/ghost/bin/ghost", line 11, in <module>
    load_entry_point('ghost', 'console_scripts', 'ghost')()
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 716, in **call**
    return self.main(_args, *_kwargs)
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, *_ctx.params)
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(_args, **kwargs)
  File "/home/nir0s/repos/nir0s/ghost/ghost.py", line 700, in init_stash
    passphrase = stash.init()
  File "/home/nir0s/repos/nir0s/ghost/ghost.py", line 145, in init
    self._storage.init()
  File "/home/nir0s/repos/nir0s/ghost/ghost.py", line 372, in init
    os.makedirs(os.path.dirname(path))
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/x'

The problem is that we're trying to parse the path wrongly in the storage.

To simplify UX, we should allow users to place the passphrase.ghost file generated when initializing a stash in different directories where it will be automatically read if it exists.

A reasonable hierarchy for retrieving a passphrase for a stash might be:

• The --passphrase flag.
• The GHOST_PASSPHRASE env var
• A passphrase.ghost file found under the cwd
• A passphrase.ghost file found under ~/.ghost
• A passphrase.ghost file found under /etc/ghost

This idea assumes you only have one active stash at any given moment and that if you're using multiple stashes, one of them is a default, which would use the file and one will require you to provide the env var/flag. We might be able to provide a mechanism for attaching a file to a stash somehow, but I wouldn't do that right now.

This will allow us to provide multiple keys for encryption and decryption if the user chooses it and can look somewhat like this (in the CLI):

ghost init --passphrase ASL*mla8fsLA* --passphrase @#IM$LIQSlll --passphrase ...

export GHOST_PASSPHRASE=ASL*mla8fsLA*;@#IM$LIQSlll;...
...

We can use the API like so:

>>> from cryptography.fernet import Fernet, MultiFernet
>>> key1 = Fernet(Fernet.generate_key())
>>> key2 = Fernet(Fernet.generate_key())
>>> f = MultiFernet([key1, key2])
>>> token = f.encrypt(b"Secret message!")
>>> token
'...'
>>> f.decrypt(token)
'Secret message!'

running ghost init stash.json results in an error:

$ ghost init stash.json
Initializing stash...
Traceback (most recent call last):
  File "/home/nir0s/.virtualenvs/ghost/bin/ghost", line 11, in <module>
    load_entry_point('ghost', 'console_scripts', 'ghost')()
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/nir0s/repos/nir0s/ghost/ghost.py", line 700, in init_stash
    passphrase = stash.init()
  File "/home/nir0s/repos/nir0s/ghost/ghost.py", line 145, in init
    self._storage.init()
  File "/home/nir0s/repos/nir0s/ghost/ghost.py", line 318, in init
    os.makedirs(os.path.dirname(self.db_path))
  File "/home/nir0s/.virtualenvs/ghost/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 2] No such file or directory: ''

This happens because we're trying to create a directory for the stash path's base, which doesn't exist.

Makes much more sense to call it that way

Assume a template boto.ini:

[Credentials]
aws_access_key_id = {{ access }}
aws_secret_access_key = {{ secret }}

and a key:

Description:   None
Uid:           08ee6102-5668-440f-b583-97a1c7a17e5a
Created_At:    2016-09-15 15:10:01
Metadata:      None
Modified_At:   2016-09-15 15:10:01
Value:         access=my_access;secret=my_secret;
Name:          aws

It would be nice to generate a file automatically using the values of that key like so:

ghost get aws --generate boto.ini

and get

[Credentials]
aws_access_key_id = my_access
aws_secret_access_key = my_secret

This will allow a user to never keep files with credentials within them, but rather templates of files only.
Of course, you can always provide the file's data as the value to encrypt, but if the file is big or if it frequently changes, it's less comfortable.

It should be possible to generate a passphrase using KMS for people who already use KMS for their systems.

Currently, they're set to 1000000. Changing to about a 100 will reduce test times to just several seconds.

Both the Consul and Vault (and later Elasticsearch, etc..) storage backends can receive additional configuration options currently not exposed via the CLI. The API comfortably exposes those options.

Creating a command which will allow to initialize each specific storage will make configuring the storage easier:

ghost init consul --stash http://localhost:8500 --directory 'my_dir' --verify --client-cert 'my_cert' --auth user:password

An alternative could be to allow the user to pass any set of kwargs to the init command like so:

ghost init http://localhost:8500 --backend consul --storage-args directory=mydir;verify=true;client_cert=my_cert;auth=user:password

Currently, the API isn't documented per backend so it's hard to know how exactly each of them acts.

Currently, we only allow to retrieve by name, but a name will not necessarily be unique in the long run.

It would be nice if a user could "lock" a certain key to prevent it from being modified or deleted.

An API like this could be provided:

stash.lock('key')
...

stash.delete('key')
# The key could is locked and therefore could not be deleted
stash.is_locked('key')
# True

stash.unlock('key')
...

Currently, after initializing a stash, users must set the GHOST_PASSPHRASE and GHOST_STASH_PATH env vars or use the --passphrase and --stash flags respectively.

The UX should be simplified by only requiring the passphrase. The default stash backend is already TinyDB and there's no reason why the default path shouldn't be the TinyDB's backend default path.

Or it might overwrite something the user did not intend.

When a stash is initialized, a passphrase.ghost file is generated for it. A user can explicitly pass a passphrase in the API and the CLI or set an environment variable else the file is searched for in different locations.

If a user works with multiple stashes, they have to explicitly pass the passphrase as only the passphrase.ghost file is looked up.

Providing an API for setting a location in the storage for where to find the passphrase file would allow users to use multiple passphrase files easily.

The API could be implemented somewhat like so:

stash.init()

stash.set_passphrase('/etc/ghost/my-stash.passphrase.ghost')

Every time a user uses a specific stash, the passphrase file will be looked up in the set position.

Currently, a key will be overridden if a put --modify command was executed on it. Allowing to create a new version of the key, which will become the default key will allow users to look back at previous versions of the key. This might prevent situations in which users unknowingly overrode keys they still need.

Versioning should be the default, but we can also allow the user to override.

As of now, the identifier for the version should be the key's name, which means that multiple keys with the same name but a different version will be possible.

If a user creates a key and the key's name wasn't found, it should give it a version of 1
If a key with the same name was found, the new key should get a version 2 and so on.
Retrieving a key should always get the latest version of it.

Currently, each storage returns a list of all keys in the storage rather than just the names. The initial premise was that a user should decide what to do with the list of keys returned to them. Thing is, that currently the Stash instance for the storage returns only the names which means we return redundant information.

WDYT?

Currently, the migrate API decrypts and re-encrypts all keys when moving from one stash to another. It would make sense to simply move all keys as they are if the passphrases are the same between the stashes.

It would be nice to be able to directly get a value from a file and put something directly into a file like so:

ghost put my_ssh_key --from-file ~/.ssh/my_ssh_key

ghost get my_ssh_key --to-file ~/.ssh/my_ssh_key

Currently, the only way to put from a file is to do something like this:

ghost put my_ssh_key ssh_key="$(cat ~/.ssh/my_ssh_key)"

and getting to a file is quite annoying unless you use ghost get ... -j, pipe to jq and then to a file somehow.

Putting to a file will result in a stash's key having a single key in its value's dict while getting to a file will verify that the dict only has a single key in its value's dict otherwise should notify and fail.