cobliteam / cassandra-migrate

Simple Cassandra schema migration tool written in Python

BRANCH CI	STATUS
master

Cassandra-migrate

Simple Cassandra schema migration tool.

Installation

Run pip install cassandra-migrate, or python ./setup.py install

Reasoning

Unlike other available tools, this one:

• Written in Python for easy installation
• Does not require cqlsh, just the Python driver
• Supports baselining existing database to given versions
• Supports partial advancement
• Supports locking for concurrent instances using Lightweight Transactions
• Verifies stored migrations against configured migrations
• Stores content, checksum, date and state of every migration
• Supports deploying with different keyspace configurations for different environments
• Supports cql and python scripts migrations

Configuration

Databases are configured through YAML files. For example:

keyspace: herbie
profiles:
  prod:
    replication:
      class: SimpleStrategy
      replication_factor: 3
migrations_path: ./migrations

Where the migrations folder (relative to the config file). contains .cql or .py files. The files are loaded in lexical order.

The default convention is to name them in the form: v001_my_migration.{cql | py}. A custom naming scheme can be specified with the new_migration_name option.

Note: new_migration_text is deprecated. The specific file type option should be used instead.

For example

# Date-based migration names
new_migration_name: "v{date:YYYYMMDDHHmmss}_{desc}"

# Default migration names
new_migration_name: "v{next_version:03d}_{desc}"

# Custom initial migration content
new_migration_text: |
  /* Cassandra migration for keyspace {keyspace}.
     Version {next_version} - {date}

     {full_desc} */

# Custom initial migration content for cql scripts
new_cql_migration_text: |
  /* Cassandra migration for keyspace {keyspace}.
     Version {next_version} - {date}

     {full_desc} */

# Custom initial migration content for python scripts
new_python_migration_text: |
  # Cassandra migration for keyspace {keyspace}.
  # Version {next_version} - {date}
  # {full_desc} */

  def execute(session, **kwargs):
      """
      Main method for your migration. Do not rename this method.

      Raise an exception of any kind to abort the migration.
      """

      print("Cassandra session: ", session)

new_migration_name is a new-style Python format string, which can use the following parameters:

• next_version: Number of the newly generated migration (as an int).
• desc: filename-clean description of the migration, as specified by the user.
• full_desc: unmodified description, possibly containing special characters.
• date: current date in UTC. Pay attention to the choice of formatting, otherwise you might include spaces in the file name. The above example should be a good starting point.
• keyspace: name of the configured keyspace.

The format string should not contain the .cql or .py extensions, as it they added automatically.

new_migraton_text is handled with the same rules outline above, but defines the initial content of the migration file, if the type-specific options below ared not set.

new_cql_migraton_text defines the initial content of CQL migration files.

new_python_migraton_text defines the initial content of Python migration files.

Profiles

Profiles can be defined in the configuration file. They can configure the replication and durable_writes parameters for CREATE KEYSPACE. A default dev profile is implicitly defined using a replication factor of 1.

Usage

Common parameters:

-H HOSTS, --hosts HOSTS
                      Comma-separated list of contact points
-p PORT, --port PORT  Connection port
-u USER, --user USER  Connection username
-P PASSWORD, --password PASSWORD
                      Connection password
-c CONFIG_FILE, --config-file CONFIG_FILE
                      Path to configuration file
-m PROFILE, --profile PROFILE
                      Name of keyspace profile to use
-s SSL_CERT, --ssl-cert SSL_CERT
                      File path of .pem or .crt containing certificate of
                      the cassandra host you are connecting to (or the
                      certificate of the CA that signed the host
                      certificate). If this option is provided, cassandra-
                      migrate will use ssl to connect to the cluster. If
                      this option is not provided, the -k and -t options
                      will be ignored.
-k SSL_CLIENT_PRIVATE_KEY, --ssl-client-private-key SSL_CLIENT_PRIVATE_KEY
                      File path of the .key file containing the private key
                      of the host on which the cassandra-migrate command is
                      run. This option must be used in conjuction with the
                      -t option. This option is ignored unless the -s option
                      is provided.
-t SSL_CLIENT_CERT, --ssl-client-cert SSL_CLIENT_CERT
                      File path of the .crt file containing the public
                      certificate of the host on which the cassandra-migrate
                      command is run. This certificate (or the CA that
                      signed it) must be trusted by the cassandra host that
                      migrations are run against. This option must be used
                      in conjuction with the -k option. This option is
                      ignored unless the -s option is provided.
-y, --assume-yes      Automatically answer "yes" for all questions

migrate

Advances a database to the latest (or chosen) version of migrations. Creates the keyspace and migrations table if necessary.

Migrate will refuse to run if a previous attempt failed. To override that after cleaning up any leftovers (as Cassandra has no DDL transactions), use the --force option.

Examples:

# Migrate to the latest database version using the default configuration file,
# connecting to Cassandra in the local machine.
cassandra-migrate -H 127.0.0.1 migrate

# Migrate to version 2 using a specific config file.
cassandra-migrate -c mydb.yml migrate 2

# Migrate to a version by name.
cassandra-migrate migrate v005_my_changes.cql

# Force migration after a failure
cassandra-migrate migrate 2 --force

reset

Reset the database by dropping an existing keyspace, then running a migration.

Examples:

# Reset the database to the latest version
cassandra-migrate reset

# Reset the database to a specifis version
cassandra-migrate reset 3

baseline

Advance an existing database version without actually running the migrations.

Useful for starting to manage a pre-existing database without recreating it from scratch.

Examples:

# Baseline the existing database to the latest version
cassandra-migrate baseline

# Baseline the existing database to a specific version
cassandra-migrate baseline 5

status

Print the current status of the database.

Example:

cassandra-migrate status

generate

Generate a new migration file with the appropriate name and a basic header template, in the configured migrations_path.

When running the command interactively, the file will be opened by the default editor. The newly-generated file name will be printed to stdout.

To generate a Python script, specify the --python option.

See the configuration section for details on migration naming.

Example:

cassandra-migrate generate "My migration description"

cassandra-migrate generate "My migration description" --python

License (MIT)

Copyright (C) 2017 Cobli

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

cassandra-migrate's Issues

Changing Replication details after keyspace is created

Has any thought been given to evaluating the replication defined in the yaml file vs the actual keyspace deployed in cassandra. We discovered its not possible to alter the replication in the yaml file and have it take effect, it only happens if creating the keyspace.

NetworkTopologyStrategy

Hi ,

Will this yml supports NetworkTopologyStrategy like below?

keyspace: migrate1_test
profiles:
migrate1_test
replication:
class: NetworkTopologyStrategy
DC1: 3
DC2: 3
migrations_path: ./migrations

Thanks

Add SSL/TLS support

Hello Cobliteam, thanks for your work on this tool.

We have hit a snug trying to run migrations against a Cassandra cluster with client_encryption enabled. The migrations fail to connect to the cluster because cassandra-migrate is trying to connect without SSL.

WARNING:cassandra.cluster:[control connection] Error connecting to [IP_ADDRESS]:
Traceback (most recent call last):
  File "cassandra\cluster.py", line 2648, in cassandra.cluster.ControlConnection._reconnect_internal (cassandra/cluster.c:48586)
    return self._try_connect(host)
  File "cassandra\cluster.py", line 2670, in cassandra.cluster.ControlConnection._try_connect (cassandra/cluster.c:49103)
    connection = self._cluster.connection_factory(host.address, is_control_connection=True)
  File "cassandra\cluster.py", line 1111, in cassandra.cluster.Cluster.connection_factory (cassandra/cluster.c:15327)
    return self.connection_class.factory(address, self.connect_timeout, *args, **kwargs)
  File "cassandra\connection.py", line 330, in cassandra.connection.Connection.factory (cassandra/connection.c:5678)
    raise conn.last_error
ConnectionShutdown: Connection to [IP_ADDRESS] was closed

The cassandra-driver supports SSL connections.

We think a fix would entail passing ssl_options to the Cluster instantiation call here, and then allowing users to provide an ssl certificate file path in the configuration yaml or as an environment variable passed to the cassandra-migrate command.

We are working on an enhancement PR to this effect but we will need your help publishing a new version pretty fast.

Please let us know if you know of a better way to do this, and if you are open to the PR.

Thanks.

Support SSL-based client_authentication

Hello,

cc @ppcuban

Currently, migrations fail when connecting to a cassandra cluster with client_encryption_options.require_client_auth: true. This PR adds some more SSL options to the cli that enable a user to pass in a .crt and .key file that are necessary for connections to auth-enabled clusters to succeed.

These have been tested and verified to work with properly configured clusters. As with this earlier PR, these options will not be respected if the -s or --ssl-cert option is not provided. The --ssl-cert option acts as the switch to turn on all things SSL.

We are working on adding more documentation to the README to help users with the newly added SSL stuff.

Let us know if this looks good.

Support specifying consistency level of the Cassandra session

While most of the time it is desirable to run DDL statements with the strongest consistency possible, in some cases it might be necessary to create a keyspace in a partially configured cluster. Very large clusters may not also deal well with a consistency level of ALL.

There should be an option in cassandra-migrate.yml to specify which consistency level to use by default in the Session.

Usage of replication_factor > 1

Maybe, it isn’t an issue, but misconfiguration or incorrect usage of replication.

cassandra-migrate-abc.yml

keyspace: abc
profiles:
  dev:
    replication:
      class: SimpleStrategy
      replication_factor: 2
migrations_path: ./migrations-abc

migrations-abc/v001-users.cql


CREATE TABLE abc.users (
    user_id text PRIMARY KEY,
    email text,
    first_name text,
    last_name text,
    phone text,
    roles text,
    status text,
    pwd_salt text,
    pwd_hash text
);

cassandra-migrate -H docker-dev -c cassandra-migrate-abc.yml -m dev migrate

And result:

Traceback (most recent call last):
  File "/usr/local/bin/cassandra-migrate", line 18, in <module>
    main()
  File "/Library/Python/2.7/site-packages/cassandra_migrate/cli.py", line 99, in main
    cmd_method(opts)
  File "/Library/Python/2.7/site-packages/cassandra_migrate/migrator.py", line 461, in migrate
    ignore_failed=opts.force)
  File "/Library/Python/2.7/site-packages/cassandra_migrate/migrator.py", line 263, in _verify_migrations
    'SELECT * FROM "{keyspace}"."{table}"'))
  File "/Library/Python/2.7/site-packages/cassandra_migrate/migrator.py", line 195, in _execute
    return self.session.execute(query, *args, **kwargs)
  File "cassandra/cluster.py", line 2016, in cassandra.cluster.Session.execute (cassandra/cluster.c:35491)
  File "cassandra/cluster.py", line 3826, in cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:74365)
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {<Host: 10.133.0.20 europe-west1>: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level ALL" info={\'required_replicas\': 2, \'alive_replicas\': 1, \'consistency\': \'ALL\'}',)})

[Feature Request] Create a cassandra session per migration script

I have a few cql scripts and a python script to migrate. The python script is something like this:

from cassandra.cluster import Session
from cassandra.query import SimpleStatement, PreparedStatement, dict_factory
import logging

log = logging.getLogger(__name__)


def execute(session: Session):
    session.row_factory = dict_factory
    __migrate_table(session, "table", newtable_schema)


def __migrate_table(session: Session, original_table: str, new_schema: str):
    log.info(f"Migrating table {original_table} to a new schema {new_schema}")
    # do stuff, deleted since it doesnt matter

After I run the migrations the migration exits with status code 1 and the following exception:

INFO:Migrator:Creating keyspace ''
INFO:Migrator:Creating table 'database_migrations' in keyspace ''
INFO:Migrator:Pending migrations found. Current version: None, Latest version: 3
INFO:Migrator:Advancing to version 1
INFO:Migrator:Writing in-progress migration version 1: Migration("v00001_initial_schema.cql")
INFO:Migrator:Applying cql migration
INFO:Migrator:Executing migration with 9 CQL statements
INFO:Migrator:Finalizing migration version with state SUCCEEDED
INFO:Migrator:Advancing to version 2
INFO:Migrator:Writing in-progress migration version 2: Migration("v00002_.cql")
INFO:Migrator:Applying cql migration
INFO:Migrator:Executing migration with 1 CQL statements
INFO:Migrator:Finalizing migration version with state SUCCEEDED
INFO:Migrator:Advancing to version 3
INFO:Migrator:Writing in-progress migration version 3: Migration("v00003_.py")
INFO:Migrator:Applying python script
INFO:v00003_nvector_partitioning:Migrating table table to a new schema 
    CREATE TABLE IF NOT EXISTS table(
        id int,
        time bigint,
        PRIMARY KEY (id, time)
    );
    
INFO:v00003_nvector_partitioning:Copying table loginnvector to tmp_loginnvector started
INFO:v00003_nvector_partitioning:Copying table loginnvector to tmp_loginnvector finished
INFO:Migrator:Finalizing migration version with state SUCCEEDED
Traceback (most recent call last):
  File "/usr/local/bin/cassandra-migrate", line 18, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/cassandra_migrate/cli.py", line 149, in main
    cmd_method(opts)
  File "/usr/local/lib/python3.6/site-packages/cassandra_migrate/migrator.py", line 100, in wrapper
    return func(self, opts, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/cassandra_migrate/migrator.py", line 542, in migrate
    force=opts.force)
  File "/usr/local/lib/python3.6/site-packages/cassandra_migrate/migrator.py", line 509, in _advance
    self._apply_migration(version, migration, skip=skip)
  File "/usr/local/lib/python3.6/site-packages/cassandra_migrate/migrator.py", line 468, in _apply_migration
    if not result or not result[0].applied:
AttributeError: 'dict' object has no attribute 'applied'
+ final_exit_code=1

You can see the python script seems to have been applied successfully, only afterwards in the migrator tool internals it crashes.

Initially I was really quite confused as to why its crashing. The issue is in the row_factory being set by my script. Ofc this was not a hard fix for me, I just have to reset it to whatever was in it before my script, but its a bit error prone. I really think that the session the migration scripts use should be kept separate from each other, and from the session that the migrator internals use. I cant imagine what kind of mischief you could cause, by accident or on purpose.

Make `click` dependency less restrictive

cassandra-migrate currently depends on click==7.1.2, which seems rather aggressive given that it doesn't even directly use click. It should ideally either be upgraded to 8.1.x or pinned less restrictively, I think.

"frozen" requirements

A package should not freeze requirements, otherwise there will be problems like bellow. I'll create a PR.

Add "force" option to baseline

It would be nice to be able to use a --force flag (or similar) when using the baseline functionality to drop the existing migrations table.

In our environment the migrations table seems to get messed up (or dropped or something) periodically. Why this is happening isn't the concern of this post (and I don't think it's the fault of this tool). To fix it, we can use baseline but I had to first drop the existing migrations table (migrate had been run and it failed on the first script (obviously) because the tables already existed).

Support automatic pre/post migration snapshots

Might be a workable solution to #2, but will be annoying to implement, requiring one of:

a) Having nodetool present in the local machine
b) Running nodetool remotely on the Cassandra machine
c) Finding/creating a basic JMX implementation in Python to communicate with Cassandra directly (I've found none yet)
d) Using a JMX proxy

Password with $

Hi,

When my password is with $ sign, it throughs code=0100 [Bad credentials], any idea how to overcome it?

Thanks

Clarification on locking while data migration

Hi

This is not an issue as such, we just wanted how this works. We are evaluating different NoSQL databases for our project and Cassandra is One of them. We wanted to evaluate Live Migration when we came across this tool. We have got a specific question about the locking.

1. I was not sure if locking was possible in cassandra, but your readme says that Supports locking for concurrent instances using Lightweight Transactions. What does that mean? Do you lock the keyspace or lock just the table while running the migration.

2. Suppose we do a add index(secondary)/add column while a lot of data is already inserted which might take some time to create indexes for all the existing data. What sort of downtime we are going to face.

   • Is it that the keyspace will not be accessible for the whole time?
   • Or the table will not be accessible for the whole time?
   • Or everything else can work but we cannot use that secondary index till all the data is indexed, which sounds obvious?

Please let us know.
Thank You

Add integration tests

There are some end-to-end tests in private Cobliteam projects, but they should also be present here.

[BUG - HIGH] Sets default keyspace to config keyspace

During my last contribution, I omitted this feature that was added based on issue #5. This PR #28 fixes this.

Why are the generated v0**.cql files blank except header comment?

@danielkza

Thank you for the great project of Cassandra Migration at first.

I'm trying to migrate data of [email protected] to cloud service of [email protected].

I followed the step below and got blank files of *.cql.
https://github.com/Cobliteam/cassandra-migrate#generate

My question is how to dump all data from KeySpace?

Thanks!

Migration prefix without collission

Thank's for your great tool.

In a team environment there might be collisions on the version prefix as it is a sequential number. Imagine 2 people evolving the schema, pulling the other's source updates only with some delay. This simultaneous work on the schema is very well possible, if each focuses on distinct tables

But with the sequential version prefix, both would generate migration files with the same prefix, right?

I would like suggest a timestamp as the prefix, as with Rails Migrations.

why cassandra-migrate installation take about 5m?

I'm trying to install cassandra-migrate by pip -vv flag and I've figured out, it tries to build cassandra-driver by gcc. why its slow, isnt there alternative to make it faster?

Flexible and extended configuration file

I would like to suggest a more flexible yaml configuration.

1. Allow every property at the root level, as well as the profile level. Do not require any property at the root level, as long as it is provided at the profile level.

2. Allow the specification of user, password and host in the yaml file

Following is a sample yaml file, and its execution

profiles:
  dev:
    keyspace: schema_dev
    host: localhost
    user: dev_user
    password: dev_pwd
  dev:
    keyspace: schema_prod
    host: db.my.dmain
    user: prod_user
migrations_path: .
replication:
  class: SimpleStrategy
  replication_factor: 3

cassandra-migrate migrate
cassandra-migrate --profile prod --password prod_pwd migrate

Thanks for cassandra-migrate.

How to recover from a failed migration?

So I'm looking at this package as a way to manage my schema changes. Seems to work quite well, until a migration fails, at which point it's unclear how to recover.

I deliberately created a migration that was intended to fail: it contained a list of 10 ALTER statements, one of which specified a table that does not exist. cassandra-migrate correctly failed to apply the migration:

The migrate operation cannot be undone. Are you sure? [y/N] y
INFO:Migrator:Pending migrations found. Current version: 2, Latest version: 3
INFO:Migrator:Advancing to version 3
INFO:Migrator:Writing in-progress migration version 3: Migration("v3_US2704_Compaction_Strategies.cql")
INFO:Migrator:Applying cql migration
INFO:Migrator:Executing migration with 165 CQL statements
ERROR:Migrator:Failed to execute migration
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/cassandra_migrate/migrator.py", line 408, in _apply_cql_migration
self.session.execute(statement)
File "cassandra/cluster.py", line 2122, in cassandra.cluster.Session.execute
return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile, paging_state).result()
File "cassandra/cluster.py", line 3982, in cassandra.cluster.ResponseFuture.result
raise self._final_exception
InvalidRequest: Error from server: code=2200 [Invalid query] message="unconfigured table attribute_type_values"
ERROR:Migrator:Failed to execute migration
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/cassandra_migrate/migrator.py", line 454, in _apply_migration
self._apply_cql_migration(version, migration)
File "/Library/Python/2.7/site-packages/cassandra_migrate/migrator.py", line 411, in _apply_cql_migration
raise FailedMigration(version, migration.name)
FailedMigration: Migration failed, cannot continue (version 3): v3_US2704_Compaction_Strategies.cql
INFO:Migrator:Finalizing migration version with state FAILED
Error: Migration failed, cannot continue (version 3): v3_US2704_Compaction_Strategies.cql

I then changed the v3_US2704_Compaction_Strategies.cql and removed the call to ALTER the unconfigured table and attempted to re-run:

Keyspace: fubar
Migrations table: database_migrations
Current DB version: 2
Latest DB version: 3

Applied migrations

Name                                 State      Date applied               Checksum

---

1 v1_First_commit.cql SUCCEEDED 2018-01-27 16:05:45+00:00 590a0967af38b59df14151e08d74817c5fa0d170b1db8efb7bf7378334d54281
2 v2_Second_commit.cql SUCCEEDED 2018-01-27 16:07:32+00:00 d12fbceede8af85cb29d6a8225d955998bb0faac65112138c4bfbe54caa0daeb
3 v3_US2704_Compaction_Strategies.cql FAILED 2018-01-27 16:28:12+00:00 66592f5e47ea87222e1f0134afe9bbf1c907cdcb0de0e8d7e66f59707dbfdc1e

Pending migrations

Name                                 Checksum

---

3 v3_US2704_Compaction_Strategies.cql 29eb0f74fdb746e3d3f4bf4aef02e20498038cfdae28b505b1b8d21e6590719b

pde@Clarence:~/src/cassandra-migrate (master●●)$ cassandra-migrate migrate
WARNING:cassandra.cluster:Cluster.init called with contact_points specified, but no load_balancing_policy. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = [u'127.0.0.1'], lbp = None)
The migrate operation cannot be undone. Are you sure? [y/N] y
Error: Migration failed, cannot continue (version 3): v3_US2704_Compaction_Strategies.cql

I then created a v4 migration, which again refuses to apply because v3 has failed.

So: if fixing the syntax of a failed migration file doesn't allow you to get past the failed migration, and subsequent migrations won't apply, how does one fix a failed migration so that work can continue? Am I expected to reset and then completely repopulate my dev cluster if I typo a migration file?

Support external credentials files/env vars

Would make it much easier to automate migrations in different environments.

Ask for confirmation before applying dangerous commands

Cannot generate migrations without specifying new_migration_name

As of version 0.2.1, we cannot run the cassandra-migrate generate command without defining a new_migration_name in the config file.
While this is not a major issue, and the workaround is pretty simple, the Readme does imply that there is a default value, which is inconsistent with the behavior.

Traceback (most recent call last):
  File "/usr/local/bin/cassandra-migrate", line 18, in <module>
    main()
  File "/usr/local/lib/python2.7/site-packages/cassandra_migrate/cli.py", line 84, in main
    new_path = Migration.generate(config=config, description=opts.description)
  File "/usr/local/lib/python2.7/site-packages/cassandra_migrate/migration.py", line 86, in generate
    fname = fname_fmt.format(**format_args) + '.cql'
KeyError: u'description'

Str vs unicode type of keyspace, migrations_table in config.py

Hi, thanks to this tool we started to import our schema to our cassandra db. But we find out that in case of str type of keyspace and migrations_table the params in sql command are not apostrophed in cassandra driver and the migration fails with message "Keyspace metadata was not refreshed. See log for details." or "Table metadata was not refreshed. See log for details."

We use:
cassandra-driver (3.22.0)
cassandra-migrate (0.3.3)
Python 2.7.5

Logs:
cassandra-migrate -H cas2 -p 9042 -u cassandra -P cassandra reset
WARNING:cassandra.cluster:Cluster.init called with contact_points specified, but no load_balancing_policy. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = [u'cas2'], lbp = None)
The reset operation cannot be undone. Are you sure? [y/N] y
INFO:Migrator:Dropping existing keyspace 'ods'
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
INFO:cassandra.cluster:New Cassandra host <Host: 10.180.137.167:9042 SearchAnalytics> discovered
INFO:cassandra.cluster:New Cassandra host <Host: 10.180.137.166:9042 SearchAnalytics> discovered
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
INFO:Migrator:Creating keyspace 'ods'
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
INFO:Migrator:_ensure_keyspace 'ods'
INFO:Migrator:_ensure_keyspace type '<class 'future.types.newstr.newstr'>'
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
(My comment: Here is the problem, the keyspace param comes here as str and then in the where_clause it is not single quoted, I assume)
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<class 'future.types.newstr.newstr'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = ods'
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
Traceback (most recent call last):
File "/usr/bin/cassandra-migrate", line 18, in
main()
File "/usr/lib/python2.7/site-packages/cassandra_migrate/cli.py", line 149, in main
cmd_method(opts)
File "/usr/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 100, in wrapper
return func(self, opts, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 558, in reset
self.migrate(opts)
File "/usr/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 100, in wrapper
return func(self, opts, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 536, in migrate
self._ensure_keyspace()
File "/usr/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 259, in _ensure_keyspace
self.cluster.refresh_keyspace_metadata(self.config.keyspace)
File "cassandra/cluster.py", line 2155, in cassandra.cluster.Cluster.refresh_keyspace_metadata
cassandra.DriverException: Keyspace metadata was not refreshed. See log for details.

If I make the following changes (the keyspace and migrations_table converted to unicode) in config.py, then the migration passes.

cassandra-migrate/cassandra_migrate/config.py

Line 40 in 62df1d8

def _assert_type(data, key, tpe, default=None):

def _assert_type(data, key, tpe, default=None):
"""Extract and verify if a key in a dictionary has a given type"""
value = data.get(key, default)
if tpe == str:
value = str(value)
if tpe == unicode:
value = unicode(value)
if not isinstance(value, tpe):
raise ValueError("Config error: {}: expected {}, found {}".format(
key, tpe, type(value)))
return value

cassandra-migrate/cassandra_migrate/config.py

Line 77 in 62df1d8

self.keyspace = _assert_type(data, 'keyspace', str)

self.keyspace = _assert_type(data, 'keyspace', unicode)

cassandra-migrate/cassandra_migrate/config.py

Line 94 in 62df1d8

self.migrations_table = _assert_type(data, 'migrations_table', str,

self.migrations_table = _assert_type(data, 'migrations_table', unicode,
default='database_migrations')

Logs:
cassandra-migrate -H cas2 -p 9042 -u cassandra -P cassandra reset
WARNING:cassandra.cluster:Cluster.init called with contact_points specified, but no load_balancing_policy. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = [u'cas2'], lbp = None)
The reset operation cannot be undone. Are you sure? [y/N] y
INFO:Migrator:Dropping existing keyspace 'ods'
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
INFO:cassandra.cluster:New Cassandra host <Host: 10.180.137.167:9042 SearchAnalytics> discovered
INFO:cassandra.cluster:New Cassandra host <Host: 10.180.137.166:9042 SearchAnalytics> discovered
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
WARNING:cassandra.connection:An authentication challenge was not sent, this is suspicious because the driver expects authentication (configured authenticator = PlainTextAuthenticator)
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
INFO:Migrator:Creating keyspace 'ods'
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
INFO:Migrator:_ensure_keyspace 'ods'
INFO:Migrator:_ensure_keyspace type '<type 'unicode'>'
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
INFO:Migrator:Creating table 'database_migrations' in keyspace 'ods'
INFO:cassandra.metadata:2018,get_keyspace, type keyspace = '<type 'unicode'>'
INFO:cassandra.metadata:2019,get_keyspace, where_clause ' WHERE keyspace_name = 'ods''
INFO:Migrator:Pending migrations found. Current version: None, Latest version: 13
INFO:Migrator:Advancing to version 1
INFO:Migrator:Writing in-progress migration version 1: Migration("1.0.0-odsCassandraSchema.cql")
...

I assume this behavior is because Py2 vs Py3. Is this tool compatible with Py2 and Py3 too? Or how otherwise can be fixed this issue?
Thanks in advance & regards
Gabo.

Support Python migrations (instead of just CQL)

Load-balancing policy missing

I am getting the following WARNING log from cassandra-migrate migrate command:
WARNING:cassandra.cluster:Cluster.__init__ called with contact_points specified, but no load_balancing_policy. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = ['cassandra.cs-dev.svc.cluster.local'], lbp = None)

Would be possible to do something about it?

NetworkTopologyStrategy In Config

Hi,

Does your tool support schema creation using NetworkTopologyStrategy?

I've tried a config like:

profiles:
  prod:
    replication:
      class: NetworkTopologyStrategy
      dc1: 3
      dc2: 3

This config doesn't seem to work, the tool just fallsback to SimpleStrategy with RF 1.

consistency level ALL

Hi

Can we change consistency level to QUORUM, as one node is down from my 9 Nodes cluster i get error message.

message="Cannot achieve consistency level ALL"

Please advice

Thanks

Error: missing keyspace in migration statement

I have a migration file, referencing its table without a keyspace. The migration fails, with the following error message:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 380, in _apply_migration
InvalidRequest: Error from server: code=2200 [Invalid query] message="No keyspace has been specified. USE a keyspace, or explicitly specify keyspace.tablename"

cassandra-migrate could issue a USE statement before the first migration, considering the keyspace configured in its configuration file. If we hardcode the schema in every migration we loose the ability to switch between dev and prod keyspaces.

I really appreciate cassandra-migrate, and am trying to establish it at my company. Unfortunately I am not experienced in Python, to submit a pull request, instead of opening the issues.

Rollback support

It would be great to be able to rollback to previous migration instead of having to forcefully drop the keyspace. How difficult would it be to implement something like this?

Error when no profiles are defined in the conf file

As of version 0.2.3, if no profiles are defined in the main configuration file, the program crashes when trying to read the config:

$ cassandra-migrate -c migrations.yaml migrate
Traceback (most recent call last):
  File "/path_redacted/venv/bin/cassandra-migrate", line 18, in <module>
    main()
  File "/path_redacted/venv/lib/python3.6/site-packages/cassandra_migrate/cli.py", line 90, in main
    config = MigrationConfig.load(opts.config_file)
  File "/path_redacted/venv/lib/python3.6/site-packages/cassandra_migrate/config.py", line 90, in load
    return cls(config, os.path.dirname(path))
  File "/path_redacted/venv/lib/python3.6/site-packages/cassandra_migrate/config.py", line 60, in __init__
    profiles = _assert_type(data, 'profiles', dict)
  File "/path_redacted/venv/lib/python3.6/site-packages/cassandra_migrate/config.py", line 26, in _assert_type
    key, tpe, type(value)))
ValueError: Config error: profiles: expected <class 'dict'>, found <class 'NoneType'>

Write unit tests

Error when no profiles are defined in the conf file, pt. 2

Hi, it's me again :)
With v0.2.4 I get this:

INFO:Migrator:Dropping existing keyspace 'dqw'
INFO:Migrator:Creating keyspace 'dqw'
Traceback (most recent call last):
  File "/path_redacted/venv/bin/cassandra-migrate", line 18, in <module>
    main()
  File "/path_redacted/venv/lib/python3.6/site-packages/cassandra_migrate/cli.py", line 109, in main
    cmd_method(opts)
  File "/path_redacted/venv/lib/python3.6/site-packages/cassandra_migrate/migrator.py", line 480, in reset
    self.migrate(opts)
  File "/path_redacted/venv/lib/python3.6/site-packages/cassandra_migrate/migrator.py", line 459, in migrate
    self._ensure_keyspace()
  File "/path_redacted/venv/lib/python3.6/site-packages/cassandra_migrate/migrator.py", line 217, in _ensure_keyspace
    durable_writes=cassandra_ddl_repr(profile['durable_writes'])))
KeyError: 'durable_writes'

To clarify, the migrations file looks like this:

keyspace: dqw
migrations_path: ./migrations

If I add the default dev profile explicitly, it all works.

Support for AWS Keyspaces with SIGV4 plugin for Cassandra driver

It is possible to connect to AWS Keyspaces (managed Cassandra) with temporary credentials thanks to SIGV4 cassandra plugin (https://github.com/aws/aws-sigv4-auth-cassandra-python-driver-plugin).

As cassandra-migrate uses the python cassandra driver (fully compatible with the SIGV4 plugin) it would be great to add AWS support for temporary credentials (generated from AWS tokens) so we would not be in the need to define a user / password in keyspaces (more secured).

getting error with cassandra-migrate status

I have installed cassandra-migrate and created some default config file with migrations folder.
Config file:

#cassandra-migrate.yml
keyspace: mail_storage
profiles:
  dev:
    replication:
      class: SimpleStrategy
      replication_factor: 2
migrations_path: ./migrations 

And when i am trying commands like cassandra-migrate migrate <my .cql file> or cassandra-migrate status i am getting this error:
WARNING:cassandra.cluster:Cluster.init called with contact_points specified, but no load_balancing_policy. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = [u'127.0.0.1'], lbp = None)
Traceback (most recent call last):
File "/home/aleksey-chudak/.local/bin/cassandra-migrate", line 18, in
main()
File "/home/aleksey-chudak/.local/lib/python2.7/site-packages/cassandra_migrate/cli.py", line 149, in main
cmd_method(opts)
File "/home/aleksey-chudak/.local/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 580, in status
ignore_concurrent=True)
File "/home/aleksey-chudak/.local/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 304, in _verify_migrations
'SELECT * FROM "{keyspace}"."{table}"'))
File "/home/aleksey-chudak/.local/lib/python2.7/site-packages/cassandra_migrate/migrator.py", line 236, in _execute
return self.session.execute(query, *args, **kwargs)
File "cassandra/cluster.py", line 2141, in cassandra.cluster.Session.execute
File "cassandra/cluster.py", line 4033, in cassandra.cluster.ResponseFuture.result
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {<Host: 127.0.0.1 datacenter1>: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level ALL" info={'required_replicas': 2, 'alive_replicas': 1, 'consistency': 'ALL'}',)})

May be i am missing some configs in my cassandra-migrate.yml?
By the way i am getting a warning like in an issue #32

Installing cassandra-migrate in ubuntu

After running pip install cassandra-migrate, I do not see command cassandra-migrate to manage migrations. Am I missing any steps during installation?

make a release

Hello people,

many thanks for the work you're doin. Do you think it would be possible to release latest changes from trunk? We are having dependency issues around pyYAML library:

The conflict is caused by:
cassandra-migrate 0.3.3 depends on PyYAML<5.0
pyaml-env 1.1.1 depends on PyYAML==5.4.1

would be cool if we can upgrade to cassandra-migrate 0.3.4 which should solve this problem. Also one pr linked to the issue:

#59

Many thanks!

Support migration reset to version 0

At this moment it is not possible to undo all migrations.

For example, if version 1 migration is to create table books. And version 2 migration is to add column author to books. You can only undo version 2 migration by running cassandra-migrate reset 1. But you cannot undo version 1 migration by running cassandra-migrate reset 0. Since 0 causes a ValueError

ValueError: Invalid database version, must be a number > 0 or the name of an existing migration