rackerlabs / cdeploy

Simple Python program for managing Cassandra schema migrations

cdeploy

cdeploy is a simple tool to manage your Cassandra schema migrations in the style of dbdeploy

Installation

pip install cdeploy

Command Line Usage

By default, cdeploy will look for migrations in './migrations'. This can be overridden by passing the path to your migrations directory as the first argument:

cdeploy db/migrations

The migrations directory should contain CQL scripts using the following naming convention: [version]_migration_description.cql

For example:

migrations/
    001_create_orders_table.cql
    002_create_customers_table.cql

Version numbers should begin from 1. Migration scripts may contain multiple semicolon terminated CQL statements. Comment lines can begin with either "--" or "//".

Migrations can also specify how to revert the changes by including additional statements following a line containing "--//@UNDO". For example:

CREATE TABLE orders(
    order_id uuid PRIMARY KEY,
    price text
);

--//@UNDO

DROP TABLE orders;

To undo the most recently applied migration, run:

cdeploy --undo

Python Library Usage

    from cdeploy import migrator

    schema_migrator = migrator.Migrator('/path/to/migrations/directory', cassandra_session)
    schema_migrator.run_migrations()

Configuration

cdeploy will look in your migrations directory for a configuration file named cassandra.yml, in a subdirectory named config:

migrations/
   config/
       cassandra.yml

The configuration file specifies the hosts to connect to and the keyspace name, and supports multiple environments. For example:

development:
    hosts: [host1]
    keyspace: keyspace_name

production:
    hosts: [host1, host2, host3]
    keyspace: keyspace_name

The environment can be set via the ENV shell variable, and defaults to development if not specified:

ENV=production cdeploy

Additional configuration parameters for Cassandra are available:

development:
    hosts: [host1]
    keyspace: keyspace_name
    auth_enabled: true
    auth_username: username
    auth_password: password
    port: 9042
    ssl_enabled: true
    ssl_ca_certs: /path/to/ca/certs
    consistency_level: ALL
    timeout: 10

cdeploy's Issues

Configurable Cassandra timeouts

Make the following Cassandra driver options configurable:

• cassandra.cluster.Cluster.control_connection_timeout
• cassandra.cluster.Session.default_timeout

Use Travis CI's Cassandra service

http://docs.travis-ci.com/user/database-setup/#Cassandra

This would enable us to run the integration tests in Travis CI.

Provide a dry run capability

It would be ideal to have a dry run capability that would allow you to see what versions are deployed, and what can be deployed without actually making any changes.

cdeploy --dry-run
Current version is 2
Target version is 4

Verify schema agreement after changes

After performing a migration step, verify all peers agree on the schema before moving on and updating the schema_migrations table.

While the Java driver has a nice method to check this for you, looks like we'll need to manually query the system.peers table.

Timestamp-Based Versions

Incrementing integers as version numbers can be slightly problematic when many develops are working on a project at the same time. I've always though the Ruby on Rails timestamp-based versions are great, because they dramatically reduce the chance of versioning collisions. Would you be open to this style of versioning?

Make cassandra port configurable

The cassandra port is not configurable. It should be added as an optional parameter to the cluster configuration.

Better Logging

While trying to debug a scenario that turned out to be a case of #25 I found that the logging did not help anything since cdeploy was not reporting what file name it was trying to operate on. Some better logging would have (a) saved time and (b) made #25 obvious.

While I think #25 should certainly be resolved, better logging should also be added.

Cassandra cluster not getting shutdown after running migrations in command line mode

We have a get_session method which takes config and creates a cluster and hence return session out of it. Ideally in any DB connection we should close the connection. Here in this case cassandra cluster has method "shutdown()" which should be called after we are done with the migrations. Since its not called there is no guarantee that the command will exit in a definite time. It will keep waiting for the sessions to die.

cdeploy breaks when swap files are present in the migrations directory

I was writing a new migration in vim, and then ran the migrations, and it broke because of the swapfile.