Coder Social home page Coder Social logo

guilhermechapiewski / simple-db-migrate Goto Github PK

View Code? Open in Web Editor NEW
196.0 11.0 54.0 1.15 MB

simple-db-migrate is a generic database migration tool inpired on Rails migrations.

Home Page: http://guilhermechapiewski.github.com/simple-db-migrate

License: Apache License 2.0

Python 99.64% Makefile 0.36%

simple-db-migrate's Introduction

simple-db-migrate “quick” documentation

Quick start

simple-db-migrate is damn simple. The best way to understand how it works is by installing and using it.

You can install it by typing:

    $ pip install simple-db-migrate

After installing, for usage tips type:

    $ db-migrate --help

Upgrading simple-db-migrate version

The 1.5.0 version removed some legacy code which updates the version table from the old format to the actual.
To whom use simple-db-migrate for MySQL or Oracle with version less than 1.4.1.1, and the Bruno Caimar version for MS-SQL Server should use the version 1.4.4 before use the new version, to have the version table updated.

Understanding how it works

The first thing you’ll need is a migration file. There are some example migration files in the “example” directory. The migration files have the following format:

    SQL_UP = """
    CREATE TABLE aleatory (
      id int(11) NOT NULL auto_increment,
      name varchar(255) default NULL,
      PRIMARY KEY  (id)
    );
    """
    SQL_DOWN = """
    DROP TABLE aleatory;
    """

... where SQL_UP and SQL_DOWN are two strings that contains respectively the SQL statements to upgrade and downgrade the database schema.

You can use db-migrate to create new migrations by typing:

    $ db-migrate --create create_table_users

The file names need to respect the format “YYYYMMDDHHMMSS_migration_description.migration”. simple-db-migrate uses the YYYYMMDDHHMMSS information to track the database schema version and to decide the order of execution of the scripts. simple-db-migrate will go through all .migration files in your directory and execute all of them in their creation (date) order.

Second, you have to configure access to your database so simple-db-migrate can execute DDL. Just create a file named “simple-db-migrate.conf”, with the following content (there is also an example in the “example” directory):

    DATABASE_HOST = "localhost"
    DATABASE_USER = "root"
    DATABASE_PASSWORD = ""
    DATABASE_NAME = "migration_example"
    DATABASE_MIGRATIONS_DIR = "."

*** The MIGRATIONS_DIR directive may have relative (from the location of this file) or absolute paths separated by ':' pointing to the migrations directories.

You don’t need to create the database. simple-db-migrate will create it for you.

After these two things you are ready to go. Just navigate to the directory where you created your configuration file and type:

    $ db-migrate

If you don’t want to navigate to the directory, you can specify it path instead. In this case you will also need to specify the path to the config file. Note that this also makes it possible to use any name you like for the config file:

    $ db-migrate --config=path/to/file.conf

Migrating to a specific version

If you want, you can migrate your database schema to a specific version by supplying the —migration (or -m) parameter. The version id is the YYYYMMDDHHMMSS identifier used at the migration file:

    $ db-migrate --migration=20090227000129

If you don’t specify any version, simple-db-migrate will migrate the schema to the latest version available in the migrations directories specified in the config file.

Configuring multiple database environments

If you want to use the same configuration file for multiple environments you can prefix the names with the name of your environment and specify it when executing, like the example below.

    DATABASE_HOST = "localhost"                              # default database host
    STAGING_DATABASE_HOST = "staging.host.com"               # staging database host
    PRODUCTION_DATABASE_HOST = "production.host.com"         # production database host
    DATABASE_USER = "root"
    DATABASE_PASSWORD = ""
    DATABASE_NAME = "migration_example"
    DATABASE_MIGRATIONS_DIR = "."

    $ db-migrate --config=path/to/file.conf                  # will use default configurations
    $ db-migrate --config=path/to/file.conf --env=staging    # will use staging configurations, and default to keys not prefixed
    $ db-migrate --config=path/to/file.conf --env=production # will use production configurations, and default to keys not prefixed

Available configurations

You can set default values for internal configurations in your configuration file and overwrite (some of them) using the command line parameters. Below is a list of all configuration options.

Configuration description default value possible values
DATABASE_HOST hostname where database is located
DATABASE_PORT port where database is located
DATABASE_USER username used to connect to database and execute the commands
DATABASE_PASSWORD password used to connect to database and execute the commands
DATABASE_NAME database name used where the commands will be executed
DATABASE_ENGINE the database type where migrations will be executed mysql oracle,mysql,mssql
DATABASE_VERSION_TABLE the table name used to save database versions db_version any name supported by the database
UTC_TIMESTAMP create migration files using UTC time to format the name False True,False
DATABASE_MIGRATIONS_DIR directories to look for migration files separated by :
DATABASE_ENCODING encoding used on database utf-8 any valid encoding
DATABASE_SCRIPT_ENCODING encoding used on migration files utf-8 any valid encoding
drop_db_first if True drop the database before executing migrations False True,False
force_execute_old_migrations_versions if True and current and destination database versions are equal, execute any old migrations not executed yet False True,False
force_use_files_on_down if True use SQL_DOWN from migration files instead of that present on version table False True,False
label_version label to be applied to all executed migrations when doing a upgrade on database
log_dir directory where a file will be created with a full log of the process, with the current time as name
new_migration name for the migration to be created any alpha numeric word, without spaces
paused_mode execute migrations, pausing after finishing each one False True,False
schema_version the desired version of the database, will do a upgrade or a downgrade to be sure that this will be the current version of database
show_sql if True show executed SQL commands False True,False
show_sql_only if True only show the SQL, but do not execute them False True,False

Supported databases engines

You can use this project to run migrations on MySQL, Oracle and MS-SQL server databases.
The default database engine is MySQL. To use the other databases set the DATABASE_ENGINE constant in the configuration file.

Procedure, Function, Trigger (Oracle and MySQL), and Packages (Oracle) support

You can use db-migrate to manage procedures, functions, triggers and packages using “/” as final delimiter.
Examples:

```
— Oracle
CREATE OR REPLACE FUNCTION simple
RETURN VARCHAR2 IS
BEGIN
RETURN ‘Simple Function’;
END simple;
/

— MySQL
CREATE PROCEDURE country_hos
(IN con CHAR)
BEGIN
SELECT Name, HeadOfState FROM Country
WHERE Continent = con;
END
/
```

Roadmap, bug reporting and feature requests

For detailed info about future versions, bug reporting and feature requests, go to issues page.

Other questions

Mail me at “guilherme.chapiewski at gmail.com” for further questions.

simple-db-migrate's People

Contributors

andrewsmedina avatar cyberelfo avatar dependabot[bot] avatar gabrielfalcao avatar guilhermechapiewski avatar hltbra avatar ichernev avatar ricobl avatar sh0nk avatar thulio avatar wandenberg avatar wesrog avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

simple-db-migrate's Issues

SSL/TLS support

I'm trying to upgrade a MariaDB database on Azure and it has the SSL enforcement enabled.

Simple-db-migrate fails with the following error:
[ERROR] could not connect to database: (9002, 'SSL connection is required. Please specify SSL options and retry.')

It would be great to be able to pass the SSL options to the driver. They could be stored in the config file for instance like the other parameters.

Feature request: skip on SQL error.

If migration file causes an SQL error (for example column already exist), and we still continue with the rest of migration files? That could be optional parameter (such as --continue-on-error).

Resetting the migration to old ones

 /usr/local/bin/db-migrate 20150721105731 --force-old-migrations --config="sample_simple-db-migrate.conf"
- Current version is: 20151216191128
- Destination version is: 20151216191128

Nothing to do.


Done.

I can't get back to previous migrations.

Tag de versão inconsistente

Uma migration foi executada mesmo estando com 1 character a mais no timestamp.

Exemplo : 201112152000000_associando_comentario_a_galeria.migration
o timestamp : 2011-12-15 20:00:000

Installation failure in python 3.8

Installation fails in python3.8.

$ docker run -it python:3.8 /bin/bash
root@4f07de037284:/# python --version
Python 3.8.2
root@4f07de037284:/# pip install simple-db-migrate
Collecting simple-db-migrate
Downloading simple-db-migrate-3.0.0.tar.gz (42 kB)
|████████████████████████████████| 42 kB 402 kB/s
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-k6peapnk/simple-db-migrate/setup.py'"'"'; file='"'"'/tmp/pip-install-k6peapnk/simple-db-migrate/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-k6peapnk/simple-db-migrate/pip-egg-info
cwd: /tmp/pip-install-k6peapnk/simple-db-migrate/
Complete output (21 lines):
/tmp/pip-install-k6peapnk/simple-db-migrate/simple_db_migrate/main.py:136: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if destination_version is not '0' and not (self.db_migrate.check_if_version_exists(destination_version) or self.sgdb.get_version_id_from_version_number(destination_version)):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-k6peapnk/simple-db-migrate/setup.py", line 4, in
setup(
File "/usr/local/lib/python3.8/site-packages/setuptools/init.py", line 144, in setup
return distutils.core.setup(**attrs)
File "/usr/local/lib/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/lib/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/local/lib/python3.8/distutils/dist.py", line 982, in run_command
log.info("running %s", command)
File "/usr/local/lib/python3.8/distutils/log.py", line 46, in info
self._log(INFO, msg, args)
File "/usr/local/lib/python3.8/distutils/log.py", line 31, in _log
stream.write('%s\n' % msg)
File "/usr/local/lib/python3.8/codecs.py", line 378, in write
self.stream.write(data)
TypeError: write() argument must be str, not bytes
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
root@4f07de037284:/#

Backslashes stored incorrectly in __db_version__

The backslashes get unescaped twice before entering the database.

For example, if I have
INSERT INTOtableNameVALUES('\\\\my\\\\namespace');
then it will enter db_version.sql_up or sql_down as
INSERT INTOtableNameVALUES('\my\namespace');

This is especially problematic if backslashes appear in sql_down, as running with --force-files will cause different queries to be executed even when there are no changes to the migration file.

db-migrate skiping sql's which are older than last one executed

Example:
I have 4 SQL to execute:
20150112144707_sql1.migration
20150112144709_sql2.migration
20150112144710_sql3.migration
20150112144711_sql4.migration

ON first deployment only 3 of migration were added to master branch, so lets say 20150112144710_sql3.migration was not added.

Migration folder were looking like this:
20150112144707_sql1.migration
20150112144709_sql2.migration
20150112144711_sql4.migration

On next deployment 20150112144710_sql3.migration was added.
So, migration folder was looking like:
20150112144707_sql1.migration
20150112144709_sql2.migration
20150112144710_sql3.migration
20150112144711_sql4.migration

But, as 20150112144711_sql4.migration was already executed and it is younger than 20150112144710_sql3.migration db-migrate did ignore 20150112144710_sql3.migration

Can it be fixed?
How do you handle this situations?

Type error in python 3.6

I get an error when I run migration in python-3.6

Traceback (most recent call last):
  File "/Users/rajeev/Workspace/procure/procure-backend/lib/python3.6/site-packages/simple_db_migrate/__init__.py", line 92, in run
    Main(config).execute()
  File "/Users/rajeev/Workspace/procure/procure-backend/lib/python3.6/site-packages/simple_db_migrate/main.py", line 50, in execute
    self._execution_log('\nStarting DB migration on host/database "%s/%s" with user "%s"...' % (self.config.get('database_host'), self.config.get('database_name'), self.config.get('database_user')), "PINK", log_level_limit=1)
  File "/Users/rajeev/Workspace/procure/procure-backend/lib/python3.6/site-packages/simple_db_migrate/main.py", line 254, in _execution_log
    CLI.msg(msg, color)
  File "/Users/rajeev/Workspace/procure/procure-backend/lib/python3.6/site-packages/simple_db_migrate/cli.py", line 243, in msg
    print("{}{}{}".format(cls.color[color], msg, cls.color["END"]))
  File "/Users/rajeev/Workspace/procure/procure-backend/bin/../lib/python3.6/codecs.py", line 377, in write
    self.stream.write(data)
TypeError: write() argument must be str, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rajeev/Workspace/procure/procure-backend/bin/db-migrate", line 11, in <module>
    sys.exit(run_from_argv())
  File "/Users/rajeev/Workspace/procure/procure-backend/lib/python3.6/site-packages/simple_db_migrate/__init__.py", line 18, in run_from_argv
    run(options.__dict__)
  File "/Users/rajeev/Workspace/procure/procure-backend/lib/python3.6/site-packages/simple_db_migrate/__init__.py", line 96, in run
    CLI.error_and_exit(str(e))
  File "/Users/rajeev/Workspace/procure/procure-backend/lib/python3.6/site-packages/simple_db_migrate/cli.py", line 233, in error_and_exit
    cls.msg("[ERROR] %s\n" % msg, "RED")
  File "/Users/rajeev/Workspace/procure/procure-backend/lib/python3.6/site-packages/simple_db_migrate/cli.py", line 243, in msg
    print("{}{}{}".format(cls.color[color], msg, cls.color["END"]))
  File "/Users/rajeev/Workspace/procure/procure-backend/bin/../lib/python3.6/codecs.py", line 377, in write
    self.stream.write(data)
TypeError: write() argument must be str, not bytes

BUG: db-migrate does not search for config file

This bug was found in version 1.5.3 but is likely in the new version as well. Steps to reproduce:

In a new project directory, install simple-db-migrate and create an example file named "simple-db-migrate.conf" with valid settings. The following command succeeds just fine:

$ db-migrate --config="./simple-db-migrate.conf" -n foo_migration_name

However if the --config directive is removed, the command fails:

$ db-migrate -n bar_migration_name
[ERROR] invalid key ('database_migrations_dir')

It seems obvious that the db-migrate command is not picking up the default config file name, as specified in the help:

--config=CONFIG_FILE
                    Use a specific config file. If not provided, will
                    search for 'simple-db-migrate.conf' in the current
                    directory.

BUG: port number default no longer works

@wandenberg this is an issue in version 1.6.0

The db-migrate command now seems to require a DATABASE_PORT setting to be explicitly defined in the configuration file, rather than automatically defaulting to 3306 in absence of a setting.

If this setting is not defined, then the command fails with:
[ERROR] could not connect to database: an integer is required

Package is out of date, doesn't work with python 3

It looks like you all added python 3 support with this commit: 8e72149

However, when I install the package with pip, it looks like this code is not present.

Downloading simple-db-migrate-2.2.0.tar.gz (41kB)
  Complete output from command python setup.py egg_info:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-udo31o1j/simple-db-migrate/setup.py", line 2, in <module>
      import simple_db_migrate
    File "/tmp/pip-build-udo31o1j/simple-db-migrate/simple_db_migrate/__init__.py", line 95
      except Exception, e:
                      ^
  SyntaxError: invalid syntax

Migration not able to handle hindi text

I have create a .migration file which contains hindi text. db-migrate command fails when I run it.

[ERROR] error logging migration: (1366, "Incorrect string value: '\xE0\xA4\xB6\xE0\xA5\x81...' for column 'sql_up' at row 1")

[ERROR DETAILS] SQL command was:
20190313161739_user.migration

Migration file:
CREATE TABLE users
(
id INT UNSIGNED PRIMARY KEY NOT NULL AUTO_INCREMENT,
name VARCHAR(128) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO users (id, name) VALUES (1, 'Rajeev'), (2, 'राजीव');

Does simple-db-migrate support adding stored procedure?

In order for SP to work, I will need to set DELIMITER and the simple-db-migrate parser parses the string by ";" which won't work.
Also, it gives syntax error, even I put
DELIMITER$$
by itself in the script.

simple_db_migrate.core.exceptions.MigrationException: error executing migration: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER$$' at line 1")

[ERROR DETAILS] SQL command was:
DELIMITER$$

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.