Coder Social home page Coder Social logo

cfmigrations's Introduction

cfmigrations

Keep track and run your database migrations with CFML.

Overview

Migrations are a way of providing version control for your application's schema. Changes to schema are kept in timestamped files that are ran in order up and down.In the up function, you describe the changes to apply your migration. In the down function, you describe the changes to undo your migration.

Here's a simple example of that using simple queryExecute:

component {

    function up() {
        queryExecute( "
            CREATE TABLE `users` (
                `id` INT UNSIGNED AUTO_INCREMENT,
                `email` VARCHAR(255) NOT NULL,
                `password` VARCHAR(255) NOT NULL
            )
        " );
    }

    function down() {
        queryExecute( "
            DROP TABLE `users`
        " );
    }

}

The name of this file could be something like 2017_09_03_043150_create_users_table.cfc. The first 17 characters of this file represent the timestamp of the migration and need to be in this format: YYYY_MM_DD_HHMISS. The reason for this is so cfmigrations can run the migrations in the correct order. You may have migrations that add columns to a table, so you need to make sure the table exists first. In this case, just make sure the timestamp for adding the new column comes after the timestamp for creating the table, like so:

2017_09_03_043150_create_users_table.cfc
2017_10_03_010406_add_is_subscribed_column.cfc

An easy way to generate these files is to use commandbox-migrations and the migrate create command.

Installation and Uninstallation

In order to track which migrations have been ran, cfmigrations needs to install a tracking mechanism. For database migrations this creates a table in your database called cfmigrations by default. You can do this by calling the install() method or by running the migrate install command from commandbox-migrations.

If you find a need to, you can uninstall the migrations tracker by calling the uninstall() method or by running migrate uninstall from commandbox-migrations. Running this method will rollback all ran migrations before removing the migrations tracker.

Configuration

The module is configured by default with a single migration service that interact with your database, optionally using qb. Multiple migration services with different managers may also be configured. The default manager for the cfmigrations is QBMigrationManager, but you may use others, such as those included with the cbmongodb and cbelasticsearch modules or roll your own.

The default configuration for the module settings are:

moduleSettings = {
    "cfmigrations" : {
        "managers" : {
            "default" : {
                // The manager handling and executing the migration files
                "manager" : "cfmigrations.models.QBMigrationManager",
                // The directory containing the migration files
                "migrationsDirectory" : "/resources/database/migrations",
                // The directory containing any seeds, if applicable
                "seedsDirectory" : "/resources/database/seeds",
                // A comma-delimited list of environments which are allowed to run seeds
                "seedEnvironments" : "development",
                "properties" : {
                    "defaultGrammar" : "BaseGrammar@qb"
                }
            }
        }
    }
}

With this configuration, the default migration manager may be retrieved via WireBox at migrationService:default.

Here is an example of a multi-manager migrations system. Each separate manager will require their own configuration properties.

moduleSettings = {
    "cfmigrations": {
        "managers": {
            "db1": {
                "manager": "cfmigrations.models.QBMigrationManager",
                "migrationsDirectory": "/resources/database/db1/migrations",
                "seedsDirectory": "/resources/database/db1/seeds",
                "properties": {
                    "defaultGrammar": "MySQLGrammar@qb",
                    "datasource": "db1",
                    "useTransactions": "false",    
                }
            },
            "db2": {
                "manager": "cfmigrations.models.QBMigrationManager",
                "migrationsDirectory": "/resources/database/db2/migrations",
                "seedsDirectory": "/resources/database/db2/seeds",
                "properties": {
                    "defaultGrammar": "PostgresGrammar@qb",
                    "datasource": "db2",
                    "useTransactions": "true"    
                }
            },
            "elasticsearch": {
                "manager": "cbelasticearch.models.migrations.Manager",
                "migrationsDirectory": "/resources/elasticsearch/migrations"
            }
        }
    }
};

With this configuration the individual migration managers would be retreived as such:

  • db1 - getInstance( "migrationService:db1" )
  • db2 - getInstance( "migrationService:db2" )
  • elasticsearch - getInstance( "migrationService:elasticsearch" )

Migration Files

A migration file is a component with two methods up and down. The function up should define how to apply the migration. The function down should define how to undo the change down in up. For QBMigrationManager migrations (which is the default), the up and down functions are passed an instance of SchemaBuilder@qb and QueryBuilder@qb as arguments. To learn more about the functionality and benefits of SchemaBuilder, QueryBuilder, and qb, please read the QB documentation here. In brief, qb offers a fluent, expressive syntax that can be compiled to many different database grammars, providing both readability and flexibility.

Here's the same example as above using qb's SchemaBuilder:

component {

    function up( schema, qb ) {
    	schema.create( "users", function( t ) {
            t.increments( "id" );
            t.string( "email" );
            t.string( "password" );
        } );
    }

    function down( schema, qb ) {
        schema.drop( "users" );
    }

}

Migration files need to follow a specific naming convention — YYYY_MM_DD_HHMISS_[describe_your_changes_here].cfc. This is how cfmigrations knows in what order to run your migrations. Generating these files is made easier with the migrate create command from commandbox-migrations.

Using the injected qb instance, you can insert or update required data for your application. If you want to create test data for your application, take a look at seeders below instead.

There is no limit to what you can do in a migration. It is recommended that you separate changes to different tables to separate migration files to keep things readable.

Running Migrations

There are a few methods for working with migrations. (Each of these methods has a related command in commandbox-migrations.)

These methods can be run by injecting MigrationService@cfmigrations - for example: getInstance( "MigrationService@cfmigrations" ).runAllMigrations( "up" ) will run all migrations.

runNextMigration

Run the next available migration in the desired direction.

Name Type Required Default Description
direction String true The direction in which to look for the next available migration — up or down.
postProcessHook function false function() {} A callback to run after running the migration.
preProcessHook function false function() {} A callback to run before running the migration.

runAllMigrations

Run all available migrations in the desired direction.

Name Type Required Default Description
direction String true The direction for which to run the available migrations — up or down.
postProcessHook function false function() {} A callback to run after running each migration.
preProcessHook function false function() {} A callback to run before running each migration.

reset

Returns the database to an empty state by dropping all objects.

findAll

Returns an array of all migrations:

[{
	fileName = "2019_12_18_195831_create-users-table.cfc",
	componentName = "2019_12_18_195831_create-users-table",
	absolutePath = "/var/www/html/app/resources/migrations/2019_12_18_195831_create-users-table.cfc",
	componentPath = "/app/resources/migrations/2019_12_18_195831_create-users-table.cfc",
	timestamp = 123455555,
	migrated = false,
	canMigrateUp = true,
	canMigrateDown = false,
	migratedDate = "2019-03-22"
}]

hasMigrationsToRun

Returns true if there are available migrations which can be run in the provided order.

Name Type Required Default Description
direction String true The direction for which to run the available migrations — up or down.

Seeders

Seeding your database is an optional step that allows you to add data to your database in mass. It is usually used in development to create a populated environment for testing. Seeders should not be used for data required to run your application or to migrate data between columns. Seeders should be seen as entirely optional. If a seeder is never ran, your application should still work.

Seeders can be ran by calling the seed method on a MigrationService. It takes an optional seedName string to only run a specific seeder. Additionally, you can run all your seeders when migrating your database by passing seed = true to the up method.

By default, seeders can only be ran in development environments. This can be configured on each manager by setting a seedEnvironments key to either a list or array of allowed environments to run in.

A seeder is a cfc file with a single required method - run. For the QBMigrationManager, it is passed a QueryBuilder instance and a MockData instance, useful for creating fake data to insert into your database. (Other Migration Managers will have other arguments passed. Please refer to the documentation for your specific manager.)

component {

    function run( qb, mockdata ) {
        qb.table( "users" ).insert(
            mockdata.mock(
                $num = 25,
                "firstName": "firstName",
                "lastName": "lastName",
                "email": "email",
                "password": "string-secure"
            )
        );
    }

}

Tips and tricks

Setting Schema

It's important to set the schema attribute for cfmigrations. Without it, cfmigrations can't tell the difference between a migration table installed in the schema you want and any other schema on the same database. You can set the schema by calling the setSchema( string schema ) method.

Default values in MS SQL server

MS SQL server requires some special treatment when removing columns with default values. Even though syntax is almost the same, MS SQL creates a special default constraint like DF_tablename_columname. When migrating down, this constraint has to be removed before dropping the column. In other grammars no special named constraint is created.

Example:

component {

    function up( schema, query   ) {
        schema.alter( "users", function ( table ) {
            table.addColumn( table.boolean( "hassuperpowers").default(0) );
        });
    }

    function down( schema, query  ) {
        schema.alter( "users", function( table ) {
            table.dropConstraint( "DF_users_hassuperpowers");
            table.dropColumn( "hassuperpowers" ) ;
        } );
    }

}

Updating database content in a migration file

Sometimes you want to do multiple content updates or inserts in a migration. In this case you can use the QueryBuilder for the updates. When doing your second update you have to reset the Querybuilder object by using the newQuery method.

Example:

component {

    function up( SchemaBuilder schema, QueryBuilder query ) {
	query.from('users')
	    .where( "username", "superuser")
	    .update( {"hassuperpowers" = true} )
	query.newQuery().from('users')
	    .where('username','RandomUser')
	    .update( {"hassuperpowers" = false} )
    }

    function down( SchemaBuilder schema, QueryBuilder query ) {
        ......
    }

}

cfmigrations's People

Contributors

daemach avatar elpete avatar grantcopley avatar jclausen avatar joeltobey avatar lmajano avatar michaelborn avatar mordantwastrel avatar ryanalbrecht avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

cfmigrations's Issues

Running migrations "down" randomly fails

While testing cfmigrations, I've been running migrations down then back up. I assume that "down" means starting with the last migration run, run the down function then remove the record from the cfmigrations table and keep doing this until none are left....

Down occasionally does not fire for some reason, verified by adding cfdump("Got this far");abort; in my migration file, inside the "down" method.

function onAppInit(event,rc,prc){
		getInstance( "MigrationService@cfmigrations" ).runAllMigrations( "down" );
		getInstance( "MigrationService@cfmigrations" ).runAllMigrations( "up" );
}

Install command errors with no information

When trying to run the command box migrate install I get the following output

ERROR (5.6.1+00618)

java.lang.NullPointerException

My datasource configuration works fine within my application - which is running in the ACF commandbox image. My database is the mcr.microsoft.com/azure-sql-edge:latest image. Running box config set verboseErrors=true does not change the output of the command.

Here is my .cfmigrations.json configuration

{
    "default": {
        "manager": "cfmigrations.models.QBMigrationManager",
        "migrationsDirectory": "resources/database/migrations/",
        "seedsDirectory": "resources/database/seeds/",
        "properties": {
            "defaultGrammar": "SqlServerGrammar@qb",
            "schema": "dbo",
            "migrationsTable": "cfmigrations",
            "connectionInfo": {
                "password": "mypassword",
                "connectionString": "jdbc:mssql://localhost:1433?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC&useLegacyDatetimeCode=true",
                "class": "com.microsoft.sqlserver.jdbc.SQLServerDriver",
                "username": "coldbox",
                "bundleName": "mssqljdbc4",
                "bundleVersion": "4.0.2206.100"
            }
        }
    }
}

Can anyone help me figure out what I'm doing wrong?

Add Multi-Tenant Support

It would be awesome if you could define an array of datasources in the configuration of the module and each of these datasources would get the migration operations applied to it. We have a multi-tenant application where the database for each tenant needs to be migrated with each release and this is the only thing keeping us from using the tool. Would we love to start using it if this is something that could be easily added.

Add transaction around the migration

A migration can contain any number of SQL statements. Right now, if one of the statements fail, any previous statements that executed successfully are not rolled back. This leaves the database is a state that the migration is not expected and cannot rollback.

We should add a transaction around the migration function to prevent this issue.

Multiple datasources

Hello - this module looks very promising. Our application has 3 datasources - main (ui/billing/etc), logging (where devices send their data) and alerts. I would like to use cfmigrations to keep them all up to date. How do I set up the environment variables so that migrate install works? Any other "gotcha's" to be aware of?

open not working on windows after migration file is created.

I created a migration file that did not create properly but did not run properly.

CommandBox> migrate create tbltest_create --open
Created /C_drive\dev\project\ols\ols_api\resources\database\migrations/2022_09_29_101349_tbltest_create.cfc

I would be glad to update and do a pull request but did not see "open" within the code when searching for it in VSCode.

Cfmigration table detected in all available schemes instead of current database

Ok, I just tested this in MySQL
the point is compileTableExists fires this query
SELECT 1 FROM information_schema.tablesWHEREtable_name = 'cfmigrations'
if Laur and you are both using the same credentials for your databases you will indeed see each others db. If you would use separate credentials for each db, which is more secure anyway, you will not have this problem

elpete [4:53 PM]
qb doesn’t see the connection string. It just uses it. There may be a way to get it from Java, but I think the easier path is to have hasTable take an optional schema parameter.

cfmigrations runs hundreds of duplicate SQL queries on app init

Running migrationService.hasMigrationsToRun( "up" ) to check for migrations to run causes cfmigrations to run a SQL query for every migration cfc in the migrations directory. So if you have a large app with hundreds or thousands of migrations, cfmigrations will run SELECT name from cfmigrations hundreds or thousands of times!

To fix this, we should cache the query results in the variables scope:

https://github.com/coldbox-modules/cfmigrations/blob/v2.0.10/models/MigrationService.cfc#L311-L318

var migrations = queryExecute(
    "
        SELECT name
        FROM #getMigrationsTable()#
    ",
    {},
    { datasource : getDatasource() }
);

Add option to skip transactions

We recently had a very unusual edge case in which we had a migration that ran an ALTER DATABASE command that could not be processed inside a transaction.

We made a slight change to MigrationService.cfc to enable this:

               preProcessHook( migrationStruct );

		if ( structKeyExists( migration, 'transaction' ) && !migration.transaction ) {
			invoke(
				migration,
				direction,
				[ schema, query ]
			);
			logMigration( direction, migrationStruct.componentName );
		}
		else {
			transaction action="begin" {
				try {
					invoke(
						migration,
						direction,
						[ schema, query ]
					);
					logMigration( direction, migrationStruct.componentName );
				} catch ( any e ) {
					transaction action="rollback";
					rethrow;
				}
			}
		}

		postProcessHook( migrationStruct );

And then on the migration itself we just set this.transaction = false;.

Would you take this as a PR?

Please change QB to provider

I use QB in provider mode everywhere. In migrations, I get bit because I forget to use newQuery() when creating and populating multiple tables (usually lookups - states, countries, tax rates, etc). Changing the QB injection to be a provider would still allow old code using newQuery() to operate as normal while alleviating this problem.

Environment safe settings

There is an excellent setting for seedEnvironments. It occurred to me there might be some !environments for certain features like "fresh". I will put this in my future submissions, but if someone gets to it before me, that is also awesome.

Error with H2 database

Hi,

I'm testing h2 database in a ColdBox application (on Lucee) ...
I can connect to database with Quick (I've created a Model and an Handler) but I'm not able to run migrations...

this is the error displayed in CommandBox after migration init and migration creation

> migrate init
cfmigrations configured!

> * > CLI v5.2.1 > 04:25 PM > ~/coldbox-examples/testApp/ > testApp (1.0.0) > lucee 5.3.7 (running) >
> migrate up

ERROR (5.2.1+00295)                                                                                                                                                                                                                                                                                                       Schema "information_schema" not found; SQL statement:SELECT 1 FROM "information_schema"."tables" WHERE "table_name" = ? AND "table_schema" = ? [90079-172]   

MigrationService.cfc

    SELECT 1 FROM "information_schema"."tables" WHERE "table_name" = 'cfmigrations' AND "table_schema" = 'testAppDB'

Both Application.cfc and box.json use ENVIRONMENT settings:

# ColdBox Name and Environment
APPNAME=tastApp
ENVIRONMENT=development

DB_BUNDLEVERSION=1.3.172
DB_PORT=
DB_USER=sa
DB_BUNDLENAME=org.h2
DB_CLASS=org.h2.Driver
DB_HOST=127.0.0.1
DB_DRIVER=org.h2.Driver
DB_SCHEMA=appDB
DB_DATABASE=appDB
DB_CONNECTIONSTRING=jdbc:h2:/Users/elena/coldbox-examples/testApp/database/appDB;MODE=MySQL;
DB_PASSWORD=

h2 was downloaded from Lucee download page and saved in /lib folder (I've added dependencies in box.json)

this is the grammar settings:

"defaultGrammar":"AutoDiscover@qb"

What am I missing?
Kind regards,
Elena

testApp.zip

ACF11: MigrationsService.cfc throws java.lang.ClassCastException exception.

On bootstrap:

The CFML compiler encountered an unexpected java.lang.ClassCastException exception.

The reason for this was: coldfusion.compiler.ASTtagAttribute cannot be cast to coldfusion.compiler.ExprNode Occurred at:java.lang.ClassCastException:

coldfusion.compiler.ASTtagAttribute cannot be cast to coldfusion.compiler.ExprNode at coldfusion.compiler.ExprVisitor.insertCast(ExprVisitor.java:210) at coldfusion.compiler.ExprVisitor.call(ExprVisitor.java:610) at coldfusion.compiler.ExprVisitor.transform(ExprVisitor.java:83) at coldfusion.compiler.Treewalker.postorder(Treewalker.java:100) at coldfusion.compiler.Treewalker.postorder(Treewalker.java:82) at coldfusion.compiler.Treewalker.postorder(Treewalker.java:27) at coldfusion.compiler.Treewalker.postorder(Treewalker.java:27) at coldfusion.compiler.Treewalker.postorder(Treewalker.java:27) at coldfusion.compiler.NeoTranslator.parseAndTransform(NeoTranslator.java:459) at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:391) at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:160) at coldfusion.runtime.TemplateClassLoader$TemplateCache$1.fetch(TemplateClassLoader.java:467) at coldfusion.util.LruCache.get(LruCache.java:180) at col...
 
The error occurred in /app/inleague/modules/cfmigrations/models/MigrationService.cfc: line 60Called from /app/inleague/coldbox/system/core/util/Util.cfc: line 287Called from /app/inleague/coldbox/system/ioc/config/Mapping.cfc: line 604Called from /app/inleague/coldbox/system/ioc/config/Mapping.cfc: line 614Called from /app/inleague/coldbox/system/ioc/config/Binder.cfc: line 1013Called from /app/inleague/coldbox/system/ioc/config/Binder.cfc: line 1008Called from /app/inleague/coldbox/system/ioc/config/Binder.cfc: line 319Called from /app/inleague/coldbox/system/web/services/ModuleService.cfc: line 527Called from /app/inleague/coldbox/system/web/services/ModuleService.cfc: line 481Called from /app/inleague/coldbox/system/web/services/ModuleService.cfc: line 476Called from /app/inleague/coldbox/system/web/services/ModuleService.cfc: line 422Called from /app/inleague/coldbox/system/web/services/LoaderService.cfc: line 57Called from /app/inleague/coldbox/system/bootstrap.cfc: line 98Called from /app/inleague/Application.cfc: line 158Called from /app/inleague/coldbox/system/core/util/Util.cfc: line 287Called from /app/inleague/coldbox/system/ioc/config/Mapping.cfc: line 604Called from /app/inleague/coldbox/system/ioc/config/Mapping.cfc: line 614Called from /app/inleague/coldbox/system/ioc/config/Binder.cfc: line 1013Called from /app/inleague/coldbox/system/ioc/config/Binder.cfc: line 1008Called from /app/inleague/coldbox/system/ioc/config/Binder.cfc: line 319Called from /app/inleague/coldbox/system/web/services/ModuleService.cfc: line 527Called from /app/inleague/coldbox/system/web/services/ModuleService.cfc: line 481Called from /app/inleague/coldbox/system/web/services/ModuleService.cfc: line 476Called from /app/inleague/coldbox/system/web/services/ModuleService.cfc: line 422Called from /app/inleague/coldbox/system/web/services/LoaderService.cfc: line 57Called from /app/inleague/coldbox/system/bootstrap.cfc: line 98Called from /app/inleague/Application.cfc: line 158

Cannot find migration cfc paths in Windows environment

The latest version of Wirebox is unable to parse a full Windows file path due to a check in the injector that dumps out a blank string when passed anything with a : character. https://github.com/Ortus-Solutions/commandbox/blob/8d12ac9b13b7ff95af4cd1f7b5bac7a92785ddc6/src/cfml/system/wirebox/system/ioc/Injector.cfc#L542

The findAll() function at

public array function findAll() {
is using a directoryList call to retrieve a list of migration cfcs, and while it seems to attempt to generate a relative path at one point on line 97, when run inside CommandBox at least, the working directory is not the same and the resulting component path is invalid.

Wirebox then errors out when running migrate up.

testbox jquery issue

cfmigrations' test harness has the same missing jquery.js file that I've seen in a couple other Coldbox modules. Pushing a fix.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.