weavejester / ragtime

Database-independent migration library

Ragtime

Ragtime is a Clojure library for migrating structured data in a way that's database independent. It defines a common interface for expressing migrations, much like Ring defines a common interface for expressing web applications.

Installation

Add the following dependency to your deps.edn file:

dev.weavejester/ragtime {:mvn/version "0.9.4"}

Or to your Leiningen project file:

[dev.weavejester/ragtime "0.9.4"]

Overview

Ragtime needs three pieces of data to work:

1. A migratable data store
2. An ordered sequence of migrations
3. A strategy on how to deal with conflicts

A data store is an implementation of the DataStore protocol, and tells Ragtime how to record which migrations are applied to an arbitrary store of data, such as a database.

Since 0.5.0, migrations are implementations of the Migration protocol, which has three methods:

• id - returns a unique ID for the migration
• run-up! - applies the migration to a database
• run-down! - rolls back the migration in a database

Ragtime comes with a way of loading SQL migrations from files, and applying them to a SQL database.

Documentation

• Getting Started
• Upgrading from 0.3.x
• Upgrading from 0.4.x
• Concepts
• SQL Migrations
• Leiningen Integration
• API docs
• Third-party Libraries

License

Copyright © 2024 James Reeves

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version

ragtime's Issues

Exception while running migrations

Hi,

I am getting the following error when running the migrate function:

Exception Conflict! Expected ragtime.repl$wrap_reporting$reify__719@33ee656f but ragtime.repl$wrap_reporting$reify__719@f3c1ebe was applied.  ragtime.strategy/raise-error (strategy.clj:39)

Anything I can do to solve this?

postgres insert is rolled back

Hi,

I have a postgres migrator with an insert statement. The migrator runs fine but the new rows are not committed. (i can see the table's sequence has incremented). I have turned on debug logging in the posgres driver and can see the statements being executed and committed. However, after inspecting the table, no new rows have been added. The sql creates a simple table with two columns in one migration then tries to insert a record in the next. Any clues to what could be the cause?

Thanks

about the nil migrations

I use this

(def config
  {:datastore  (jdbc/sql-database {:connection-uri (make-uri)})
   :migrations (jdbc/load-resources "migrations")})

and when I run this

(config :migrations)

I got the nil seq

clojars has version 0.3.7, github has version 0.3.6

Some commits are missing from github?

https://clojars.org/ragtime/ragtime.core says that latest version is 0.3.7 "Pushed by weavejester on Mar 31, 2014.".

Github has version 0.3.6 https://github.com/weavejester/ragtime/blob/master/project.clj#L1 Latest commit 2014-02-22T17:59:37Z

How to deal with production env

Obviously, to run lein ragtime migrate I have to install leiningen on production server. Is it the recommended way? Or call ragtime programmatically as suggested by #13?

Remove ragtime.lein plugin

The Leiningen plugin has a number of problems inherent to it:

1. Database connection details cannot be easily reused from the environment or config files
2. If packaging as an uberjar, migrate and rollback commands are needed in the project itself
3. Creating a database-independent configuration results an overuse of function references
4. Leiningen is slow to start

Instead, we should prefer REPL based functions:

user=> (migrate)
Applying 20150612194314-create-table-foo
user=> (rollback)
Rolling back 20150612194314-create-table-foo

If we really want Leiningen commands, we can create aliases for these:

:aliases {"migrate"  ["run" "-m" "user/migrate"]
          "rollback" ["run" "-m" "user/rollback"]}

See issues #13, #15, #46 and #66.

Optional Down Migrations?

Just checking before I submit a pull request - would you accept a patch to add optional down migrations in ragtime.sql.files? In production apps, I never find the concept of rollbacks feasible but yet have to create empty down migrations to get Ragtime to run.

Add hash to migrations IDs by default

If the ID for a migration is not specified, the ID should default to the filename plus a hash of the file's content. This will prevent migrations with different content but the same filename from confusing the migration stack.

It will also allow for automatic rollbacks during development, if the rebase strategy is chosen. Changing the file in any way will change its ID, and Ragtime will be able to rollback the old migration and then apply the new one.

This may require some way of performing a manual rollback and discard of a migration if the migration errors when it's rolled back.

Add a new arity to logging-reporter to include the DataStore

The logging-reporter only receives the name of the migration and the operation. This means that if I want to run migrations in parallel across many databases, the migrations logs would be mixed up between databases with no way of correlating which database a log message applies to. What are your thoughts about passing in the DataStore or some other database identifier to be used for logging purposes?

I can't up or down single magration

I do this:

(:require  [ragtime.core :as rcore])
.....
(rcore/migrate config 20160104) 

or

(rcore/rollback config 20160104)

And I got the result as follow:

java.lang.IllegalArgumentException: No implementation of method: :run-up! of protocol: #'ragtime.protocols/Migration found for class: java.lang.Long

or

 java.lang.IllegalArgumentException: No implementation of method: :run-down! of protocol: #'ragtime.protocols/Migration found for class: java.lang.Long

Must I implement the :run-up! or :run-down! ?

error with sqlite

I'm having issues running ragtime and using SQLite. I'm 99% sure its an issue on my end but I can't find any docs on how to achieve what I want. Alternatively I'm misusing ragtime, but again I can't confirm.

Below is the output I am receiving and my project.clj

#errors from cli

$ lein ragtime migrate
Exception in thread "main" java.sql.SQLException: No suitable driver found for jdbc:sqlite:db/database.db, compiling:(/private/var/folders/37/h3zw02kx1hv2kh
3gy7s7gh180000gn/T/form-init5077759087180200969.clj:1:124)
        at clojure.lang.Compiler.load(Compiler.java:7142)
        at clojure.lang.Compiler.loadFile(Compiler.java:7086)
        at clojure.main$load_script.invoke(main.clj:274)
        at clojure.main$init_opt.invoke(main.clj:279)
        at clojure.main$initialize.invoke(main.clj:307)
        at clojure.main$null_opt.invoke(main.clj:342)
        at clojure.main$main.doInvoke(main.clj:420)
        at clojure.lang.RestFn.invoke(RestFn.java:421)
        at clojure.lang.Var.invoke(Var.java:383)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.Var.applyTo(Var.java:700)
        at clojure.main.main(main.java:37)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:sqlite:db/database.db
        at java.sql.DriverManager.getConnection(DriverManager.java:689)
        at java.sql.DriverManager.getConnection(DriverManager.java:270)
        at clojure.java.jdbc.deprecated$get_connection.invoke(deprecated.clj:175)
        at clojure.java.jdbc.deprecated$with_connection_STAR_.invoke(deprecated.clj:305)
        at ragtime.sql.database.SqlDatabase.applied_migration_ids(database.clj:48)
        at ragtime.core$applied_migrations.invoke(core.clj:31)
        at ragtime.core$migrate_all.invoke(core.clj:57)
        at ragtime.core$migrate_all.invoke(core.clj:53)
        at ragtime.main$migrate.invoke(main.clj:32)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invoke(core.clj:626)
        at ragtime.main$_main.doInvoke(main.clj:67)
        at clojure.lang.RestFn.invoke(RestFn.java:551)
        at clojure.lang.Var.invoke(Var.java:419)
        at user$eval5.invoke(form-init5077759087180200969.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:6703)
        at clojure.lang.Compiler.eval(Compiler.java:6693)
        at clojure.lang.Compiler.load(Compiler.java:7130)
        ... 11 more

;; Project.clj
(defproject qna "0.1.0-SNAPSHOT"                                                                                                                        
   :description "FIXME: write description"                                                                                                               
   :url "http://example.com/FIXME"                                                                                                                       
   :dependencies [[org.clojure/clojure "1.6.0"]                                                                                                          
                  [compojure "1.1.6"]                                                                                                                    
                  [hiccup "1.0.5"]                                                                                                                       
                  [ring-server "0.3.1"]                                                                                                                  

                  [org.clojure/java.jdbc "0.3.6"]                                                                                                        
                  [org.xerial/sqlite-jdbc "3.7.2"]                                                                                                       
                  [ragtime "0.3.7"]]                                                                                                                     
   :ragtime {:migrations ragtime.sql.files/migrations                                                                                                    
             :database "jdbc:sqlite:db/database.db"}                                                                                                     
   :plugins [[lein-ring "0.8.12"]                                                                                                                        
             [ragtime/ragtime.lein "0.3.7"]]                                                                                                             
   :ring {:handler qna.handler/app                                                                                                                       
          :init qna.handler/init                                                                                                                         
          :destroy qna.handler/destroy}                                                                                                                  
   :profiles                                                                                                                                             
   {:uberjar {:aot :all}                                                                                                                                 
    :production                                                                                                                                          
    {:ring                                                                                                                                               
     {:open-browser? false, :stacktraces? false, :auto-reload? false}}                                                                                   
    :dev                                                                                                                                                 
    {:dependencies [[ring-mock "0.1.5"] [ring/ring-devel "1.3.1"]]} }) 

• db/database.db exists
• migrations/ and sql files exist
• I can connect to the DB with code
• I've tried jdbc:sqlite:db/database.db and jdbc:sqlite://db/database.db for the database URL

What am I doing wrong?

Remove connection multimethod

Current Ragtime versions use a connection string to determine how to connect to the database. This limits customisability (for example #49), and has little benefit besides conciseness.

Statement execution breaks when creating stored procedures

In cases where you want to create a stored procedure (temporarily changing the delimiter), the method used to split sql strings in the following function breaks execution:
https://github.com/weavejester/ragtime/blob/master/ragtime.sql.files/src/ragtime/sql/files.clj#L51

Looking into a possible fix right now (perhaps utilizing jdbc's batch statement execution directly); might take me a bit as I'm new to clojure/leiningen; just wanted to post this issue here in the meantime.

Example sql statement:

CREATE OR REPLACE FUNCTION increment(i integer) RETURNS integer AS $$
        BEGIN
                RETURN i + 1;
        END;
$$ LANGUAGE plpgsql;

New release

@weavejester I've ported a couple of fixes that prevented Joplin from working for one of my systems. How should we go about doing releases? I can do a signed release to Clojars if you'd prefer it.

@martintrojer thoughts?

Using SELECT and side-effect functions in a migration

I'm using Ragtime for some postgres/postgis migrations, and it's been great so far!

the postgis docs suggest using SELECT some_function to add a spatial column, but trying to run a migration like this

CREATE TABLE parks (
  park_id    INTEGER,
  park_name  VARCHAR,
  park_date  DATE,
  park_type  VARCHAR
);
SELECT AddGeometryColumn('parks', 'park_geom', 128, 'MULTIPOLYGON', 2 );

(copy/pasted from the PostGIS docs). Turns up an error saying A result was returned when none was expected., which is understandable; selecting during a migration normally wouldn't make much sense.

Is there a workaround for using side-effect functions in migrations?

Extra (perhaps not-so-useful) info:

For what it's worth, a shallow dive turned up the PERFORM statement, which looks specific to PL/PgSQL scripts.

A weak attempt at combining this with ragtime

do $$
    PERFORM AddGeometryColumn('parks', 'park_geom', 128, 'MULTIPOLYGON', 2 );
$$ LANGUAGE plpgsql;

was (perhaps expectedly) fruitless.

Migrate Required before rollback.

Currently it seems that you have to migrate before your rollback in the same JVM session. This is because of the use of the "defined-migration" atom, which stores applied migrations, and then filters the applied migrations returned from the Migratable.
The culprit is the function "applied-migrations". Perhaps I'm missing something but I don't see the utility in this. If the Migratable interface returns applied migrations what is the purpose of filtering again on an in memory map? It seems like an odd choice considering any relevant info should come from a database.

SQL files based migrations does not skip hidden files

Using the SQL files support, if for some reason a hidden file is created in the migrations directory, ragtime will try to use it in the migration resulting in weirdness. I ran into this when Emacs created a .# temporary file for migration.

Problem with 0.3.7 and postgresql

This happens in 0.3.7 and not 0.3.6. I assume this is related to this commit: 7173605

Any clue? Thanks!

$ cat migrations/0001-create-extensions.up.sql 
CREATE EXTENSION IF NOT EXISTS hstore;
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
$ lein with-profile ragtime ragtime migrate -d "jdbc:postgresql:redacted"
Applying 0001-create-extensions
org.postgresql.util.PSQLException: Too many update results were returned.
    at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleCommandStatus(AbstractJdbc2Statement.java:2733)
    at org.postgresql.core.v3.QueryExecutorImpl$1.handleCommandStatus(QueryExecutorImpl.java:452)
    at org.postgresql.core.v3.QueryExecutorImpl.interpretCommandStatus(QueryExecutorImpl.java:2215)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1847)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2909)
    at clojure.java.jdbc.deprecated$execute_batch.invoke(deprecated.clj:423)
    at clojure.java.jdbc.deprecated$do_commands$fn__231.invoke(deprecated.clj:435)
    at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:389)
    at clojure.java.jdbc.deprecated$do_commands.doInvoke(deprecated.clj:434)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at ragtime.sql.files$run_sql_fn$fn__364$fn__365$fn__366.invoke(files.clj:80)
    at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:377)
    at ragtime.sql.files$run_sql_fn$fn__364$fn__365.invoke(files.clj:77)
    at clojure.java.jdbc.deprecated$with_connection_STAR_.invoke(deprecated.clj:307)
    at ragtime.sql.files$run_sql_fn$fn__364.invoke(files.clj:76)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invoke(core.clj:624)
    at ragtime.main$wrap_println$fn__298.doInvoke(main.clj:19)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at ragtime.core$migrate.invoke(core.clj:38)
    at ragtime.core$migrate_all.invoke(core.clj:60)
    at ragtime.core$migrate_all.invoke(core.clj:53)
    at ragtime.main$migrate.invoke(main.clj:32)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invoke(core.clj:626)
    at ragtime.main$_main.doInvoke(main.clj:67)
    at clojure.lang.RestFn.invoke(RestFn.java:703)
    at clojure.lang.Var.invoke(Var.java:442)
    at user$eval5$fn__7.invoke(form-init3619964908012902520.clj:1)
    at user$eval5.invoke(form-init3619964908012902520.clj:1)
    at clojure.lang.Compiler.eval(Compiler.java:6703)
    at clojure.lang.Compiler.eval(Compiler.java:6693)
    at clojure.lang.Compiler.load(Compiler.java:7130)
    at clojure.lang.Compiler.loadFile(Compiler.java:7086)
    at clojure.main$load_script.invoke(main.clj:274)
    at clojure.main$init_opt.invoke(main.clj:279)
    at clojure.main$initialize.invoke(main.clj:307)
    at clojure.main$null_opt.invoke(main.clj:342)
    at clojure.main$main.doInvoke(main.clj:420)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:383)
    at clojure.lang.AFn.applyToHelper(AFn.java:156)
    at clojure.lang.Var.applyTo(Var.java:700)
    at clojure.main.main(main.java:37)
Exception in thread "main" java.sql.BatchUpdateException: Batch entry 1 <unknown> was aborted.  Call getNextException to see the cause., compiling:(/tmp/form-init3619964908012902520.clj:1:90)
    at clojure.lang.Compiler.load(Compiler.java:7142)
    at clojure.lang.Compiler.loadFile(Compiler.java:7086)
    at clojure.main$load_script.invoke(main.clj:274)
    at clojure.main$init_opt.invoke(main.clj:279)
    at clojure.main$initialize.invoke(main.clj:307)
    at clojure.main$null_opt.invoke(main.clj:342)
    at clojure.main$main.doInvoke(main.clj:420)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:383)
    at clojure.lang.AFn.applyToHelper(AFn.java:156)
    at clojure.lang.Var.applyTo(Var.java:700)
    at clojure.main.main(main.java:37)
Caused by: java.sql.BatchUpdateException: Batch entry 1 <unknown> was aborted.  Call getNextException to see the cause.
    at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2762)
    at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleCommandStatus(AbstractJdbc2Statement.java:2733)
    at org.postgresql.core.v3.QueryExecutorImpl$1.handleCommandStatus(QueryExecutorImpl.java:452)
    at org.postgresql.core.v3.QueryExecutorImpl.interpretCommandStatus(QueryExecutorImpl.java:2215)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1847)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2909)
    at clojure.java.jdbc.deprecated$execute_batch.invoke(deprecated.clj:423)
    at clojure.java.jdbc.deprecated$do_commands$fn__231.invoke(deprecated.clj:435)
    at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:389)
    at clojure.java.jdbc.deprecated$do_commands.doInvoke(deprecated.clj:434)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at ragtime.sql.files$run_sql_fn$fn__364$fn__365$fn__366.invoke(files.clj:80)
    at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:377)
    at ragtime.sql.files$run_sql_fn$fn__364$fn__365.invoke(files.clj:77)
    at clojure.java.jdbc.deprecated$with_connection_STAR_.invoke(deprecated.clj:307)
    at ragtime.sql.files$run_sql_fn$fn__364.invoke(files.clj:76)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invoke(core.clj:624)
    at ragtime.main$wrap_println$fn__298.doInvoke(main.clj:19)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at ragtime.core$migrate.invoke(core.clj:38)
    at ragtime.core$migrate_all.invoke(core.clj:60)
    at ragtime.core$migrate_all.invoke(core.clj:53)
    at ragtime.main$migrate.invoke(main.clj:32)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invoke(core.clj:626)
    at ragtime.main$_main.doInvoke(main.clj:67)
    at clojure.lang.RestFn.invoke(RestFn.java:703)
    at clojure.lang.Var.invoke(Var.java:442)
    at user$eval5$fn__7.invoke(form-init3619964908012902520.clj:1)
    at user$eval5.invoke(form-init3619964908012902520.clj:1)
    at clojure.lang.Compiler.eval(Compiler.java:6703)
    at clojure.lang.Compiler.eval(Compiler.java:6693)
    at clojure.lang.Compiler.load(Compiler.java:7130)
    ... 11 more
Error encountered performing task 'ragtime' with profile(s): 'ragtime'
Suppressed exit

Understanding the ordering of migrations

I'm probably being dense, but I'm struggling to get migrations to execute in the correct order. here is my simple project:

;src/testragtime/core.clj
(ns testragtime.core
  (:gen-class)
  (:require [ragtime.jdbc :as jdbc]
            [ragtime.repl :as repl]))

(def migration-config
  {:database (jdbc/sql-database {:connection-uri "jdbc:postgresql://localhost:5432/testdb?user=nic&password=password"})
   :migrations (jdbc/load-resources "migrations")})

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (println "Hello, World!"))

;project.clj
(defproject testragtime "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [ragtime "0.4.1"]
                 [postgresql "9.3-1102.jdbc41"]]
  :main ^:skip-aot testragtime.core
  :target-path "target/%s"
  :profiles {:uberjar {:aot :all}})

-- resources/migrations/001-thing.up.sql
CREATE TABLE thing (id int);
-- resources/migrations/001-thing.down.sql
DROP TABLE thing;

-- resources/migrations/002-stuff.up.sql
ALTER TABLE thing ADD stuff date;
-- resources/migrations/002-stuff.down.sql
ALTER TABLE thing DROP stuff;

-- resources/migrations/003-misc.up.sql
CREATE TABLE misc (id int);
-- resources/migrations/003-misc.down.sql
DROP TABLE misc;

Then in the repl:

testragtime.core=> migration-config
{:database #ragtime.jdbc.SqlDatabase{:db-spec {:connection-uri "jdbc:postgresql://localhost:5432/testdb?user=nic&password=nic"}, :migrations-table "ragtime_migrations"}, :migrations ({:id "003-misc", :up #<jdbc$sql_migration$fn__739 ragtime.jdbc$sql_migration$fn__739@61f4bdad>, :down #<jdbc$sql_migration$fn__741 ragtime.jdbc$sql_migration$fn__741@15e04bdb>} {:id "002-stuff", :up #<jdbc$sql_migration$fn__739 ragtime.jdbc$sql_migration$fn__739@38942215>, :down #<jdbc$sql_migration$fn__741 ragtime.jdbc$sql_migration$fn__741@549adb8>} {:id "001-thing", :up #<jdbc$sql_migration$fn__739 ragtime.jdbc$sql_migration$fn__739@282c0dbe>, :down #<jdbc$sql_migration$fn__741 ragtime.jdbc$sql_migration$fn__741@643a498c>})}
testragtime.core=> (repl/migrate migration-config)
Applying 003-misc
Applying 002-stuff

PSQLException ERROR: relation "thing" does not exist  org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse (QueryExecutorImpl.java:2198)

I would expect it to execute 001-things then 002-stuff then 003-misc, but it seems to be doing it in reverse order.

strangely if I rename 001-things..sql to 004-things..sql, it has no effect at all.

I'm clearly missing something here.

Merge ragtime.sql and ragtime.sql.files projects

These projects have the same dependencies, and the additional code for loading SQL from files is potentially small. Therefore these libraries will be merged into a "ragtime.jdbc" project.

Migrations table with postgres schema

Hello,

I tried to change the migrations table to one using a postgres schema:

{:datastore  (jdbc/sql-database (System/getenv "CONN")
                   {:migrations-table "directory.ragtime_migrations"})
 :migrations (jdbc/load-directory "migrations")})

And I get:

Exception in thread "main" org.postgresql.util.PSQLException: ERROR: relation "directory.ragtime_migrations" does not exist

If I change, for example to directory_ragtime_migrations, everything works as expected.

Difficult to securely set DB password

Our database has a password, which for security reasons I don't want to store raw in project.clj or in an environment variable. What I really want is to be able to set the DB password with arbitrary Clojure code, but this is impossible in project.clj since most libraries are unavailable. I couldn't find documentation about how to accomplish this—is it possible?

Thanks!

namespace for migrations?

Would it be possible and advisable to add namespaces for the migrations? I have a somewhat odd use case for them where I'm in the process of writing a CMS. One of the goals is to provide modular functionality that can be added and removed as needed. This causes a problem with migrations since the migrations would be in different files/jars and they could be added at different times. Namespacing the migrations (with a default namespace) seems to me to be a possible solution.

No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil

In the migration folder I have a file called 201412211016-add-recipes-table.up.sql with the contents:

CREATE TABLE test_table (id INT);

The version of ragtime I'm using is:

{:dependencies [... [ragtime/ragtime.sql.files "0.3.7"]]
 :plugins [... [ragtime/ragtime.lein "0.3.7"]]}

When I want to run the migration I get No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil.

Stacktrace:

$ lein ragtime migrate
Applying 201412211016-add-recipes-table
Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil, compiling:(/private/var/folders/7t/gpzjxhts24d5wvqys1dfvrd80000gn/T/form-init6540913809318527864.clj:1:124)
    at clojure.lang.Compiler.load(Compiler.java:7142)
    at clojure.lang.Compiler.loadFile(Compiler.java:7086)
    at clojure.main$load_script.invoke(main.clj:274)
    at clojure.main$init_opt.invoke(main.clj:279)
    at clojure.main$initialize.invoke(main.clj:307)
    at clojure.main$null_opt.invoke(main.clj:342)
    at clojure.main$main.doInvoke(main.clj:420)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:383)
    at clojure.lang.AFn.applyToHelper(AFn.java:156)
    at clojure.lang.Var.applyTo(Var.java:700)
    at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil
    at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:544)
    at clojure.java.io$fn__8628$G__8610__8635.invoke(io.clj:69)
    at clojure.java.io$reader.doInvoke(io.clj:102)
    at clojure.lang.RestFn.invoke(RestFn.java:410)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at clojure.core$apply.invoke(core.clj:626)
    at clojure.core$slurp.doInvoke(core.clj:6390)
    at clojure.lang.RestFn.invoke(RestFn.java:410)
    at ragtime.sql.files$run_sql_fn$fn__364$fn__365$fn__366.invoke(files.clj:80)
    at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:377)
    at ragtime.sql.files$run_sql_fn$fn__364$fn__365.invoke(files.clj:77)
    at clojure.java.jdbc.deprecated$with_connection_STAR_.invoke(deprecated.clj:307)
    at ragtime.sql.files$run_sql_fn$fn__364.invoke(files.clj:76)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invoke(core.clj:624)
    at ragtime.main$wrap_println$fn__298.doInvoke(main.clj:19)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at ragtime.core$migrate.invoke(core.clj:38)
    at ragtime.core$migrate_all.invoke(core.clj:60)
    at ragtime.core$migrate_all.invoke(core.clj:53)
    at ragtime.main$migrate.invoke(main.clj:32)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invoke(core.clj:626)
    at ragtime.main$_main.doInvoke(main.clj:67)
    at clojure.lang.RestFn.invoke(RestFn.java:551)
    at clojure.lang.Var.invoke(Var.java:419)
    at user$eval5.invoke(form-init6540913809318527864.clj:1)
    at clojure.lang.Compiler.eval(Compiler.java:6703)
    at clojure.lang.Compiler.eval(Compiler.java:6693)
    at clojure.lang.Compiler.load(Compiler.java:7130)
    ... 11 more

Can't use alternate strategy with the lein plugin

There doesn't seem to be a way to use any of the alternative strategies with the lein plugin. I'd have thought I'd be able to do something like

lein ragtime migrate -s apply-new

...but no dice. A quick look at the source code confirms that this option does not seem to be exposed.

"lein ragtime migrate" does not migrate pending migrations

I had two migrations for user create and conversation create, but for some reason it only creates table only for users, the first migratio.

the file names are

201405180855-create-users-table.up.sql
201405180911-create-conversations-table.up.sql

Transaction support

Databases like Postgres support wrapping most DDL statements inside transactions. That would allow for transactional application of migrations, which among other things would reduce the hassle associated with initially writing and testing migrations.

SQL lexer doesn't understand dollar-quoted values

Hey there. I'm working on a postgres-based application that makes use of stored procedures, and I have encountered what I believe is a limitation of ragtime's lexer. In particular, the lexer does not seem to be able to properly split on dollar-quoted bodies.

Take for example this migration file:

DROP FUNCTION IF EXISTS add_thing_to_partition(int, int);

CREATE OR REPLACE FUNCTION add_thing_to_partition(pid int, tid int)
 RETURNS VOID AS
$$
DECLARE
  existing_partition   RECORD;
BEGIN

    SELECT * into existing_partition FROM partition_thingss WHERE partition_id = pid AND thing_id = tid;

    IF not found THEN
       INSERT INTO partition_things values(pid, tid);
    END IF;

END;
$$
LANGUAGE plpgsql;

As it currently stands, mark-sql-statement-ends ignores $$ and splits on ;, causing the following broken SQL statements:

• "DROP FUNCTION IF EXISTS add_thing_to_partition(int, int)"
• "CREATE OR REPLACE FUNCTION add_thing_to_partition(pid int, bid int)\n RETURNS VOID AS\n$$ \\\nDECLARE \\\n existing_partition RECORD"
• "\\\nBEGIN \\\n \\\n SELECT * into existing_partition FROM partition_things WHERE partition_id = pid AND thing_id = bid"
• "\\\n \\\n IF not found THEN \\\n INSERT INTO partition_things values(pid, bid)"
• "\\\n END IF"
• "\\\n \\\nEND"
• "\\\n$$\nLANGUAGE plpgsql"

---

Replacing $$ with ' corrects the problem, but this is not ideal. Single-quotes lack some of the capabilities for escaping and nesting that dollar-quotes allow for. For example, dollar-quotes can be named and nested (docs).

jdbc: pattern broken in windows

Hi,

I just figured out that this line:(re-pattern (str "([^\" File/separator "]*)" File/separator "?$"))
in https://github.com/weavejester/ragtime/blob/master/ragtime.jdbc/src/ragtime/jdbc.clj#L83

On windows File/separator returns "" which lets this function break with this exception:

foo.bar.user=> (re-pattern (str "([^\\]*)\\?$"))
PatternSyntaxException Unclosed character class near index 9
([^\]*)\?$
         ^  java.util.regex.Pattern.error (Pattern.java:1955)

I am not sure what you need this for, so I cannot provide a fix for now I guess. But I could test a snapshot or a fix if you want me to and have no access to windows.

Thanks,
Sven

Ragtime silenty fails when migrating sql files containing apostrophe in comment.

If migration SQL file contains character ' in comments migration looks like performed sucessfully (ragtime_migrations table is updated) but migration is not performed.
Observed this bug on PostgreSQL, using ragtime 0.3.6. I don't know if it occurs on other DBMSes.

ragtime.core/rollback-last should not assume there is a migration to roll back

ragtime.core/rollback-last currently assumes there is always a migration to roll back:

(defn rollback-last
  "Rollback the last n previous migrations from the database. If n is not
  specified, only the very last migration is rolled back."
  ([db]
     (rollback-last db 1))
  ([db n]
     (doseq [migration (take n (reverse (applied-migrations db)))]
         (rollback db migration))))

In case of a blank database, when you try to roll back and then migrate forward (e.g. as part of running a test suite), the migration local will be nil and the rollback call will explode with an NPE.

Version 0.3.7 on clojars?

There's a 0.3.7 version up on clojars. For example, https://clojars.org/ragtime/ragtime.sql.files/versions/0.3.7. I don't see any evidence of this version in this repository. Is the 0.3.7 version a mistake? I've encountered a problem with 0.3.7 that doesn't exist in 0.3.6 so I wonder if I should spend any time on it. Thanks.

Programmatic Migrations?

Is there an officially supported way to run migrations programmatically? We have been using ragtime.main#migrate to do the trick which is easy enough. However, the method is totally undocumented and so I'm unsure whether we're relying on Ragtime implementation details to make this work, or if this method is a contract to the outside world.

If this is by design, would you accept a documentation-only pull request to clarify the arguments and usage?

(If you're curious why we do this, we don't store our configuration in Leiningen so the lein ragtime * commands don't do much for us.)

Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil

When I run ragtime in development and on my CI, it works perfectly. I use database jdbc as follow:

jdbc:postgresql://localhost:5432/wheelcrowd?user=user

without a password.

however, when I do 'lein ragtime migrate' on production (heroku),

jdbc:postgresql://ec2-23-21-209-58.compute-1.amazonaws.com:5432/databasename?user=username&password=password

I get:

Copying 46 files to /app/lib
Applying 20121123100000-add-rating-table
Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil
at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:495)
at clojure.java.io$fn__7795$G__7790__7802.invoke(io.clj:63)
at clojure.java.io$reader.doInvoke(io.clj:96)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:602)
at clojure.core$slurp.doInvoke(core.clj:6024)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at ragtime.sql.files$run_sql_fn$fn__258$fn__259$fn__260.invoke(files.clj:73)
at clojure.java.jdbc$transaction_STAR_.invoke(jdbc.clj:372)
at ragtime.sql.files$run_sql_fn$fn__258$fn__259.invoke(files.clj:72)
at clojure.java.jdbc$with_connection_STAR_.invoke(jdbc.clj:302)
at ragtime.sql.files$run_sql_fn$fn__258.invoke(files.clj:71)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:600)
at ragtime.main$wrap_println$fn__42.doInvoke(main.clj:19)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at ragtime.core$migrate.invoke(core.clj:38)
at ragtime.core$migrate_all.invoke(core.clj:60)
at ragtime.core$migrate_all.invoke(core.clj:53)
at ragtime.main$migrate.invoke(main.clj:32)

I use leiningen 1.7.1 and clojure 1.3.0. Should I upgrade any or all of these for this to work?
I use the last stable version of ragtime (my whole project is https://github.com/elisehuard/wheelcrowd if that helps)

:dependencies [[ragtime "0.3.2"]]
:plugins [[ragtime/ragtime.lein "0.3.2"]]

ensure-migrations-table-exists isn't compatible with postgres

There isn't any "datetime" field type in postgres so sql/create-table doesn't succeed

Triggers are parsed as prepared statements lacking parameter.

For example,

CREATE TRIGGER TRG_FOO_BIER
BEFORE INSERT ON FOO
FOR EACH ROW
BEGIN
  SELECT FOO_SEQ.NEXTVAL
  INTO :new.id FROM DUAL
END​

yields:

SQLException Missing IN or OUT parameter at index:: 1  oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow (OraclePreparedStatement.java:1821​)

core/applied-migrations may produce a collection with nils

When applied migration ids are present but @defined-migrations is not, the result is

(map {} [1])
;= (nil)
This is an edge case with Joplin which has a feature that migrates down first, then up. I'm afraid Joplin cannot work around this, so the only
solution I see is to filter out nils in Ragtime.

A PR is coming.

New contributors

@michaelklishin, @martintrojer, I wanted to write a few quick notes on Ragtime before opening it up to contribution.

I tend to treat the Clojure dependency as the minimum version. This ensures I won't accidentally use newer features that would make it incompatible with older versions. Currently Ragtime specifies a minimum of Clojure 1.3.0, but we could up that to 1.5.1.

Git commits follow the usual guidelines, i.e. imperative ("Add foo" rather than "Added foo"), short subject line to avoid truncation (under 72 characters), and the commits should present a readable history. Past commits have been in past tense, but I've been trying to move to imperative tense instead, as that's more the standard.

PRs are likely a good tool for collaborating, so I think we should prefer them over committing straight to master (I'm probably the worst offender for doing this).

Regarding future changes I've been considering...

Splitting up SQL files into executable statements turns out to be non-trivial for a lot of databases, so I've been considering an approach where there's an explicit delimiter comment (like ----), or just converting the whole thing to an edn document in the form:

;; {:up [String], :down [String]}
{:up   ["INSERT ..." "UPDATE ..."]
 :down ["UPDATE ..." "DELETE ..."]}

Which while perhaps not as neat as a pure SQL file, has at least the advantage that the statements are unambiguously separated.

I've also been meaning to add a generator for creating new migrations.

These changes are not set in stone, but just things I've been considering based on the feedback I've had so far on the project. I'd like to get a second opinion on them, and I'd like to hear any pain points you've run into.

Okay, that's enough talk. I'm adding you two as contributors. :)

ragtime.sql.database doesn't work with latest clojure.java.jdbc

ragtime doesn't work with latest clojure.java.jdbc 0.3.0-beta2, because of breaking API changes.

For example https://github.com/weavejester/ragtime/blob/master/ragtime.sql/src/ragtime/sql/database.clj uses with-connection macro that no longer exists.

But migrating 0.5.0

Hello, just tried 0.5.0 from 0.4.2and got the following exception. I'm using SQL files.

Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :applied-migration-ids of protocol: #'ragtime.protocols/DataStore found for class: nil, compiling:(/tmp/form-init8004482708322009980.clj:1:73)
    at clojure.lang.Compiler.load(Compiler.java:7239)
    at clojure.lang.Compiler.loadFile(Compiler.java:7165)
    at clojure.main$load_script.invoke(main.clj:275)
    at clojure.main$init_opt.invoke(main.clj:280)
    at clojure.main$initialize.invoke(main.clj:308)
    at clojure.main$null_opt.invoke(main.clj:343)
    at clojure.main$main.doInvoke(main.clj:421)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:383)
    at clojure.lang.AFn.applyToHelper(AFn.java:156)
    at clojure.lang.Var.applyTo(Var.java:700)
    at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: No implementation of method: :applied-migration-ids of protocol: #'ragtime.protocols/DataStore found for class: nil
    at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:554)
    at ragtime.protocols$eval542$fn__543$G__533__548.invoke(protocols.clj:12)
    at ragtime.core$applied_migrations.invoke(core.clj:16)
    at ragtime.core$migrate_all.invoke(core.clj:42)
    at ragtime.repl$migrate.invoke(repl.clj:49)
    at clanhr.absences_api.config.database$migrate.invoke(database.clj:10)
    at clojure.lang.Var.invoke(Var.java:375)
    at user$eval9.invoke(form-init8004482708322009980.clj:1)
    at clojure.lang.Compiler.eval(Compiler.java:6782)
    at clojure.lang.Compiler.eval(Compiler.java:6772)
    at clojure.lang.Compiler.load(Compiler.java:7227)
    ... 11 more

Compilation Error (Windows)

Hi, I am trying your duct template with the +ragtime flag and I am getting the following exception while starting the repl:

java.lang.ExceptionInInitializerError
        at clojure.main.<clinit>(main.java:20)
Caused by: java.util.regex.PatternSyntaxException: Unclosed character class near i
([^\]*)\?$
         ^, compiling:(ragtime/jdbc.clj:83:1)
        at clojure.lang.Compiler.load(Compiler.java:7239)
        at clojure.lang.RT.loadResourceScript(RT.java:371)
        at clojure.lang.RT.loadResourceScript(RT.java:362)
        at clojure.lang.RT.load(RT.java:446)
        at clojure.lang.RT.load(RT.java:412)
        at clojure.core$load$fn__5448.invoke(core.clj:5866)
        at clojure.core$load.doInvoke(core.clj:5865)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5671)
        at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
        at clojure.core$load_lib.doInvoke(core.clj:5710)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$load_libs.doInvoke(core.clj:5749)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$require.doInvoke(core.clj:5832)
        at clojure.lang.RestFn.invoke(RestFn.java:482)
        at duct.component.ragtime$eval3668$loading__5340__auto____3669.invoke(ragt
        at duct.component.ragtime$eval3668.invoke(ragtime.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:6782)
        at clojure.lang.Compiler.eval(Compiler.java:6771)
        at clojure.lang.Compiler.load(Compiler.java:7227)
        at clojure.lang.RT.loadResourceScript(RT.java:371)
        at clojure.lang.RT.loadResourceScript(RT.java:362)
        at clojure.lang.RT.load(RT.java:446)
        at clojure.lang.RT.load(RT.java:412)
        at clojure.core$load$fn__5448.invoke(core.clj:5866)
        at clojure.core$load.doInvoke(core.clj:5865)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invoke(core.clj:5671)
        at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
        at clojure.core$load_lib.doInvoke(core.clj:5710)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$load_libs.doInvoke(core.clj:5749)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invoke(core.clj:632)
        at clojure.core$require.doInvoke(core.clj:5832)
        at clojure.lang.RestFn.invoke(RestFn.java:1289)
        at user$eval3$loading__5340__auto____4.invoke(user.clj:1)
        at user$eval3.invoke(user.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:6782)
        at clojure.lang.Compiler.eval(Compiler.java:6771)
        at clojure.lang.Compiler.load(Compiler.java:7227)
        at clojure.lang.RT.loadResourceScript(RT.java:371)
        at clojure.lang.RT.loadResourceScript(RT.java:358)
        at clojure.lang.RT.maybeLoadResourceScript(RT.java:354)
        at clojure.lang.RT.doInit(RT.java:468)
        at clojure.lang.RT.<clinit>(RT.java:330)
        ... 1 more
Caused by: java.util.regex.PatternSyntaxException: Unclosed character class near i
([^\]*)\?$
         ^
        at java.util.regex.Pattern.error(Pattern.java:1955)
        at java.util.regex.Pattern.clazz(Pattern.java:2548)
        at java.util.regex.Pattern.sequence(Pattern.java:2063)
        at java.util.regex.Pattern.expr(Pattern.java:1996)
        at java.util.regex.Pattern.group0(Pattern.java:2905)
        at java.util.regex.Pattern.sequence(Pattern.java:2051)
        at java.util.regex.Pattern.expr(Pattern.java:1996)
        at java.util.regex.Pattern.compile(Pattern.java:1696)
        at java.util.regex.Pattern.<init>(Pattern.java:1351)
        at java.util.regex.Pattern.compile(Pattern.java:1028)
        at clojure.core$re_pattern.invoke(core.clj:4641)
        at ragtime.jdbc$eval4485.invoke(jdbc.clj:83)
        at clojure.lang.Compiler.eval(Compiler.java:6782)
        at clojure.lang.Compiler.load(Compiler.java:7227)
        ... 50 more
Exception in thread "main" Exception in thread "Thread-3" clojure.lang.ExceptionIn
        at clojure.core$ex_info.invoke(core.clj:4593)
        at leiningen.core.eval$fn__5830.invoke(eval.clj:255)
        at clojure.lang.MultiFn.invoke(MultiFn.java:233)
        at leiningen.core.eval$eval_in_project.invoke(eval.clj:356)
        at leiningen.repl$server$fn__11848.invoke(repl.clj:243)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invoke(core.clj:630)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1868)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invoke(core.clj:634)
        at clojure.core$bound_fn_STAR_$fn__4439.doInvoke(core.clj:1890)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.lang.Thread.run(Thread.java:745)

I suppose is related to this line:

83: (let [pattern (re-pattern (str "([^\\" File/separator "]*)\\" File/separator "?$"))]

Anyways I am in a Windows 8 machine using Java8.

Add function to core that migrates like repl

The repl version of migration takes a DataStore and a sequence of Migration. The core version requires index, which is not clear as to what it does. The repl verison provides a default implementation of this missing thing.
Could you add a similar version to core as repl?

Issues with Ragtime and Heroku

I've recently started a new project (Luminus based, nothing added) and added ragtime 0.3.7 to perform migrations. This worked (past tense see later) locally as expected however when I deploy to Heroku and run heroku run "lein ragtime migrate" it fails with the following error,

Exception in thread "main" java.lang.RuntimeException: No such var: sql/with-connection, compiling:(ragtime/sql/database.clj:22:3)

I suspect this is related to #23 but I haven't been able to find a solution. In a bid to diagnose the issue locally I blew away my ~/.m2 directory and ran the command locally and now I'm getting the same issue.

UPDATE It seems I can workaround the issue both locally and on heroku by issuing

lein do clean, ragtime migrate

However I'm not sure if this is just covering up a dependency problem so thought it worth mentioning.

Allow SQL migrations to take multiple commands

Some SQL databases won't allow certain combinations of SQL to be sent in the same command, and separating SQL statements in a cross-platform way turns out to be an intractable problem. Therefore the format for migrations will be changed to make the commands explicit, via edn:

{:up   ["CREATE TABLE foo (id int)"]
 :down ["DROP TABLE foo"]}

edit: SQL migrations will also support the format outlined by @ragnard, i.e.

<id>.up.<n>.sql
<id>.down.<n>.sql

And the old SQL format:

<id>.up.sql
<id>.down.sql

See issues #40, #38 and #17.

feature request for mutiple migrations per file

The current recommended approach is to split up each command in its own separate file. This can result in large amounts of files for each migration for many non-trivial cases. This incurs an overhead on the users having to manage multiple files with related commands, and can make it harder to see the entirety of the migration being applied.

The rationale behind this appears to be that using a heuristic to determine the split between commands is error prone. I agree with that, and my suggestion would be to prove an ability to use an optional separator token the way Yesql and Migratus do.

For example, Migratus uses --;; user supplied tokens to split multiple migration commands in the same file. This removes the ambiguity of having to infer the split between the commands without necessitating the use of a file per command.

The feature could be entirely optional and the token could be a key in the config map, so if its not present then the default behavior would not be affected.

Adding a notification when no migrations need to be applied

Sometimes it is just as important to know when no migrations need to be applied as when migrations are applied. Currently Ragtime silently does nothing which is fine in some situations, but not always. One possible route for this would be to add a new op for logging-reporter to handle. Perhaps a :no-op to go with :up and :down?

What are your thoughts?

Lein plugin / generate migration

I was thinking of adding an additional command to the lein plug to create the migration files:

lein ragtime new "add-some-field-to-some-table"

Which would generate:

migrations/20130114154000-add-some-field-to-some-table.up.sql
migrations/20130114154000-add-some-field-to-some-table.down.sql

Would this be useful?

Happy to produce this patch if so.

Alternate 'migrations' directory

I am curious to know if you would be for or against having an additional (optional) field in the ragtime map to change the default directory that ragtime looks in for migrations?

I looked through the code and see that its currently set as a private symbol here https://github.com/weavejester/ragtime/blob/master/ragtime.sql.files/src/ragtime/sql/files.clj#L110

After digging through the code, it looks like the best long term option would be to have an option field that holds data to be passed to the migration method so that other migrators could use more than just a string. So i would do that.

Running migrations with H2

We would like to use Ragtime in conjunction with H2. We can establish a connection, and Ragtime finds our migrations successfully, but we get an IllegalArgumentException when we try to actually run the migration.

Here is a repl session showing what happens when we try to migrate:
https://gist.github.com/skatenerd/8119cd3c35b78b8a7066

Did ragtime break compatibility with H2, or are we migrating incorrectly?

Exception in thread "main" java.lang.Exception: Conflict! Where no conflict

After the initial db creation all the migrations run cleanly. However if I am to either roll back one migration and roll forward (migrations that have nothing to do with this table) I get the following error:

Exception in thread "main" java.lang.Exception: Conflict! Expected {:id "20130530131310.add-share-recipient-table", :up #<main$wrap_println$fn__42 ragtime.main$wrap_println$fn__42@7f2ea1dd>, :down #<main$wrap_println$fn__42 ragtime.main$wrap_println$fn__42@4e84f566>} but {:id "20130619161301.share-key-unique-index", :up #<main$wrap_println$fn__42 ragtime.main$wrap_println$fn__42@7b0b23cf>, :down #<main$wrap_println$fn__42 ragtime.main$wrap_println$fn__42@185c2a25>} was applied.

The tables migrations in question are:

20130530131310.add-share-recipient-table.up
CREATE TABLE share_recipients (
id INT PRIMARY KEY AUTO_INCREMENT,
share_id INT,
share_key VARCHAR(50),
recipient VARCHAR(50),
FOREIGN KEY (share_id) REFERENCES shares(id)
)

20130530131310.add-share-recipient-table.down
DROP TABLE share_recipients;

20130619161301.share-key-unique-index.up
ALTER TABLE share_recipients ADD UNIQUE INDEX (share_key);

20130619161301.share-key-unique-index.down
ALTER TABLE share_recipients DROP INDEX share_key;