weavejester / ragtime Goto Github PK
View Code? Open in Web Editor NEWDatabase-independent migration library
License: Eclipse Public License 1.0
Database-independent migration library
License: Eclipse Public License 1.0
There's a 0.3.7 version up on clojars. For example, https://clojars.org/ragtime/ragtime.sql.files/versions/0.3.7. I don't see any evidence of this version in this repository. Is the 0.3.7 version a mistake? I've encountered a problem with 0.3.7 that doesn't exist in 0.3.6 so I wonder if I should spend any time on it. Thanks.
The Leiningen plugin has a number of problems inherent to it:
Instead, we should prefer REPL based functions:
user=> (migrate)
Applying 20150612194314-create-table-foo
user=> (rollback)
Rolling back 20150612194314-create-table-foo
If we really want Leiningen commands, we can create aliases for these:
:aliases {"migrate" ["run" "-m" "user/migrate"]
"rollback" ["run" "-m" "user/rollback"]}
Is there an officially supported way to run migrations programmatically? We have been using ragtime.main#migrate
to do the trick which is easy enough. However, the method is totally undocumented and so I'm unsure whether we're relying on Ragtime implementation details to make this work, or if this method is a contract to the outside world.
If this is by design, would you accept a documentation-only pull request to clarify the arguments and usage?
(If you're curious why we do this, we don't store our configuration in Leiningen so the lein ragtime *
commands don't do much for us.)
I use this
(def config
{:datastore (jdbc/sql-database {:connection-uri (make-uri)})
:migrations (jdbc/load-resources "migrations")})
and when I run this
(config :migrations)
I got the nil seq
The logging-reporter only receives the name of the migration and the operation. This means that if I want to run migrations in parallel across many databases, the migrations logs would be mixed up between databases with no way of correlating which database a log message applies to. What are your thoughts about passing in the DataStore or some other database identifier to be used for logging purposes?
Hi,
I am getting the following error when running the migrate function:
Exception Conflict! Expected ragtime.repl$wrap_reporting$reify__719@33ee656f but ragtime.repl$wrap_reporting$reify__719@f3c1ebe was applied. ragtime.strategy/raise-error (strategy.clj:39)
Anything I can do to solve this?
Using the SQL files support, if for some reason a hidden file is created in the migrations directory, ragtime will try to use it in the migration resulting in weirdness. I ran into this when Emacs created a .#
temporary file for migration.
Hi,
I just figured out that this line:(re-pattern (str "([^\" File/separator "]*)" File/separator "?$"))
in https://github.com/weavejester/ragtime/blob/master/ragtime.jdbc/src/ragtime/jdbc.clj#L83
On windows File/separator returns "" which lets this function break with this exception:
foo.bar.user=> (re-pattern (str "([^\\]*)\\?$"))
PatternSyntaxException Unclosed character class near index 9
([^\]*)\?$
^ java.util.regex.Pattern.error (Pattern.java:1955)
I am not sure what you need this for, so I cannot provide a fix for now I guess. But I could test a snapshot or a fix if you want me to and have no access to windows.
Thanks,
Sven
I'm having issues running ragtime and using SQLite. I'm 99% sure its an issue on my end but I can't find any docs on how to achieve what I want. Alternatively I'm misusing ragtime, but again I can't confirm.
Below is the output I am receiving and my project.clj
#errors from cli
$ lein ragtime migrate
Exception in thread "main" java.sql.SQLException: No suitable driver found for jdbc:sqlite:db/database.db, compiling:(/private/var/folders/37/h3zw02kx1hv2kh
3gy7s7gh180000gn/T/form-init5077759087180200969.clj:1:124)
at clojure.lang.Compiler.load(Compiler.java:7142)
at clojure.lang.Compiler.loadFile(Compiler.java:7086)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:sqlite:db/database.db
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at clojure.java.jdbc.deprecated$get_connection.invoke(deprecated.clj:175)
at clojure.java.jdbc.deprecated$with_connection_STAR_.invoke(deprecated.clj:305)
at ragtime.sql.database.SqlDatabase.applied_migration_ids(database.clj:48)
at ragtime.core$applied_migrations.invoke(core.clj:31)
at ragtime.core$migrate_all.invoke(core.clj:57)
at ragtime.core$migrate_all.invoke(core.clj:53)
at ragtime.main$migrate.invoke(main.clj:32)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:626)
at ragtime.main$_main.doInvoke(main.clj:67)
at clojure.lang.RestFn.invoke(RestFn.java:551)
at clojure.lang.Var.invoke(Var.java:419)
at user$eval5.invoke(form-init5077759087180200969.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6703)
at clojure.lang.Compiler.eval(Compiler.java:6693)
at clojure.lang.Compiler.load(Compiler.java:7130)
... 11 more
;; Project.clj
(defproject qna "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:dependencies [[org.clojure/clojure "1.6.0"]
[compojure "1.1.6"]
[hiccup "1.0.5"]
[ring-server "0.3.1"]
[org.clojure/java.jdbc "0.3.6"]
[org.xerial/sqlite-jdbc "3.7.2"]
[ragtime "0.3.7"]]
:ragtime {:migrations ragtime.sql.files/migrations
:database "jdbc:sqlite:db/database.db"}
:plugins [[lein-ring "0.8.12"]
[ragtime/ragtime.lein "0.3.7"]]
:ring {:handler qna.handler/app
:init qna.handler/init
:destroy qna.handler/destroy}
:profiles
{:uberjar {:aot :all}
:production
{:ring
{:open-browser? false, :stacktraces? false, :auto-reload? false}}
:dev
{:dependencies [[ring-mock "0.1.5"] [ring/ring-devel "1.3.1"]]} })
What am I doing wrong?
@michaelklishin, @martintrojer, I wanted to write a few quick notes on Ragtime before opening it up to contribution.
I tend to treat the Clojure dependency as the minimum version. This ensures I won't accidentally use newer features that would make it incompatible with older versions. Currently Ragtime specifies a minimum of Clojure 1.3.0, but we could up that to 1.5.1.
Git commits follow the usual guidelines, i.e. imperative ("Add foo" rather than "Added foo"), short subject line to avoid truncation (under 72 characters), and the commits should present a readable history. Past commits have been in past tense, but I've been trying to move to imperative tense instead, as that's more the standard.
PRs are likely a good tool for collaborating, so I think we should prefer them over committing straight to master (I'm probably the worst offender for doing this).
Regarding future changes I've been considering...
Splitting up SQL files into executable statements turns out to be non-trivial for a lot of databases, so I've been considering an approach where there's an explicit delimiter comment (like ----
), or just converting the whole thing to an edn document in the form:
;; {:up [String], :down [String]}
{:up ["INSERT ..." "UPDATE ..."]
:down ["UPDATE ..." "DELETE ..."]}
Which while perhaps not as neat as a pure SQL file, has at least the advantage that the statements are unambiguously separated.
I've also been meaning to add a generator for creating new migrations.
These changes are not set in stone, but just things I've been considering based on the feedback I've had so far on the project. I'd like to get a second opinion on them, and I'd like to hear any pain points you've run into.
Okay, that's enough talk. I'm adding you two as contributors. :)
Sometimes it is just as important to know when no migrations need to be applied as when migrations are applied. Currently Ragtime silently does nothing which is fine in some situations, but not always. One possible route for this would be to add a new op
for logging-reporter
to handle. Perhaps a :no-op
to go with :up
and :down
?
What are your thoughts?
I was thinking of adding an additional command to the lein plug to create the migration files:
lein ragtime new "add-some-field-to-some-table"
Which would generate:
migrations/20130114154000-add-some-field-to-some-table.up.sql
migrations/20130114154000-add-some-field-to-some-table.down.sql
Would this be useful?
Happy to produce this patch if so.
Hi, I am trying your duct template with the +ragtime
flag and I am getting the following exception while starting the repl:
java.lang.ExceptionInInitializerError
at clojure.main.<clinit>(main.java:20)
Caused by: java.util.regex.PatternSyntaxException: Unclosed character class near i
([^\]*)\?$
^, compiling:(ragtime/jdbc.clj:83:1)
at clojure.lang.Compiler.load(Compiler.java:7239)
at clojure.lang.RT.loadResourceScript(RT.java:371)
at clojure.lang.RT.loadResourceScript(RT.java:362)
at clojure.lang.RT.load(RT.java:446)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5671)
at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
at clojure.core$load_lib.doInvoke(core.clj:5710)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$load_libs.doInvoke(core.clj:5749)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$require.doInvoke(core.clj:5832)
at clojure.lang.RestFn.invoke(RestFn.java:482)
at duct.component.ragtime$eval3668$loading__5340__auto____3669.invoke(ragt
at duct.component.ragtime$eval3668.invoke(ragtime.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6782)
at clojure.lang.Compiler.eval(Compiler.java:6771)
at clojure.lang.Compiler.load(Compiler.java:7227)
at clojure.lang.RT.loadResourceScript(RT.java:371)
at clojure.lang.RT.loadResourceScript(RT.java:362)
at clojure.lang.RT.load(RT.java:446)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5671)
at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
at clojure.core$load_lib.doInvoke(core.clj:5710)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$load_libs.doInvoke(core.clj:5749)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$require.doInvoke(core.clj:5832)
at clojure.lang.RestFn.invoke(RestFn.java:1289)
at user$eval3$loading__5340__auto____4.invoke(user.clj:1)
at user$eval3.invoke(user.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6782)
at clojure.lang.Compiler.eval(Compiler.java:6771)
at clojure.lang.Compiler.load(Compiler.java:7227)
at clojure.lang.RT.loadResourceScript(RT.java:371)
at clojure.lang.RT.loadResourceScript(RT.java:358)
at clojure.lang.RT.maybeLoadResourceScript(RT.java:354)
at clojure.lang.RT.doInit(RT.java:468)
at clojure.lang.RT.<clinit>(RT.java:330)
... 1 more
Caused by: java.util.regex.PatternSyntaxException: Unclosed character class near i
([^\]*)\?$
^
at java.util.regex.Pattern.error(Pattern.java:1955)
at java.util.regex.Pattern.clazz(Pattern.java:2548)
at java.util.regex.Pattern.sequence(Pattern.java:2063)
at java.util.regex.Pattern.expr(Pattern.java:1996)
at java.util.regex.Pattern.group0(Pattern.java:2905)
at java.util.regex.Pattern.sequence(Pattern.java:2051)
at java.util.regex.Pattern.expr(Pattern.java:1996)
at java.util.regex.Pattern.compile(Pattern.java:1696)
at java.util.regex.Pattern.<init>(Pattern.java:1351)
at java.util.regex.Pattern.compile(Pattern.java:1028)
at clojure.core$re_pattern.invoke(core.clj:4641)
at ragtime.jdbc$eval4485.invoke(jdbc.clj:83)
at clojure.lang.Compiler.eval(Compiler.java:6782)
at clojure.lang.Compiler.load(Compiler.java:7227)
... 50 more
Exception in thread "main" Exception in thread "Thread-3" clojure.lang.ExceptionIn
at clojure.core$ex_info.invoke(core.clj:4593)
at leiningen.core.eval$fn__5830.invoke(eval.clj:255)
at clojure.lang.MultiFn.invoke(MultiFn.java:233)
at leiningen.core.eval$eval_in_project.invoke(eval.clj:356)
at leiningen.repl$server$fn__11848.invoke(repl.clj:243)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:630)
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1868)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:634)
at clojure.core$bound_fn_STAR_$fn__4439.doInvoke(core.clj:1890)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.run(AFn.java:22)
at java.lang.Thread.run(Thread.java:745)
I suppose is related to this line:
83: (let [pattern (re-pattern (str "([^\\" File/separator "]*)\\" File/separator "?$"))]
Anyways I am in a Windows 8 machine using Java8.
Hello,
I tried to change the migrations table to one using a postgres schema:
{:datastore (jdbc/sql-database (System/getenv "CONN")
{:migrations-table "directory.ragtime_migrations"})
:migrations (jdbc/load-directory "migrations")})
And I get:
Exception in thread "main" org.postgresql.util.PSQLException: ERROR: relation "directory.ragtime_migrations" does not exist
If I change, for example to directory_ragtime_migrations
, everything works as expected.
If the ID for a migration is not specified, the ID should default to the filename plus a hash of the file's content. This will prevent migrations with different content but the same filename from confusing the migration stack.
It will also allow for automatic rollbacks during development, if the rebase strategy is chosen. Changing the file in any way will change its ID, and Ragtime will be able to rollback the old migration and then apply the new one.
This may require some way of performing a manual rollback and discard of a migration if the migration errors when it's rolled back.
Hey there. I'm working on a postgres-based application that makes use of stored procedures, and I have encountered what I believe is a limitation of ragtime's lexer. In particular, the lexer does not seem to be able to properly split on dollar-quoted bodies.
Take for example this migration file:
DROP FUNCTION IF EXISTS add_thing_to_partition(int, int);
CREATE OR REPLACE FUNCTION add_thing_to_partition(pid int, tid int)
RETURNS VOID AS
$$
DECLARE
existing_partition RECORD;
BEGIN
SELECT * into existing_partition FROM partition_thingss WHERE partition_id = pid AND thing_id = tid;
IF not found THEN
INSERT INTO partition_things values(pid, tid);
END IF;
END;
$$
LANGUAGE plpgsql;
As it currently stands, mark-sql-statement-ends ignores $$
and splits on ;
, causing the following broken SQL statements:
"DROP FUNCTION IF EXISTS add_thing_to_partition(int, int)"
"CREATE OR REPLACE FUNCTION add_thing_to_partition(pid int, bid int)\n RETURNS VOID AS\n$$ \\\nDECLARE \\\n existing_partition RECORD"
"\\\nBEGIN \\\n \\\n SELECT * into existing_partition FROM partition_things WHERE partition_id = pid AND thing_id = bid"
"\\\n \\\n IF not found THEN \\\n INSERT INTO partition_things values(pid, bid)"
"\\\n END IF"
"\\\n \\\nEND"
"\\\n$$\nLANGUAGE plpgsql"
Replacing $$
with '
corrects the problem, but this is not ideal. Single-quotes lack some of the capabilities for escaping and nesting that dollar-quotes allow for. For example, dollar-quotes can be named and nested (docs).
The repl version of migration takes a DataStore and a sequence of Migration. The core version requires index, which is not clear as to what it does. The repl verison provides a default implementation of this missing thing.
Could you add a similar version to core as repl?
Databases like Postgres support wrapping most DDL statements inside transactions. That would allow for transactional application of migrations, which among other things would reduce the hassle associated with initially writing and testing migrations.
For example,
CREATE TRIGGER TRG_FOO_BIER
BEFORE INSERT ON FOO
FOR EACH ROW
BEGIN
SELECT FOO_SEQ.NEXTVAL
INTO :new.id FROM DUAL
END
yields:
SQLException Missing IN or OUT parameter at index:: 1 oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow (OraclePreparedStatement.java:1821)
In the migration folder I have a file called 201412211016-add-recipes-table.up.sql with the contents:
CREATE TABLE test_table (id INT);
The version of ragtime I'm using is:
{:dependencies [... [ragtime/ragtime.sql.files "0.3.7"]]
:plugins [... [ragtime/ragtime.lein "0.3.7"]]}
When I want to run the migration I get No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil.
Stacktrace:
$ lein ragtime migrate
Applying 201412211016-add-recipes-table
Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil, compiling:(/private/var/folders/7t/gpzjxhts24d5wvqys1dfvrd80000gn/T/form-init6540913809318527864.clj:1:124)
at clojure.lang.Compiler.load(Compiler.java:7142)
at clojure.lang.Compiler.loadFile(Compiler.java:7086)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil
at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:544)
at clojure.java.io$fn__8628$G__8610__8635.invoke(io.clj:69)
at clojure.java.io$reader.doInvoke(io.clj:102)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:626)
at clojure.core$slurp.doInvoke(core.clj:6390)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at ragtime.sql.files$run_sql_fn$fn__364$fn__365$fn__366.invoke(files.clj:80)
at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:377)
at ragtime.sql.files$run_sql_fn$fn__364$fn__365.invoke(files.clj:77)
at clojure.java.jdbc.deprecated$with_connection_STAR_.invoke(deprecated.clj:307)
at ragtime.sql.files$run_sql_fn$fn__364.invoke(files.clj:76)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:624)
at ragtime.main$wrap_println$fn__298.doInvoke(main.clj:19)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at ragtime.core$migrate.invoke(core.clj:38)
at ragtime.core$migrate_all.invoke(core.clj:60)
at ragtime.core$migrate_all.invoke(core.clj:53)
at ragtime.main$migrate.invoke(main.clj:32)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:626)
at ragtime.main$_main.doInvoke(main.clj:67)
at clojure.lang.RestFn.invoke(RestFn.java:551)
at clojure.lang.Var.invoke(Var.java:419)
at user$eval5.invoke(form-init6540913809318527864.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6703)
at clojure.lang.Compiler.eval(Compiler.java:6693)
at clojure.lang.Compiler.load(Compiler.java:7130)
... 11 more
If migration SQL file contains character ' in comments migration looks like performed sucessfully (ragtime_migrations table is updated) but migration is not performed.
Observed this bug on PostgreSQL, using ragtime 0.3.6. I don't know if it occurs on other DBMSes.
Some SQL databases won't allow certain combinations of SQL to be sent in the same command, and separating SQL statements in a cross-platform way turns out to be an intractable problem. Therefore the format for migrations will be changed to make the commands explicit, via edn:
{:up ["CREATE TABLE foo (id int)"]
:down ["DROP TABLE foo"]}
edit: SQL migrations will also support the format outlined by @ragnard, i.e.
<id>.up.<n>.sql
<id>.down.<n>.sql
And the old SQL format:
<id>.up.sql
<id>.down.sql
Hello, just tried 0.5.0
from 0.4.2
and got the following exception. I'm using SQL files.
Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :applied-migration-ids of protocol: #'ragtime.protocols/DataStore found for class: nil, compiling:(/tmp/form-init8004482708322009980.clj:1:73)
at clojure.lang.Compiler.load(Compiler.java:7239)
at clojure.lang.Compiler.loadFile(Compiler.java:7165)
at clojure.main$load_script.invoke(main.clj:275)
at clojure.main$init_opt.invoke(main.clj:280)
at clojure.main$initialize.invoke(main.clj:308)
at clojure.main$null_opt.invoke(main.clj:343)
at clojure.main$main.doInvoke(main.clj:421)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: No implementation of method: :applied-migration-ids of protocol: #'ragtime.protocols/DataStore found for class: nil
at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:554)
at ragtime.protocols$eval542$fn__543$G__533__548.invoke(protocols.clj:12)
at ragtime.core$applied_migrations.invoke(core.clj:16)
at ragtime.core$migrate_all.invoke(core.clj:42)
at ragtime.repl$migrate.invoke(repl.clj:49)
at clanhr.absences_api.config.database$migrate.invoke(database.clj:10)
at clojure.lang.Var.invoke(Var.java:375)
at user$eval9.invoke(form-init8004482708322009980.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6782)
at clojure.lang.Compiler.eval(Compiler.java:6772)
at clojure.lang.Compiler.load(Compiler.java:7227)
... 11 more
I'm using Ragtime for some postgres/postgis migrations, and it's been great so far!
the postgis docs suggest using SELECT some_function
to add a spatial column, but trying to run a migration like this
CREATE TABLE parks (
park_id INTEGER,
park_name VARCHAR,
park_date DATE,
park_type VARCHAR
);
SELECT AddGeometryColumn('parks', 'park_geom', 128, 'MULTIPOLYGON', 2 );
(copy/pasted from the PostGIS docs). Turns up an error saying A result was returned when none was expected.
, which is understandable; select
ing during a migration normally wouldn't make much sense.
Is there a workaround for using side-effect functions in migrations?
Extra (perhaps not-so-useful) info:
For what it's worth, a shallow dive turned up the PERFORM
statement, which looks specific to PL/PgSQL scripts.
A weak attempt at combining this with ragtime
do $$
PERFORM AddGeometryColumn('parks', 'park_geom', 128, 'MULTIPOLYGON', 2 );
$$ LANGUAGE plpgsql;
was (perhaps expectedly) fruitless.
The current recommended approach is to split up each command in its own separate file. This can result in large amounts of files for each migration for many non-trivial cases. This incurs an overhead on the users having to manage multiple files with related commands, and can make it harder to see the entirety of the migration being applied.
The rationale behind this appears to be that using a heuristic to determine the split between commands is error prone. I agree with that, and my suggestion would be to prove an ability to use an optional separator token the way Yesql and Migratus do.
For example, Migratus uses --;;
user supplied tokens to split multiple migration commands in the same file. This removes the ambiguity of having to infer the split between the commands without necessitating the use of a file per command.
The feature could be entirely optional and the token could be a key in the config map, so if its not present then the default behavior would not be affected.
Just checking before I submit a pull request - would you accept a patch to add optional down migrations in ragtime.sql.files? In production apps, I never find the concept of rollbacks feasible but yet have to create empty down migrations to get Ragtime to run.
@weavejester I've ported a couple of fixes that prevented Joplin from working for one of my systems. How should we go about doing releases? I can do a signed release to Clojars if you'd prefer it.
@martintrojer thoughts?
Hi,
I have a postgres migrator with an insert statement. The migrator runs fine but the new rows are not committed. (i can see the table's sequence has incremented). I have turned on debug logging in the posgres driver and can see the statements being executed and committed. However, after inspecting the table, no new rows have been added. The sql creates a simple table with two columns in one migration then tries to insert a record in the next. Any clues to what could be the cause?
Thanks
ragtime.core/rollback-last
currently assumes there is always a migration to roll back:
(defn rollback-last
"Rollback the last n previous migrations from the database. If n is not
specified, only the very last migration is rolled back."
([db]
(rollback-last db 1))
([db n]
(doseq [migration (take n (reverse (applied-migrations db)))]
(rollback db migration))))
In case of a blank database, when you try to roll back and then migrate forward (e.g. as part of running a test suite), the migration local will be nil
and the rollback call will explode with an NPE.
In cases where you want to create a stored procedure (temporarily changing the delimiter), the method used to split sql strings in the following function breaks execution:
https://github.com/weavejester/ragtime/blob/master/ragtime.sql.files/src/ragtime/sql/files.clj#L51
Looking into a possible fix right now (perhaps utilizing jdbc's batch statement execution directly); might take me a bit as I'm new to clojure/leiningen; just wanted to post this issue here in the meantime.
Example sql statement:
CREATE OR REPLACE FUNCTION increment(i integer) RETURNS integer AS $$
BEGIN
RETURN i + 1;
END;
$$ LANGUAGE plpgsql;
Obviously, to run lein ragtime migrate
I have to install leiningen on production server. Is it the recommended way? Or call ragtime programmatically as suggested by #13?
When applied migration ids are present but @defined-migrations is not, the result is
(map {} [1])
;= (nil)
This is an edge case with Joplin which has a feature that migrates down first, then up. I'm afraid Joplin cannot work around this, so the only
solution I see is to filter out nils in Ragtime.
A PR is coming.
There doesn't seem to be a way to use any of the alternative strategies with the lein plugin. I'd have thought I'd be able to do something like
lein ragtime migrate -s apply-new
...but no dice. A quick look at the source code confirms that this option does not seem to be exposed.
Currently it seems that you have to migrate before your rollback in the same JVM session. This is because of the use of the "defined-migration" atom, which stores applied migrations, and then filters the applied migrations returned from the Migratable.
The culprit is the function "applied-migrations". Perhaps I'm missing something but I don't see the utility in this. If the Migratable interface returns applied migrations what is the purpose of filtering again on an in memory map? It seems like an odd choice considering any relevant info should come from a database.
I've recently started a new project (Luminus based, nothing added) and added ragtime 0.3.7
to perform migrations. This worked (past tense see later) locally as expected however when I deploy to Heroku and run heroku run "lein ragtime migrate"
it fails with the following error,
Exception in thread "main" java.lang.RuntimeException: No such var: sql/with-connection, compiling:(ragtime/sql/database.clj:22:3)
I suspect this is related to #23 but I haven't been able to find a solution. In a bid to diagnose the issue locally I blew away my ~/.m2
directory and ran the command locally and now I'm getting the same issue.
UPDATE It seems I can workaround the issue both locally and on heroku by issuing
lein do clean, ragtime migrate
However I'm not sure if this is just covering up a dependency problem so thought it worth mentioning.
Current Ragtime versions use a connection string to determine how to connect to the database. This limits customisability (for example #49), and has little benefit besides conciseness.
Our database has a password, which for security reasons I don't want to store raw in project.clj
or in an environment variable. What I really want is to be able to set the DB password with arbitrary Clojure code, but this is impossible in project.clj
since most libraries are unavailable. I couldn't find documentation about how to accomplish this—is it possible?
Thanks!
I had two migrations for user create and conversation create, but for some reason it only creates table only for users, the first migratio.
the file names are
201405180855-create-users-table.up.sql
201405180911-create-conversations-table.up.sql
These projects have the same dependencies, and the additional code for loading SQL from files is potentially small. Therefore these libraries will be merged into a "ragtime.jdbc" project.
I'm probably being dense, but I'm struggling to get migrations to execute in the correct order. here is my simple project:
;src/testragtime/core.clj
(ns testragtime.core
(:gen-class)
(:require [ragtime.jdbc :as jdbc]
[ragtime.repl :as repl]))
(def migration-config
{:database (jdbc/sql-database {:connection-uri "jdbc:postgresql://localhost:5432/testdb?user=nic&password=password"})
:migrations (jdbc/load-resources "migrations")})
(defn -main
"I don't do a whole lot ... yet."
[& args]
(println "Hello, World!"))
;project.clj
(defproject testragtime "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.6.0"]
[ragtime "0.4.1"]
[postgresql "9.3-1102.jdbc41"]]
:main ^:skip-aot testragtime.core
:target-path "target/%s"
:profiles {:uberjar {:aot :all}})
-- resources/migrations/001-thing.up.sql
CREATE TABLE thing (id int);
-- resources/migrations/001-thing.down.sql
DROP TABLE thing;
-- resources/migrations/002-stuff.up.sql
ALTER TABLE thing ADD stuff date;
-- resources/migrations/002-stuff.down.sql
ALTER TABLE thing DROP stuff;
-- resources/migrations/003-misc.up.sql
CREATE TABLE misc (id int);
-- resources/migrations/003-misc.down.sql
DROP TABLE misc;
Then in the repl:
testragtime.core=> migration-config
{:database #ragtime.jdbc.SqlDatabase{:db-spec {:connection-uri "jdbc:postgresql://localhost:5432/testdb?user=nic&password=nic"}, :migrations-table "ragtime_migrations"}, :migrations ({:id "003-misc", :up #<jdbc$sql_migration$fn__739 ragtime.jdbc$sql_migration$fn__739@61f4bdad>, :down #<jdbc$sql_migration$fn__741 ragtime.jdbc$sql_migration$fn__741@15e04bdb>} {:id "002-stuff", :up #<jdbc$sql_migration$fn__739 ragtime.jdbc$sql_migration$fn__739@38942215>, :down #<jdbc$sql_migration$fn__741 ragtime.jdbc$sql_migration$fn__741@549adb8>} {:id "001-thing", :up #<jdbc$sql_migration$fn__739 ragtime.jdbc$sql_migration$fn__739@282c0dbe>, :down #<jdbc$sql_migration$fn__741 ragtime.jdbc$sql_migration$fn__741@643a498c>})}
testragtime.core=> (repl/migrate migration-config)
Applying 003-misc
Applying 002-stuff
PSQLException ERROR: relation "thing" does not exist org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse (QueryExecutorImpl.java:2198)
I would expect it to execute 001-things then 002-stuff then 003-misc, but it seems to be doing it in reverse order.
strangely if I rename 001-things..sql to 004-things..sql, it has no effect at all.
I'm clearly missing something here.
I am curious to know if you would be for or against having an additional (optional) field in the ragtime map to change the default directory that ragtime looks in for migrations?
I looked through the code and see that its currently set as a private symbol here https://github.com/weavejester/ragtime/blob/master/ragtime.sql.files/src/ragtime/sql/files.clj#L110
After digging through the code, it looks like the best long term option would be to have an option field that holds data to be passed to the migration method so that other migrators could use more than just a string. So i would do that.
Some commits are missing from github?
https://clojars.org/ragtime/ragtime.core says that latest version is 0.3.7 "Pushed by weavejester on Mar 31, 2014.".
Github has version 0.3.6 https://github.com/weavejester/ragtime/blob/master/project.clj#L1 Latest commit 2014-02-22T17:59:37Z
I do this:
(:require [ragtime.core :as rcore])
.....
(rcore/migrate config 20160104)
or
(rcore/rollback config 20160104)
And I got the result as follow:
java.lang.IllegalArgumentException: No implementation of method: :run-up! of protocol: #'ragtime.protocols/Migration found for class: java.lang.Long
or
java.lang.IllegalArgumentException: No implementation of method: :run-down! of protocol: #'ragtime.protocols/Migration found for class: java.lang.Long
Must I implement the :run-up! or :run-down! ?
When I run ragtime in development and on my CI, it works perfectly. I use database jdbc as follow:
jdbc:postgresql://localhost:5432/wheelcrowd?user=user
without a password.
however, when I do 'lein ragtime migrate' on production (heroku),
jdbc:postgresql://ec2-23-21-209-58.compute-1.amazonaws.com:5432/databasename?user=username&password=password
I get:
Copying 46 files to /app/lib
Applying 20121123100000-add-rating-table
Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil
at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:495)
at clojure.java.io$fn__7795$G__7790__7802.invoke(io.clj:63)
at clojure.java.io$reader.doInvoke(io.clj:96)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:602)
at clojure.core$slurp.doInvoke(core.clj:6024)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at ragtime.sql.files$run_sql_fn$fn__258$fn__259$fn__260.invoke(files.clj:73)
at clojure.java.jdbc$transaction_STAR_.invoke(jdbc.clj:372)
at ragtime.sql.files$run_sql_fn$fn__258$fn__259.invoke(files.clj:72)
at clojure.java.jdbc$with_connection_STAR_.invoke(jdbc.clj:302)
at ragtime.sql.files$run_sql_fn$fn__258.invoke(files.clj:71)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:600)
at ragtime.main$wrap_println$fn__42.doInvoke(main.clj:19)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at ragtime.core$migrate.invoke(core.clj:38)
at ragtime.core$migrate_all.invoke(core.clj:60)
at ragtime.core$migrate_all.invoke(core.clj:53)
at ragtime.main$migrate.invoke(main.clj:32)
I use leiningen 1.7.1 and clojure 1.3.0. Should I upgrade any or all of these for this to work?
I use the last stable version of ragtime (my whole project is https://github.com/elisehuard/wheelcrowd if that helps)
:dependencies [[ragtime "0.3.2"]]
:plugins [[ragtime/ragtime.lein "0.3.2"]]
We would like to use Ragtime in conjunction with H2. We can establish a connection, and Ragtime finds our migrations successfully, but we get an IllegalArgumentException
when we try to actually run the migration.
Here is a repl session showing what happens when we try to migrate:
https://gist.github.com/skatenerd/8119cd3c35b78b8a7066
Did ragtime break compatibility with H2, or are we migrating incorrectly?
This happens in 0.3.7 and not 0.3.6. I assume this is related to this commit: 7173605
Any clue? Thanks!
$ cat migrations/0001-create-extensions.up.sql
CREATE EXTENSION IF NOT EXISTS hstore;
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
$ lein with-profile ragtime ragtime migrate -d "jdbc:postgresql:redacted"
Applying 0001-create-extensions
org.postgresql.util.PSQLException: Too many update results were returned.
at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleCommandStatus(AbstractJdbc2Statement.java:2733)
at org.postgresql.core.v3.QueryExecutorImpl$1.handleCommandStatus(QueryExecutorImpl.java:452)
at org.postgresql.core.v3.QueryExecutorImpl.interpretCommandStatus(QueryExecutorImpl.java:2215)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1847)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2909)
at clojure.java.jdbc.deprecated$execute_batch.invoke(deprecated.clj:423)
at clojure.java.jdbc.deprecated$do_commands$fn__231.invoke(deprecated.clj:435)
at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:389)
at clojure.java.jdbc.deprecated$do_commands.doInvoke(deprecated.clj:434)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at ragtime.sql.files$run_sql_fn$fn__364$fn__365$fn__366.invoke(files.clj:80)
at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:377)
at ragtime.sql.files$run_sql_fn$fn__364$fn__365.invoke(files.clj:77)
at clojure.java.jdbc.deprecated$with_connection_STAR_.invoke(deprecated.clj:307)
at ragtime.sql.files$run_sql_fn$fn__364.invoke(files.clj:76)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:624)
at ragtime.main$wrap_println$fn__298.doInvoke(main.clj:19)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at ragtime.core$migrate.invoke(core.clj:38)
at ragtime.core$migrate_all.invoke(core.clj:60)
at ragtime.core$migrate_all.invoke(core.clj:53)
at ragtime.main$migrate.invoke(main.clj:32)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:626)
at ragtime.main$_main.doInvoke(main.clj:67)
at clojure.lang.RestFn.invoke(RestFn.java:703)
at clojure.lang.Var.invoke(Var.java:442)
at user$eval5$fn__7.invoke(form-init3619964908012902520.clj:1)
at user$eval5.invoke(form-init3619964908012902520.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6703)
at clojure.lang.Compiler.eval(Compiler.java:6693)
at clojure.lang.Compiler.load(Compiler.java:7130)
at clojure.lang.Compiler.loadFile(Compiler.java:7086)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Exception in thread "main" java.sql.BatchUpdateException: Batch entry 1 <unknown> was aborted. Call getNextException to see the cause., compiling:(/tmp/form-init3619964908012902520.clj:1:90)
at clojure.lang.Compiler.load(Compiler.java:7142)
at clojure.lang.Compiler.loadFile(Compiler.java:7086)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.sql.BatchUpdateException: Batch entry 1 <unknown> was aborted. Call getNextException to see the cause.
at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2762)
at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleCommandStatus(AbstractJdbc2Statement.java:2733)
at org.postgresql.core.v3.QueryExecutorImpl$1.handleCommandStatus(QueryExecutorImpl.java:452)
at org.postgresql.core.v3.QueryExecutorImpl.interpretCommandStatus(QueryExecutorImpl.java:2215)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1847)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2909)
at clojure.java.jdbc.deprecated$execute_batch.invoke(deprecated.clj:423)
at clojure.java.jdbc.deprecated$do_commands$fn__231.invoke(deprecated.clj:435)
at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:389)
at clojure.java.jdbc.deprecated$do_commands.doInvoke(deprecated.clj:434)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at ragtime.sql.files$run_sql_fn$fn__364$fn__365$fn__366.invoke(files.clj:80)
at clojure.java.jdbc.deprecated$transaction_STAR_.invoke(deprecated.clj:377)
at ragtime.sql.files$run_sql_fn$fn__364$fn__365.invoke(files.clj:77)
at clojure.java.jdbc.deprecated$with_connection_STAR_.invoke(deprecated.clj:307)
at ragtime.sql.files$run_sql_fn$fn__364.invoke(files.clj:76)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:624)
at ragtime.main$wrap_println$fn__298.doInvoke(main.clj:19)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at ragtime.core$migrate.invoke(core.clj:38)
at ragtime.core$migrate_all.invoke(core.clj:60)
at ragtime.core$migrate_all.invoke(core.clj:53)
at ragtime.main$migrate.invoke(main.clj:32)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:626)
at ragtime.main$_main.doInvoke(main.clj:67)
at clojure.lang.RestFn.invoke(RestFn.java:703)
at clojure.lang.Var.invoke(Var.java:442)
at user$eval5$fn__7.invoke(form-init3619964908012902520.clj:1)
at user$eval5.invoke(form-init3619964908012902520.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6703)
at clojure.lang.Compiler.eval(Compiler.java:6693)
at clojure.lang.Compiler.load(Compiler.java:7130)
... 11 more
Error encountered performing task 'ragtime' with profile(s): 'ragtime'
Suppressed exit
After the initial db creation all the migrations run cleanly. However if I am to either roll back one migration and roll forward (migrations that have nothing to do with this table) I get the following error:
Exception in thread "main" java.lang.Exception: Conflict! Expected {:id "20130530131310.add-share-recipient-table", :up #<main$wrap_println$fn__42 ragtime.main$wrap_println$fn__42@7f2ea1dd>, :down #<main$wrap_println$fn__42 ragtime.main$wrap_println$fn__42@4e84f566>} but {:id "20130619161301.share-key-unique-index", :up #<main$wrap_println$fn__42 ragtime.main$wrap_println$fn__42@7b0b23cf>, :down #<main$wrap_println$fn__42 ragtime.main$wrap_println$fn__42@185c2a25>} was applied.
The tables migrations in question are:
20130530131310.add-share-recipient-table.up
CREATE TABLE share_recipients (
id INT PRIMARY KEY AUTO_INCREMENT,
share_id INT,
share_key VARCHAR(50),
recipient VARCHAR(50),
FOREIGN KEY (share_id) REFERENCES shares
(id)
)
20130530131310.add-share-recipient-table.down
DROP TABLE share_recipients;
20130619161301.share-key-unique-index.up
ALTER TABLE share_recipients
ADD UNIQUE INDEX (share_key
);
20130619161301.share-key-unique-index.down
ALTER TABLE share_recipients
DROP INDEX share_key
;
ragtime doesn't work with latest clojure.java.jdbc 0.3.0-beta2, because of breaking API changes.
For example https://github.com/weavejester/ragtime/blob/master/ragtime.sql/src/ragtime/sql/database.clj uses with-connection
macro that no longer exists.
There isn't any "datetime" field type in postgres so sql/create-table doesn't succeed
Would it be possible and advisable to add namespaces for the migrations? I have a somewhat odd use case for them where I'm in the process of writing a CMS. One of the goals is to provide modular functionality that can be added and removed as needed. This causes a problem with migrations since the migrations would be in different files/jars and they could be added at different times. Namespacing the migrations (with a default namespace) seems to me to be a possible solution.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.