lastland / scala-forklift

Type-safe data migration tool for Slick, Git and beyond.

Scala-Forklift

Scala-Forklift helps manage and apply database migrations for your Scala project.

Write your migrations in plain SQL:

MyMigrations.migrations = MyMigrations.migrations :+ SqlMigration(1)(List(
  sqlu"""create table "users" ("id" INTEGER NOT NULL PRIMARY KEY,"first" VARCHAR NOT NULL,"last" VARCHAR NOT NULL)"""
))

Or type-safe Slick queries:

MyMigrations.migrations = MyMigrations.migrations :+ DBIOMigration(2)(
  DBIO.seq(Users ++= Seq(
    UsersRow(1, "Chris","Vogt"),
    UsersRow(2, "Yao","Li")
  )))

Or use slick-migration-api:

MyMigrations.migrations = MyMigrations.migrations :+ APIMigration(3)(
  TableMigration(Users).
    renameColumn(_.first, "firstname").
    renameColumn(_.last, "lastname"))

(Note: APIMigration is not supported in versions prior to v0.2.3)

Don't worry about keeping the Scala code and your database schema consistent. Our source code generator will have it generated for you.

Key Features:

• Supports for type-safe database migration with Slick and slick-migration-api.
• A source code generator to generate and manage Scala models from your database schemas.
• A tool to help you manage your dev db with git, with supports for branching and merging.
• High customizability.

How to Use

Scala-Forklift supports both Slick 3.1 and Slick 3.2. The latest versions of Scala-Forklift are given below:

Scala Version	Slick Version	SBT dependency
2.11.x	3.1.x	libraryDependencies += "com.liyaos" %% "scala-forklift-slick" % "0.2.3"
2.12.x, 2.11.x	3.2.x	libraryDependencies += "com.liyaos" %% "scala-forklift-slick" % "0.3.1"
2.13.x, 2.12.x	3.3.x	libraryDependencies += "com.liyaos" %% "scala-forklift-slick" % "0.3.2"

For tutorial and example code, please check example.

Here is also a wonderful tutorial written by Andreas Burkard and Julian Tragé.

Quick Start

You can use our start template on GitHub to quickly start a project with Scala-Forklift:

git clone https://github.com/lastland/scala-forklift-start-template.git

More Examples

• Demo with Play and Slick
• Demo with Quill

Known Issues

• The reset command may not correctly handle database schemas with foreign keys.

scala-forklift's Issues

Get the tests working on CI

[Help] Running multiple migratedb tool in parallel

Hello,

Recently, we've introduced scala-forklift in our cloud applications, that are running at kubernetes.

Some applications have replicas and this means that migratedb-tool (using forklift) runs (when application is restarted), in parallel, among other replicas to work on same database.

We have some issues trying to run migratedb-tool in parallel - it seams that it lacks support for parallel runs on same database.

There is a way to lock database during migration ?

Support Slick 3.3.0

Now that Slick 3.3.0 was released it would be nice to support it. Something especially interesting is the code gen of case classes with more than 22 elements link

Support Slick 3.1.0

Play framework evolutions

How is it possible to use forklift with Play framework evolutions?

Provide SQL migration via SQL file

I would like to use my SQL files that I have syntax checking on for plain SQL migrations rather than embedding them in sqlu interpolation.

This may already be possible, but I could not figure out how to turn a file resources into a SqlAction, and could not find a better venue than issues to pose this questions.

Problem using forklift with VCS

Mercurial doesn't support absolute symlinks, so I ignored all files in path migrations/src/.
But in this case I should also exclude MigrationSummary from MyMigrations.
Finally I cannot roll migrations on deploy, because all mechanism is broken.
Can you explain, why is necessary to store rolled migrations in MigrationSummary?

Make `APIMigration` the subtype of `slick.migration.api.Migration`

It is currently slick.migration.api.SqlMigration.

Add slick-migration-api in the slick3.2 version

nafg/slick-migration-api#10 is a prerequisite.

Fix the test failure in SQLiteMigrationTests

The case "up should deprecate object models of previous versions" fails to throw SQLException as expected.

The problem is caused by Slick 3.1.0 and SQLite. See more in: slick/slick#1402

Tests for cross Scala versions

Now we have cross build for Forklift. Unfortunately, there is no cross test. The reason is that part of Forklift's test is copying an example Scala project and run tests against it. The example project has its own build definition, thus a +test in Forklift will not be able to test different versions of the example project.

The solution I have in mind is to utilize parallelization in CircleCI. We need to have a script to change the Scala version in the build definition of the example project. Then on one specific machine, the script is executed before running tests.

Test SlickMigrationManager on other Slick drivers than H2

Right now the SlickMigrationManager is only tested on h2 database. We must test it on other Slick drivers (e.g. MySQL, Postgres) to prove it's solid.

The automatic tests that will be implemented for #7 will also need to include these tests.

Slick Drivers:

• H2Driver
• SQLiteDriver
• MySQLDriver
• PostgresDriver
• HsqldbDriver
• AccessDriver (will not support this since it's no long supported in Slick)
• DerbyDriver

Test SlickMigrationManager with migrations on more complicated scenarios

For example, with tables having foreign keys.

Supports for Casbah

Slick 3.4.1

Slick 3.4 is out for a while now, and forklift doesn't support it.

Cannot find JDBC Derby Driver with Derby 10.12.1.1

With Derby 10.12.1.1:

Exception encountered when attempting to run a suite with class name: com.liyaos.forklift.slick.tests.unittests.DerbyMigrationTests *** ABORTED ***
[info]   java.lang.NoClassDefFoundError: Could not initialize class org.apache.derby.jdbc.EmbeddedDriver
[info]   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[info]   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[info]   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info]   at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
[info]   at java.lang.Class.newInstance(Class.java:442)
[info]   at com.zaxxer.hikari.AbstractHikariConfig.setDriverClassName(AbstractHikariConfig.java:366)
[info]   at slick.jdbc.hikaricp.HikariCPJdbcDataSource$$anonfun$forConfig$12.apply(HikariCPJdbcDataSource.scala:28)
[info]   at slick.jdbc.hikaricp.HikariCPJdbcDataSource$$anonfun$forConfig$12.apply(HikariCPJdbcDataSource.scala:28)
[info]   at scala.Option.map(Option.scala:146)
[info]   at slick.jdbc.hikaricp.HikariCPJdbcDataSource$.forConfig(HikariCPJdbcDataSource.scala:28)
[info]   ...

However, it's fine with Derby 10.11.1.1. Not sure why yet.

Scala 2.13 Support

2.13 has been out for a while now

will first need #47 implemented, slick 3.3.2 has Scala 2.13 support

Scala 2.12 Support

Hi there. This library looks like exactly what we'd like to use to manage our Slick migrations. Is it possible to push a build for Scala 2.12?

Thank you very much!

Provide Scaladoc

Overriding of Slick Code generation

I would like to use Joda DateTimes in the Slick generated models. It supports this via overriding the SourceCodeGenerator, here's an example: https://github.com/tminglei/slick-pg/blob/master/examples/codegen-customization/codegen/src/main/scala/demo/CustomizedCodeGenerator.scala#L43

Is it possible to override/configure this?

Support Quil

Would be interesting to investigate Quil and perhaps support it in Scala-Forklift: https://github.com/getquill/quill

"build.scala" is deprecated

@lastland Hi,
thanks a lot, "Scala-Forklift" is really great!

I have one suggestion, the "build.scala" is deprecated now as far as I know.
I would suggest to only use the "build.sbt" for the template and the example. It also works with the sub-projects, I tried it already.
If you agree, I can make a commit for both of them.

Have a nice day!

Don't swallow exceptions silently in `mg new s`

https://github.com/lastland/scala-forklift/blob/develop/migrations/slick/src/main/scala/Commands.scala#L252

> mg new
[info] Running MyMigrations new
you must enter a proper parameter!

When the real problem was that the application.conf was not parseable. This is very confusing.

Packaging up migrations

Hi there.

We've been working to add forklift to a fairly new project recently, and for the most part it's been going well enough. I worked around the scala binary incompatibility by building my own JARs and making them available in the class path.

However, I'm now running into a new problem: we intend to ship our project in a docker container using sbt-native-packager, and we're discovering that (at least to my fairly-new-to-the-java-ecosystem understanding) this entire concept is incompatible with Forklift. Symlinks with absolute paths combined with the Summary object have been rather difficult obstacles for me to overcome.

Have you encountered this, or do you have any insight into how one can package up the code such that we'd be able to bundle up the migrations and run them against our production databases? For what it's worth, running the migrations against the production database from a local machine is not an option.

Thanks very much for your help. I'd be glad to answer any further questions to the best of my ability.

Add more tests

• Tests for the mg new command.
• Tests for the following command sequence: mg init, ~mg migrate, delete the db, ~mg migrate (maybe on SQLite alone would be sufficient).

Scala-Forklift 0.3.0 for Scala 2.11

Now some tests fail on Scala-Forklift 0.3.0 with Scala 2.11. See https://circleci.com/gh/lastland/scala-forklift/154

[Feature] maybeInit

atm forklift has to be manually initialized, please correct me if that is wrong.

there should be a maybeInit imo which only inits if needed (maybe using MTable.getTables?)

the current init fails if the table already exists

I could make a PR for this

Change License to Apache 2.0

@cvogt suggests that we may want to consider changing the license to Apache 2.0 (our current license is BSD).

For the major difference between two licenses, here's a comment from @fommil, and I quote:

Apache is definitely not more restrictive, it's safer. The primary difference is that nobody can contribute their patented work to the project without also giving a free licence to all future users.
(from cvogt/cbt#27)

For more information you can check his talk here: http://fommil.github.io/scalasphere16/

I would like to see if our other contributors (cc @kshepard @DerLump) agree with this change?

Supports for slick-migration-api

slick-migration-api: https://github.com/nafg/slick-migration-api

Tests for Slick migrations part

mg reset fails due to constraints

I have a number of tables with foreign key constraints. When I try to run mg reset it fails because the drop needs to either happen in reverse order of create or specify CASCADE. I'd say the former is better, since CASCADE would mean later drops might encounter missing tables that were already removed.

Of course, right now there is no tracking of creation order, unless each table was created as a separate migration, so not sure how this should be handled

SlickMigrationManager not very modular

I think an AbstractSlickMigrationManager would be useful, which takes dbConfig as parameter. That way I could inject / reuse my configuration (for example inside a PlayModule)

I could make a PR for this