Comments (16)
Glad this was helpful @totallymike and thank you for sharing your solution! (Forked it here for the record https://gist.github.com/cvogt/f94c579ff8dad164d61a2be238f7e2d7)
Don't hesitate to reach out with further questions or experience reports, etc.
from scala-forklift.
@totallymike Thanks very much for sharing your solution! I will try to add it to Forklift.
from scala-forklift.
Hi,
Thanks for reporting this! This is indeed a problem since Forklift requires changing the code (it changes the generated code, which application code depend on). The best way I can think of is to run the source code with sbt on the server (can be automated by using git hooks?). However, I don't have an easy solution if that is not an option. Perhaps we can come up with some way to support packaging, but I'm not sure if we can support that in near future.
@cvogt I wonder if you would have any suggestions?
from scala-forklift.
I haven't followed forklifts details well enough to have a super informed opinion about this, but I believe that is is probably a very good idea for Forklift to document a suggested way of handling production deployments (and support all necessary steps). Here is one way to do it, which could server as an initial draft:
- Developer writes migration
- Developer applies migration to his local dev db
- Forklift updates local Scala schemas
- User compiles and tests user code against new schema and fixes errors.
- Optional: User reverts migration and also tests new code against old schema.
- User commits migration code, changes to code base and newly generated schema code.
- User pushes the commits, which triggers integration server to act (circleci or whatever).
- Integration server compiles the code, bundles it as a jar, tests it against a database with the old schema before the migration (can be a mirror of production if tests are read-only).
- Integration server applies the migration against a test database and tests the code against this database with the new schema, too.
- On success, integration server deploys new code and triggers the production database migration. The easiest would be running the migration on the integration server via sbt. If that's really impossible for some reasons (e.g. migration duration), it might be easier to invoke the migration via the bundled jar, which Forklift would have to allow and require the migrations to be in bundles the jar.
Variations:
- Instead of committing generated code, you can let the deployment server generate it against a test database.
- Integration server can deploy to a staging environment first. Only after deploy to production.
- Integration server can first migrate and deploy afterwards, but that requires the old code to work on the new schema, which also needs to be tested.
- If migrating the production database fails for some reason the previous version of the code should probably be re-deployed.
Happy to discuss this further with you guys and flesh out the above to something clear and easily implementable which can be added to the documentation.
from scala-forklift.
@cvogt That looks great, though I'm not sure why there's a step such that "integration server compiles the code, bundles it as a jar, tests it against a database with the old schema before the migration"?
from scala-forklift.
Somebody got to produce the class files, package them into a deployable format (e.g. a jar) and deploy them to the production server. A lot of people work in the way that they have a continuous integration server setup for this, that compiles the code, packages it into the jar and deploys it (i.e. copies it to the production server in some way), plus also runs the tests before or after packaging.
Not sure this answers your question?
from scala-forklift.
@totallymike Just think about it a little bit more. Maybe modifying code is not required. We only need a way to apply all the migration files to the database: so the difficulty concerns the symbolic links. I think maybe the discussion in #28 would be helpful to you?
In particular, you can try customizing the part we handled migration files by making copys instead of symlinks. If you want to modify how migration files and migration summary are handled, you can try to override traits in this file(In particular, handleMigrationFiles
for creating symlinks for migration files; writeSummary
for putting migrations into the summary file). Does this make sense to you?
Let me know if there are any other problems and I would be happy to help figure out a way.
from scala-forklift.
@cvogt but why "tests it against a database with the old schema before the migration"?
from scala-forklift.
@lastland you have to first deploy the code to your production server and then migrate the production db or first migrate and then deploy, right? In either case, between the two steps, either the old production code has to run against the new schema or the new production code has to run against the old schema, right? And I'd say, better test that that actually works by letting the code fire a bunch of queries against the other db version.
from scala-forklift.
Thanks for this feedback. This is tremendously useful for helping me isolate the problems and work around them 👍
from scala-forklift.
I want to thank you again. Using the comments in this and the other issue referenced, I was able to get this working and usable in a couple hours. For what it's worth, here's a gist that encapsulates the changes I made: https://gist.github.com/totallymike/d969ecd61c3e134471e7c5492f722cc2
The only additional thing in this gist is that I allowed for adding package name to the generated MigrationSummary object. This means that ammonite can find my migration objects with its weird import magic, and in theory I'll be able to programmatically run my migrations from elsewhere in my project.
from scala-forklift.
I'm glad you both found it useful. Thanks again for your help and patience :)
from scala-forklift.
@lastland might be worth re-opening this as a documentation request or feature request
from scala-forklift.
you have to first deploy the code to your production server and then migrate the production db or first migrate and then deploy, right? In either case, between the two steps, either the old production code has to run against the new schema or the new production code has to run against the old schema, right?
If you can tell me how to achieve that with Slick and types, or with fundamentally backwards-incompatible changes, I would love to hear.
I currently use Flyway, and it's a non-problem. I include Flyway as a library and on boot the application calls the migrate method. So the application is not running while migrations are being applied. Of course this doesn't scale horizontally, or allow blue-green deployments. I'm not sure the answer though --- see above.
I confess I'm not familiar enough with forklift to comment, but this seems like a fundamental requirement.
And running SBT on the server is just wrong. It's a build tool. Some of my servers don't even have enough memory to run SBT.
Running migrations from the outside as part of a deploy process might work for some, but relying on that would be really limiting.
from scala-forklift.
Reopen this because it seems more discussions are necessary, and some related features/documentations are required.
from scala-forklift.
On success, integration server deploys new code and triggers the production database migration. The easiest would be running the migration on the integration server via sbt. If that's really impossible for some reasons (e.g. migration duration), it might be easier to invoke the migration via the bundled jar, which Forklift would have to allow and require the migrations to be in bundles the jar.
And running SBT on the server is just wrong. It's a build tool. Some of my servers don't even have enough memory to run SBT.
I couldn't find more info about these points.
Is there any way to run the migrations without SBT, from the bundled jar?
from scala-forklift.
Related Issues (20)
- Overriding of Slick Code generation HOT 5
- "build.scala" is deprecated HOT 4
- Change License to Apache 2.0 HOT 1
- Add more tests HOT 1
- Add slick-migration-api in the slick3.2 version
- Problem using forklift with VCS HOT 11
- Don't swallow exceptions silently in `mg new s` HOT 4
- Scala 2.12 Support HOT 11
- Scala-Forklift 0.3.0 for Scala 2.11
- Tests for cross Scala versions
- Provide SQL migration via SQL file HOT 5
- Make `APIMigration` the subtype of `slick.migration.api.Migration`
- mg reset fails due to constraints HOT 4
- [Feature] maybeInit HOT 1
- SlickMigrationManager not very modular HOT 1
- Support Slick 3.3.0 HOT 2
- Scala 2.13 Support HOT 3
- [Help] Running multiple migratedb tool in parallel HOT 4
- Slick 3.4.1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scala-forklift.