timlkelly / onesie Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 4.0 110 KB

A Rails-based Task runner for one time scripts and data migrations

License: MIT License

Ruby 99.51% Shell 0.49%

ruby ruby-gem ruby-on-rails

onesie's People

Contributors

Stargazers

Watchers

Forkers

watermelonexpress joinhandshake gejustin jbanass

onesie's Issues

Priority Tasks

To meet feature parity with the current onesie task runner we need to ensure that certain tasks will be ran before others.

To do this, the current system separates tasks of high priority in a different directory. Would this approach work here? Or do we need more specificity with the ordering of Tasks?

Ideas:

High/normal priority directories
Task dependencies

@DrkCloudStrife @jbanass @millerjs I am curious for your thoughts/input.

I ran bundle exec rake onesie:new['blah_blah_blah_css'], which generated class BlahBlahBlahCss, which could not be found by the onesie runner, which was looking for BlahBlahBlahCSS. Can these be made consistent so the runner can always find the class?

Full trace:

BlahBlahBlahCSS: NameError uninitialized constant Onesie::Tasks::BlahBlahBlahCSS
Traceback (most recent call last):
        49: from /opt/rubies/ruby-2.7.8/bin/bundle:23:in `<main>'
        48: from /opt/rubies/ruby-2.7.8/bin/bundle:23:in `load'
        47: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/exe/bundle:36:in `<top (required)>'
        46: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/friendly_errors.rb:120:in `with_friendly_errors'
        45: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/exe/bundle:48:in `block in <top (required)>'
        44: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/cli.rb:25:in `start'
        43: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        42: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/cli.rb:31:in `dispatch'
        41: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
        40: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        39: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
        38: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/cli.rb:483:in `exec'
        37: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/cli/exec.rb:23:in `run'
        36: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
        35: from /opt/rubies/ruby-2.7.8/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/cli/exec.rb:58:in `load'
        34: from /usr/local/bundle/bin/rake:23:in `<top (required)>'
        33: from /usr/local/bundle/bin/rake:23:in `load'
        32: from /usr/local/bundle/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'
        31: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:80:in `run'
        30: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:208:in `standard_exception_handling'
        29: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:83:in `block in run'
        28: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:132:in `top_level'
        27: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:147:in `run_with_threads'
        26: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:138:in `block in top_level'
        25: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:138:in `each'
        24: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:138:in `block (2 levels) in top_level'
        23: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:182:in `invoke_task'
        22: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:188:in `invoke'
        21: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
        20: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
        19: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
        18: from /usr/local/bundle/gems/airbrake-13.0.2/lib/airbrake/rake.rb:17:in `execute'
        17: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:279:in `execute'
        16: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:279:in `each'
        15: from /usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:279:in `block in execute'
        14: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/tasks/onesie.rake:42:in `block (2 levels) in <main>'
        13: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/manager.rb:38:in `run_task'
        12: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/runner.rb:14:in `perform'
        11: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/runner.rb:26:in `perform'
        10: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/runner.rb:26:in `each'
         9: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/runner.rb:27:in `block in perform'
         8: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/task_proxy.rb:12:in `run'
         7: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/task_proxy.rb:21:in `task'
         6: from /usr/local/bundle/bundler/gems/onesie-29a19b95c79e/lib/onesie/task_proxy.rb:35:in `load_task'
         5: from /usr/local/bundle/gems/activesupport-5.2.8.1/lib/active_support/core_ext/string/inflections.rb:68:in `constantize'
         4: from /usr/local/bundle/gems/activesupport-5.2.8.1/lib/active_support/inflector/methods.rb:281:in `constantize'
         3: from /usr/local/bundle/gems/activesupport-5.2.8.1/lib/active_support/inflector/methods.rb:281:in `inject'
         2: from /usr/local/bundle/gems/activesupport-5.2.8.1/lib/active_support/inflector/methods.rb:281:in `each'
         1: from /usr/local/bundle/gems/activesupport-5.2.8.1/lib/active_support/inflector/methods.rb:285:in `block in constantize'
/usr/local/bundle/gems/activesupport-5.2.8.1/lib/active_support/inflector/methods.rb:285:in `const_get': uninitialized constant Onesie::Tasks::BlahBlahBlahCSS (NameError)

Describe does not consider environment

The rake onesie:describe task does not consider the active environment when listing available tasks to run. For instance, if there's a task intended only for production through the allowed_enivornments macro, the DescribeTasks class will still consider it a pending task if you check in development. Which technically it is. However, it will never run in development.

The current design designates the allowed environments in the task class itself and the DescribeTasks uses Manager#pending_tasks to check the available tasks to run, which uses the TaskProxy and so the class itself isn't loaded yet. To solve this, I think we'll need to load the task code to filter the tasks.

Add custom templates

Hi!

Another configurable piece I think would add value is allowing folks to pass in an argument that specifies a custom template that the consumer would maintain.

General thought:

rake onesie:new['task_name', 'manual', 'consumer_template']

Where consumer template would be a file that contains the inside of the run method inside the onesie folder.

For ex, say consumers had a common pattern develop for using onesies to create a new Foo record, but it's complex to do it and multiple teams may have to create these onesies, so consistency may be difficult to attain. Using a template approach:

new_foo = Foo.where(
  x: VALUE_HERE,
  y: VALUE_HERE,
  ...etc
).first_or_initialize

Would allow any team needing to consume foo and not forget commonly-forgotten items.

I'd also be up for developing this if you feel this is a worthwhile addition.

Development config onesie

Occasionally there will be a change to a configuration file in v2 that is required and break v2 if it isn't present. Previously we used to communicate these changes through the dev channel. However, as we grow, and work with contractors who are in different time zones and do not have access to all communication channels, this approach is no longer feasible.

A possible solution is to create a onesie add the new configuration file or key to an existing YAML config file.

Add a possible config onesie template?
Add helpers to make this process easier.

DRY the TASKS_DIR constant

Do we have an issue to track this so we don't lose track of it?

Originally posted by @jbanass in https://github.com/watermelonexpress/onesie/pull/14#r762598899

Task Wrapper doesn't state which onesie finished running

From convo in https://benchprep.slack.com/archives/GTZNJAWV9/p1642735524023400?thread_ts=1642182531.012200&cid=GTZNJAWV9

Our logging separates out each log message as its own line. To make it simpler to parse which onesie finished running, we should add the task name once it finishes.

Add hook system

Problem

Given the purpose for Onesie is to run scripts that largely account for changing data, there are times where an artifact is necessary that records actions taken, and subsequently uploaded for later analysis. This is especially useful in continuous delivery circumstances where these onesies are ran without human intervention.

With that said, it's also understood that this gems purpose is to not handle uploading of artifacts.

Proposal

Add support for a hooks system (akin to RSpec's before(:each)) that allows for either onesie set up or tear down that enables consumers to do things such as:

prep storage providers for artifact upload
clean up things that a onesie may have created
Post a slack message stating this onesie's about to run

Could possibly take advantage of https://github.com/rails/rails/tree/main/activesupport for callback?

Add ability to re-run tasks

Especially while developing tasks it would be helpful to have api support for re-running a task. Something along the lines of:

onesie:force re-run last task
onesie:force STEP=3 re-run last 3 tasks
onesie:force['20220105140152_my_task'] re-run specific task

One thing to bear in mind is whether tasks that have not been run should be run along with the tasks that you wish to re-run. Also, I am not stuck on force as a name.

Task class incorrectly referenced in error reporting.

The error reporting is saying that an error occurred in Onesie::TaskProxy when instead it should list the class name of the task being run.

Running TestError...
An error occurred with Onesie::TaskProxy
The following Tasks contained errors


Onesie::TaskProxy: StandardError dangit

Error Handling

Should an error halt all further Onesie Tasks like Rails migrations or rescue and continue with further tasks?

Note that further tasks could anyways be run manually if need be.

Update singular task specificity

Highly unlikely, but not impossible, what would happen if there is more than one onesie with the same task_version?

Originally posted by @DrkCloudStrife in https://github.com/watermelonexpress/onesie/pull/10#discussion_r747006855

Recurring Tasks

Is there a need for some Tasks to be able to be run multiple times?

Database table name

I'm curious if this table name makes more sense as onesie_schema or onesie_history since it seems that is mostly used for tracking which onesies have been run versus actual logs.

Originally posted by @DrkCloudStrife in https://github.com/watermelonexpress/onesie/pull/10#discussion_r747004824

The database table name does not seem as relevant with some of the recent changes to the project.