Coder Social home page Coder Social logo

godaddy / activerecord-delay_touching Goto Github PK

View Code? Open in Web Editor NEW
111.0 31.0 18.0 27 KB

Batch up your ActiveRecord "touch" operations for better performance. ActiveRecord::Base.delay_touching do ... end. When "end" is reached, all accumulated "touch" calls will be consolidated into as few database round trips as possible.

Home Page: https://rubygems.org/gems/activerecord-delay_touching

License: MIT License

Ruby 100.00%

activerecord-delay_touching's Introduction

Activerecord::DelayTouching

Note: this version requires ActiveRecord 4.2 or higher. To use ActiveRecord 3.2 through 4.1, use the branch https://github.com/godaddy/activerecord-delay_touching/tree/pre-activerecord-4.2.

Batch up your ActiveRecord "touch" operations for better performance.

When you want to invalidate a cache in Rails, you use touch: true. But when you modify a bunch of records that all belong_to the same owning record, that record will be touched N times. It's incredibly slow.

With this gem, all touch operations are consolidated into as few database round-trips as possible. Instead of N touches you get 1 touch.

Installation

Add this line to your application's Gemfile:

gem 'activerecord-delay_touching'

And then execute:

$ bundle

Or install it yourself:

$ gem install activerecord-delay_touching

Usage

The setup:

class Person < ActiveRecord::Base
  has_many :pets
  accepts_nested_attributes_for :pets
end

class Pet < ActiveRecord::Base
  belongs_to :person, touch: true
end

Without delay_touching, this simple update in the controller calls @person.touch N times, where N is the number of pets that were updated via nested attributes. That's N-1 unnecessary round-trips to the database:

class PeopleController < ApplicationController
  def update
    ...
    #
    @person.update(person_params)
    ...
  end
end

# SQL (0.1ms)  UPDATE "people" SET "updated_at" = '2014-07-09 19:48:07.137158' WHERE "people"."id" = 1
# SQL (0.1ms)  UPDATE "people" SET "updated_at" = '2014-07-09 19:48:07.138457' WHERE "people"."id" = 1
# SQL (0.1ms)  UPDATE "people" SET "updated_at" = '2014-07-09 19:48:07.140088' WHERE "people"."id" = 1

With delay_touching, @person is touched only once:

ActiveRecord::Base.delay_touching do
  @person.update(person_params)
end

# SQL (0.1ms)  UPDATE "people" SET "updated_at" = '2014-07-09 19:48:07.140088' WHERE "people"."id" = 1

Consolidates Touches Per Table

In the following example, a person gives his pet to another person. ActiveRecord automatically touches the old person and the new person. With delay_touching, this will only make a single round-trip to the database, setting updated_at for all Person records in a single SQL UPDATE statement. Not a big deal when there are only two touches, but when you're updating records en masse and have a cascade of hundreds touches, it really is a big deal.

class Pet < ActiveRecord::Base
  belongs_to :person, touch: true

  def give(to_person)
    ActiveRecord::Base.delay_touching do
      self.person = to_person
      save! # touches old person and new person in a single SQL UPDATE.
    end
  end
end

Cascading Touches

When delay_touch runs through and touches everything, it captures additional touch calls that might be called as side-effects. (E.g., in after_touch handlers.) Then it makes a second pass, batching up those touches as well.

It keeps doing this until there are no more touches, or until the sun swallows up the earth. Whichever comes first.

Gotchas

Things to note:

  • after_touch callbacks are still fired for every instance, but not until the block is exited. And they won't happen in the same order as they would if you weren't batching up your touches.
  • If you call person1.touch and then person2.touch, and they are two separate instances with the same id, only person1's after_touch handler will be called.

Contributing

  1. Fork it ( https://github.com/godaddy/activerecord-delay_touching/fork )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

activerecord-delay_touching's People

Contributors

athal7 avatar bmorearty avatar krainboltgreene avatar mjaniszewski-godaddy avatar tjschuck avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

activerecord-delay_touching's Issues

can't write unknown attribute :time in Rails 5

I have the following in my ApplicationController

around_action :delay_touching

def delay_touching
  ActiveRecord::Base.delay_touching { yield }
end

In Rails 4.2, everything works but after the update to Rails 5 I get the following error in lots of rspec tests:

Failure/Error: ActiveRecord::Base.delay_touching { yield }

ActiveModel::MissingAttributeError:
  can't write unknown attribute `{:time=>2016-07-19 13:46:06 UTC}

When I remove the around_action, all specs run as expected.

Would it be possible to skip touching altogether?

Thanks for this awesome gem! I'm using it to speed up a CSV import script in a Rails app that does a lot of touching. Without this gem, the script takes about 31 minutes with a particular data set. With the gem, it takes a little less than 18 minutes.

Ideally, touching would be bypassed completely during the import because the touching is only needed to populate a single Postgres tsvector column used for full-text search. If I remove all touching code from my app, the script takes about 8 minutes. Then, populating the search index takes an additional 3 minutes via Location.find_each(&:touch). That's a total of 11 minutes, which is significantly faster.

Since touching is not necessary during the intial data import, I was wondering if there is a way to bypass the touching. Something like ActiveRecord::Base.skip_touching do.

Thoughts?

Infinite loop when one or more failed new records present

When using this on a block of record creates where one or more of the creates failed (and so have no id value -- they are still nil), delay_touching goes into an infinite loop.

Looking at the code, it appears that the problem is that the key for the failed record is nil and in the State classes updated() method, attempts to delete that from the @records hash/set are ignored and so the hash/set never empties out. As a result, the loop that attempts to iteratively process all pending touches never completes.

Perhaps a check on .persisted? should be done and if false, silently skipped (since the problem is an unending attempt to set the time stamps on a record with an ID of NULL).

Rollback with delay_touching on relationship table may cause "update where NULL" query

Sorry for the delay here @mtuckergd; thanks for merging a fix and cutting a new release.

After testing with the new version, there's no infinite loop (๐Ÿ‘ ), but for our case at least there is still an erroneous update query that fires in the case of rollbacks to "has_many through" relationships. I was able to replicate this on old versions, so thankfully it's not new behavior, and it's an even narrower edge case than what triggered the infinite loop.

I've done some testing in various scenarios and narrowed it to rolling back after writing both a new record and a new relationship record, as may happens with custom "has_many through" associations. NOTE: I was NOT able to replicate with "has_and_belongs_to_many" relationships.

In these cases, there's no infinite loop, but there is an attempt to touch the removed record from the class that joins it:

D, [2018-08-26T18:16:17.761495 #73358] DEBUG -- :    (0.0ms)  begin transaction
D, [2018-08-26T18:16:17.762345 #73358] DEBUG -- :   SQL (0.1ms)  INSERT INTO "posts" ("created_at", "updated_at") VALUES (?, ?)  [["created_at", "2018-08-27 01:16:17.761768"], ["updated_at", "2018-08-27 01:16:17.761768"]]
D, [2018-08-26T18:16:17.769743 #73358] DEBUG -- :   SQL (0.1ms)  INSERT INTO "tags" ("created_at", "updated_at") VALUES (?, ?)  [["created_at", "2018-08-27 01:16:17.769232"], ["updated_at", "2018-08-27 01:16:17.769232"]]
D, [2018-08-26T18:16:17.772731 #73358] DEBUG -- :   SQL (0.1ms)  INSERT INTO "tag_relationships" ("post_id", "tag_id") VALUES (?, ?)  [["post_id", 2], ["tag_id", 1]]
D, [2018-08-26T18:16:17.773108 #73358] DEBUG -- :    (0.0ms)  rollback transaction
D, [2018-08-26T18:16:17.773287 #73358] DEBUG -- :    (0.0ms)  begin transaction
D, [2018-08-26T18:16:17.773849 #73358] DEBUG -- :   SQL (0.1ms)  UPDATE "tags" SET "updated_at" = '2018-08-27 01:16:17.773340' WHERE "tags"."id" IS NULL
D, [2018-08-26T18:16:17.773994 #73358] DEBUG -- :    (0.0ms)  commit transaction

I have some hunches, but please take a look at the PR with the spec and let me know what you think.

Also, I'm sorry for the late response; if I recall, this was some of the thinking that went into the proposed infinite loop fix.

ActiveModel::MissingAttributeError: can't write unknown attribute ``

Just pulled in this gem to try it out and running into this error. This is running on Rails 4.2.

   (4.3ms)  ROLLBACK
ActiveModel::MissingAttributeError: can't write unknown attribute ``
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/attribute.rb:124:in `with_value_from_database'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/attribute_set.rb:39:in `write_from_user'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/attribute_methods/write.rb:74:in `write_attribute_with_type_cast'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/attribute_methods/write.rb:56:in `write_attribute'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/attribute_methods/dirty.rb:96:in `write_attribute'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/persistence.rb:464:in `block in touch'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/persistence.rb:462:in `each'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/persistence.rb:462:in `touch'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activerecord/lib/active_record/callbacks.rb:296:in `block in touch'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/callbacks.rb:117:in `call'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/callbacks.rb:117:in `call'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/callbacks.rb:505:in `call'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/callbacks.rb:505:in `call'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/callbacks.rb:92:in `_run_callbacks'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/callbacks.rb:776:in `_run_touch_callbacks'
... 13 levels...
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/railties/lib/rails/commands/console.rb:9:in `start'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/railties/lib/rails/commands/commands_tasks.rb:68:in `console'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/railties/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/railties/lib/rails/commands.rb:17:in `<top (required)>'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/dependencies.rb:274:in `require'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/dependencies.rb:274:in `block in require'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/dependencies.rb:240:in `load_dependency'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/dependencies.rb:274:in `require'
    from /Users/nathan/Source/CrowdOx/Api/bin/rails:8:in `<top (required)>'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/dependencies.rb:268:in `load'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/dependencies.rb:268:in `block in load'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/dependencies.rb:240:in `load_dependency'
    from /Users/nathan/.rvm/gems/ruby-2.1.5/bundler/gems/rails-79728982aa75/activesupport/lib/active_support/dependencies.rb:268:in `load'
    from /Users/nathan/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/nathan/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'

This happens right after I try to run touch on a model. Thoughts? I haven't gone through the source code verses what is in ActiveRecord but I'm curious if you're running on an older version of Rails than I.

not working with rails 5.2

in the code @changed_attributes.except!(*changes.keys) rails is not filling @changed_attributes any more

Add around_action-based implementation

This is really great work and very easy to implement! One thing that could make it even easier is to add a method into ActionController::Base that could be implemented in an around_action to wrap the entire controller action. The method would look something like this:

def delay_touching
  ActiveRecord::Base.delay_touching { yield }
end

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.