ryanb / populator

Mass populate an Active Record database.

Unmaintained¶ ↑

The Populator gem is no longer maintained. Feel free to fork this project.

Populator¶ ↑

RDocs | Screencast

Populate an Active Record database with mass insertion.

Installation¶ ↑

Rails 3 support is currently being worked on. Stay tuned.

In Rails 2, install the gem.

gem install populator

And then load it in a rake task or elsewhere.

require "populator"

Usage¶ ↑

This gem adds a “populate” method to all Active Record models. Pass the number of records you want to create along with a block. In the block you can set the column values for each record.

Person.populate(3000) do |person|
  person.first_name = "John"
  person.last_name = "Smith"
end

This will do a mass insert into the database so it is very fast. The person object contains the “id” so you can set up associations.

Person.populate(3000) do |person|
  person.first_name = "John"
  person.last_name = "Smith"
  Project.populate(30) do |project|
    project.person_id = person.id
  end
end

That will create 30 projects for each person.

Passing a range or array of values will randomly select one.

Person.populate(1000..5000) do |person|
  person.gender = ['male', 'female']
  person.annual_income = 10000..200000
end

This will create 1000 to 5000 men or women with the annual income between 10,000 and 200,000.

You can pass a :per_query option to limit how many records are saved per query. This defaults to 1000.

Person.populate(2000, :per_query => 100)

If you need to generate fake data, there are a few methods to do this.

Populator.words(3) # generates 3 random words separated by spaces
Populator.words(10..20) # generates between 10 and 20 random words
Populator.sentences(5) # generates 5 sentences
Populator.paragraphs(3) # generates 3 paragraphs

For fancier data generation, try the Faker gem.

Important¶ ↑

For performance reasons, this gem does not use actual instances of the model. This means validations and callbacks are bypassed. It is up to you to ensure you’re adding valid data.

Development¶ ↑

Problems or questions? Add an issue on GitHub or fork the project and send a pull request.

See spec/README for instructions on running specs.

Special Thanks¶ ↑

Special thanks to Zach Dennis for his ar-extensions gem which some of this code is based on. Also many thanks to the contributors. See the CHANGELOG for the full list.

populator's Issues

password and password_confirmation aren't populated

When populating a model which extends authlogic, it does not populate the password. I have to enter the password hash and salt to make it work. Not sure if you knew this or not.

Great job on this!

Eric Berry

uninitialized constant Object::ActiveRecod in model_additions.rb

I'm using: Ruby 1.9.2, populator 1.0.0, Rails 3.0.3, and bundler 1.0.9

When I tried starting a bundler console this error happend. It seems that AR isn't required yet so the dependency isn't met. It works when I added require 'active_record' at the top of the file but not sure that's the best thing to do.

$> bundle console
/Users/.rvm/gems/ruby-1.9.2-p136/gems/populator-1.0.0/lib/populator/model_additions.rb:30:in <top (required)>': uninitialized constant Object::ActiveRecord (NameError) from /Users/.rvm/gems/ruby-1.9.2-p136/gems/populator-1.0.0/lib/populator.rb:2:inrequire'
from /Users/.rvm/gems/ruby-1.9.2-p136/gems/populator-1.0.0/lib/populator.rb:2:in <top (required)>' from /Users/.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/runtime.rb:68:inrequire'
from /Users/.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/runtime.rb:68:in block (2 levels) in require' from /Users/.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/runtime.rb:66:ineach'
from /Users/.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/runtime.rb:66:in block in require' from /Users/.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/runtime.rb:55:ineach'
from /Users/.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/runtime.rb:55:in require' from /Users/.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler.rb:114:inrequire'
from /Users/.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/cli.rb:436:in console' from /Users//.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/vendor/thor/task.rb:22:inrun'
from /Users//.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/vendor/thor/invocation.rb:118:in invoke_task' from /Users//.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/vendor/thor.rb:246:indispatch'
from /Users//.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/lib/bundler/vendor/thor/base.rb:389:in start' from /Users//.rvm/gems/ruby-1.9.2-p136/gems/bundler-1.0.9/bin/bundle:13:in<top (required)>'
from /Users//.rvm/gems/ruby-1.9.2-p136/bin/bundle:19:in load' from /Users//.rvm/gems/ruby-1.9.2-p136/bin/bundle:19:in

'

Unable to call rake task twice, with postgresql

Hello,

I have a rake task that looks like this:

namespace :db do
desc "Erase and fill database"
task :populate => :environment do

[A,B].each(&:delete_all)

A.populate 15 do |a|
  a.title = Populator.words(5..8)

    B.populate 3 do |b|
      b.a_id = a.id
      b.title = Populator.words(5..8)
    end#B

  end#A

end#task

end#namespace

Using sqlite3, whenever I call the rake task multiple times, everything works.
Using postgresql, first time works like a charm (asuming I just migrated the database) but the second+ times it fails. When looking at
A.all.to_a (using Rails 4)
B.all.to_a

I have data for both models but each record in b has an invalid id for a_id and associations are lost (ie. I cannot do a.bs)

obvious here a, b are places holders.

This seems to happen with postgresql and not sqlite3. Getting the same error on heroku (uses postresql) and in local machine

no such file to load -- populator

I am trying out the Railscast populator example. I installed the populator gem, but Rby is telling me that it cannot find it. I'm not sure why this is happening.

Incorrect primary key used in child populate objects

Using Postgres-8.3, Ruby-1.8.7-OSX, Rails-3.0.4 and Populator-1.0.0:

User.populate 500 do |user|
     ....
     PhoneNumber.populate 1..3 do |p|
         p.user_id = user.id
         ....
     end
end

All Users will have no phone number, because strangely my phone_numbers table will start counting user_ids from 1 (or 0?) while my users table will start counting users from 6700 (approx).

This worked with populate-0.3 and Rails 2.3.10.

Support HABTM

Currently there is no way to populate has_and_belongs_to_many because there is no model set apart for the join table. There should be additional methods one can call on either side of the association to setup the join.

no such file to load -- populator

Hi. I created another issue with this title, but closed it because I thought it was resolved. When I run the db:populate task, which is referenced in Railscast episode #128, I still see this error message. I restarted my server several times to see if it would be resolved on its own, but it hasn't.

Postgres auto-increment key errors on Rails 3

Rails 3.0.3
pg adapter 0.10.0
Postgres 9.0.1

It appears that using populator on a model in my app breaks the sequencing on the table. When I go to save another instance of that model through my app, I get errors along the lines of:

PGError: ERROR: duplicate key value violates unique constraint "entries_pkey"
DETAIL: Key (id)=(12) already exists.

Populator doesn't work in Rails 3

Hi ryan,

I try to use populator in rails 3 and it didn't work. this is the runtime stack

/usr/local/lib/ruby/gems/1.9.1/gems/populator-0.2.5/lib/populator/model_additions.rb:30:in `<top (required)>': uninitialized constant ActiveRecord (NameError)
from /usr/local/lib/ruby/gems/1.9.1/gems/populator-0.2.5/lib/populator.rb:2:in `require'
from /usr/local/lib/ruby/gems/1.9.1/gems/populator-0.2.5/lib/populator.rb:2:in `<top (required)>'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-0.9.7/lib/bundler/runtime.rb:41:in `require'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-0.9.7/lib/bundler/runtime.rb:41:in `block (2 levels) in require'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-0.9.7/lib/bundler/runtime.rb:36:in `each'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-0.9.7/lib/bundler/runtime.rb:36:in `block in require'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-0.9.7/lib/bundler/runtime.rb:35:in `each'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-0.9.7/lib/bundler/runtime.rb:35:in `require'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-0.9.7/lib/bundler.rb:71:in `require'
from /home/ecleel/codes/myproject/config/boot.rb:20:in `<top (required)>'
from /home/ecleel/codes/myproject/script/rails:9:in `require'
from /home/ecleel/codes/myproject/script/rails:9:in `<main>'

Build up items in an array of hashes

Sometimes it's not applicable to do a loop because it is limited in how the nesting works. Instead one should be able to build up an array of hashes and assign it in one call.

Item.populate [{:name => "foo"}, {:name => "bar"}]

What to do when generating a model with double belongs_to ?

Let's say we have Book, Shelf and Author models. We can generate Authors and Books inside the loop, but how could we link Shelves?

Shelf.populate 20 do |shelf|
shelf.name = Populator.words(1..3).titleize
end
Author.populate 40 do |author|
Book.populate 0..40 do |book|
book.author_id = author.id
end
end

Any idea or suggestion ?

Errors with Populator and Rails 3

I have just upgraded to Rails 3 and I am having significant trouble getting rake db:populate to work.

I'm getting instances of unexpected tIDENTIFIER, unexpected tSTRING_BEG, unexpected tCONSTANT, unexpected tLBRACK, unexpected keyword_end, unexpected $undefined, unexpected tIVAR, etc, all in long sequences of errors with each rake db:populate. These are most prevalent wherever there are strings involved, and errors often (but not always) point to the " at the end of a particular string.

The other issue I'm having is that single quotes inside my strings (such as the quote inside "it's" raise the 'invalid multibyte char' error regarding ASCII characters. I want my users to be able to type freely in these spaces so the failure causes concern.

Any help would be very much appreciated. Thank you.
My new set up is Ruby 1.9.2, Rails 3, and MySQL running on Vista.

The echoe gem is still required

I just tried to run tests on this project in Ruby 1.9.1 and it's complaining about not finding the 'echoe' gem. When I installed the 'echoe' gem the error message went away. I'm guessing that either your fix for 0.2.4 didn't work or it wasn't committed properly. :(

Populater doesn't use after create callbacks

I used populator for first time. Its cool and fills my database with test data very well. One of my models has an create after callback to create ten default assocciated datasets. My RSpec tests will test the presence of these default datasets. I can create these datas with populator. Perhaps there is a better way so I not have to change my populater rake task every time the after create method changes or other models get a callback method.

undefined method 'catch_schema_changes' in Sqlite3 adapter

Using Rails 3 RC, when I try rake db:populate, I get the below error. Up until a week ago, it worked. I don't think I've changed anything in my environment, but I could be wrong...

rake aborted!
undefined method catch_schema_changes' for #<ActiveRecord::ConnectionAdapters::SQLite3Adapter:0x10534b308> /Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/adapters/sqlite.rb:6:inexecute_batch'
/Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/adapters/sqlite.rb:18:in populate' /Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/factory.rb:61:insave_records'
/Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/factory.rb:19:in save_remaining_records' /Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/factory.rb:18:ineach'
/Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/factory.rb:18:in save_remaining_records' /Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/factory.rb:31:inremember_depth'
/Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/factory.rb:42:in populate' /Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator/model_additions.rb:25:inpopulate'
/Users/jack/Work/sportoften/lib/tasks/populate.rake:10
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in call' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:inexecute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in each' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:inexecute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in invoke_with_call_chain' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/monitor.rb:242:insynchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in invoke_with_call_chain' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:ininvoke'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in invoke_task' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:intop_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in each' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:intop_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in standard_exception_handling' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:intop_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in run' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:instandard_exception_handling'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in run' /Library/Ruby/Gems/1.8/gems/rake-0.8.7/bin/rake:31 /usr/bin/rake:19:inload'
/usr/bin/rake:19

"superclass mismatch for class SQLiteAdapter" error

i am getting "superclass mismatch for class SQLiteAdapter" when using this gem with jruby 1.6.3

Stack trace:

superclass mismatch for class SQLiteAdapter
/home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/populator-1.0.0/lib/populator/adapters/sqlite.rb:25:in ConnectionAdapters' /home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/populator-1.0.0/lib/populator/adapters/sqlite.rb:24:inActiveRecord'
/home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/populator-1.0.0/lib/populator/adapters/sqlite.rb:23:in (root)' org/jruby/RubyKernel.java:1038:inrequire'
/home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/activesupport-3.0.5/lib/active_support/dependencies.rb:239:in require' /home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/activesupport-3.0.5/lib/active_support/dependencies.rb:225:inload_dependency'
/home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/activesupport-3.0.5/lib/active_support/dependencies.rb:596:in new_constants_in' /home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/activesupport-3.0.5/lib/active_support/dependencies.rb:595:innew_constants_in'
/home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/activesupport-3.0.5/lib/active_support/dependencies.rb:225:in load_dependency' /home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/activesupport-3.0.5/lib/active_support/dependencies.rb:239:inrequire'
/home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/populator-1.0.0/lib/populator/adapters/sqlite.rb:8:in (root)' org/jruby/RubyKernel.java:1038:inrequire'
/home/ec2-user/.rvm/gems/jruby-1.6.3@dm_app/gems/populator-1.0.0/lib/populator.rb:68:in require' org/jruby/RubyArray.java:1603:ineach'
/home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/bundler-1.0.21/lib/bundler/runtime.rb:66:in require' org/jruby/RubyArray.java:1603:ineach'
/home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/bundler-1.0.21/lib/bundler/runtime.rb:55:in require' /home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/bundler-1.0.21/lib/bundler.rb:122:inrequire'
/home/ec2-user/tmp/dm_app/config/application.rb:9:in (root)' org/jruby/RubyKernel.java:1038:inrequire'
/home/ec2-user/.rvm/rubies/jruby-1.6.3/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:29:in require' /home/ec2-user/tmp/dm_app/config/application.rb:4:in(root)'
org/jruby/RubyKernel.java:1063:in load' /home/ec2-user/tmp/dm_app/Rakefile:25:inload_rakefile'
/home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/rake-0.9.2/lib/rake/application.rb:495:in raw_load_rakefile' /home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/rake-0.9.2/lib/rake/application.rb:78:inload_rakefile'
/home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/rake-0.9.2/lib/rake/application.rb:129:in standard_exception_handling' /home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/rake-0.9.2/lib/rake/application.rb:77:inload_rakefile'
/home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/rake-0.9.2/lib/rake/application.rb:61:in run' /home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/rake-0.9.2/lib/rake/application.rb:129:instandard_exception_handling'
/home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/rake-0.9.2/lib/rake/application.rb:59:in run' /home/ec2-user/.rvm/gems/jruby-1.6.3@global/gems/rake-0.9.2/bin/rake:32:in(root)'
org/jruby/RubyKernel.java:1063:in load' /home/ec2-user/.rvm/gems/jruby-1.6.3@global/bin/rake:19:in(root)'

Fix auto incrementing id in PostgreSQL

The id value should not be set directly under PostgreSQL because it messes up the sequence. See phlawski/populator for a solution.

NoMethodError: undefined method `sanitize' with Rails 5.1.0.rc2

I got below error.

$ rails db:seed

rails aborted!
NoMethodError: undefined method `sanitize' for #<Class:0x000000034e8ba0>
/home/k2/.rvm/gems/ruby-2.4.0/gems/activerecord-5.1.0.rc2/lib/active_record/dynamic_matchers.rb:22:i
n `method_missing'
/home/k2/.rvm/gems/ruby-2.4.0/gems/populator-1.0.0/lib/populator/factory.rb:85:in `block (2 levels) 
in rows_sql_arr'
/home/k2/.rvm/gems/ruby-2.4.0/gems/populator-1.0.0/lib/populator/factory.rb:85:in `map'
/home/k2/.rvm/gems/ruby-2.4.0/gems/populator-1.0.0/lib/populator/factory.rb:85:in `block in rows_sq$
_arr'
/home/k2/.rvm/gems/ruby-2.4.0/gems/populator-1.0.0/lib/populator/factory.rb:84:in `map'
/home/k2/.rvm/gems/ruby-2.4.0/gems/populator-1.0.0/lib/populator/factory.rb:84:in `rows_sql_arr'
...

I'm using rails that versioned 5.1.0.rc2 .

and I tried with 5.0.2, db:seed passed.
this problem occurs only rails 5.1.0.rc2.

Attaching populator 1.0.0 gem

Hello,
As soon as I attach Populator (which should be compatible with Rails3, shouldn't it?), I get the following error:
undefined method execute' for classActiveRecord::ConnectionAdapters::OracleAdapter'
I use sqlite

Incorrect FK in nested models under Rails 3.2/PostgreSQL

I am currently on Rails 3.2 with PostgreSQL as database, and the following example does not work as expected:

Foo.populate 10 do |foo|
    Bar.populate 10 do |bar|
        bar.foo_id = foo.id
    end
end

The entries for Bar should be linked to Foo through a foreign key. Instead the value of foo.id inside Bar seems to start from 0 always, thus, running a rake db:populate multiple times would cause the id of Foo to move forward (as it does not reset by default) and lead to Bar linking to the incorrect Foo.

This commit from route/populator@5c1da8c3784e6322c3d26f3b1c4511daeecb7a80 seems to fix this - might be a good idea to merge?

"Command not found" errors

Hey Ryan,

I've been trying out the populator gem on Rails 3 and have been getting "command not found" errors. I thought it might be a permissions issue at first, but I reinstalled using sudo and that didn't make any difference. I do gem which populator and try to run the file directly from commandline but that causes the same error.

Freenaut:bio-clinical newuser$ sudo /Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator

sudo: /Library/Ruby/Gems/1.8/gems/populator-0.2.5/lib/populator: command not found

Doesn't respect column defaults

If I've set a column default in my schema, populator won't set the attribute value and will try saving it as NULL. This causes errors when a column is not nullable.

Looking at the code, Populator::Record only looks at the column names, and anything that's not explicitly set by the user will be set to NULL. How about if we make it use the defaults from the schema too?

My associations aren't working!

Restaurant.populate 10..25 do |restaurant|
  restaurant.name = Populator.words(1..4).titleize
  restaurant.street_address = Faker::Address.street_address
  restaurant.description = Populator.sentences(2..8)
  restaurant.zipcode = Faker::Address.zip_code   
  restaurant.state = "MO"      
  restaurant.phone = Faker::PhoneNumber.phone_number
  Tagging.populate 6 do |tg|
    tg.restaurant_id = restaurant.id 
    tag = Tag.all
    tg.tag_id = tag[rand(tag.length)].id
  end
end

But when I go to my Tagging table, the 'restaurant_id' fields in each record is populated with an ID that is not associated with any of the Restaurant IDs that this method created. Any idea why? When I run it in Rails console, I get the desired effect.

Populating a serialized Array

I've got a column that stores a serialized array, that I want to populate:

class Pitch < ActiveRecord::Base
  MEDIA_TYPES= %w{tvc online tv_show dvd }.freeze
  serialize :media_types, Array
  ...

The simple idea to populate it obviously does not work:

pitch.media_types  = Pitch::MEDIA_TYPES.sample(3)

as the sampling just generates a random array, from which populator randomly picks an entry to insert as a string. This results in a type-mismatch when I read the media_type (as it bypasses the serialization with the mass-assignment).

Any ideas how to achieve this? Is there a way to create the array with random picks and call the serialize manually?!