knowncitizen / schlepp

Schlepp is a gem for normalizing and importing data from various formats. It can currently normalize CSV data and pull from RDBMS sources by using ActiveRecord transparently. Schlepp is based on a simple DSL.

Schlepp¶ ↑

Description¶ ↑

Schlepp makes it easy to normalize data from various sources like CSV. Need to go from flattened CSV to relational tables? Schlepp provides a simple DSL for mapping, grouping and importing data from and to various formats.

At least that’s the plan. Currently handles access to CSV files.

Usage¶ ↑

Schlepp::Burden.new :products do
  cd 'data/'

  glob 'catalogs/*' do |dir|
    file 'Csv' do |csv|
      csv.name = 'products.csv'
      csv.load_mapping 'config/products.yml'
      csv.groups = [3, 4]

      csv.map do |product|
        product_record = Product.create(:name => product[:name])
        product.children.each do |color|
          color_record = Color.create(:name => color[:name])
          product_record.colors << color_record
          color.children.each do |size|
            size_record = Size.create(:name => size[:name])
            color.sizes << size_record
          end
        end
      end

    end
  end

  # Schlepp::Db uses ActiveRecord internally, so AR style configuration and
  # manipulation.
  db do |database|
    database.config = {
      adapter:  'mysql',
      host:     'localhost',
      username: 'root',
      password: 'root',
      database: 'test_db'
    }

    database.table :posts do |posts|
      posts.default_scope do
        where(:is_deleted => false)
      end

      # association options are passed to the corresponding AR method
      posts.has_many :comments, :class_name => 'CoolComments' do |comments|
        comments.default_scope do
          limit(2)
        end
      end

      posts.each do |post|
        # post is now an instance of a model descended from ActiveRecord::Base
        # meaning post.comments works fine, or any other AR stuff.
        # you would do something with the post here.
      end
    end

  end

end