cocoapods / humus

Handles database migrations and states for all CP web projects.

Humus

Handles database migrations and states for all CP web projects.

Humus is the data ground on which we build beautiful trees, for example the cocoa palm.

Goals

This project:

• handles the CocoaPods production DB state by managing trunk/metrics migrations.
• helps setup a local CocoaPods development/test DB.
• can pull snapshots from the production DB for testing purposes.

Getting a copy of the Test Database set up for using with CocoaPods web projects

Set up the repo:

git clone https://github.com/CocoaPods/Humus.git
cd Humus
bundle install

Create the DB for the development environment:

RACK_ENV=development bundle exec rake db:create

Update it to latest:

RACK_ENV=development bundle exec rake db:migrate

Dependencies

Ubuntu:

If bundle install fails, then you probably need to install the following:

sudo apt-get install postgresql libpqdev postgresql-server-dev-9.6 

Run Migrations in Production

First of all, be super careful.

Add the remote once:

a. heroku git:remote -a cocoapods-humus-service

Ask for permissions if you need access.

Then:

1. Set the versions in migrate.rb to where you'd like them to be in production.
2. Test the migrations vigorously locally: bundle exec rake db:migrate.
3. Use automated tests. There is an migration spec example here: spec/integration/delete_cascade_spec.rb.
4. Verify that you are happy with the migration.
5. Push the code to Heroku: git push heroku master.
6. Consider making a manual backup on Heroku.
7. Run the migrations in production: heroku run bundle exec rake db:migrate
8. Verify that everything went well. If not, depending on the situation, consider:

• Reverting to the backup.
• Adding another migration.

9. If all went well: Congratulations!

Answer all verification questions.

If it goes badly wrong, we have daily automated backups in place. See the Trunk Heroku app for infos.

Running tests

You need to have access (be on the core team).

1. List the available snapshots

bundle exec rake db:test:dump
trunk-201510021730-b154.dump
trunk-201709-11349-a857.dump

2. Provide the selected ID (omitting the trunk- prefix) to the same command

bundle exec rake db:test:dump[201510021730-b154]

3. bundle exec rake

Using Humus Snapshots for Integration Tests

A Humus snapshot is just a sanitized Trunk DB Heroku snapshot (dump). They allow you to write test that run against the actual prod DB.

1. Add the following to your project's Gemfile:

    gem 'cocoapods-humus', :require => false
   

2. Add the following Ruby code where you'd like to seed the DB:

    require 'cocoapods-humus'
    Humus.with_snapshot('201510021730-b154')
   

That snapshot identifier is the only one currently available (Dump from 17:30, Oct 2, 2015, from Heroku snapshot b154). Note: When loading the DB, you may see some warnings.

Using Humus Snapshots for CI

See explanation in the last section.

1. Set up Humus snapshots for your tests as in the last section.

2. Add to the following to .travis.yml:

    sudo: false
    addons:
      postgresql: "9.4"
      apt:
        packages:
        - postgresql-server-dev-9.4
    env:
      secure: "MeyjQ3gyQBQhvTxHC/HQSUG2LmvGdeD9aizM+pNxF8ae+0Rf4yPXKhJdQW8iNnl1QnQdNEHv/6y4mitR2UJ4wllSW/kvk6SBPQShXSmvrQIAX//R8hR4vZzRnLkEZmfL8al1ZazPABOeinQg6vEL1+AYjOLk2UAfHvcyUlUTcpM="
   

3. Replace the secure text above (insert S3 access keys) with the result of travis encrypt (http://docs.travis-ci.com/user/environment-variables/#Encrypted-Variables):

    travis encrypt 'DUMP_ACCESS_KEY_ID=... DUMP_SECRET_ACCESS_KEY=...'
   

4. Push the project. Check Travis.

Generating a sanitized Trunk DB Heroku snapshot

So you've added a new table to the DB and you'd like it included in the dump. That is, the old dump is not good enough anymore.

1. Create a new Heroku snapshot

   heroku pg:backups:capture --app cocoapods-trunk-service
   

2. Download that dump.

   heroku pg:backups:download --app cocoapods-trunk-service
   

3. Load it into a DB.

4. Run this SQL on that DB to sanitize the Trunk data:

    -- Add helper function.
    create or replace function random_string(length integer) returns text as
    $$
    declare
      chars text[] := '{A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z}';
      result text := '';
      i integer := 0;
    begin
      if length < 0 then
        raise exception 'Given length cannot be less than 0';
      end if;
      for i in 1..length loop
        result := result || chars[1+random()*(array_length(chars, 1)-1)];
      end loop;
      return result;
    end;
    $$ language plpgsql;
    
    -- Anonymize owners.
    UPDATE owners
    SET
    	name = concat(random_string(1), lower(random_string(5)), ' ', random_string(1), lower(random_string(7))),
    	email = lower(concat(random_string(15), '@', random_string(10), '.', random_string(3)));
    
    -- Empty sessions table.
    TRUNCATE TABLE sessions;
   

5. Export the Trunk DB data by running (-Fc compresses):

    pg_dump -Fc dbname > data.dump
   

6. Rename the data.dump file to:

    trunk-201501021234-bXXX.dump (trunk-%Y%m%d%H%M-<Heroku manual dump name>.dump)
   

7. Upload to our S3 DB dump storage (Only Florian knows how, currently - will update this).

humus's Issues

Update tests to not rely on outdated DB dump.

Document Heroku migration process

Before Humus goes live, I'd also love to see some documentation on how the migration process will work on Heroku

Sanity Check

Before this goes live, and before migrations are removed from Trunk and metrics, we need to sanity check the migrations such that the result corresponds to the state we have online.

Add pod_versions.deleted.

(Or deleted_at, actually – with a default of NULL, or false, respectively)

See CocoaPods/trunk.cocoapods.org#144 (comment).

Make it play nice with Trunk, metrics, search.

Plays nice with:

• Trunk
• metrics
• search

Add pod_version_id to cocoadocs_pod_metrics table

Related to: CocoaPods/cocoadocs-api#24 (comment)

This additional field will allow tracking the recency of cocoadocs_pod_metrics data, which is necessary for reliable continuous integration.

Prepare Humus for use in CI

• Upload sanitized/size-reduced DB dump to e.g. S3. (Using pg_dump -Fc)
• Store the dump's location in Humus.
• Enable the snapshot (dump) to be downloaded and loaded into the DB.
• Add s3 as dependency on the Humus gem.
• Gemify Humus.
• Re-enable working snapshots to be used in search.
• Polishing.
• Release gem.
• Add s3 credentials to … Admin?
• Set up CI for search on Travis or Circle (@kylef).

Example on gem cocoapods-humus usage

For the following example to work, both ENV['DUMP_ACCESS_KEY_ID'] and ENV['DUMP_SECRET_ACCESS_KEY'] need to be set.

In Gemfile:

gem 'cocoapods-humus', :require => false

In your e.g. spec_helper.rb / integration test:

require 'cocoapods-humus'

# Non-block version.
Humus.with_snapshot('201510021730-b154')

# Block version.
Humus.with_snapshot('201510021730-b154') do
  # …
end

The gem is currently using the file system, and writing to a fixed DB with name trunk_cocoapods_org_test.
@kylef Let me know if that is an issue and we'll improve.

Real world example

1. Clone https://github.com/CocoaPods/search.cocoapods.org.
2. bundle install
3. bundle exec rake

Code is here: https://github.com/CocoaPods/search.cocoapods.org/blob/5d1781722e58cbb42d9dd0723f4852680aaf5070/spec/db_seed.rb#L3-L6