tumblr / jetpants

MySQL toolkit for managing billions of rows and hundreds of database machines

Jetpants¶ ↑

OVERVIEW:¶ ↑

Jetpants is an automation toolkit for handling monstrously large MySQL database topologies. It is geared towards common operational tasks like cloning slaves, rebalancing shards, and performing master promotions. It features a command suite for easy use by operations staff, though it’s also a full Ruby library for use in developing custom migration scripts and database automation.

Jetpants supports a range-based sharding scheme for MySQL by providing a fast way to split shards that are approaching capacity or I/O limitations. Jetpants is able to accomplish this without any locking, downtime, data inconsistency, or query failures. Dynamically resizable range-based sharding allows you to scale MySQL horizontally in a sane manner, without any need for a central lookup service or massive pre-allocation of tiny shards.

MOTIVATION:¶ ↑

Jetpants was created by Tumblr to help manage our database infrastructure. It handles automation tasks for our entire database topology, which as of May 2013 consists of over 215 dedicated database servers and nearly 200 billion total distinct relational rows.

One of the primary requirements for Jetpants was speed. On our hardware, Jetpants can divide a 750GB, billion-row shard in half in about six hours – or even faster if you’re diving into thirds or fourths.

For more background on the motivations behind Jetpants, please see Evan Elias’s presentation at Percona Live 2013.

COMMAND SUITE FEATURES:¶ ↑

The Jetpants command suite offers easy command-line interaction with complex MySQL automation tasks.

• Clone slaves efficiently, including to multiple targets simultaneously

• Split a range-based shard into N new shards with zero downtime and no failed queries

• Perform master promotions and other pool topology changes

• Defragment tables quickly in parallelized chunks

• Interact with your database topology in a REPL environment via jetpants console mode

For more information on the command suite, please see doc/commands.rdoc (view on GitHub).

LIBRARY FEATURES:¶ ↑

Jetpants is also a Ruby module which you can use to build complex database migration scripts and other customized automation. It provides object modeling for databases, hosts, global/functional pools, sharded pools, and your database topology as a whole.

• Utilize scriptable versions of all command suite functionality

• Crawl replication topology programmatically

• Import or export arbitrary portions of a data set

• Copy large files quickly and efficiently, including to multiple simultaneous destinations. Flexibility in terms of single threaded copy or multi-threaded copy is provided based on the resources available (large network bandwidth, PCIe flash devices). Supporting scripts (jetpants_clone_sender and jetpants_clone_receiver) are present in bin/ directory.

• Manipulate server settings or concurrently execute arbitrary UNIX commands / administrative MySQL queries on multiple servers

ASSUMPTIONS AND REQUIREMENTS:¶ ↑

Percona Toolkit of at least 2.2.15.

The base classes of Jetpants currently make a number of assumptions about your environment and database topology. Please see doc/requirements.rdoc (view on GitHub).

CONFIGURATION:¶ ↑

Jetpants supports a global configuration file at /etc/jetpants.yaml, as well as per-user configuration files at ~/.jetpants.yaml.

At least one of these files must exist for Jetpants to function properly, since certain options (database schema name, database credentials, etc) are mandatory and cannot be inferred.

Please see doc/configuration.rdoc (view on GitHub) for information on configuring Jetpants.

PLUGINS:¶ ↑

Jetpants offers an extensible plugin system. Plugins are Ruby code (such as stand-alone gems) that add to Jetpants by supplying callback methods, and/or overriding core methods.

It is highly recommended that you tie Jetpants into your site’s asset tracker / hardware management system by writing a custom plugin. This will allow Jetpants to automatically know what database pools and shards are present, and to make topological changes immediately be reflected in your site’s configuration. Several complex Jetpants features (including shard splits) actually require an asset tracker plugin in order to function, since these processes involve obtaining spare nodes and manipulating multiple pools in your database topology.

Other recommended uses of plugins include integration with your site’s monitoring system, trending system, query killers, and environment-specific overrides to various core methods.

If you are using Collins for asset management, Jetpants now ships with a plugin that offers integration. Please see doc/jetpants_collins.rdoc (view on GitHub) for usage.

For more information on how to write plugins and use the Jetpants::CallbackHandler system, please see doc/plugins.rdoc (view on GitHub)

FREQUENTLY ASKED QUESTIONS:¶ ↑

Please see doc/faq.rdoc (view on GitHub) for answers to common questions.

If you have a question that isn’t covered here, please feel free to email the authors at the addresses listed in jetpants.gemspec.

CREDITS:¶ ↑

• Evan Elias: Creator and developer

• Dallas Marlow: Developer

• Bob Patterson Jr: Developer

• Tom Christ: Developer

• Tyler Neely: Developer

• Kiril Angov: Developer

• Amar Mudrankit: Developer

• Duan Wang: Developer

• Akshay Suryawanshi: Developer

• Graham Christensen: Developer

Special thanks to Tim Ellis for testing and bug reports.

LICENSE:¶ ↑

Copyright 2013 Tumblr, Inc.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

jetpants's Issues

Support cloning a slave from an online master

Hi guys. Thx for u'r great soft.
Tell me pls, is there a solution in jetpants for adding (not clone) slave for single master (without any other slaves). (maybe, by using Percona XtraBackup or some other "hot" backup utilities)?
I didn't find any stuff like this in jetpants.

update topology scale #'s so I can brag the fuck out of this

Is it still an active project?

Folks, jetpants looks very promising and I'd really like to use it for shards automation. Thanks! However, there haven't been any activity for an year already, so I'm wondering about current project status...

Missing Method - debug_exceptions

Hi :)

Any idea on how i can solve this ?
(or telling me what i do wrong...)

$ ~/jetpants/bin# ./jetpants
Notice: no plugin has overridden Topology#load_pools, so no pools are imported automatically
Heads-up: You're not using screen or tmux.
/var/lib/gems/2.3.0/gems/jetpants-0.8.2/lib/jetpants.rb:104:in `method_missing': undefined method `debug_exceptions' for Jetpants:Module (NoMethodError)
	from ./jetpants:1031:in `<main>'

Is this project still active?

I see lots of stars, but almost no issues. The commits also dropped off sharply...

Jetpants sharded_tables example in configuration needs further nesting.

https://github.com/tumblr/jetpants/blob/master/etc/jetpants.yaml.sample#L46-L59

The example results in errors when doing a shard cutover, digging reveals it should probably be:

sharded_tables:
    shardname:
        tablename:
            sharding_key: user_id
            chunks: 200

No unit/functional/e2e tests

I know you guys are using this in production but it's really hard to contribute from the outside without knowing if something is broken.
Unless I'm missing something this project has no tests at all.
This project needs coverage.
In order to start doing that we should enable travis and coveralls to track when and if tests fail.