odewahn / dds-field-guide

Project to track the distributed development stack (DDS).

Introduction

This project began while we were developing Atlas, O'Reilly Media's next-generation publishing tool. It seemed like every day we were finding interesting new tools in the DevOps space, so I started a "Sticky" for the most interesting-looking tools so I could explore them later.

At first, this worked fine. I was content to simply keep a list, where my only ordering criteria was "Huh, that looks cool. Someday when I have time, I'll take a look at that," in the same way you might buy an exercise DVD and then only occasionally pull it out and think "Huh, someday I'll get to that." But, as anyone who has watched DevOps for any length of time can tell you, it's a space bursting with interesting and exciting new tools, so my list and guilt quickly got out of hand.

Once I reached the limits of the Sticky as a medium, I started to look for patterns in my list. Some were obvious. For example, many of the tools, like Ansible, Salt, or (to a certain extent) Dockerfiles, fit into a clear infrastructure-automation group pioneered by Chef, CFEngine, and Puppet. So, too, the many cloud services.

But where would something like CoreOS, Docker, or Mesos fit? As I thought about how to group them, they seemed somehow tied up with the notion of containerization, but that just seemed too narrow. Rather, these projects and tools were part of a much larger trend — enabling clustering and distributed computing—and containerization was just a piece. So, rather than group by technology, it made sense to me to group by trend—in other words, what did the tool enable, and why was that trend important?

Simultaneously, other people at O'Reilly were also exploring this same question, but from a different perspective. In "Everything is distributed," Courtney Nash, the chair of Velocity, was asking "how do we manage systems that are too large to understand, too complex to control, and that fail in unpredictable ways." In "Beyond the stack," Mike Loukides was thinking about how "a new toolset has grown up to support the development of massively distributed applications," and described the profound consequences that the shift from well-tended, internal servers to disposable VMs was having on the traditional "LAMP" stack. (As well as its hipster cousin, the MEAN stack.)

So, it's from this context that my Sticky list grew into this Field Guide to the Distributed Development Stack. The Guide is organized into buckets based on a general observation, such as:

• The cloud is the default platform
• CI servers deploy code, not ops
• The codebase is in git
• The entire application runs locally in development
• The environment is automated in the code
• The monitoring infrastructure is critical
• Tests done in code, not by a QA department

In addition to being a (hopefully) useful framework, the Guide is also meant to be a living resource. So, we've put the source on GitHub and invite you to contribute. If you feel like we've missed a tool (which we most certainly have, since new things are popping up every day) or a major theme, then fork the repo and send me a pull request. We'll be keeping this document up to date and republishing it as we watch this trend continue to grow. We'll use O'Reilly Atlas to pull in the contributions and periodically republish the guide.

This is still very much a work-in-progress, but I hope it will be a resource you'll add to your own Sticky collection.

How to Contribute

To contribute to the DDS field guide:

• Fork this repo
• Agree to the O'Reilly Contributor License Agreement
• Add your tool / contribution
• Submit a pull request

If your request is accepted, we'll add you to the Contributors page.

Making a larger contribution

If you want to make a suggestion or contribution that is larger than just a single tool, it might make sense to begin the conversation as a GitHub issue, rather than a pull request. For example, if you want to add a new theme, or want to add a major narrative section, it would be good to discuss that first to make sure it's suitable for the guide. While I certainly don't want to limit what people contribute in any way, it's also the case that this guide will be centrally curated by me and other O'Reilly contributors.

dds-field-guide's Issues

I don't know it we should cite Twisted in Python framework in application stack

I think that saying that Twisted is a web framework is not false, but I think It's far away from what someone might expect.

Do not misunderstand me, I like Twisted, I made projects using Twisted in the past, but I guess target audience for this content is waiting for de-facto / most popular frameworks. I don't say Twisted for web dev is obscure, just that it's not the straightforward way nowadays.

See http://twistedmatrix.com/trac/wiki/WebDevelopmentWithTwisted for reference.

PS : I made this an issue to discuss it, because I was surprised when reading this. You can tell me that the more we have, the more exhaustive we are and that's fine :)

Guidelines for contributions?

I'm not really sure where this should go, but figured an issue is as good of a place as any for now. What I'm wondering is what form non-tool contributions should take. Writing something up and then throwing it over the wall may not be ideal if there's significant effort in writing it up and it just gets turned down. Perhaps collecting a set of TODOs and suggested topics or areas needing additional elaboration would be helpful.

One thing I'd like to see (and would be happy to contribute to, if time permits) would be adding some example scenarios of how these tools might be used in practice. E.g., for the monitoring chapter, it would be good to describe a couple situations using some of the tools mentioned and how they might fit into an overall infrastructure/architecture following the "patterns" (using that term very loosely here) described in the book.

"fork me on github" ribbon is not clickable

The "fork me on github" ribbon at http://sites.oreilly.com/odewahn/dds-field-guide/ is not clickable.

Item 2) of the "How to contribute" list works, though.

Consider new sections that are not tools

This guide is mostly focused on tools and technical practices, but there are a lot of aspects that go to business practices. Should there be a section for this?

For example, here are a few ideas:

• Building a DevOps culture
• Design and UX
• Non-Functional requirements is the competitive advantage (per @ragsns)

There are probably many, many others.