yelp / undebt

A fast, straightforward, reliable tool for performing massive, automated code refactoring

Undebt

Undebt is a fast, straightforward, reliable tool for performing massive, automated code refactoring used @Yelp. Undebt lets you define complex find-and-replace rules using standard, straightforward Python that can be applied quickly to an entire code base with a simple command.

To learn about what Undebt is and why we created it, check out our post on the Yelp Engineering Blog.

Get Started

To get started using Undebt, install with

pip install undebt

then head over to our documentation for more information.

Example

While the full list of examples can be found in the documentation, to show you how it's done we'll go in-depth into one example in particular, class_inherit_object.py. Like most of the examples, this pattern is built for Python, but in theory Undebt could be used with any language. The idea of this pattern is to convert any usage of old-style classes to new-style classes by making all classes that don't inherit from anything else inherit from object. The code for this pattern is incredibly simple—a mere four lines not counting imports:

grammar = INDENT + Keyword("class").suppress() + NAME + (Optional(LPAREN + RPAREN) + COLON).suppress()

@tokens_as_list(assert_len=2)
def replace(tokens):
    return tokens[0] + "class " + tokens[1] + "(object):"

What's going on here? The basic idea is that grammar defines what to look for, and replace defines how to change it. Undebt scans your files looking for grammar, tokenizes the matching text, passes the tokens to replace, and replaces the original text with the return value. For a more in-depth explanation of how to use grammar and replace in a pattern file, see the pattern files documentation.

In this particular case, grammar is defined to match an indent (INDENT), followed by a class definition (+ Keyword("class") + NAME) that doesn't inherit from anything (+ Optional(LPAREN + RPAREN) + COLON). Along the way, all the tokens except for the indent and the class name are suppressed, that way replace only gets those two tokens, which it reassembles into a class definition that inherits from object. For a full specification of all of the helper objects used here, see the pattern utilities documentation.

To run this pattern on your code:

(1) Install Undebt by entering into your command line

pip install undebt

(2) Run undebt with class_inherit_object as the pattern

undebt --pattern undebt.examples.class_inherit_object <file to undebt> ...

For a complete command line example and the full command line help, see the command line documentation, which includes tips and tricks to show how you how to use Undebt with other common Unix utilities.

undebt's Issues

Refactor to avoid needing to patch `sys.argv`

As per discussion here

Add some contribution documentation

It could help 🍻

bin/__main__.py

undebt-0.2.2 installs bin/__main__.py -- it shouldn't pollute the bin directory that way. (Distfile from pypi in case it matters.)

readthedocs is out of date

it still shows that undebt.pattern.python is a thing, and has no reference to the new undebt.pattern.lang module or its submodules.

Adding a 'stdlib' of anti-patterns fixes

Something that quantified code (will shutdown in october) was trying to achieve, was a tool that could detect bad python code and rewrite it for you. Writing patterns involved using their DSL but with undebt, we can write them in python!

They had a rather sizeable list of anti-patterns:

• https://www.quantifiedcode.com/knowledge-base/

Do you imagine the undebt repository as a place for a 'stdlib' of python code fixups that it could run automatically on a code base, based on the above anti-patterns? We could start to port the anti-patterns into grammar definitions.

unintended replacements in remove_needless_u_specifier example

I gave undebt a try using remove_needless_u_specifier.py and noticed some instances where it replaced more than expected when a string ended with a u character:

     try:
         subprocess.check_call(('mysql',
-                               '-u', 'test_user',
+                               '-', 'test_user',
…)

 TITLE_LABELS = {
     "en": ("Subject", ),
-    "ru": ("Тема", ),
+    "r": ("Тема", ),
     "fr": ("Objet", "Sujet"),
…

Create a submodule under undebt.pattern for languages?

Theoretically, we will one day have common patterns for a variety of languages. Rather than littering undebt.pattern with {python, ruby, java, ...} modules, why don't we create a module called undebt.pattern.lang to house all of the patterns for the various languages?

Unfortunately, undebt/pattern/python.py will have to be moved there as well, which means all of the code that uses it will have to be updated. But, it's probably easier to do this now rather than in the future.

Automate generation of CLI usage

Apparently there's some way we've done this in the past.

Let's do it again, so I stop forgetting things.

Example test case for Swift 2.2 to Swift 3.0

first question, is this undebt only work for python code? if it work for other language that is "pyparsing grammar object"

for example in current swift 2.2, it is written like below
if let x = y, a = b where a == x {

how to use "undebt" to parse above line to below line?
if let x = y, let a = b, a == x {

Include a "Tips" section in the README

Ideally it would have a bunch of usage tips for making use of other bash tools (git, grep, xargs, etc) to make better use of undebt

Set up travis-ci

Improve test coverage to 100%

We can work on this after #37

pypi distfile does not include tests

The pypi distfile does not include the tests. Please add them.

How can I match only standalone NAMEs?

I expected that DOTTED_NAME will matched only standalone identifiers (identifiers that start a new expression) but it also matches trailing ones.

For instance I wrote a recipe to apply some patterns like util.asbool(request.form.get('flag', False)) => request.form.get('flag', False, type=util.asbool) and it unexpectedly made a change like below:

- a = session.query(Model).get(request.form.get('id'))
+ a = session.query(Model).request.form.get('id', type=get)

(By the way, this tool is awesome! 👍 🍰 👍 Thanks.)

Add a helper function to match exactly N repititions of a pattern

This could be a useful addition to our collection of helpers.

Something like:

def sequence(g, n):
    return reduce(lambda x, y: x + y, [g] * n)

@evhub thoughts?

Replace deprecated `<>` operator with `!=`

Related to #21.

I'll create an issue as I go, to make it more visible which ones I are finished.

Ref: Avoid deprecated <> operator

Should the assert_keys argument to tokens_as_dict check for subset?

Consider the following grammar:

grammar = (
     Optional(Keyword("mymodule") + DOT.suppress())("module")
     + condense(NAME + ZeroOrMore(DOT + ~Keyword("foo") + NAME))("classname")
     + DOT.suppress()
     + Keyword("foo")("method")
     + LPAREN.suppress()
     + NAME("arg")
     + RPAREN.suppress()
)

I would like to use the assert_keys argument to tokens_as_list to verify that classname, method and arg are all present, but since it checks for strict set equality, this will fail when the pattern captured by module appears in the input.

Should we check that the list of keys provided is a subset of the actual keys instead?

`_file_processor` no longer needs to be an object

Since we removed multiprocessing support, the file processor thingy doesn't actually need to be pickleable, so it doesn't have to be an object.

Perhaps there is a better way to structure this part of the code now?

Dry run option

Is there an option to not have it write in place?

If I just want to dry run this application without actually changing anything is that possible?

Remove support for path-like module names

In #41, Undebt was changed to only support importing pattern modules in sys.path, and to support writing those module names either how they would be imported, or as if they are a path. I propose removing the support for path-like module names, since the fact that they look like paths seems to imply that any path is valid, when in fact that's not the case.

This is especially true in the context of the README, which is now outdated and, most importantly, the example in which will no longer work, since it says to pass the path to the pattern file you create instead of installing it as a module and passing the module name.

Dry run option should have variant to show diff

From #5 (comment)

The dry run option should have a variant command that will print a standard unified diff to stdout instead of printing the new file version.

For a test case, we should be able to assert dry-run-stdout == (dry-run-with-diff-stdout-piped-to-patch)

tox is not in requirements-dev.txt

Allow pattern to be a module instead of a file

Would enable a pre-commit repo that defines a bunch of modules to just have a config like:

- id: fix-fixer-1
  entry: undebt
  language: python
  args: [-p, mypkg.fixer_1]

Modify CLI to support input paths as positional arguments

Quoting @asottile in #20:

Then I'd suggest changing the CLI to instead be:

undebt -p pattern [-p pattern ...] FILENAME [FILENAME ...]

This is pretty standard for most linting / rewriting tools and certainly helps with xargs as above (and would make undebt compatible to use as a pre-commit hook)

Support --grep option to only undebt the files that contain a particular string

Most of the time when I use undebt, I grep a particular string to find (then cut/pipe) the filenames I want to run on instead of running on a directory with massive number of files.
Wonder if it worthy to incorporate this feature into undebt?

For example, I normally do:
git grep | cut -d ":" -f 1 | sort | uniq > filenames.txt
cat filenames.txt | xargs -n 1 undebt -p <my_pattern> -i

It would be nice to have something like:
undebt -p <my_pattern> --grep

Remove directory traversal and multiprocess

Now that we have documented examples explaining how to use undebt in conjunction with tools like grep, find and xargs, there is no reason to include directory traversal or multiprocessing. These functionalities are completely contained within these other tools. Let's remove this code.