Somewhere between python2.0 and python2.4 the L prefix became unnecessary for long literals

Ideally handle these different cases:

# input
from six import PY2
if PY2:
   x = 5
else:
   x = 6
# output
x = 6

But also for

sys.version_info[0] == 2
sys.version_info < (3,)

etc.

• six.PY2
• six.PY3
• not six.PY2
• not six.PY3
• sys.version_info >= (3,)
• sys.version_info < (3,)
• sys.version_info[0] == 2
• sys.version_info[0] == 3

The following code which I tried to reduce (which I had no idea was possible):

thing = {'a': 1, 'b': 2}
print('{0[a]} and {0[b]}'.format(thing))

leads to

Traceback (most recent call last):
  File "/home/ethan/venv/bin/pyupgrade", line 10, in <module>
    sys.exit(main())
  File "/home/ethan/venv/lib/python3.7/site-packages/pyupgrade.py", line 1396, in main
    ret |= fix_file(filename, args)
  File "/home/ethan/venv/lib/python3.7/site-packages/pyupgrade.py", line 1372, in fix_file
    contents_text = _fix_fstrings(contents_text)
  File "/home/ethan/venv/lib/python3.7/site-packages/pyupgrade.py", line 1348, in _fix_fstrings
    tokens[i] = token._replace(src=_to_fstring(token.src, node))
  File "/home/ethan/venv/lib/python3.7/site-packages/pyupgrade.py", line 1310, in _to_fstring
    name = ''.join((params[k or str(i)], dot, rest))
KeyError: '0[a]'

This is a very esoteric format string usage, but 🤷‍♂️

In all my uses of pyupgrade, I want to apply it to every Python file in the repo - in general, some directory. pyupgrade $(find . -name '*.py') works OK on *nix systems, but I'd be happier if I could just write pyupgrade . and have it work on Windows too!

Personally I'd prefer to follow Black and flake8 (etc) by having recursive-by-default for directories, but if not a -r/--recursive flag is also fairly common.

For example

>>> set(x for x in range(1),)
{0}
>>> {x for x in range(1),}
  File "<stdin>", line 1
    {x for x in range(1),}
                        ^
SyntaxError: invalid syntax

I have projects that have this line at the start of every file, that I now want to upgrade to Python 3 only:

from __future__ import absolute_import, print_function, division

That's pretty common, everyone dropping Python 2 support from a codebase will want to do that cleanup, no?

It would be great if there was an option to just delete those lines (reviewing / adjusting the rest of the files would be up to me & automated tests to check).

#42 is related, but I thought I'd ask separately for this feature to remove any __future__ import lines.

2to3 can do it, but it leaves an empty line, doesn't delete the future line.
If you don't want to add something here, then maybe the 2to3 option could be mentioned in the README?

Running pyupgrade 1.5 on matplotlib's ticker.py as of, say, mpl3.0rc2 (https://github.com/matplotlib/matplotlib/blob/v3.0.0rc2/lib/matplotlib/ticker.py) gives me

Traceback (most recent call last):
  File "/usr/bin/pyupgrade", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python3.7/site-packages/pyupgrade.py", line 907, in main
    ret |= fix_file(filename, args)
  File "/usr/lib/python3.7/site-packages/pyupgrade.py", line 884, in fix_file
    contents_text = _fix_fstrings(contents_text)
  File "/usr/lib/python3.7/site-packages/pyupgrade.py", line 858, in _fix_fstrings
    tokens[i] = token._replace(src=_to_fstring(token.src, node))
  File "/usr/lib/python3.7/site-packages/pyupgrade.py", line 824, in _to_fstring
    name = ''.join((params[k or str(i)], dot, rest))
KeyError: '3'

ur literals are invalid syntax in python3.x

I have the following piece of code:

            if preview_version_string:
                actions.append(
                    'ttfix --name-version="{v}" -o "{target}" "{target}"'.format(
                        target=target_path, v=preview_version_string
                    )
                )

            if kerntable_file:
                actions.append(
                    (
                        "kerntable --ufo {ufo} --glyphset {kerntable} "
                        '"{target}" "{target}"'
                    ).format(ufo=ufo_path, target=target_path, kerntable=kerntable_file)
                )
                file_deps.append(kerntable_file)

pyupgrade --py36-plus did not upgrade those to f-strings.

on 1.5.1

do("foo {}".format(bar))
do("foo {}"
   .format(bar))

the first line gets fstringified, not the second one.
(imagine very long strings, such that it made sense to break a line before the method call)

For example:

dict(
    (
        foo,
        bar,
    )
    for foo, bar in baz
)

Is rewritten to

{
    
        foo:
        bar,
    
    for foo, bar in baz
}

% pyupgrade --py3-only --keep-percent-format src/_pytest/fixtures.py
Traceback (most recent call last):
  File "…/Vcs/pytest/.venv/bin/pyupgrade", line 11, in <module>
    sys.exit(main())
  File "…/Vcs/pytest/.venv/lib/python3.7/site-packages/pyupgrade.py", line 1428, in main
    ret |= fix_file(filename, args)
  File "…/Vcs/pytest/.venv/lib/python3.7/site-packages/pyupgrade.py", line 1402, in fix_file
    contents_text = _fix_py3_plus(contents_text)
  File "…/Vcs/pytest/.venv/lib/python3.7/site-packages/pyupgrade.py", line 1244, in _fix_py3_plus
    _replace_call(tokens, i, end, func_args, template)
  File "…/Vcs/pytest/.venv/lib/python3.7/site-packages/pyupgrade.py", line 1171, in _replace_call
    src = tmpl.format(args=arg_strs, rest=rest)
IndexError: list index out of range

print(repr(tmpl), repr(arg_strs), repr(rest)) shows:

'raise {args[1]}.with_traceback({args[2]})' ['*err'] ''

Could pyupgrade remove __future__ imports that are no longer necessary? For example, absolute_import when running in Python 3 mode.

In Python 3, source files are decoded as utf-8 by default. This allows Python 3 only codebases to drop the encoding cookie, # -*- coding: utf-8 -*-. pyupgrade could include this transformation when --py3-plus is passed.

Currently I run pyupgrade with something like find ... | xargs pyupgrade. A recursive flag to drill down directories would be nice (and in line with a lot of other formatters like black, yapf, and isort).

In Python 3, the first positional argument to .encode() and .decode() defaults to 'utf-8'. This was not the case with Python 2:

https://docs.python.org/3/library/stdtypes.html#str.encode
https://docs.python.org/3/library/stdtypes.html#bytes.decode

As this is the most common value passed, there could be a transform to simplify the calls to just .encode() or .decode() (without the first positional argument). This could help clean up and simplify modern Python code by removing noise.

Similar to #47, let's automatically fix this as it becomes a SyntaxWarning in python3.8

$ python3.8 -Werror
Python 3.8.0a3 (default, Mar 27 2019, 03:46:44) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = 5
>>> x is 5
  File "<stdin>", line 1
SyntaxError: "is" with a literal. Did you mean "=="?
>>> 

Let's try and automatically un-six python source. Here's a quick audit of what six does and how to undo it in python 3:

From the docs:

trivial replacements

• six.text_type => str
• six.binary_type => bytes
• six.class_types => (type,)
• six.string_types => (str,)
• six.integer_types => (int,)
• six.unichr => chr
• six.iterbytes => iter
• six.print_(...) => print(...)
• six.exec_(c, g, l) => exec(c, g, l)
• six.advance_iterator(it) => next(it)
• six.next(it) => next(it)
• six.callable(x) => callable(x)

trivial removals:

• python_2_unicode_compatible (remove decorator)

  @six.python_2_unicode_compatible
  class C: pass

requires parsing a function call

• six.u('foo') => 'foo'
• six.byte2int(bs) => bs[0]
• six.indexbytes(bs, i) => bs[i]
• six.iteritems(dct) => dct.items() # also {iter,view}{items,keys,values}
• six.create_unbound_method(fn, cls) => fn
• six.get_unbound_method(meth) => meth
• six.get_method_function(meth) => meth.__func__
• six.get_method_self(meth) => meth.__self__
• six.get_function_closure(fn) => fn.__closure__
• six.get_function_code(fn) => fn.__code__
• six.get_function_defaults(fn) => fn.__defaults__
• six.get_function_globals(fn) => fn.__globals__
• six.assertCountEqual(self, arg1, arg2) => self.assertCountEqual(arg1, arg2)
• six.assertRaisesRegex(self, fn, *a, **k) => self.assertRaisesRegex(fn, *a, **k)
• six.assertRegex(self, *a) => self.assertRegex(*a)

maybe replace with literal?

• six.b => replace with bytes literal?

need to know that you're not in a place where a raise statement is invalid

• six.raise_from(exc, exc_from) => raise exc from exc_from
• six.reraise(tp, exc, tb) => raise exc.with_traceback(tb)

probably can reuse the "new style class" logic

• class C(six.Iterator): => class C:

requires multiple function call parses, probably hard

• class C(six.with_metaclass(Meta, Base)): => class C(Base, metaclass=Meta):

• (rewriting add_metaclass)

  @six.add_metaclass(mcs)
  class C(...): pass

  to

  class C(..., metaclass=mcs): pass

probably can't do (requires introducing an import)

• six.wraps(fn) => functools.wraps(fn): asottile/reorder-python-imports#59

Note that for six.moves I'm planning on handling that in reorder-python-imports

Python 3.7.0
pyupgrade==1.4.0

Before

bad = 1

print ("Test FAIL: ".format(bad))

Run

$ pyupgrade 1.py --py36-plus
Rewriting 1.py

After

bad = 1

print (f"Test FAIL: ")

E.g. our style guide forces printf-like syntax (we switch to C-like languages often and cut the corners) and we have to revert the corresponding upgrades as in src-d/ml#322 (review)

So would be nice to enable/disable features besides the master switch by Python version.

No need for async for support according to this

Here's an ast parser to detect this, just need the tokenization to rewrite it:

class Visitor(ast.NodeVisitor):
    def __init__(self, filename):
        self.filename = filename

    def visit_For(self, node: ast.For):
        if (
            isinstance(node.target, ast.Name) and
            len(node.body) == 1 and
            isinstance(node.body[0], ast.Expr) and
            isinstance(node.body[0].value, ast.Yield) and
            isinstance(node.body[0].value.value, ast.Name) and
            node.target.id == node.body[0].value.value.id
        ):
            print(f'{self.filename}:{node.lineno}: could be yield from')

        self.generic_visit(node)

This change hugovk/cartopy@f1c5b7a fails unit tests, which passed before:

-        return u'%.4g, %.4g (%f\u00b0%s, %f\u00b0%s)' % (x, y, abs(lat),
-                                                         ns, abs(lon), ew)
+        return u'{:.4g}, {:.4g} ({:f}\u00b0{}, {:f}\u00b0{})'.format(x, y, abs(lat),
+                                                                     ns, abs(lon), ew)

https://travis-ci.org/hugovk/cartopy/jobs/463956921#L1111

Small example:

Python 3.7.1 (default, Nov  6 2018, 18:45:35)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> x, y = np.array([-969100.0]), np.array([-4457000.0])
>>>
>>> # Before
...
>>> u"%.4g, %.4g" % (x, y)
'-9.691e+05, -4.457e+06'
>>>
>>> # After
...
>>> u"{:.4g}, {:.4g}".format(x, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported format string passed to numpy.ndarray.__format__
>>>

Hi!

Nice tool! I noticed that it's not listed at https://pre-commit.com/hooks.html although I suppose it should be? I didn't find the pre-commit website as a repository on GitHub so I thought I'd report here for a start. Any ideas how to get it listed?

Best, Sebastian

A file containing

a = 2
b = "%s" % a

is not rewritten to use .format() unless I wrap the a into a one-element tuple like b = "%s" % (a,).

Python 3.7.2
pyupgrade 1.11.0

Input:

given_keyword = "a"
and_keyword = "b"

input = '%(given)s a string with Given in it   \n%(and)s another string' % {
    'given': given_keyword,
    'and': and_keyword,
}

Output:

given_keyword = "a"
and_keyword = "b"

input = '{given} a string with Given in it   \n{and} another string'.format(
    given=given_keyword,
    and=and_keyword,
)

Running the output causes:

    and=and_keyword,
      ^
SyntaxError: invalid syntax

You mention that f-string support was planned in the README, and I'm just curious if you had a rough idea of when that might be.

Big fan of the project by the way.

This line is not upgraded:

text = 'TIME {}: {:f} '.format(label, time.time() - t)

This one is super minor but probably worth fixing

dict((a, b)for a, b in y)

gets rewritten to

{a: bfor a, b in y}

Currently:

-datetime.datetime(1991, 07, 05)
+datetime.datetime(1991, 0o7, 0o5)

Better:

-datetime.datetime(1991, 07, 05)
+datetime.datetime(1991, 7, 5)

Is this supported? If not, could it be?

Currently ony tuples and dicts used in % format are converted to .fortmat() calls. Would be nice if single parameter non tuple percentformat Strings would be converted, too.

In 2+3 code, a native literal (py2: bytes, py3: str) is often written as str('foo') -- this can be cleaned up automatically

Thanks for this handy tool!

It would be nice to be able to run pyupgrade on a directory, and have it process all .py files in that directory and subdirectories.

For example:

pyupgrade .

This would be similar to other tools like Flake8 and Black.

In the meantime, this works:

pyupgrade `find . -name "*.py"`

Code such as

class EventProcessorMeta(six.with_metaclass(abc.ABCMeta)):

is being rewritten as invalid Python

class EventProcessorMeta(, metaclass=abc.ABCMeta):

Hi.

The pyupgrade tool is introducing a raw string where a line continuation is being used.

Check this example below:

import textwrap


def test_importerror(tmpdir):
    print(tmpdir)
    p1 = tmpdir.join("test1.txt")
    p2 = tmpdir.join("test2.txt")
    p1.write(
        textwrap.dedent(
            """\
        import doesnotexist

        x = 1
        """
        )
    )
    p2.write(
        textwrap.dedent(
            """
        import doesnotexist

        x = 2
        """
        )
    )

after running pyupgrade on this file, a r""" is introduced as it's demonstrated above.

def test_importerror(tmpdir):
    print(tmpdir)
    p1 = tmpdir.join("test1.txt")
    p2 = tmpdir.join("test2.txt")
    p1.write(
        textwrap.dedent(
            r"""\
        import doesnotexist

        x = 1
        """
        )
    )
    p2.write(
        textwrap.dedent(
            """
        import doesnotexist

        x = 2
        """
        )
    )

I'm running pyupgrade (1.10.0) with python Python 3.6.2 on Windows 10, version 1709

Thank you very much for writing this tool!!

I want to run pyupgrade over a project that probably still contains code that raises a syntax error when imported in a python3 interpreter.

pyupgrade --py36-plus outputs is the following in a python3.7 conda env:

filename1.py
filename2.py
Traceback (most recent call last):
  File "/home/thomas/miniconda/envs/py37/bin/pyupgrade", line 10, in <module>
    sys.exit(main())
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/site-packages/pyupgrade.py", line 1428, in main
    ret |= fix_file(filename, args)
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/site-packages/pyupgrade.py", line 1397, in fix_file
    contents_text = _fix_format_literals(contents_text)
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/site-packages/pyupgrade.py", line 107, in _fix_format_literals
    tokens = src_to_tokens(contents_text)
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/site-packages/tokenize_rt.py", line 44, in src_to_tokens
    ) in tokenize.generate_tokens(tokenize_target.readline):
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/tokenize.py", line 579, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (350, 0))

Unfortunately this exception is not very helpful, since it doesn't really tell me which file lead to this exception. Note that filename2.py is not the culprit, because it doesn't have a line with number 350.

IMO pyupgrade should avoid showning tracebacks by defaults, but instead output filename+linum of the faulty files.

Tested it with

(py37) > $ conda list pyupgrade                                                                                 [±master ●●]
# packages in environment at /home/thomas/miniconda/envs/py37:
#
# Name                    Version                   Build  Channel
pyupgrade                 1.17.0                     py_0    conda-forge

BTW pyupgrade doesn't have a --version cli flag.

This is a syntax error in python3.x:

'\N'

I suspect something has changed in offset information in ast. See if it's a fix or a bug and report before 3.8 releases

$ pytest tests -v | grep FAILED
tests/pyupgrade_test.py::test_sets[set((1, 2))-{1, 2}] FAILED            [  7%]
tests/pyupgrade_test.py::test_sets[set([x for x in y])-{x for x in y}] FAILED [  8%]
tests/pyupgrade_test.py::test_sets[set((x for x in y))-{x for x in y}] FAILED [  8%]
tests/pyupgrade_test.py::test_sets[set(((1, 2)))-{1, 2}] FAILED          [  9%]
tests/pyupgrade_test.py::test_sets[set((a, b) for a, b in y)-{(a, b) for a, b in y}] FAILED [  9%]
tests/pyupgrade_test.py::test_sets[set(((1, 2), (3, 4)))-{(1, 2), (3, 4)}] FAILED [  9%]
tests/pyupgrade_test.py::test_sets[set((((1, 2),),))-{((1, 2),)}] FAILED [ 11%]
tests/pyupgrade_test.py::test_sets[set(\n(1, 2))-{\n1, 2}] FAILED        [ 11%]
tests/pyupgrade_test.py::test_sets[set((\n1,\n2,\n))\n-{\n1,\n2,\n}\n] FAILED [ 12%]
tests/pyupgrade_test.py::test_sets[set((frozenset(set((1, 2))), frozenset(set((3, 4)))))-{frozenset({1, 2}), frozenset({3, 4})}] FAILED [ 12%]
tests/pyupgrade_test.py::test_sets[set((1,))-{1}] FAILED                 [ 12%]
tests/pyupgrade_test.py::test_sets[set((1, ))-{1}] FAILED                [ 13%]
tests/pyupgrade_test.py::test_sets[set((x for x in y),)-{x for x in y}] FAILED [ 13%]
tests/pyupgrade_test.py::test_sets[set(\n    (x for x in y),\n)-{\n    x for x in y\n}] FAILED [ 14%]
tests/pyupgrade_test.py::test_dictcomps[dict((a, b) for a, b in y)-{a: b for a, b in y}] FAILED [ 16%]
tests/pyupgrade_test.py::test_dictcomps[dict((a, b,) for a, b in y)-{a: b for a, b in y}] FAILED [ 17%]
tests/pyupgrade_test.py::test_dictcomps[dict((a, b, ) for a, b in y)-{a: b for a, b in y}] FAILED [ 17%]
tests/pyupgrade_test.py::test_dictcomps[dict([a, b] for a, b in y)-{a: b for a, b in y}] FAILED [ 17%]
tests/pyupgrade_test.py::test_dictcomps[dict(((a, b)) for a, b in y)-{a: b for a, b in y}] FAILED [ 18%]
tests/pyupgrade_test.py::test_dictcomps[dict([(a, b) for a, b in y])-{a: b for a, b in y}] FAILED [ 18%]
tests/pyupgrade_test.py::test_dictcomps[dict([(a, b), c] for a, b, c in y)-{(a, b): c for a, b, c in y}] FAILED [ 18%]
tests/pyupgrade_test.py::test_dictcomps[dict(((a), b) for a, b in y)-{(a): b for a, b in y}] FAILED [ 19%]
tests/pyupgrade_test.py::test_dictcomps[dict((k, dict((k2, v2) for k2, v2 in y2)) for k, y2 in y)-{k: {k2: v2 for k2, v2 in y2} for k, y2 in y}] FAILED [ 19%]
tests/pyupgrade_test.py::test_dictcomps[dict((a, b)for a, b in y)-{a: b for a, b in y}] FAILED [ 19%]
tests/pyupgrade_test.py::test_dictcomps[dict(\n    (\n        a,\n        b,\n    )\n    for a, b in y\n)-{\n        a:\n        b\n    for a, b in y\n}] FAILED [ 20%]
tests/pyupgrade_test.py::test_dictcomps[x(\n    dict(\n        (a, b) for a, b in y\n    )\n)-x(\n    {\n        a: b for a, b in y\n    }\n)] FAILED [ 21%]
tests/pyupgrade_test.py::test_percent_format["%s" % ("%s" % ("nested",),)-"{}".format("{}".format("nested"))] FAILED [ 63%]
tests/pyupgrade_test.py::test_main_changes_a_file FAILED                 [ 97%]
tests/pyupgrade_test.py::test_main_keeps_line_endings FAILED             [ 97%]

Seems to be related to tuple offset calculation -- which I'm pretty sure was buggy before

Currently this is happening, but it causes a SyntaxError -- the extra set of parens is necessary here:

 def run_tests():
     url = options.url + '/getCaseCount'
     control_ws = yield websocket_connect(url, None)
-    num_tests = int((yield control_ws.read_message()))
+    num_tests = int(yield control_ws.read_message())

When the format string contains braces, converting %s to {} can be quite unwieldy:

$ echo '"{%s, %s}" % (x, y)' >! /tmp/test.py
$ pyupgrade --py36-plus test.py
$ cat test.py
f"{{{x}, {y}}}"

A real case (from the Matplotlib examples), involving the generation of TeX strings:

-        label = r'$%s_{_{\mathrm{%s}}}$' % (orig_label[0], orig_label[1:])
+        label = r'${}_{{_{{\mathrm{{{}}}}}}}$'.format(orig_label[0], orig_label[1:])

I would thus suggest adding a switch --avoid-brace-brace (name up to bikeshedding).

Mistakenly does the following:

-    return'foo'
+    retrn'foo'

Python 3.6 has an insertion order preserving dict. Python 3.7 has this defined as a language feature.

This obviates collections.OrderedDict usage, so this tool could rewrite OrderedDict as dict literals when --py36-plus is passed.

Along the same lines as the long literals

This pyupgrade change (https://github.com/hugovk/yamllint/commit/25e8e9be0838bff930b3b204360fc24385b6edbf) fails indentation-related unit tests which previously passed:

-            output += '%9s %s\n' % (token_type,
+            output += '{:9} {}\n'.format(token_type,
                                     self.format_stack(context['stack']))

https://travis-ci.org/hugovk/yamllint/jobs/459076859#L860

Small example:

Python 3.7.1 (default, Nov  6 2018, 18:45:35)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> '%9s %s\n' % ("abc", "def")
'      abc def\n'
>>> '{:9} {}\n'.format("abc", "def")
'abc       def\n'

It would have worked if they were integers:

>>> '%9s %s\n' % (123, 456)
'      123 456\n'
>>> '{:9} {}\n'.format(123, 456)
'      123 456\n'

Another issue with this example, it'd be nice if the indentation of the second line was adjusted, as this often causes flake8 errors. But I understand if this is out of scope of pyupgrade.

For example, instead of this:

-            output += '%9s %s\n' % (token_type,
+            output += '{:9} {}\n'.format(token_type,
                                     self.format_stack(context['stack']))

Do this:

-            output += '%9s %s\n' % (token_type,
-                                    self.format_stack(context['stack']))
+            output += '{:9} {}\n'.format(token_type,
+                                         self.format_stack(context['stack']))

In Python 3.3+ the following exceptions are now simple aliases of OSError: IOError, EnvironmentError, WindowsError, mmap.error, socket.error and select.error.

The aliases have been removed from the documented hierarchy: https://docs.python.org/3/library/exceptions.html#exception-hierarchy

I think pyupgrade should rewrite these aliases as the canonical OSError.

References:

https://docs.python.org/3/whatsnew/3.3.html#pep-3151-reworking-the-os-and-io-exception-hierarchy

https://docs.python.org/3/library/exceptions.html#OSError