omnilib / ufmt

Safe, atomic formatting with black and µsort

µfmt

Safe, atomic formatting with black and µsort

µfmt is a safe, atomic code formatter for Python built on top of black and µsort:

Black makes code review faster by producing the smallest diffs possible. Blackened code looks the same regardless of the project you’re reading.

μsort is a safe, minimal import sorter. Its primary goal is to make no “dangerous” changes to code, and to make no changes on code style.

µfmt formats files in-memory, first with µsort and then with black, before writing any changes back to disk. This enables a combined, atomic step in CI/CD workflows for checking or formatting files, without any chance of conflict or intermediate changes between the import sorter and the code formatter.

Install

µfmt requires Python 3.8 or newer. You can install it from PyPI:

$ pip install ufmt

If you want to prevent unexpected formatting changes that can break your CI workflow, make sure to pin your transitive dependencies–including black, µsort, and µfmt–to your preferred versions.

If you use requirements.txt, this might look like:

black==22.6.0
ufmt==2.0.0
usort==1.0.4

Usage

To format one or more files or directories in place:

$ ufmt format <path> [<path> ...]

To validate files are formatted correctly, like for CI workflows:

$ ufmt check <path> [<path> ...]

To validate formatting and generate a diff of necessary changes:

$ ufmt diff <path> [<path> ...]

Integrations

See the user guide for details on each integration.

GitHub Actions

µfmt provides a GitHub Action that can be added to an existing workflow, or as a separate workflow or job, to enforce proper formatting in pull requests:

jobs:
  ufmt:
    runs-on: ubuntu-latest
    steps:
      - uses: omnilib/ufmt@action-v1
        with:
          path: <PATH TO CHECK>
          requirements: requirements-fmt.txt
          python-version: "3.x"

pre-commit hook

µfmt provides a pre-commit hook. To format your diff before every commit, add the following to your .pre-commit-config.yaml file:

  - repo: https://github.com/omnilib/ufmt
    rev: v2.0.0
    hooks:
      - id: ufmt
        additional_dependencies: 
          - black == 22.6.0
          - usort == 1.0.4

Visual Studio Code

µfmt has an official VS Code extension to use µfmt as a formatter for Python files. Once installed, µfmt can be set as the default formatter with the following settings:

"[python]": {
  "editor.defaultFormatter": "omnilib.ufmt"
}

µfmt can automatically format when saving with the following settings:

"[python]": {
  "editor.defaultFormatter": "omnilib.ufmt",
  "editor.formatOnSave": true
}

For more details, or to install the extension, see the Visual Studio Marketplace page:

License

ufmt is copyright Amethyst Reese, and licensed under the MIT license. I am providing code in this repository to you under an open source license. This is my personal repository; the license you receive to my code is from me and not from my employer. See the LICENSE file for details.

ufmt's Issues

Drop support for 3.7

Black and flake8 no longer support it, and it's now EOL. We can go ahead and drop support.

µfmt doesn't handle exceptions from black/µsort

(.venv) jreese@butterfree ~/workspace/torch/fairseq main  » ufmt check .
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/jreese/.pyenv/versions/3.9.9/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/Users/jreese/.pyenv/versions/3.9.9/lib/python3.9/concurrent/futures/process.py", line 202, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/Users/jreese/.pyenv/versions/3.9.9/lib/python3.9/concurrent/futures/process.py", line 202, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/ufmt/core.py", line 88, in ufmt_file
    dst_contents = ufmt_string(path, src_contents, usort_config, black_config)
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/ufmt/core.py", line 50, in ufmt_string
    content = format_file_contents(content, fast=False, mode=mode)
  File "src/black/__init__.py", line 993, in format_file_contents
  File "src/black/__init__.py", line 970, in check_stability_and_equivalence
  File "src/black/__init__.py", line 1361, in assert_equivalent
AssertionError: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /var/folders/jg/hrl2p4ms1vjbn44gmmyl4p7r0000gn/T/blk_d04ehvsc.log
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jreese/workspace/torch/fairseq/.venv/bin/ufmt", line 8, in <module>
    sys.exit(main())
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/ufmt/cli.py", line 54, in check
    results = ufmt_paths(paths, dry_run=True)
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/ufmt/core.py", line 117, in ufmt_paths
    results = list(runner.run(all_paths, fn).values())
  File "/Users/jreese/workspace/torch/fairseq/.venv/lib/python3.9/site-packages/trailrunner/core.py", line 183, in run
    results = list(exe.map(func, paths))
  File "/Users/jreese/.pyenv/versions/3.9.9/lib/python3.9/concurrent/futures/process.py", line 559, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/Users/jreese/.pyenv/versions/3.9.9/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "/Users/jreese/.pyenv/versions/3.9.9/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/Users/jreese/.pyenv/versions/3.9.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
AssertionError: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /var/folders/jg/hrl2p4ms1vjbn44gmmyl4p7r0000gn/T/blk_d04ehvsc.log

Importing version from __version__.py makes building conda-forge packages awkward

First of all thank you for µfmt, it is a really useful addition to my CI pipelines. So useful, that I am in the process of packaging it for conda-forge.

While creating the package I, however, ran into an issue that makes the build process less optimal than it could be. As µfmt is using flit_core as metadata backend, the pip build process will run flit_core during build time. As a step of this, flit is trying to determine the package version from __init__.py. It does so using two passes:

1. An AST-based pass that tries to find an assignment of the form __version__ = "x.y.z"
2. An import based pass that loads the package and then inspects the __version__ variable.

For µfmt, pass 1) fails as the version variable is defined in __version__.py. This leads to pass 2) being run, which succeeds. However in the conda-forge build environment, runtime dependencies are not installed by default, leading to a build failure. Thus, as a workaround, we need to add the relevant runtime dependencies to the build step, which is non-ideal but works.

Please consider defining __version__ directly in __init__.py to allow making the conda-forge package more straightforward.

Link to the conda-forge pull request for reference: conda-forge/staged-recipes#20527

P.S.: the same behavior can be observed for trailrunner and stdlibs. For the latter no failure happens as there are no additional dependencies though. Please let me know in case you consider a change and I can open separate issues.

Syntax errors don't print the name of the file

If there's a file containing a an unclosed paren, e.g. just ( ithen python -m ufmt check pkg prints a bunch of stuff but not the name of the file:

(.venv) [tim@aerynsun wreck]$ make lint
python -m ufmt check wreck setup.py
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 151, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: TokenType(ENDMARKER)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 202, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 202, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/ufmt/core.py", line 88, in ufmt_file
    dst_contents = ufmt_string(path, src_contents, usort_config, black_config)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/ufmt/core.py", line 43, in ufmt_string
    content = usort_string(content, usort_config, path)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/usort/api.py", line 90, in usort_string
    raise result.error
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/usort/api.py", line 31, in usort
    module = try_parse(data=data, path=path)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/usort/util.py", line 76, in try_parse
    raise parse_error or Exception("unknown parse failure")
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/usort/util.py", line 65, in try_parse
    mod = cst.parse_module(
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 109, in parse_module
    result = _parse(
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 56, in _parse
    return _pure_python_parse(
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 89, in _pure_python_parse
    result = parser.parse()
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 110, in parse
    self._add_token(token)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 186, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 2:1.
Incomplete input. Unexpectedly encountered end of file (EOF).

(
 ^
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/ufmt/__main__.py", line 4, in <module>
    main()
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/ufmt/cli.py", line 54, in check
    results = ufmt_paths(paths, dry_run=True)
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/ufmt/core.py", line 117, in ufmt_paths
    results = list(runner.run(all_paths, fn).values())
  File "/home/tim/code/wreck/.venv/lib/python3.9/site-packages/trailrunner/core.py", line 193, in run
    results = list(exe.map(func, paths))
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 559, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
libcst._exceptions.ParserSyntaxError: Syntax Error @ 2:1.
Incomplete input. Unexpectedly encountered end of file (EOF).

(
 ^
make: *** [Makefile:28: lint] Error 1

ufmt reports syntax error on correct structural pattern matching expression

Hey there! First of all, thanks for ufmt, we are using it in most of our production python projects, very handy!

Today I encountered an interesting issue with ufmt in a file that had some structural pattern matching expressions.
But first, my environment:

Python 3.10.8

black==22.12.0
ufmt==2.0.1
usort==1.0.5

When i run ufmt against a file with this code in it:

def this_breaks(foo: str) -> None:
    match foo:
        case "":
            pass
        case "something else":
            pass

it fails reporting a syntax error:

$ ufmt format ufmt_spm_error.py 
Error formatting [REDACTED]/ufmt_spm_error.py: Syntax Error @ 2:11.
✨ 1 errors ✨

black itself is perfectly happy with this code (it is valid afaict and it works in the service we use this in)

$ black ufmt_spm_error.py
All done! ✨ 🍰 ✨1 file left unchanged.

so the culprit seems to be usort:

$ usort format ufmt_spm_error.py 
Error sorting ufmt_spm_error.py: Syntax Error @ 2:11.
Incomplete input. Encountered 'foo', but expected ';', or 'NEWLINE'.

    match foo:
          ^

usort has this relevant issue: facebook/usort#216

The suggested work-around (exporting LIBCST_PARSER_TYPE="native" to the environment) worked for me!

Feel free to remove this issue as it is not directly a problem of ufmt, but maybe it is useful for someone else who has the same problem.

Fails to format multiple files on windows

I'm trying to run ufmt format on multiple files on Windows, and get this exception:

Traceback (most recent call last):
  File "C:\Users\zsolz\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\zsolz\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\zsolz\projects\venvs\libcst\Scripts\ufmt.exe\__main__.py", line 7, in <module>
  File "c:\users\zsolz\projects\venvs\libcst\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\zsolz\projects\venvs\libcst\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "c:\users\zsolz\projects\venvs\libcst\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\zsolz\projects\venvs\libcst\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\zsolz\projects\venvs\libcst\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\zsolz\projects\venvs\libcst\lib\site-packages\ufmt\cli.py", line 77, in format
    results = ufmt_paths(paths)
  File "c:\users\zsolz\projects\venvs\libcst\lib\site-packages\ufmt\core.py", line 113, in ufmt_paths
    results = list(runner.run(all_paths, fn).values())
  File "c:\users\zsolz\projects\venvs\libcst\lib\site-packages\trailrunner\core.py", line 183, in run
    results = list(exe.map(func, paths))
  File "C:\Users\zsolz\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\process.py", line 559, in _chain_from_iterable_of_lists
    for element in iterable:
  File "C:\Users\zsolz\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "C:\Users\zsolz\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\_base.py", line 445, in result
    return self.__get_result()
  File "C:\Users\zsolz\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Seems to get triggered anytime there's more than one file on the command line (or a directory containing more than one). Example: ufmt -d format .\libcst\__init__.py .\libcst\testing\utils.py or just ufmt -d format . in LibCST. Version 1.3.0

Start formatting files while still walking paths

ufmt_paths currently waits until it has walked all paths from input before starting to process/format individual files. This can result in significant delays when formatting large directories or when given large lists of files to format.

Walking files should be converted into a generator, and pass that generator to trailrunner.run_iter() so that it can start processing files while the main process is still walking paths.

List index out of range when formatting empty files

(venv310) jreese@luxray ~/scratch » touch foo.py
(venv310) jreese@luxray ~/scratch » ufmt check foo.py
Error formatting /Users/jreese/scratch/foo.py: list index out of range

Error occurs when detecting newlines at https://github.com/omnilib/ufmt/blob/main/ufmt/util.py#L64

(vscode) Unnecessary new lines are added indefinitely to docstring for class fields

I discovered this while using the VSCode Extension. Sorry if this issue has already been resolved.

Doesn't respect Black's policy to ignore `fmt: off` or `fmt:skip` sections

https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#ignoring-sections

$ ufmt diff poetry_zen
Would format /workspaces/python-zen/poetry_zen/poetry_zen/__init__.py
--- /workspaces/python-zen/poetry_zen/poetry_zen/__init__.py
+++ /workspaces/python-zen/poetry_zen/poetry_zen/__init__.py
@@ -20,8 +20,7 @@
 # from . import math  # noqa
 # # fmt: on
 
-from . import hello  # fmt: skip
-from . import math  # fmt: skip
+from . import hello, math  # fmt: skip  # fmt: skip

$ black --diff poetry_zen
All done! ✨ 🍰 ✨
3 files would be left unchanged.

ufmt check fails on large directories

We have ufmt set up on a fairly large src tree and are in the process of migration. It works great for smaller folders, but on our larger folders we get the following error when running ufmt check <large_dir>. Have not had time to quantify "large", but can do so if needed

concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 151, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 156, in _add_token
    self._pop()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 221, in _pop
    new_node = self.convert_nonterminal(tos.dfa.from_rule, tos.nodes)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/python_parser.py", line 44, in convert_nonterminal
    return self.nonterminal_conversions[nonterminal](self.config, children)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/conversions/statement.py", line 532, in convert_import_as_name
    asname=AsName(
  File "<string>", line 6, in __init__
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_nodes/base.py", line 115, in __post_init__
    self._validate()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_nodes/statement.py", line 728, in _validate
    raise CSTValidationError(
  File "<string>", line None
libcst._nodes.base.CSTValidationError: There must be at least one space between 'as' and name.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/process.py", line 198, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/process.py", line 198, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/ufmt/core.py", line 84, in ufmt_file
    dst_contents = ufmt_string(path, src_contents, usort_config, black_config)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/ufmt/core.py", line 43, in ufmt_string
    content = usort_string(content, usort_config, path)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/usort/sorting.py", line 257, in usort_string
    return usort_bytes(data=data.encode(), config=config, path=path)[0].decode()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/usort/sorting.py", line 269, in usort_bytes
    mod = try_parse(data=data, path=path)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/usort/util.py", line 77, in try_parse
    raise parse_error or Exception("unknown parse failure")
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/usort/util.py", line 66, in try_parse
    mod = cst.parse_module(
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/entrypoints.py", line 70, in parse_module
    result = _parse(
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/entrypoints.py", line 50, in _parse
    result = parser.parse()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 110, in parse
    self._add_token(token)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 170, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 13:48.
Internal error: There must be at least one space between 'as' and name.

                                   conda_deploy)
                                               ^
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/calvinytong/miniconda3/envs/ctrldev/bin/ufmt", line 8, in <module>
    sys.exit(main())
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/ufmt/cli.py", line 54, in check
    results = ufmt_paths(paths, dry_run=True)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/ufmt/core.py", line 113, in ufmt_paths
    results = list(runner.run(all_paths, fn).values())
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/trailrunner/core.py", line 183, in run
    results = list(exe.map(func, paths))
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
libcst._exceptions.ParserSyntaxError: Syntax Error @ 13:48.
Internal error: There must be at least one space between 'as' and name.

                                   conda_deploy)

ufmt quiet mode

It would be nice if ufmt had a quiet mode. For example

ufmt format foo/bar.txt

will output

Formatted foo/bar.txt

It would be nice to have a -q flag to silence these

Add contributor guide

Somehow this never got copied from other projects.

ufmt_file sends wrong path to ufmt config factory

Sends path.parent instead of just path.

Support excluding files from black formatting

While usort is relatively harmless, formatting files with black can introduce a lot of vertical space that is not always wanted. Especially in the documentation such as usage examples, directives to disable it locally (# fmt: off / # fmt: on) are undesirable.

black has an --exclude option that can also be given through its configuration in the config file. Maybe we can add a flag to ufmt_string that disables black formatting. With #11 merged, this flag could be set in ufmt_file after parsing the black configuration file.

Ufmt ignore paths aren't rooted to project root

Due to "I'm used to this convention" reasons, I store projects under /Users/akx/build.

ufmt's trailrunner looks at path segments beyond the detected project root, so when a project's .gitignore file contains build/, it finds build/ in the pathname and decides nothing is worth looking at, ever.

ufmt's own .gitignore contains build/, so, with /Users/akx/build/ufmt being the project root,

/Users/akx/build/ufmt $ ufmt check .
✨ 15 files already formatted ✨
/Users/akx/build/ufmt $ ufmt check /Users/akx/build/ufmt
No files found
✨ 1 errors ✨
/Users/akx/build/ufmt $

More broadly, ufmt (or trailrunner) acts differently depending on whether the path is absolute or relative, which is a bit silly :)

This is pretty much the exact same bug I encountered in ruff: astral-sh/ruff#2034

How does one format stdin?

ufmt is awesome, thanks a bunch! I already use the pre-commit hook, but I'd like to auto-format as well.
With my editor (helix) I need to write the auto-format command with this kind of syntax:

formatter = { command = "black", args = ["-", "-q"] }

but I can't seem to do it with ufmt.

In fact, when I try with

formatter = { command = "ufmt", args = ["-q", "format"] }

it actually deletes the whole contents of the file instead of formatting.

Am I just misunderstanding something or is there no way to format text from stdin?

Project root detection logic leads to configuration duplication in some circumstances

Currently, μfmt appears to look for tool configuration in pyproject.toml files relative to the file under format, selecting the nearest ancestor. In a monorepo, or in a repo with a main package and several sub-packages, this would require duplicating black's and μsort's (and μfmt's) config in each individual package's pyproject.toml. Some alternatives I can think of are:

• Adding a command-line option for the project root
• Taking the project root to be the cwd - this is a departure from how μfmt works right now so I assume you won't want to do this
• De-prioritising pyproject.toml without a [tool.black] or [tool.usort] section, continue looking up the tree - if people do want to override the config from an outer folder they'll need to add a [tool.x] section in their pyproject.toml which is not the least bit intuitive
• Do nothing :)

Thanks for μfmt!

Missing deps

Appears to be missing

['click', 'libcst', 'moreorless', 'tomlkit', 'trailrunner', 'typing-extensions']

with more detail

ufmt/cli.py uses click but 'click' not in requirements
ufmt/cli.py uses moreorless.click.echo_color_precomputed_diff but 'moreorless' not in requirements
ufmt/config.py uses tomlkit but 'tomlkit' not in requirements
ufmt/config.py uses trailrunner.project_root but 'trailrunner' not in requirements
ufmt/core.py uses moreorless.click.unified_diff but 'moreorless' not in requirements
ufmt/core.py uses trailrunner.Trailrunner but 'trailrunner' not in requirements
ufmt/tests/cli.py uses click.testing.CliRunner but 'click' not in requirements
ufmt/tests/cli.py uses libcst.ParserSyntaxError but 'libcst' not in requirements
ufmt/tests/cli.py uses trailrunner but 'trailrunner' not in requirements
ufmt/tests/config.py uses trailrunner.tests.core.cd but 'trailrunner' not in requirements
ufmt/tests/core.py uses libcst.ParserSyntaxError but 'libcst' not in requirements
ufmt/tests/core.py uses trailrunner but 'trailrunner' not in requirements
ufmt/tests/util.py uses tomlkit but 'tomlkit' not in requirements
ufmt/types.py uses typing_extensions.Protocol but 'typing-extensions' not in requirements

Bug: diff and return_contents returns wrong values when run on files with CRLF newlines

ufmt_file() reads the file contents—with normalized/unix newlines—as well as the newline style, but does not use the newline style when generating a diff or setting before/after contents on the result object. This results in wrong bytes data for files that originally had Windows-style CRLF newlines, even though it would have done the right thing when writing changes back to disk.

Add API to format existing tree object

Eg, for use with tools like LiCST codemods or Fixit, that already have a parsed tree object, and just need to apply formatting to it, it would be nice to pass that tree directly.

<STDIN> is not a resolvable path on Windows

ufmt/core.py creates a Path('<stdin>') object when formatting stdin, but this path is not resolvable on Windows.

Document guidance about pinning black

The version constraint ufmt currently has will allow black upgrades, which historically have come with formatting changes. I'd prefer ufmt pin, or explicitly document that projects that use ufmt should pin.

VSCode, TypeGuard, and TypeVar throw error

I am using python 3.11

$ python --version
Python 3.11.4

Here is the output that I get from VSCode Output tab, any ideas?

Name: ufmt
Module: ufmt
Python extension loading
Waiting for interpreter from python extension.
Python extension loaded
Traceback (most recent call last):
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/tool/server.py", line 49, in <module>
    from pygls import protocol, server, uris, workspace
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/protocol.py", line 43, in <module>
    from pygls.feature_manager import FeatureManager, assign_help_attrs, is_thread_function
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/feature_manager.py", line 28, in <module>
    from pygls.lsp import get_method_options_type, is_instance
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/lsp/__init__.py", line 36, in <module>
    from typeguard import check_type
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typeguard/__init__.py", line 27, in <module>
    from typing_extensions import Literal
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typing_extensions.py", line 1174, in <module>
    class TypeVar(typing.TypeVar, _DefaultMixin, _root=True):
TypeError: type 'typing.TypeVar' is not an acceptable base type
[Info  - 8:34:49 AM] Connection to server got closed. Server will restart.
Traceback (most recent call last):
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/tool/server.py", line 49, in <module>
    from pygls import protocol, server, uris, workspace
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/protocol.py", line 43, in <module>
    from pygls.feature_manager import FeatureManager, assign_help_attrs, is_thread_function
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/feature_manager.py", line 28, in <module>
    from pygls.lsp import get_method_options_type, is_instance
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/lsp/__init__.py", line 36, in <module>
    from typeguard import check_type
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typeguard/__init__.py", line 27, in <module>
    from typing_extensions import Literal
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typing_extensions.py", line 1174, in <module>
    class TypeVar(typing.TypeVar, _DefaultMixin, _root=True):
TypeError: type 'typing.TypeVar' is not an acceptable base type
[Info  - 8:34:50 AM] Connection to server got closed. Server will restart.
Traceback (most recent call last):
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/tool/server.py", line 49, in <module>
    from pygls import protocol, server, uris, workspace
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/protocol.py", line 43, in <module>
    from pygls.feature_manager import FeatureManager, assign_help_attrs, is_thread_function
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/feature_manager.py", line 28, in <module>
    from pygls.lsp import get_method_options_type, is_instance
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/lsp/__init__.py", line 36, in <module>
    from typeguard import check_type
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typeguard/__init__.py", line 27, in <module>
    from typing_extensions import Literal
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typing_extensions.py", line 1174, in <module>
    class TypeVar(typing.TypeVar, _DefaultMixin, _root=True):
TypeError: type 'typing.TypeVar' is not an acceptable base type
[Info  - 8:34:50 AM] Connection to server got closed. Server will restart.
Traceback (most recent call last):
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/tool/server.py", line 49, in <module>
    from pygls import protocol, server, uris, workspace
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/protocol.py", line 43, in <module>
    from pygls.feature_manager import FeatureManager, assign_help_attrs, is_thread_function
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/feature_manager.py", line 28, in <module>
    from pygls.lsp import get_method_options_type, is_instance
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/lsp/__init__.py", line 36, in <module>
    from typeguard import check_type
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typeguard/__init__.py", line 27, in <module>
    from typing_extensions import Literal
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typing_extensions.py", line 1174, in <module>
    class TypeVar(typing.TypeVar, _DefaultMixin, _root=True):
TypeError: type 'typing.TypeVar' is not an acceptable base type
[Info  - 8:34:50 AM] Connection to server got closed. Server will restart.
Traceback (most recent call last):
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/tool/server.py", line 49, in <module>
    from pygls import protocol, server, uris, workspace
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/protocol.py", line 43, in <module>
    from pygls.feature_manager import FeatureManager, assign_help_attrs, is_thread_function
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/feature_manager.py", line 28, in <module>
    from pygls.lsp import get_method_options_type, is_instance
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/pygls/lsp/__init__.py", line 36, in <module>
    from typeguard import check_type
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typeguard/__init__.py", line 27, in <module>
    from typing_extensions import Literal
  File "/Users/USER/.vscode/extensions/omnilib.ufmt-2023.4.3-darwin-arm64/bundled/libs/typing_extensions.py", line 1174, in <module>
    class TypeVar(typing.TypeVar, _DefaultMixin, _root=True):
TypeError: type 'typing.TypeVar' is not an acceptable base type
[Error - 8:34:51 AM] Connection to server got closed. Server will not be restarted.

allow Black formatting option as pre-commit arg

Hi there,
I found this very handy aggregation of the two most used formatting tools.
Not sure if it is already feasible (if so would be nice to have it mentioned in readme) but it would be great to allow passing Black arguments such as line length --line-length=120. I have tried naive way but does not work

  - repo: https://github.com/omnilib/ufmt
    rev: v1.3.0
    hooks:
      - id: ufmt
        additional_dependencies:
          - black == 21.9b0
          - usort == 0.6.4
        args: ["--line-length=120"]

ufmt runs slower than black+usort

Running ufmt seems to be a bit slower than running black and usort successively:

(botorch) balandat@balandat-mbp botorch % time black --diff .
All done! ✨ 🍰 ✨
210 files would be left unchanged.
black --diff .  29.14s user 2.35s system 1021% cpu 3.082 total
(botorch) balandat@balandat-mbp botorch % time usort check .
Would sort botorch/acquisition/knowledge_gradient.py
usort check .  17.36s user 0.28s system 99% cpu 17.754 total
(botorch) balandat@balandat-mbp botorch % time ufmt check .
ufmt check .  32.73s user 0.39s system 99% cpu 33.294 total

This was on a vanilla clone of the botorch repo (git clone https://github.com/pytorch/botorch)

Not sure if this is expected.

"IndexError: list index out of range" when parsing line with ;

The issue can be replicated as follows with code is valid python.

$ cat > test.py
import warnings; warnings.filterwarnings("ignore")

$ ufmt check test.py 
Traceback (most recent call last):
  File "/usr/local/bin/ufmt", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/ufmt/cli.py", line 119, in check
    changed, error = echo_results(results, quiet=options.quiet)
  File "/usr/local/lib/python3.9/site-packages/ufmt/cli.py", line 41, in echo_results
    msg = lines[0]
IndexError: list index out of range

Option to preserve original and formatted content

Would like to be able to do something like ufmt_paths(paths, preserve_content=True) and then be able to check result.original_content and result.final_content when inspecting results.

Yield results as they become available

For speed and memory performance, it would be nice to have ufmt_paths yield result objects as they become available, rather than buffering all results and waiting for everything to finish formatting.

Black output

Black outputs

All done! ✨ 🍰 ✨
32 files left unchanged.

Is there a way to also get this output from ufmt?

µfmt does not tell black when formatting type stubs

See Instagram/LibCST#565: Instagram/LibCST#565 (comment)

Formatting via stdin loses original path when calling black/usort config factories

Make an exception class that pre/post processors can raise to skip formatting

Would like to have a pre/post processor function that can raise a special exception that tells ufmt to skip formatting a file without triggering an error status.

pre-commit + excludes prevent people from committing code

Currently the behavior of the CLI is to return an error code when no files can be formatted, e.g.

$ ufmt check foo.py
No files found
✨ 1 errors ✨
$ echo $?
1

This is probably okay for the interactive use case but this breaks how pre-commit works when used in conjunction with an exclusion rule in pyproject.toml. Say you have this set up

[tool.ufmt]
excludes = ["foo.py"]

Then if you do

$ echo "'a'" > foo.py
$ git add  foo.py
$ git commit -m 'test'
Format files with µfmt...................................................Failed
- hook id: ufmt
- exit code: 1

No files found
✨ 1 errors ✨

But this should absolutely not be an error! In fact it blocks people from committing things when they are totally valid.

I'm not sure what the best suggestion for a fix is, I'm happy to contribute it here if there is some agreement. Some ideas I had:

• Add a flag to the CLI to not have an error when this happens, e.g. --ignore-no-files or something
• Change the logic such that it's not an error to have no files in the CLI.
• Change the logic such that there is only an error if there are no files before the excludes
• Something else?

`flit==3.7.0` breaks `pre-commit` hook

flit==3.7.0 was released ~3 hours ago and with that version, usort errors during installation from source. This happens for example if it is installed through a pre-commit hook. See for example this CI run.

Quick fix would be to pin

ufmt/pyproject.toml

Line 2 in 0e05a95

requires = ["flit_core >=3,<4"]

to

requires = ["flit_core >=3,<3.7"] 

[Feature Request] Add --diff output to ufmt check

It would be nice to have a --diff-like setting (as in black) to print the changes to stdout if ufmt check would reformat? That way there would be a lot more signal and it would be easy to correct things from log outputs (e.g. in CI) rather than having to re-run it locally.