browniebroke / django-codemod

A tool to automatically fix Django deprecations.

---

Documentation: https://django-codemod.readthedocs.io

Source Code: https://github.com/browniebroke/django-codemod

---

A tool to help upgrade Django projects to newer version of the framework by automatically fixing deprecations.

The problem

When maintaining a Django site, over time you'll find yourself to a point where you'll need to update to the next major version of Django. When Django APIs changes, functions move or are removed, changing usages in your project might add up to many changes. Often these changes are simple to do, but sometimes a simple "find and replace" is not possible.

Take, for instance, the removal of the url() function from Django 4.0, to be replaced by re_path(). In simple cases, you might even want to switch to path(), which has a nicer API. A typical Django project easily has 100's or routes, so this simple decision becomes a much longer task when to be made for each of them.

This solution

This project solves this problem by providing codemodders for simple changes like this. A codemodder re-writes your code from the old way to the new way.

With the help of AST analysis, we're able to understand what modifications are applicable, remove imports as they become irrelevant, and add missing ones as they are needed.

To continue the example, the tool will look at the route in the url() call, and decide whether the regular expression may be replaced by one of the built-in URL converters and use path() or stick to a regex and use re_path().

Interested? Check out the documentation for usage and the full list of codemodders.

What this tool is not

• This tool is best suited for Django sites, NOT for reusable Django applications. The project needs to target a single Django version, e.g. 3.1.x.
• You do NOT need to install this tool as part of your project dependencies, it is a CLI tool, not a Django package to be installed in your site.

Similar tools

django-upgrade is a similar tool written by Adam Johnson. It's a reimplementation taking a different approach based only on standard library module.

django-codemod is based on libCST (Concrete Syntax Tree) which is a limiting factor in terms of speed. By using standard library modules, django-upgrade is a lot faster and is able to support the latest Python, however it requires Python 3.8+.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Bruno Alla 💻 📖 🤔	Aarni Koskela 💻 🤔 ⚠️	Adam Johnson 📖	Nikita Sobolev 📖	Chris VanderKolk 💻	John Vandenberg 🐛 💻	Anjishnu 🚇
Drew Winstel 🤔	Dmytro Litvinov 📖	M. Zulqarnain 📖

This project follows the all-contributors specification. Contributions of any kind welcome!

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

django-codemod's Issues

Replace `assignment_tag` by `simple_tag`

https://docs.djangoproject.com/en/dev/releases/1.9/#assignment-tag

• Deprecated: 1.9
• Removed 2.0

Codemodder: `django.utils.http.cookie_date()`

From Django 3.0 removals:

django.utils.http.cookie_date() will be removed.

Deprecated in Django 2.1:

django.utils.http.cookie_date() is deprecated in favor of django.utils.http.http_date(), which follows the format of the latest RFC.

Make codemods discoverable by libCST's list command

Description

LibCST has a list command which is supposed to list all known codemods based on the config:

python3 -m libcst.tool list

This should be listing Django Codemod when installed, but last time I tried it's not the case. It's not critical, but would be nice to have it and help with discovrability of codemods.

Removed `weak` argument to `django.dispatch.signals.Signal.disconnect()`

• Deprecated in Django 1.9
• Removed in Django 2.0

The weak argument to django.dispatch.signals.Signal.disconnect() will be removed.

It has no effect since Django 1.9.

Codemodder: `QuerySet.earliest()` and `QuerySet.latest()`

Description

From Django 2.0 release notes:

The field_name keyword argument to QuerySet.earliest() and QuerySet.latest() is deprecated in favor of passing the field names as arguments. Write .earliest('pub_date') instead of .earliest(field_name='pub_date').

This is removed in Django 3.0:

The field_name keyword argument of QuerySet.earliest() and latest() will be removed.

codemod goes all over the place and triggers OOM

I was building my first fixer, found at its current horrid state at #123 (4803651), then I ran the following on a file which definitely needs the update

djcodemod --removed-in 2.0 redirect_urls/utils.py

The file in question can be found at https://github.com/pmac/django-redirect-urls/

I go get a coffee, and get otherwise distracted, etc, etc. A few hours later I come back and find my machine is out of memory, and codemod is having fun in my pnpm global node_modules , which is rather large.

I am sure I will find out how and why, but it isnt immediately clear to me, and I dare say something needs to prevent this happening to other people.

I had thousands of these:

Codemodding /home/jayvdb/.asdf/installs/nodejs/8.14.0/.npm/pnpm-global/1/node_modules/.registry.npmjs.org/node-gyp/3.8.0/node_modules/node-gyp/gyp/pylib/gyp/xcode_ninja.py
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 152, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(,)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/libcst/codemod/_cli.py", line 283, in _parallel_exec_process_stub
    input_tree = parse_module(
  File "/usr/lib/python3.8/site-packages/libcst/_parser/entrypoints.py", line 71, in parse_module
    result = _parse(
  File "/usr/lib/python3.8/site-packages/libcst/_parser/entrypoints.py", line 51, in _parse
    result = parser.parse()
  File "/usr/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 111, in parse
    self._add_token(token)
  File "/usr/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 187, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 31:17.
Incomplete input. Encountered ',', but expected ':'.

  except OSError, e:
                ^

Failed to codemod /home/jayvdb/.asdf/installs/nodejs/8.14.0/.npm/pnpm-global/1/node_modules/.registry.npmjs.org/node-gyp/3.8.0/node_modules/node-gyp/gyp/pylib/gyp/xcode_ninja.py

Here is the output at the end when it was killed (kinda nice that my oom killer did kill this and not my desktop apps, which it often targets first)

38m 42s 0.0540% complete, 1194h 25m 20s estimated for 1362939 files to go...Terminated

Invalid .pyup.yml detected

The bot encountered an error in your .pyup.yml config file:

  in "<unicode string>", line 6, column 10:
          pin: False
             ^

You can validate it with this online YAML parser or by taking a look at the Documentation.

Switch to `version_toml` for semantic release

Python semantic release has added proper support for version coming from pyproject.toml via a new version_toml option, try to use the new config instead of version_variable:

django-codemod/pyproject.toml

Lines 65 to 68 in f581585

	version_variable = [
	"django_codemod/__init__.py:__version__",
	"pyproject.toml:version",
	]

Not sure if it would work, maybe we don't need this line anymore:

django-codemod/django_codemod/__init__.py

Line 1 in f581585

__version__ = "1.3.4"

Attribute access to `url` incorrectly replaced

Describe the bug
Issue with URL Transformer, converting an attribute access of the HttpResponseRedirectBase.url to HttpResponseRedirectBase.re_path if django.conf.urls.url is imported

To Reproduce
Consider this code (from a test case):

from django.conf.urls import url

# Define some URL patterns for tests
urlpatterns = [
    url(r'^something', Mock(), name='something'),
]

# Later: 
assert isinstance(response, HttpResponseRedirectBase)
assert response.url == expected_path

The last assertion is incorrectly converted to:

assert response.re_path == expected_path

Additional context
This is potentially affecting other rename transformers.

Django 3.0: update super call to InlineModelAdmin.has_add_permissions()

Currently, the transformer for InlineModelAdmin.has_add_permissions() updates method definition in subclasses of InlineModelAdmin, however if the implementation calls the base method, the call isn't updated.

Example of input code:

from django.contrib import admin


class MyInlineInline(admin.TabularInline):
	def has_add_permission(self, request):
		if condition:
			return False
		return super().has_add_permission(request)

Current output:

from django.contrib import admin


class MyInlineInline(admin.TabularInline):
	def has_add_permission(self, request, obj=None):
		if condition:
			return False
		return super().has_add_permission(request)

Expected output:

from django.contrib import admin


class MyInlineInline(admin.TabularInline):
	def has_add_permission(self, request, obj=None):
		if condition:
			return False
		return super().has_add_permission(request, obj=obj)

Notice the last line in each snippet, should add the keyword argument obj=obj.

Resolve depreactions earlier: new CLI option `--deprecated-in`

Use case:

Given a project running with warning enabled on CI, right after upgrading to Django 3.0, one will start to see all the new deprecations appear (e.g. Removed in Django 4.0). Once on Django 3.0, I might be tempted to resolve all known deprecations now, even before I upgrade to Django 3.1 or 3.2.

Problem

However, some other things removed in Django 4.0 might not be deprecated in Django 3.0 yet, so the current option --removed-in would potential apply some changes before my project is compatible.

Suggested solution

Add a new option --deprecated-in to allow users to apply deprecations based on the Django version they have been deprecated in.

The workflow would look like this:

1. Fix all Django depreacation warning
2. Upgrade to Django 3.0 and ignore all new Django deprecations warnings
3. Once the upgrade is done, enable all deprecation warnings again run djcodmod --deprecated-in 3.0 to make the CI less noisy again

ImportError on an Apple M1 Mac related to libcst

Describe the bug
A clear and concise description of what the bug is.
ImportError on an Apple M1 Mac.

libcst seems to be installed for x86_64, not the required arm64e.

To Reproduce
Steps to reproduce the behavior:

% pipx --version

1.0.0

% pipx install django-codemod

  installed package django-codemod 1.10.0, Python 3.9.12
  These apps are now globally available
    - djcodemod
done! ✨ 🌟 ✨

% djcodemod run --deprecated-in 1.3 .

Running codemods: ActionCheckboxNameTransformer
Executing codemod...
Traceback (most recent call last):
  File "/Users/cclauss/.local/bin/djcodemod", line 8, in <module>
    sys.exit(djcodemod())
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/rich_click/rich_group.py", line 21, in main
    return super().main(*args, standalone_mode=False, **kwargs)
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/django_codemod/cli.py", line 188, in run
    call_command(command_instance, files)
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/django_codemod/cli.py", line 195, in call_command
    result = parallel_exec_transform_with_prettyprint(
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/libcst/codemod/_cli.py", line 623, in parallel_exec_transform_with_prettyprint
    parse_module(
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 109, in parse_module
    result = _parse(
  File "/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 42, in _parse
    from libcst.native import parse_expression, parse_module, parse_statement
ImportError: dlopen(/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/libcst/native.cpython-39-darwin.so, 0x0002):
    tried: '/Users/cclauss/.local/pipx/venvs/django-codemod/lib/python3.9/site-packages/libcst/native.cpython-39-darwin.so'
    (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))

Additional context
Add any other context about the problem here.

Ability to use a specific codemodder to the CLI

We're listing codemodders in the list command. It would be nice to add a new CLI option accepting a codemodder name and execute just that one.

Ideally, one could specify the option multiple time to run only the codemodders they want.

Django 4 django.dispatch.Signal unexpected keyword argument providing_args

SUSE Tumbleweed has updated django to 4.0, and while fixing django packages I've noticed a lot of packages are failing with this.
e.g. https://build.opensuse.org/package/show/home:jayvdb:branches:devel:languages:python:django/python-django-avatar

https://docs.djangoproject.com/en/dev/releases/4.0/

See Features deprecated in 3.1 for details on these changes, including how to remove usage of these features.
..
The providing_args argument for django.dispatch.Signal is removed.

Deprecation timeline

• Deprecated in: 3.1
• Removed in: 4.0

Add an option to skip paths (e.g. node_modules)

Is your feature request related to a problem? Please describe.
I use django-tailwind for CSS in a project. One of the JS libraries it imports has some Python 2 code that it puts inside my theme/static_src/node_modules/ directory, which is already in my .gitignore. Because my project is using Python 3.8, the code modder throws syntax errors for those 23 files (which is technically correct).

Describe the solution you'd like
I'd like the ability to exclude paths (via a command-line switch or other means) or have the option to have django-codemod skip files ignored by git.

Additional context
N/A, but this tool is excellent!

Add tests to sdist

Describe the feature
Currently the tests are not included in the PyPI sdist. As a result in order to do packaging for openSUSE at https://build.opensuse.org/package/show/devel:languages:python:django/python-django-codemod , I need to fetch the sdist from Github, which is a few extra steps.

Please can the tests be added to the sdist.

To Reproduce
Download sdist from PyPI, untar and see that now tests are present.

AttributeError: 'Call' object has no attribute 'value'

When running ForceTextToStrCommand:

  File "...python3.7/site-packages/libcst/_visitors.py", line 72, in on_leave
    updated_node = leave_func(original_node, updated_node)
  File "...python3.7/site-packages/django_codemod/codemods/django_40.py", line 47, in leave_Call
    if updated_node.func.value == "force_text":
AttributeError: 'Call' object has no attribute 'value'

Looks like this type of code:

def func():
   return "Hello"

def factory():
	return func

factory()()  # problem here?

Option to show diffs

Is your feature request related to a problem? Please describe.
When running djcodemod in a build worker, it would be useful to show the changes it made so the logs record the transformations made, and also so it is easy to copy the diffs to submit upstream.

This is especially useful when working from tarballs, as there is no VCS to be able to generate diffs.

Describe the solution you'd like
A disabled-by-default flag like --show-diffs which emits the changes which are being written to disk as patch-compatible stdout .

Additional context
e.g. it takes a couple of secs to add
`djcodemod run --removed-in 4.0 test_project/
to the build scripts for https://build.opensuse.org/package/show/home:jayvdb:branches:devel:languages:python:django/python-django-debreach , to allow me to provide a more detailed issue at lpomfrey/django-debreach#21

But I really should check the diffs to make sure djcodemod is not introducing bugs in the process, and if there were diffs I could have easily added to the upstream issue an inline patch for the first which djcodemod knows how to fix, which gets them closer to the harder second problem which djcodemod doesnt know how to solve.

Add a first-class command line interface

Description

The current usage requires to setup libCST and invoke the Django Codemod commands from it.

It would be nicer if users could have a first class CLI specific to the use case of upgrading Django.

Examples of usages I've thoyght of, which I think would be nice:

# Run all codemods to jump from 2.2 to 3.1
django_codemod --from 2.2 [--to] 3.1

# Run all codemods to jump from the current installed version to 3.1
django_codemod [--to] 3.1

# Run just codemods for 3.1
django_codemod --only 3.1

# Run a specific modifier
django_codemod --mod django.shortcuts.render_to_response

The CLI should call libCST under the hood, without the end user having to worry about implementation detail.

iterable ConnectionHandler

Occurs in the test suite of https://github.com/pmac/django-redirect-urls/ , but I have seen it frequently in test suites which havent been updated to use pytest.

cls = <class 'tests.test_middleware.TestRedirectsMiddleware'>

    @classmethod
    def _validate_databases(cls):
        if cls.databases == '__all__':
            return frozenset(connections)
        for alias in cls.databases:
>           if alias not in connections:
E           TypeError: argument of type 'ConnectionHandler' is not iterable

Replace `Field.rel` by `Field.remote_field`

https://docs.djangoproject.com/en/dev/releases/1.9/#field-rel-changes

Field.rel and its methods and attributes have changed to match the related fields API. The Field.rel attribute is renamed to remote_field and many of its methods and attributes are either changed or renamed.

To note from Django 2.0 removals:

Field.remote_field.to attribute will be removed.

• Deprecated: 1.9
• Removed: 2.0

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Use matchDepNames instead of matchPackageNames

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update dependency libcst to v1.3.1

Detected dependencies

github-actions

.github/workflows/ci.yml

• actions/checkout v4
• snok/install-poetry v1.3.4
• actions/setup-python v5
• codecov/codecov-action v4.3.1
• actions/checkout v4
• actions/setup-python v5
• actions/checkout v4
• wagoid/commitlint-github-action v6.0.1
• actions/checkout v4
• python-semantic-release/python-semantic-release v9.7.1
• python-semantic-release/python-semantic-release v9.7.1

.github/workflows/codeql.yml

• actions/checkout v4
• github/codeql-action v3
• github/codeql-action v3
• github/codeql-action v3

.github/workflows/hacktoberfest.yml

• browniebroke/hacktoberfest-labeler-action v2.3.0

.github/workflows/issue-manager.yml

• tiangolo/issue-manager 0.5.0

.github/workflows/labels.yml

• actions/checkout v4
• actions/setup-python v5

.github/workflows/poetry-upgrade.yml

• browniebroke/github-actions v1

pep621

pyproject.toml

• poetry-core >=1.0.0

poetry

pyproject.toml

• python ^3.8
• click <9
• libcst ==1.1.0
• pathspec >=0.6,<1
• rich >=10
• rich-click >=1.0
• parameterized ^0.9.0
• pytest ^8.0.0
• pytest-cov ^5.0.0
• pytest-mock ^3.3
• myst-parser >=0.16
• sphinx >=4.0
• furo >=2023.5.20

pre-commit

.pre-commit-config.yaml

• pre-commit/pre-commit-hooks v4.6.0
• python-poetry/poetry 1.8.3
• pre-commit/mirrors-prettier v3.1.0
• astral-sh/ruff-pre-commit v0.4.4
• psf/black 24.4.2
• codespell-project/codespell v2.2.6
• commitizen-tools/commitizen v3.25.0
• pre-commit/mirrors-mypy v1.10.0

---

• Check this box to trigger a request for Renovate to run again on this repository

Django 1.11 to 2.2 codemods.

Description

Hello @browniebroke thanks for this project it looks promising! I believe it can greatly accelerate the speed of adoptions of newly released versions of Django.

I was wondering if you'd be interested in receiving contributions for migration to versions between Django 1.11 LTS and 2.2 LTS. Even if these versions are not supported anymore I think it could be of interest to a lot of developers who are still in the process of upgrading.

I currently work for a company that is the process of doing so and also has a few Django 1.9/1.10 deprecation warnings that need to be addressed on a large project. Since we'll have to automate the process anyway I was wondering if you'd be interested in receiving contributions for these versions as well.

Removed compatibility imports in 3.1

Link to Django docs
Link to release note section where the change and suitable replacement is mentioned:

https://docs.djangoproject.com/en/3.1/releases/3.1/#id1

Quote of the relevant paragraph

In case links gets outdated/removed:

The compatibility imports of django.core.exceptions.EmptyResultSet in django.db.models.query, django.db.models.sql, and django.db.models.sql.datastructures are removed.

The compatibility import of django.core.exceptions.FieldDoesNotExist in django.db.models.fields is removed.

The compatibility imports of django.forms.utils.pretty_name() and django.forms.boundfield.BoundField in django.forms.forms are removed.

The compatibility imports of Context, ContextPopException, and RequestContext in django.template.base are removed.

The compatibility import of django.contrib.admin.helpers.ACTION_CHECKBOX_NAME in django.contrib.admin is removed.

Deprecation timeline

• Removed in: 3.1
• EmptyResultSet - deprecated in: 1.11
• FieldDoesNotExist - deprecated in: 1.8
• pretty_name - deprecated in: 1.9
• BoundField - deprecated in: 1.9
• Context - deprecated in: 1.7
• ContextPopException - deprecated in: 1.7
• RequestContext - deprecated in: 1.7
• ACTION_CHECKBOX_NAME - deprecated in: 1.3

Avoid try: .. except ImportError: blocks

The following occurs when running on https://github.com/stephenmcd/django-forms-builder with various deprecation/removed args.

Several problems can be seen here, but IMO the main one is that both branches should just be ignored unless the parser properly understands what ImportError exceptions will occur in each branch, and avoid changes when they are supposed to trip an exception and are handled correctly in the exception block.

git diff forms_builder/example_project/urls.py
diff --git a/forms_builder/example_project/urls.py b/forms_builder/example_project/urls.py
index 78627dd..a944703 100644
--- a/forms_builder/example_project/urls.py
+++ b/forms_builder/example_project/urls.py
@@ -1,10 +1,10 @@
 from __future__ import unicode_literals
 
 try:
-    from django.urls import re_path, include
+    from django.urls import re_path as re_path, re_path, include
 except ImportError:
     # For Django 1.8 compatibility
-    from django.conf.urls import url as re_path, include
+    from django.conf.urls import include
 from django.contrib import admin
 from django.shortcuts import render
 

Extra trailing comma when import removed is the last one

Description

If the import to be removed is the last one of the line, but the new import already exists, and extra tariling comma is left at the end:

from django.utils.translation import ngettext_lazy, ungettext_lazy

Run UNGetTextLazyToNGetTextLazyCommand will give:

from django.utils.translation import ngettext_lazy,

We should not have the comma hanging at the end.

Simplify import checking with LibCST's FullyQualifiedNameProvider

Is your feature request related to a problem? Please describe.

LibCST 0.3.18 added a FullyQualifiedNameProvider which seems like it could help us simplify our checks for imported symbols:

https://libcst.readthedocs.io/en/latest/metadata.html#libcst.metadata.FullyQualifiedNameProvider

Describe the solution you'd like

https://github.com/browniebroke/django-codemod/blob/main/django_codemod/visitors/base.py

Especially, the code from BaseRenameTransformer.leave_ImportFrom seems like a good candidate.

Codemodder: remove `context` argument of `Field.from_db_value()` and `Expression.convert_value()`

Description

From Django 2.0 release notes:

The context argument of Field.from_db_value() and Expression.convert_value() is unused as it’s always an empty dictionary.

The signature of both methods is now:

(self, value, expression, connection)

instead of:

(self, value, expression, connection, context)

We could probably update the usages of these methods.

This is removed in Django 3.0.

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>whitesource/merge-confidence:beta)

Merge test, lint and commitlint workflows

We have 3 workflows that run during CI at the moment: test.yml, lint.yml and commitlint.yml. We could simplify this by merging them into a single ci.yml workflow, with 3 separate jobs test, lint and commitlint.

Support renames when parent module is imported

Currently the simple renaming from BaseSimpleFuncRenameTransformer doesn't detect when the parent module is imported, and we call the function with the module as prefix. It would be nice to add support for that. An example might speak better, so here is one with force_text (although other subclasses of BaseSimpleFuncRenameTransformer have the same problem).

Example input

from django.utils import encoding

encoding.force_text(something)

Expected output

from django.utils import encoding

encoding.force_str(something)

Extra imports added with libCST 0.3.14 and above

Describe the bug
When installing this library fresh with the latest version of libCST (0.3.16 at time of writing), some extra imports are being added to modules which should not be changed.

If I downgrade to libCST 0.3.13, the codemodders work as expected, but from libCST 0.3.14 I can see this odd behaviour.

To Reproduce
On the readthedocs.org codebase:

• Install Django codemod with libCST >= 0.3.14
• checkout source code from https://github.com/readthedocs/readthedocs.org (running on Django 2.2 at the moment)
• Run djcodemod run --deprecated-in 3.0 . from the root of the repo

Observe that some empty __init__.py files have some imports being added e.g.: from django.utils.translation import gettext as _ .

Additional context

Here is the diff between the 2 releases: Instagram/LibCST@v0.3.13...v0.3.14

One thing that stands out: Instagram/LibCST#402

Workaround
We should pin libCST to 0.3.13.

List supported codemod from command line

The current CLI allows doesn't do a great job of listing what is supported and what is not. The help text is basic and doesn't say which versions are accepted for each option or what modifications will be applied.

• For each option, we should at least give the accepted values
• Add a list option or subcommand to list all supported modifications, with version deprecated and removed. A table sounds appropriate, a package I came across recently which look nice and I want to try is rich.

AttributeError: 'Lambda' object has no attribute 'value'

Error leads to:

  File ".../python3.7/site-packages/libcst/_visitors.py", line 72, in on_leave
    updated_node = leave_func(original_node, updated_node)
  File ".../python3.7/site-packages/django_codemod/codemods/django_40.py", line 47, in leave_Call
    if updated_node.func.value == "force_text":
AttributeError: 'Lambda' object has no attribute 'value'

Suspect that this is caused by:

(lambda x: lambda self: x)(constant_name)

django.contrib.postgres.fields.jsonb not captured by codemod

Link to Django docs
Link to release note section where the change and suitable replacement is mentioned:

https://docs.djangoproject.com/en/dev/releases/3.1/#postgresql-jsonfield

Quote of the relevant paragraph

In case links gets outdated/removed:

PostgreSQL JSONField¶
django.contrib.postgres.fields.JSONField and django.contrib.postgres.forms.JSONField are deprecated in favor of models.JSONField and forms.JSONField.

The undocumented django.contrib.postgres.fields.jsonb.KeyTransform and django.contrib.postgres.fields.jsonb.KeyTextTransform are also deprecated in favor of the transforms in django.db.models.fields.json.

The new JSONFields, KeyTransform, and KeyTextTransform can be used on all supported database backends.

Deprecation timeline

• Deprecated in: 3.1
• Removed in: 4.0

My thought is that this scenario should be covered by djcodemod although the case is not explicitly covered in the release notes for 3.1.

It shows up as a deprecation when doing checks but djcodemod doesn't fix:

mint_develop_django | pool.Pool.properties: (fields.W904) django.contrib.postgres.fields.JSONField is deprecated. Support for it (except in historical migrations) will be removed in Django 4.0.
mint_develop_django |   HINT: Use django.db.models.JSONField instead.

The offending import in models.py is:

from django.contrib.postgres.fields.jsonb import JSONField

Probably an artifact of the way Pycharm offers imports.

Local variable shadowing import from outer scope

Describe the bug

In case something is being renamed by a codemodder in the current file, and if a local variable shadows this name from the outer scope, the shadowed variable gets renamed as well.

The code to change should avoid these kind of shadowing, but ideally we should not be tricked by it.

To Reproduce

from django.conf.urls import url
from django.urls import reverse_lazy


class MyModelAdmin(admin.ModelAdmin):
    def get_urls(self):
        return [
            url(r'^some-route/$', some_view, name=self.store_value_view_name),
        ] + super().get_urls()

    def get_result_url(self):
        url = reverse_lazy('some-url-name')
        return url

When running URLTransformer on this code, the variable url inside get_result_url gets incorrectly renamed to re_path.

Additional context
Add any other context about the problem here.

Recommended command recurses into my virtualenv

Is your feature request related to a problem? Please describe.
I ran the recommended command in usage, djcodemod run --deprecated-in 3.0 . . I wondered how there were 10k files to modify before realizing it's recursing into my virtualenv and rewriting everything there.

Describe the solution you'd like
A minimal solution would be to say not recommend . as the path.

A more complete solution would be to check if a folder looks like a virtualenv (e.g. has a "bin/activate") before recursing into it.

Additional context
n/a

ParserSyntaxError when a module have a "from django.utils import timezone" import

Describe the bug

I'm migrating a medium-sized codebase, and I found this strange behavior: if the code has a "from django.utils import timezone" import, a ParserSyntaxError is raised

$ pre-commit run -a
djcodemod................................................................Failed
- hook id: djcodemod
- exit code: 1

Running codemods: AssignmentTagTransformer, CookieDateTransformer, FixedOffsetTransformer, FloatRangeFormFieldTransformer, FloatRangeModelFieldTransformer, InlineHasAddPermissionsTransformer, ModelsPermalinkTransformer, OnDeleteTransformer, QuerySetPaginatorTransformer, SignalDisconnectWeakTransformer, URLResolversTransformer
Executing codemod...
Finished codemodding 62 files!
 - Transformed 62 files successfully.
 - Skipped 0 files.
 - Failed to codemod 0 files.
 - 0 warnings were generated.
Running codemods: AssignmentTagTransformer, CookieDateTransformer, FixedOffsetTransformer, FloatRangeFormFieldTransformer, FloatRangeModelFieldTransformer, InlineHasAddPermissionsTransformer, ModelsPermalinkTransformer, OnDeleteTransformer, QuerySetPaginatorTransformer, SignalDisconnectWeakTransformer, URLResolversTransformer
Executing codemod...
Codemodding /home/luzfcb/projects/big_project/foobar/tests/test_managers.py
Traceback (most recent call last):
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 152, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(import)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_cli.py", line 295, in _execute_transform
    output_tree = transformer.transform_module(input_tree)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_command.py", line 88, in transform_module
    tree = self._instantiate_and_run(transform, tree)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_command.py", line 58, in _instantiate_and_run
    return inst.transform_module(tree)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_codemod.py", line 108, in transform_module
    return self.transform_module_impl(tree_with_metadata)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_visitor.py", line 32, in transform_module_impl
    return tree.visit(self)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_nodes/module.py", line 91, in visit
    result = super(Module, self).visit(visitor)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_nodes/base.py", line 235, in visit
    leave_result = visitor.on_leave(self, with_updated_children)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/matchers/_visitors.py", line 512, in on_leave
    retval = CSTTransformer.on_leave(self, original_node, updated_node)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_visitors.py", line 72, in on_leave
    updated_node = leave_func(original_node, updated_node)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/visitors/_add_imports.py", line 393, in leave_Module
    *[
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/visitors/_add_imports.py", line 394, in <listcomp>
    parse_statement(
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 100, in parse_statement
    result = _parse(
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 51, in _parse
    result = parser.parse()
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 111, in parse
    self._add_token(token)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 187, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 1:7.
Incomplete input. Encountered 'import', but expected '.', '...', or 'NAME'.

from  import datetime
      ^

Failed to codemod /home/luzfcb/projects/big_project/foobar/tests/test_managers.py

To Reproduce

pipx install "django-codemod==1.5.5"

echo "from django.utils import timezone" > foobar.py

djcodemod run --deprecated-in 1.9 --deprecated-in 1.10 --deprecated-in 1.11 --deprecated-in 2.0 --deprecated-in 2.1 --deprecated-in 2.2 foobar.py

Additional context

Django 3.0: HttpRequest.xreadlines()

Removed in Django 3.0:

HttpRequest.xreadlines() will be removed.

Deprecated since Django 2.0:

HttpRequest.xreadlines() is deprecated in favor of iterating over the request.