Patches Django to reconnect on a failed database connection once before failing. Helping with running Django ORM through HAProxy, for example.
License: BSD 3-Clause "New" or "Revised" License
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Seems the most straight forward thing would to not attempt retries when are in transaction, as that can only lead to some terrible bugs.
psycopg3 is not supported at the moment;
Seems like this (and similar issues in the future) can be avoided if with self.wrap_database_errors: will be used inside try-except instead of outside. I can create a PR to that effect if it makes sense to you as well :)
I'm running Django and Postgres in two separate Docker containers. If I restart the Postgres container to simulate a connection problem, the connection is only reestablished in a Django shell on the second attempt to run a query.
>>> User.objects.first() # This works.
<User: admin>
>>> # restart database to force loss of connection
>>> User.objects.first() # This first attempt fails.
Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/venv/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 653, in first
for obj in (self if self.ordered else self.order_by('pk'))[:1]:
File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/venv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/venv/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
>>> User.objects.first() # This second attempt succeeds.
<User: admin>
>>> django.__version__
'2.2.5'
Using this app causes an error during teardown in one of our tests.
The test is triggering warnings like the following, which are likely causing the atomic block to be left:
2018-03-06 20:18:13,527:WARNING::drf_yasg.inspectors.base: 'default' on schema for HiddenField(default=CurrentUserDefault()) will not be set because to_representation raised an exception
Traceback (most recent call last):
File "…/project/.venv/lib/python3.6/site-packages/drf_yasg/inspectors/base.py", line 223, in SwaggerType
default = field.to_representation(default)
File "…/Vcs/django-rest-framework/rest_framework/fields.py", line 573, in to_representation
field_name=self.field_name,
NotImplementedError: HiddenField.to_representation() must be implemented for field own_field. If you do not need to support write operations you probably want to subclass `ReadOnlyField` instead.
The error during teardown (using pytest and pytest-django):
ERROR project/app/tests/test_api_docs.py::test_docs_media_types_and_index
_____ ERROR at teardown of test_docs_media_types_and_index _____
../../../Vcs/django/django/test/testcases.py:893: in _post_teardown
self._fixture_teardown()
../../../Vcs/django/django/test/testcases.py:1043: in _fixture_teardown
self._rollback_atomics(self.atomics)
../../../Vcs/django/django/test/testcases.py:984: in _rollback_atomics
transaction.set_rollback(True, using=db_name)
../../../Vcs/django/django/db/transaction.py:92: in set_rollback
return get_connection(using).set_rollback(rollback)
../../../Vcs/django/django/db/backends/base/base.py:425: in set_rollback
"The rollback flag doesn't work outside of an 'atomic' block.")
E django.db.transaction.TransactionManagementError: The rollback flag doesn't work outside of an 'atomic' block.
Hi, thanks for the project. I'm trying to use it instead of wait-for-it script with docker-compose.
runserver doesn't fail because of an unavailable db - and that's right! However, when trying to apply migrations, the command fails:
migrator_1 | Reconnecting to the database didn't help could not connect to server: Connection refused
migrator_1 | Is the server running on host "db" (172.22.0.4) and accepting
migrator_1 | TCP/IP connections on port 5432?
migrator_1 |
migrator_1 | Traceback (most recent call last):
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 53, in ensure_connection_with_retries
migrator_1 | self.connect()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
migrator_1 | self.connection = self.get_new_connection(conn_params)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
migrator_1 | connection = Database.connect(**conn_params)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
migrator_1 | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
migrator_1 | psycopg2.OperationalError: could not connect to server: Connection refused
migrator_1 | Is the server running on host "db" (172.22.0.4) and accepting
migrator_1 | TCP/IP connections on port 5432?
migrator_1 |
migrator_1 |
migrator_1 | During handling of the above exception, another exception occurred:
migrator_1 |
migrator_1 | Traceback (most recent call last):
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 53, in ensure_connection_with_retries
migrator_1 | self.connect()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
migrator_1 | self.connection = self.get_new_connection(conn_params)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
migrator_1 | connection = Database.connect(**conn_params)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
migrator_1 | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
migrator_1 | psycopg2.OperationalError: could not connect to server: Connection refused
migrator_1 | Is the server running on host "db" (172.22.0.4) and accepting
migrator_1 | TCP/IP connections on port 5432?
migrator_1 |
migrator_1 |
migrator_1 | The above exception was the direct cause of the following exception:
migrator_1 |
migrator_1 | Traceback (most recent call last):
migrator_1 | File "./manage.py", line 15, in <module>
migrator_1 | execute_from_command_line(sys.argv)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
migrator_1 | utility.execute()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
migrator_1 | self.fetch_command(subcommand).run_from_argv(self.argv)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
migrator_1 | self.execute(*args, **cmd_options)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
migrator_1 | output = self.handle(*args, **options)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 83, in wrapped
migrator_1 | res = handle_func(*args, **kwargs)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 82, in handle
migrator_1 | executor = MigrationExecutor(connection, self.migration_progress_callback)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
migrator_1 | self.loader = MigrationLoader(self.connection)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in __init__
migrator_1 | self.build_graph()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 212, in build_graph
migrator_1 | self.applied_migrations = recorder.applied_migrations()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 61, in applied_migrations
migrator_1 | if self.has_table():
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 44, in has_table
migrator_1 | return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor
migrator_1 | return self._cursor()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor
migrator_1 | self.ensure_connection()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 71, in ensure_connection_with_retries
migrator_1 | self.ensure_connection()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 81, in ensure_connection_with_retries
migrator_1 | del self._in_connecting
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
migrator_1 | raise dj_exc_value.with_traceback(traceback) from exc_value
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django_dbconn_retry/__init__.py", line 53, in ensure_connection_with_retries
migrator_1 | self.connect()
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
migrator_1 | self.connection = self.get_new_connection(conn_params)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
migrator_1 | connection = Database.connect(**conn_params)
migrator_1 | File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
migrator_1 | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
migrator_1 | django.db.utils.OperationalError: could not connect to server: Connection refused
migrator_1 | Is the server running on host "db" (172.22.0.4) and accepting
migrator_1 | TCP/IP connections on port 5432?
Here is the relevant docker-compose.yml part:
migrator:
build: ./backend
command: ./manage.py migrate
working_dir: /app
volumes:
- ./backend/:/app
env_file:
- .env
db:
image: postgres:10-alpine
ports:
- 5432:5432
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.