fix: CI Postgres teardown — RobustCompressorTestRunner.teardown_databases now force-closes lingering connections before Django's DROP. CI step test-UTs-n-ITs (python manage.py test apps, full suite incl. channels-tagged tests) was failing post-test even when all 1165 tests passed — psycopg2.errors.ObjectInUse: database "test_python_tdd_test" is being accessed by other users / DETAIL: There is 1 other session using the database. Two-step leak: (1) core.settings.DATABASES['default']['conn_max_age']=600 keeps Postgres connections alive in the per-thread pool for 10 min (prod-perf default); (2) Channels' database_sync_to_async (16 call sites across apps.epic.tests.integrated.test_consumers's CursorMoveConsumerTest + SigHoverConsumerTest) runs in a process-wide asgiref threadpool — each worker thread accumulates its own DB connection that outlives the test + sits idle in the pool when teardown fires. Postgres refuses DROP while ANY session targets the row. Local dev unaffected: --exclude-tag=channels skips the consumer tests + SQLite has no DROP step. **Fix** lives entirely in core/runner.py's teardown override — connections.close_all() covers the main thread's runner connection; iterating old_config + running SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = %s AND pid <> pg_backend_pid() against each test DB kicks any worker-thread session still pinning the row. Both safe on a green run (DB about to be dropped anyway) + scoped to vendor == "postgresql" so SQLite local-dev is a clean no-op. Prod CONN_MAX_AGE=600 untouched — fix lives in the test runner, NOT in settings. 19/19 lyric UTs green via the new runner path (smoke verify the override is benign on SQLite); Postgres-side validated next CI run. Trap captured: [[feedback-test-teardown-conn-leak]] — symptom signature Ran NNNN tests / OK / Destroying.../ObjectInUse: ... belongs in CI-fail triage notes so future flakes get diagnosed in seconds instead of test-hunting
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -58,3 +58,42 @@ class RobustCompressorTestRunner(DiscoverRunner):
|
|||||||
except PermissionError: time.sleep(0.05)
|
except PermissionError: time.sleep(0.05)
|
||||||
raise
|
raise
|
||||||
CompressorFileStorage.save = _robust_save
|
CompressorFileStorage.save = _robust_save
|
||||||
|
|
||||||
|
def teardown_databases(self, old_config, **kwargs):
|
||||||
|
"""Force-close any lingering DB connections before Django issues
|
||||||
|
`DROP DATABASE`. Without this, CI fails post-test w. Postgres'
|
||||||
|
`ObjectInUse: database "..." is being accessed by other users`
|
||||||
|
even on a fully-green run.
|
||||||
|
|
||||||
|
Root cause: `core.settings.DATABASES['default']['conn_max_age']
|
||||||
|
= 600` keeps Postgres connections alive in the per-thread pool
|
||||||
|
for 10 min. Channels' `database_sync_to_async` (used in
|
||||||
|
`apps.epic.tests.integrated.test_consumers`'s `CursorMoveConsumerTest`
|
||||||
|
+ `SigHoverConsumerTest`) runs in a global asgiref threadpool;
|
||||||
|
each worker thread accumulates its own DB connection that
|
||||||
|
outlives the test + sits idle in the pool when teardown fires.
|
||||||
|
Postgres refuses the drop while ANY session targets the row.
|
||||||
|
|
||||||
|
Two-step fix — `connections.close_all()` covers the main
|
||||||
|
thread (test runner's own connection); `pg_terminate_backend`
|
||||||
|
targets any worker-thread session still pinning the test DB.
|
||||||
|
Both are safe on a green test run: the DB is about to be
|
||||||
|
dropped anyway, and prod's `CONN_MAX_AGE=600` stays untouched
|
||||||
|
(the fix lives in the test runner, not in `settings.py`).
|
||||||
|
|
||||||
|
Local dev (SQLite, no DROP step) is unaffected; CI Postgres
|
||||||
|
is where the leak was visible. See [[feedback-test-teardown-conn-leak]].
|
||||||
|
"""
|
||||||
|
from django.db import connections
|
||||||
|
connections.close_all()
|
||||||
|
for connection, _, _ in old_config:
|
||||||
|
if connection.vendor == "postgresql":
|
||||||
|
test_db_name = connection.settings_dict["NAME"]
|
||||||
|
with connection._nodb_cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT pg_terminate_backend(pid) "
|
||||||
|
"FROM pg_stat_activity "
|
||||||
|
"WHERE datname = %s AND pid <> pg_backend_pid()",
|
||||||
|
[test_db_name],
|
||||||
|
)
|
||||||
|
super().teardown_databases(old_config, **kwargs)
|
||||||
|
|||||||
Reference in New Issue
Block a user