health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception #376
Description
Background
I've been using django-health-check for ages, with success. I'm running in Kubernetes with persistent storage on TrueNAS, via democratic-csi. The persistent volumes are ReadWriteMany and formatted as ext4.
I've recently switched my persistent storage to Rook/Ceph, reprovisioned all my volumes and redeployed the app. The Ceph volumes are otherwise working (I can create files & dirs on them) but django-health-check seems unable to create dirs.
Logs
unavailable: Unknown exception
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 63, in check_status
file_name = self.check_save(file_name, file_content)
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 42, in check_save
file_name = storage.save(file_name, ContentFile(content=file_content))
File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 54, in save
name = self._save(name, content)
File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 260, in _save
os.makedirs(directory, exist_ok=True)
File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/camerahub/media/health_check_storage_test'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
self.check_status()
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 67, in check_status
raise ServiceUnavailable("Unknown exception") from e
health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception
Internal Server Error: /health/
[pid: 8|app: 0|req: 1/1] 192.168.0.58 () {30 vars in 363 bytes} [Sun Sep 3 18:57:44 2023] GET /health/ => generated 1767 bytes in 344 msecs (HTTP/1.1 500) 7 headers in 298 bytes (1 switches on core 0)
unavailable: Unknown exception
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 63, in check_status
file_name = self.check_save(file_name, file_content)
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 42, in check_save
file_name = storage.save(file_name, ContentFile(content=file_content))
File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 54, in save
name = self._save(name, content)
File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 260, in _save
os.makedirs(directory, exist_ok=True)
File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/camerahub/media/health_check_storage_test'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
self.check_status()
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 67, in check_status
raise ServiceUnavailable("Unknown exception") from e
health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception
Internal Server Error: /health/
[pid: 7|app: 0|req: 1/2] 192.168.0.58 () {30 vars in 363 bytes} [Sun Sep 3 18:58:00 2023] GET /health/ => generated 1767 bytes in 379 msecs (HTTP/1.1 500) 7 headers in 298 bytes (1 switches on core 0)
unavailable: Unknown exception
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 63, in check_status
file_name = self.check_save(file_name, file_content)
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 42, in check_save
file_name = storage.save(file_name, ContentFile(content=file_content))
File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 54, in save
name = self._save(name, content)
File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 260, in _save
os.makedirs(directory, exist_ok=True)
File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/camerahub/media/health_check_storage_test'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
self.check_status()
File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 67, in check_status
raise ServiceUnavailable("Unknown exception") from e
health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception
Internal Server Error: /health/
Debug
My app is running as a root container, permissions on the volume look fine and I can create a dir manually. The rest of the app deployment has the same permissions as the media subdir.
root@camerahub-dev-app-5bc6645cf8-7j46z:/media# ls -la
total 8
drwxr-xr-x 2 root root 4096 May 22 00:00 .
drwxr-xr-x 1 root root 4096 Sep 3 18:57 ..
root@camerahub-dev-app-5bc6645cf8-7j46z:/media# mkdir test
root@camerahub-dev-app-5bc6645cf8-7j46z:/media# ls -la
total 12
drwxr-xr-x 1 root root 4096 Sep 3 19:49 .
drwxr-xr-x 1 root root 4096 Sep 3 18:57 ..
drwxr-xr-x 2 root root 4096 Sep 3 19:49 test
I can't see any reason why the dir can't be created by django-health-check. Any clues?