The consumer locks up when Redis is unavailable

[abompard]

This may be an upstream bug, but we got this in the log:

[2023-06-12 12:08:55,710] [ERROR cashews.backends.redis.client] redis: can not execute command: SET
Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/connection.py", line 843, in read_response
    response = await self._parser.read_response(
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/connection.py", line 265, in read_response
    response = await self._read_response(disable_decoding=disable_decoding)
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/connection.py", line 273, in _read_response
    raw = await self._readline()
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/connection.py", line 347, in _readline
    data = await self._stream.readline()
  File "/usr/lib64/python3.10/asyncio/streams.py", line 524, in readline
    line = await self.readuntil(sep)
  File "/usr/lib64/python3.10/asyncio/streams.py", line 616, in readuntil
    await self._wait_for_data('readuntil')
  File "/usr/lib64/python3.10/asyncio/streams.py", line 501, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/connection.py", line 842, in read_response
    async with async_timeout(read_timeout):
  File "/opt/app-root/lib64/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/opt/app-root/lib64/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.10/site-packages/cashews/backends/redis/client.py", line 26, in execute_command
    return await super().execute_command(command, *args, **kwargs)
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/client.py", line 518, in execute_command
    return await conn.retry.call_with_retry(
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/retry.py", line 62, in call_with_retry
    await fail(error)
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/client.py", line 505, in _disconnect_raise
    raise error
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
    return await do()
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/client.py", line 492, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/client.py", line 539, in parse_response
    response = await connection.read_response()
  File "/opt/app-root/lib64/python3.10/site-packages/redis/asyncio/connection.py", line 857, in read_response
    raise TimeoutError(f"Timeout reading from {self.host}:{self.port}")
redis.exceptions.TimeoutError: Timeout reading from redis.fmn.svc.cluster.local:6379

The consumer was frozen afterwards, which is bad. It should either ignore the error, retry, or die properly.