Improve AMDA handling

[Beforerr]

US-based users sometimes experience poor connectivity to AMDA. When this happens, speasy would error, which can be disruptive.

I'm just wondering if we could improve to just log a warning instead. This would provide a smoother user experience and reduce confusion when AMDA is temporarily unreachable.

InitError: Python: TypeError: argument of type 'NoneType' is not iterable
Python stacktrace:
 [1] _send_indirect_request
   @ speasy.core.impex.client ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/impex/client.py:154
 [2] get_obs_data_tree
   @ speasy.core.impex.client ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/impex/client.py:85
 [3] _get_obs_data_tree
   @ speasy.core.impex ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/impex/__init__.py:944
 [4] _get_obs_data_tree
   @ speasy.data_providers.amda.ws ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/data_providers/amda/ws.py:354
 [5] wrapped
   @ speasy.core.cache._function_cache ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/cache/_function_cache.py:52
 [6] build_inventory
   @ speasy.core.impex ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/impex/__init__.py:104
 [7] _inventory
   @ speasy.core.dataprovider ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/dataprovider.py:55
 [8] wrapped
   @ speasy.core.proxy ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/proxy/__init__.py:156
 [9] update_inventory
   @ speasy.core.dataprovider ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/dataprovider.py:65
 [10] __init__
   @ speasy.core.dataprovider ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/dataprovider.py:50
 [11] __init__
   @ speasy.core.impex ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/impex/__init__.py:63
 [12] __init__
   @ speasy.data_providers.amda.ws ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/data_providers/amda/ws.py:124
 [13] init_amda
   @ ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/requests_scheduling/request_dispatch.py:39
 [14] init_providers
   @ ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/requests_scheduling/request_dispatch.py:91
 [15] <module>
   @ ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/requests_scheduling/request_dispatch.py:99
 [16] <module>
   @ ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/core/requests_scheduling/__init__.py:2
 [17] <module>
   @ ~/src/Speasy.jl/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/speasy/__init__.py:21

[Beforerr]

See PR #229

[jeandet]

Actually, it's not only for US users only, we have the same issue in France (true everywhere). The main reason behind this is the way AMDA handles sessions, they have a finite allowed number of opened sessions. When using the webservice we have one session per IP and each session remain opened for 24 hours (IIRC), we tend to reach the limit easily when pushing a lot on GH and triggering many CI builds that end up on different VMs.
That a known issue inherited from legacy AMDA code, AMDA devs are aware of this and planing to fix this in the future.

[Beforerr]

Thanks for the detailed explanation! That makes sense. Do you know if the cache server failures I’ve seen from time to time are due to the same session limit issue?

[jeandet]

For the cache server, I have more insights since I set it up. It is simply a REST API on-top of Speasy, that means, if Speasy fails to load for any reason it crashes the cache server. That also mean that each correction we push on speasy land in the server too :).
Speasy works when the cache server is down but would behave as the cache server unless if you disable the faulty provider (in this case AMDA).
Note also that is runs with FastAPI (multi threads ans processes) inside a Podman container with automatic restarts, that makes it quite robust.
To answer your question, that's possible.
There is one other weird issue on GH actions, it seems that sometimes some runners are not able to resolve polytechnique.fr addresses, this one is really rare.

[brenard-irap]

It wasn't an issue related to session management in AMDA.
In fact, we have a disk volume allocated to AMDA workspaces, dedicated to managing user spaces and storing files generated by the WebService.
The problem was that during the night from Friday to Saturday, an automatic update on one of the datasets was executed, and it wrote a large number of files to this same workspace (which it shouldn't have), filling it up completely.
This workspace isn't originally intended for that kind of usage.
As a result, some of the application's working files couldn't be updated on the system, and even worse: they became corrupted.
The workspace was then automatically cleaned during the night from Saturday to Sunday, freeing up disk space.
However, the corrupted files remained corrupted, meaning AMDA couldn't properly handle process execution.
Everything was restored this morning. The AMDA application, as well as its access via WebService (and therefore via Speasy), are now fully operational again.