Speasy offline mode

[jeandet]

We should be able to use Speasy even offline or when a server is down (#236)

1. Option 1, implicit cache and automatic offline mode, this implies:

   • we should not set expiration date on cache entries anymore (for data)
   • we should warn if there is a missing fragment inside a requested interval
   • users depend on previous requests and luck

2. Option 2, explicit data index, as described in Option to replace cache by an exportable persistent index #122, users could explicitly save data into a dedicated index for later usage.

   • this has to be simple to create, use and extend
   • it should be simple to update data inside an index
   • when extending an index, we might add data new fragments that have been reprocessed next to older ones

[Beforerr]

I’d like to bring up CDAWeb.jl recently developed as a related example. It takes an approach that’s somewhat in between the two: using a metadata table to track all downloaded files and their associated information. This way:
• You retain offline usability and reproducibility (the table is directly portable and inspectable), since the metadata table serves as a persistent record of what’s already available.
• Create, use and extend "index" is simply table operations.
• The approach stays relatively simple for users, while still offering fine-grained control for those who want it.

Might be worth considering whether something similar could work here—combining the convenience of Option 1 with the explicitness and reproducibility of Option 2.

Also, thanks to Speasy for being such a great source of inspiration.

P.S: For similar tasks like get_data and loading it into memory, CDAWeb.jl runs about 3–20× faster for cached data compared to speasy, which makes me wonder whether we really need a very complicated caching mechanism for performance reasons. I’m not entirely sure how caching works in Speasy (e.g., whether compressed data is decompressed and stored in a different format), but in my case simply storing/loading CDF files hasn’t been a bottleneck.

[jeandet]

I’d like to bring up CDAWeb.jl recently developed as a related example. It takes an approach that’s somewhat in between the two: using a metadata table to track all downloaded files and their associated information. This way: • You retain offline usability and reproducibility (the table is directly portable and inspectable), since the metadata table serves as a persistent record of what’s already available. • Create, use and extend "index" is simply table operations. • The approach stays relatively simple for users, while still offering fine-grained control for those who want it.

Looks nice! I don't know Julia so I wasn't able to see if your current implementation is thread and process safe. The nice thing with our current cache implementation is that is ensure all threads and processes share the same cache state. It is also quite fast, one SQLite request and one pickle load.

Might be worth considering whether something similar could work here—combining the convenience of Option 1 with the explicitness and reproducibility of Option 2.

Yes, all options are on the table, it just won't be implemented in one shot to avoid big breaking changes unless its is for a 2.0.0 release. Note that we also started to implement a custom cache replacement for Speasy.

Also, thanks to Speasy for being such a great source of inspiration.

P.S: For similar tasks like get_data and loading it into memory, CDAWeb.jl runs about 3–20× faster for cached data compared to speasy, which makes me wonder whether we really need a very complicated caching mechanism for performance reasons. I’m not entirely sure how caching works in Speasy (e.g., whether compressed data is decompressed and stored in a different format), but in my case simply storing/loading CDF files hasn’t been a bottleneck.

I suspect, the cache is not the issue here, more the way Speasy works. Even if a remote file is cached, Speasy will always send a request to the server to check if the file changed, sadly that HEAD request is not that fast (~10 to 100ms). This is one of the reason for that new to be documented prefer_cache argument. If you have some time, I would be happy to have the numbers using prefer_cache=True to confirm it gets closer to your implementation.
The big issue here is that speed, reproducibility and ensuring users have the best data (up to date with best calibration) are totally antagonist. As for making Speasy easy to use and giving control to users.
So far we tried to prioritize up to date data and simple interface with some opt-out options.

[Beforerr]

Thanks a lot for the detailed explanation — that makes sense. My current simple implementation is not thread- or process-safe, it was mainly intended as a lightweight prototype.

The new cache API you’ve designed looks very clean! I agree with your point: “the cache is not the issue.” That matches what I see as well. My main thought was that for reproducibility, it may sometimes be simpler to just store the CDF file directly in the filesystem rather than relying on pickle serialization.

Below are the benchmark numbers I obtained (calling Python overhead is very small ~50 ns in general):

julia> @b Speasy.get_data("cda/AC_H0_MFI/BGSEc", "2023-01-01", "2023-01-02"; prefer_cache=true)
4.612 ms (193 allocs: 6.641 KiB)

julia> @b Speasy.get_data("cda/AC_H0_MFI/BGSEc", "2023-01-01", "2023-01-02"; prefer_cache=false)
4.902 ms (193 allocs: 6.641 KiB)

# just retrieve variable without loading it
julia> @b CDAWeb.get_data("AC_H0_MFI/BGSEc", "2023-01-01", "2023-01-02")
24.875 μs (170 allocs: 7.594 KiB)

# retrieve and load data to memory
julia> @b Array(CDAWeb.get_data("AC_H0_MFI/BGSEc", "2023-01-01", "2023-01-02"))
194.875 μs (236 allocs: 76.938 KiB)

[jeandet]

@Beforerr , I didn't expect the first one to be that slow, I have to investigate this, 4ms for ~100kB is way too slow for me.
Thanks for pointing that out!

[jeandet]

I found the main reason Speasy is slow, specially here, even with prefer_cache=True. It will still list all files and parse names extracting dates to find which files are overlapping the requested interval. In this case there are 365 files per folder, Speasy will parse all 365 file names each time you request some data:

Fixing this should get us under 1ms. I'll work on this once I'm done with #241.

[jeandet]

@Beforerr #243 should help here, it's not as fast as I would like but still better.

[Beforerr]

Just checked. It is much faster now (10x times), truly amazing!

julia> @b Speasy.get_data("cda/AC_H0_MFI/BGSEc", "2023-01-01", "2023-01-02"; prefer_cache=true)
477.166 μs (236 allocs: 8.375 KiB)