Make lazy parsing (defer_iteration_parsing) more discoverable by giving a hint on the command line

[franzpoeschel]

Close #1799.

Warning: Parsing Iterations is taking a long time. Consider setting {"defer_iteration_parsing": true} for lazy opening of the Series and then explicitly opening Iterations with Iteration::open(). Refer also to the documentation at https://openpmd-api.readthedocs.io. Suppress this warning by setting {"hint_lazy_parsing_timeout": 0}.
Elapsed time: 5s, parsed 991 of 10030 Iterations.
Elapsed time: 10s, parsed 1951 of 10030 Iterations.
Elapsed time: 15s, parsed 2871 of 10030 Iterations.
Elapsed time: 20s, parsed 3755 of 10030 Iterations.
Elapsed time: 25s, parsed 4606 of 10030 Iterations.
Elapsed time: 30s, parsed 5429 of 10030 Iterations.
Elapsed time: 35s, parsed 6224 of 10030 Iterations.
Elapsed time: 40s, parsed 6993 of 10030 Iterations.
Elapsed time: 45s, parsed 7737 of 10030 Iterations.
Elapsed time: 50s, parsed 8461 of 10030 Iterations.
Elapsed time: 55s, parsed 9155 of 10030 Iterations.
Elapsed time: 60s, parsed 9841 of 10030 Iterations.

TODO:

• docs
  • Python Series options: https://openpmd-api.readthedocs.io/en/0.16.1/details/python.html, showing concrete series-level options in Python snippets
• add env var for suppressing the warning / setting the timeout
• extend documentation, adapt examples, maybe https://openpmd-api.readthedocs.io/en/0.16.1/details/python.html

[ax3l]

That is a neat idea.

[pordyna]

Nice idea, I still find it always hard to find how to pass that option when opening a series in python and this comment doesn't help much. So maybe add an example like series = io.Series(path, io.Access.read_only, options='{"defer_iteration_parsing": true}') to the readthedocs documentation and link the paragraph in the warning? I think most users don't actually know how to use the json configuration.

[franzpoeschel]

Nice idea, I still find it always hard to find how to pass that option when opening a series in python and this comment doesn't help much. So maybe add an example like series = io.Series(path, io.Access.read_only, options='{"defer_iteration_parsing": true}') to the readthedocs documentation and link the paragraph in the warning? I think most users don't actually know how to use the json configuration.

I guess that the easiest way would be to write the example directly into the error message?

[pordyna]

Nice idea, I still find it always hard to find how to pass that option when opening a series in python and this comment doesn't help much. So maybe add an example like series = io.Series(path, io.Access.read_only, options='{"defer_iteration_parsing": true}') to the readthedocs documentation and link the paragraph in the warning? I think most users don't actually know how to use the json configuration.

I guess that the easiest way would be to write the example directly into the error message?

Sure!

[franzpoeschel]

Now with a more extensive warning:

[openPMD] WARNING: Parsing Iterations is taking a long time.
Consider using deferred Iteration parsing in order to open the Series lazily.
This can be achieved by either setting an environment variable:

> export OPENPMD_DEFER_ITERATION_PARSING=1

Or by specifying it as part of a JSON/TOML configuration:

> // C++:
> Series simData("my_data_%T.%E", R"({"defer_iteration_parsing": true})");
> // Python:
> simData = opmd.Series("my_data_%T.%E", {"defer_iteration_parsing": True})

Iterations will then be parsed only upon explicit user request:

> series.snapshots()[100].open()  // new API
> series.iterations[100].open()   // old API

Alternatively, Iterations will be opened implicitly when iterating in
READ_LINEAR access mode.
Refer also to the documentation at https://openpmd-api.readthedocs.io

This warning can be suppressed also by either specifying
an environment variable:

> export OPENPMD_HINT_LAZY_PARSING_TIMEOUT=0

Or by the JSON/TOML option {"hint_lazy_parsing_timeout": 0}.

Elapsed time: 1s, parsed 592 of 10000 Iterations.
Elapsed time: 2s, parsed 1171 of 10000 Iterations.
Elapsed time: 3s, parsed 1734 of 10000 Iterations.
Elapsed time: 4s, parsed 2281 of 10000 Iterations.
Elapsed time: 5s, parsed 2816 of 10000 Iterations.