Handle issues from other packages side?

[tanglef]

The CI is currently red and one of the reasons is a problem from a package that is used (see ).
Technically, this is not a BenchOpt error, and it can impact multiple repositories if the same library is used for different problems to solve.

So where is the limit (if there is one) between a warning in the CI that there is a problem on some library side and a real error from BenchOpt that we can deal with?

(poke @josephsalmon)

[tomMoral]

I would say it is fine to have the CI broken when a package change/is not installed correctly anymore.
This allows to have a good idea on whether or not a benchmark can be run.

That being said, if the CI breaks due to an external package and there is no fix on the way, we can also skip this particular test using the test_config.py file that allows to skip this particular solver in the tests.

[tanglef]

All right. Thank you for the clarification

So I am wondering would it make more sense from a user pov to have at the end of the readme a table with the status of each "dependency" instead of having only one status per problem? (not that I know how to make it right now)
That would also be easier for us to debug and skip the test if needed.

[tomMoral]

Interesting idea. I think this is easy to do in the context of running the tests separately but the issue is morer how to report this.
This feels like either lot of CI dev work to hack this into github actions or we need a database to store the results.

I would say it is not a priority right now but it could be fun to try.
Let's keep the issue open and if someone has an idea, feel free to propose a solution.