Arviz To Scoring Vignette

[O957]

This issue covers creating a vignette in this repository that goes from an arviz InferenceData object to scored forecasts (using scoringutils).

@zsusswein @SamuelBrand1 @dylanhmorris

Sam, feel free to remove yourself as an assignee.

[zsusswein]

Thanks for starting this issue!

I have a few notes from yesterday's meeting and some asks for this vignette for things that I'd personally find helpful.

I liked the work you all did on scoring your models in testing and I'd like to see this vignette go beyond scored forecasts to some default plots and diagnostics that are helpful when interpreting and visualizing the performance of a series of model fits. That is to say, the PR associated with this vignette could include¹:

• Documentation for the functions that are here that you all use and an explanation of how they fit together (i.e., what passes to what?)
• How do I feed in multiple geographies? I have cases where these come from independent models and cases where these come from the same model. How would your code handle these cases?
• Visuals breaking out scores by horizon (I'm thinking something like nowcast period, 7-day ahead, 14-day ahead). I think this could look like slide 18?
• The visuals from slide 8 or slide 9 of score relative to a simple, reasonable baseline
• The visual of the WIS breakdown from multiple geographies (slide 18)
• Something like the diagnostics on slides 15 and 16.

As I see it, the hard work in this ask is distilling how you all have thought about visualizing and explaining model forecast performance and how you have laid that out in your code.

Footnotes

1. I don't want to paste figures in here because we're on the CDCGov side, but I'm going to reference figures by slide number from here. ↩

[O957]

Thank you very much Zach for this detailed message and your thoughts.

As I implement, I will remark on any problems or notable decisions necessitating user-feedback and, subsequently, will provide updates on this issue's scope.

[kgostic]

Hey guys, just jumping in here. I think that what @zsusswein asked for is the right outline, but I think that the real meat of what we need is basically a script that producess the set of outputs that you guys are using to evaluate your models scientifically.

We don't necessarily need help writing the code to use scoringutils or to transform between data formats, although it would be nice to see what you guys are doing if we're able to use the script to move in the same direction.

But I think the milestone that we really want to get to here is to see the suite of outputs that you guys are actually generating when you run your pipelines, and then to use that to:

1. Plug in our own outputs
2. Have a scientific discussion about whether this is the right set of evaluation outputs to cover both our team's needs

I think you guys probably already have this script somewhere to run in production. If you're able to just share that, or drop it here into a vignette, I think that is all we need to move forward.

[O957]

Re: #73 (comment)

Your comment is clarifying, thank you for making it, Katie.

Having read this comment, I can see that there are somewhat more divergent action-lines that this present issue could have covered.

Some points:

• Re: "I think you guys probably already have this script somewhere to run in production".
  • Tagging @dylanhmorris to link pyrenew-hew files relevant to this discussion.
  • A comment from DHM on the feasibility of "dropping [the files] into a vignette" would be appreciated.
• Re: "We don't necessarily need help writing the code to use scoringutils or to transform between data formats"
  • There are other elements in forecasttools right now for these tasks.
  • I am confirming here with this comment that I understand that the tasks outlined in your comment are not the same as these other, existing elements (e.g. Connect scoringutils To forecasttools #9 , Function To Cast InferenceData Into tidy_draws Format #36 , Hubverse Table To Scoring #72 ).
• Re: "Have a scientific discussion about whether this is the right set of evaluation outputs to cover both our team's needs"
  • Such a discussion should occur soon, in my view.
  • I am not sure if this issue is the best location.
  • The when and where can be decided by DHM and KG.

[dylanhmorris]

Summarizing f2f discussion I just had with @kgostic and @pulliam-cdc:

Going from scoringutils::score() output (as, e.g. in the forecasttools vignette or within pyrenew-hew here) to the viz you saw in our presentation happens in these this script :https://github.com/CDCgov/pyrenew-hew/blob/main/pipelines/summarize_visualize_scores.R

There are also some examples in this more lightweight script for quick scoring of hubverse tables: https://github.com/CDCgov/pyrenew-hew/blob/main/pipelines/hubverse_score.R

Neither of these is a generic tool, but both reflect the kinds of postprocessing and visualization we're doing for pyrenew-hew eval (and also COVIDHub eval). We would love to make some of the functions more generic and contribute them to forecasttools / add them to the vignette. We'll try to do it ourselves but we also welcome contributions.

[kgostic]

Summarizing a follow-up discussion about what's needed:

@zsusswein would like to be able to call main() from https://github.com/CDCgov/pyrenew-hew/blob/main/pipelines/summarize_visualize_scores.R, but the inputs are not clearly documented and would be time consuming for him to reverse engineer.

Is it possible for @AFg6K7h4fhy2 to build on this vignette to add a worked example with documented inputs, of that call to main()?

@dylanhmorris points out that there is some documentation already if you run Rscript summarize_visualize_scores.R --help, but we need a little more info. E.g. what is the expected output format of "collate_score_tables.R", etc.

[O957]

Re: #73 (comment)

@kgostic I am pretty sure I can (1) add more documentation and (2) produce a vignette. A guiding comment from @dylanhmorris would help me, though; I am not sure how much this task is relevant to forecasttools-py and would benefit from some clarification.

Also, thank you for the screenshot and links.