[ENH] Create design document for model output and metrics standardization v2

[PranavBhatP]

Is your feature request related to a problem? Please describe.
Create a design document for outlining the current status quo and changes required for standardising model output and metrics for v2.

Draft: https://hackmd.io/DL0_NjbERIy65laob22GlA?view

Addtional context

PR #1895 #1897
Issue #1894

[fkiraly]

Suggestion of structure:

• summary
  • description of the aim
  • background/context
  • very high-level summary of change
• description of status quo
  • use case vignettes
    • list all important cases!
    • isolated and integrated
  • model output formats
  • metrics
• proposed change
  • high-level summary
  • model output formats
  • metrics
• implementation plan
  • pull request list - planned
  • testing
  • downwards compatibility

[phoeenniixx]

Copied from the discussion on HackMD

• AS

  • Do we really need a 4D output in models? I mean we are using the lsit of tensors for Multiloss then why do we need the extra dim for num_targets? Whenever we would need more than one target we can simply use list of tensors?
    • The main reason for my proposal is that would make even the current implementation of metrics compatible (Pranav please correct me if I am wrong) and we dont need to make changes to v1 models then?

• pb:

  • Could you maybe compile what the loss functions currently expect. If a 4D tensor input is requiring too many changes.

    • AS: Well we tried doing so in [ENH] Refactor metrics to be polymorphic #1897 but debugging is really hard in Metrics and models also I donot see any reason to keep the extra dim for num_targets if we are using list of tensors

  • PB: Ok, I get what you are suggesting, but I'm not fully sure how efficient it would be to use a list of tensor for every case and restrict the tensor size to 3D for all cases i.e list size will be 4th dimension right? I was assuming pure tensors are always better to use in pytorch rather than looping through targets whenever we have more than one?

    • AS: I understand that but changing the legacy code is really hard, maybe we should move this discussion to issue so that others can also comment?

FYI @fkiraly, @agobbifbk

Point of discussion:

• The proposal aims to standardise all the outputs of models in PTF v1 and provide a single output strategy for all models in v2.

• From the above observations

  • It is evident that there can be standardisation of output tensors, only subject to the number of loss functions used.
    • The standard output can be (batch_size, timesteps, n_targets, distribution_params) for a single loss case.
      • n_targets = 1 for single target and n_targets > 1 for multi-targets
      • distribution_params = 1 for point predictions and distribution_params > 1 for quantile or probabilistic predictions (based on the loss function).
    • The standard output can be list[torch.Tensor] for multi-loss (MultiLoss) case where the tensors within the list are standardised as per single loss case i.e each tensor is treated as a single target forecast and cast to (batch_size, timesteps, 1, distribution_params).

[PranavBhatP]

@phoeenniixx

AS: Well we tried doing so in #1897 but debugging is really hard in Metrics and models also I donot see any reason to keep the extra dim for num_targets if we are using list of tensors

So, if we have multi-targets (with same loss for all targets) we still use MultiLoss and basically call the same loss on each target, like below?

target = ["target1","target2"]

# instead of calling 
# loss = MAE() or QuantileLoss()

loss = MultiLoss([MAE()]*len(targets))

[phoeenniixx]

Yes exactly! Actually I did the same thing here in the nb that I used to show the vignette

[PranavBhatP]

Ok it's good that we are on the same page now :). I think we need a small discussion with others on which is better, since we are clear about the two approaches we can take. Can you add this approach as a separate vignette in the hackmd?

Currently I am not sure which approach is the right way...

[fkiraly]

From a planning perspective:

• list the approaches in the design doc, both - with function signatures, vignettes.
• think about pros/cons, from multiple perspectives like user comfort, software design desiderata, maintainability, efficieny, etc
• that brings the doc in a format where it is just "picking an option". Very important, we need to be in the position of making an informed decision, that is, the basis for the decision should be solid (and not conjectural or ad-hoc).

There will be multiple decisions and sets of options, such as:

• how to parameterize multi-target in terms of metrics, if the same metric is used for all targets
• 4D vs list of 3D

Etc. - please name these decision points and discuss them.
This is "normal" procedure for senior design tasks, the core devs do this routinely!