[Feature]: Model FLOPs Utilization Reporting

[bwasti]

🚀 The feature, motivation and pitch

I'm working on inference optimization and would like Model FLOPs Utilization (MFU) to be more readily reported by vLLM. It knows the model and it knows the hardware -- that's all you need!

I'd like to open this issue for discussion on the best way to report MFU. Some open questions include:

• where does this get dumped? periodically printed like the speculative decoding metrics?
• should it be always on or require something like "/start_profile"?
• should care be taken to make FLOP counts exact (and unreported if the model isn't supported) or can simple approximations be used? (e.g. 2x # params)
• for hardware peak should we compile + check in vendor metrics? should we use simple approximations (e.g. number of tensorcores * clock)?

In general I would like to volunteer to help build out a calculator.

Alternatives

manual calculation

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[zhuohan123]

I think from vLLM's perspective, let's get the numerator of the MFU correct. It should be easy for us to compute the actual flop used by each inference step, then we can report the flops by dividing the actual flop by the step time. We only need to report the flops and any user can compute the MFU by finding the proper denominator through their specific hardware spec.

[bwasti]

let's get the numerator of the MFU correct

this can be quite hard to do precisely. some questions about that:

• do you think if we don't have new modules supported should we just bail and not report MFU?
• how should we handle speculative decoding, which "wastes" compute to some degree (but certainly improves utilization)?

I like the splitting up the denominator -- which allows us to ignore things like quantization (although this might be something the user wants in the future, since only parts of the model will run fp8 for example).

[zhuohan123]

where does this get dumped? periodically printed like the speculative decoding metrics?

It should be dumped together with where we dump our throughput numbers?

should it be always on or require something like "/start_profile"?

I think the cost of this is very low and it should be always on.

should care be taken to make FLOP counts exact (and unreported if the model isn't supported) or can simple approximations be used? (e.g. 2x # params)

If getting the exact count is too much work then we can start from params approach and we can report both eventually. But even just counting params we still need to have special handling for MoE layers to make sure we only count active params.

for hardware peak should we compile + check in vendor metrics? should we use simple approximations (e.g. number of tensorcores * clock)?

As above we can ignore this for now and get the numerator out first.

do you think if we don't have new modules supported should we just bail and not report MFU?

We can return infinity or negative infinity for these layers so people won't take the wrong MFU.

how should we handle speculative decoding, which "wastes" compute to some degree (but certainly improves utilization)?

I think let's focus on getting actual hardware flops first.

[youkaichao]

FYI we can get tensor core utilizations from GPU monitors, this is very close to MFU.

calculating MFU directly from model config would be quite complicated.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[markmc]

#30738 has been merged