[v1] Add cross-attention KV cache support for encoder-decoder models

[russellb]

• Implement CrossAttentionManager for managing encoder states in KV cache
• Add num_encoder_tokens parameter to allocation methods for cross-attention blocks
• Update scheduler to handle encoder token allocation for Whisper models
• Disable prefix caching for cross-attention blocks since encoder states are request-specific
• Add encoder-decoder compatibility checks with KV connectors

This is a subset of the changes from #21088. It includes the changes
to the KV cache manager and scheduler for supporting cross-attention
for Whisper.

Signed-off-by: Russell Bryant [email protected]

[gemini-code-assist]

Code Review

This pull request introduces support for cross-attention KV cache in encoder-decoder models, with an initial focus on Whisper. The changes to the KV cache coordinator and scheduler to handle encoder token allocation are logical. However, I've identified three critical issues that would lead to runtime errors. The new CrossAttentionManager incorrectly raises errors in find_longest_cache_hit and cache_blocks instead of handling the cases gracefully. Additionally, there's a call to a non-existent method get_encdec_max_encoder_len in MULTIMODAL_REGISTRY. These issues need to be addressed to ensure the functionality works as intended.

[heheda12345]

LGTM!