Dynamic batching and dynamic padding

[FlorentRamb]

Hello,

I'd like to deploy a Transformers model and use the dynamic batching feature. However the inputs shape may vary from one request to another (different sequences length).

Is there a way to perform dynamic padding (pad to the longest sequence length) within the dynamic batch created by Triton? What would be the best approach?

Thanks in advance!

[stekiri]

Same question here, any advice would be highly appreciated 🙏

@FlorentRamb, have you figured out a solution in the meantime?