[QUESTION] About Optimizer & Params Offload

[shh2000]

Hello, I wonder if there's any plan to support offload optimizer states even params like ZeRO-offload? DeepSpeed has offload but without mcore's parallel or TE. I hope there's any possibility to use offload with TP/PP/CP and TE, achieving high performance, especially when h2d/d2h bandwidth is higher like those mentioned in https://openreview.net/pdf?id=rqn2v1Ltgn0.

Looking forward to your reply and thanks!

[deepakn94]

CPU offloading is not on our roadmap right now.