Add multimodal flow matching example#83
Add multimodal flow matching example#83amorehead wants to merge 47 commits intofacebookresearch:mainfrom
Conversation
|
Thanks @amorehead! Can you please give a more detailed explanation on the multimodal model? |
|
Hi, @itaigat. The multimodal model consists of similar MLPs as used for the discrete and continuous (checkerboard) flow matching examples. However, now they are used for generating both input and output embeddings of each respective modality, and they each share a common |
Improve performance of multimodal model by integrating all modalities into a single model and solver
|
@itaigat, I've added a new |
This reverts commit bed9ec6.
…for discrete sampling more robust; initialize solver once; and assert for all continuous floating-point types
flow_matchingpackage's components.flow_matching/utils/multimodal.pymodule andflow_matching/solver/multimodal_solver.pyare introduced to construct instances of the new multimodalFlowclass and sample from it.