Running your demo (chat.py)

[guillaumejs2403]

Hi,

I've been trying to run your model, but ran into some problems. Installation completed without issues, but when running the chat.py script I got the following error:

No module named 'torch.distributed.device_mesh'

From what I found, device_mesh was introduced in torch>=2.2, while the repository specifies torch==2.1.2. Could you clarify which PyTorch version is required?

Also, the README refers to a VideoGLaMM/requirements_sam2_cluster.txt file, but this file doesn’t exist. Is it meant to be requirements.txt instead?

Finally, which version of flash-attn are you using?

Thank you for your help!

[guillaumejs2403]

I managed to install the env by changing some of the requirements. If someone is interested, I did it like this:

conda create --name=videoglamm python=3.11
conda activate videoglamm

pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorch.org/whl/cu118
pip install transformers==4.41.0
DS_BUILD_FUSED_ADAM=1 pip install deepspeed==0.14.0

pip install packaging
pip install sentencepiece
pip install einops==0.4.1
pip install fastapi==0.100.1
pip install markdown2==2.4.10
pip install numpy
pip install openai==1.51.2
pip install opencv_python
pip install Pillow==11.0.0
pip install pycocotools==2.0.6
pip install ray==2.6.1
pip install Requests==2.32.5
pip install shortuuid==1.0.11
pip install tqdm
pip install uvicorn==0.23.2
pip install scipy==1.11.2
pip install bitsandbytes==0.41.1
pip install timm
pip install blobfile
pip install mypy
pip install pytest
pip install requests
pip install tensorboardX
pip install ftfy
pip install opencv-python
pip install pyarrow
pip install torchmetrics==0.7.3
pip install pycocoevalcap
pip install torchscale==0.2.0
pip install hydra-core
pip install accelerate
pip install triton==2.2.0
pip install peft==0.12.0
pip install gradio==3.39.0
pip install tensorboard
pip install scikit-image
pip install decord
pip install h5py
pip install jsonlines
pip install ffmpeg-python
pip install py-cpuinfo
pip install scikit-learn
pip install flash-attn==2.4.3.post1
cd VideoGLaMM/model/segment_anything_2
TORCH_CUDA_ARCH_LIST="9.0 8.0" python setup.py build_ext --inplace
cd ../../..

I can run the code now but I have other problems now. When loading the weights, I always get some warning about unexpected keys. Is this normal?