Skip to content

[CI/Build] Update causal-conv1d and lm-eval #22141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions .buildkite/test-pipeline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -535,8 +535,6 @@ steps:
- vllm/
- tests/models/language
commands:
# Install causal-conv1d for plamo2 models here, as it is not compatible with pip-compile.
- pip install 'git+https://github.com/Dao-AILab/[email protected]'
- pip freeze | grep -E 'torch'
- pytest -v -s models/language -m core_model

Expand All @@ -547,8 +545,6 @@ steps:
- vllm/
- tests/models/language/generation
commands:
# Install causal-conv1d for plamo2 models here, as it is not compatible with pip-compile.
- pip install 'git+https://github.com/Dao-AILab/[email protected]'
- pytest -v -s models/language/generation -m hybrid_model

- label: Language Models Test (Extended Generation) # 1hr20min
Expand All @@ -558,8 +554,6 @@ steps:
- vllm/
- tests/models/language/generation
commands:
# Install causal-conv1d for plamo2 models here, as it is not compatible with pip-compile.
- pip install 'git+https://github.com/Dao-AILab/[email protected]'
- pytest -v -s models/language/generation -m '(not core_model) and (not hybrid_model)'

- label: Language Models Test (Extended Pooling) # 36min
Expand Down
1 change: 1 addition & 0 deletions docker/Dockerfile.cpu
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ WORKDIR /workspace/vllm

RUN --mount=type=bind,src=requirements/test.in,target=requirements/test.in \
cp requirements/test.in requirements/cpu-test.in && \
sed -i '/causal_conv1d/d' requirements/cpu-test.in && \
sed -i '/mamba_ssm/d' requirements/cpu-test.in && \
sed -i 's/^torch==.*/torch==2.6.0/g' requirements/cpu-test.in && \
sed -i 's/torchaudio.*/torchaudio/g' requirements/cpu-test.in && \
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.rocm
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm /vllm-workspace
RUN cd /vllm-workspace \
&& rm -rf vllm \
&& python3 -m pip install -e tests/vllm_test_utils \
&& python3 -m pip install lm-eval[api]==0.4.4 \
&& python3 -m pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git@bc811365ef3917a15da6aac9b7feafb4ba0fb42f#egg=lm-eval[api] \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Pinning a dependency to a specific commit hash can introduce maintenance and security risks. It makes the build less reproducible if the commit history is changed (e.g., force-push) and bypasses the security vetting of official releases. If this specific commit is necessary to unblock a temporary issue, please add a comment explaining the reason for this pin and which issue/PR it relates to. This will help with future maintenance and make it clear when this can be reverted to a stable, versioned release.

&& python3 -m pip install pytest-shard

# -----------------------
Expand Down
2 changes: 1 addition & 1 deletion docs/contributing/ci/update_pytorch_version.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ uv pip install --system \
### causal-conv1d

```bash
uv pip install 'git+https://github.com/Dao-AILab/[email protected].0.post8'
uv pip install 'git+https://github.com/Dao-AILab/[email protected].2'
```

## Update all the different vLLM platforms
Expand Down
3 changes: 2 additions & 1 deletion requirements/test.in
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,15 @@ torch==2.7.1
torchaudio==2.7.1
torchvision==0.22.1
transformers_stream_generator # required for qwen-vl test
causal_conv1d==1.5.2 # required for plamo2 test
mamba_ssm==2.2.5 # required for plamo2 test
matplotlib # required for qwen-vl test
mistral_common[image,audio] >= 1.8.2 # required for voxtral test
num2words # required for smolvlm test
open_clip_torch==2.32.0 # Required for nemotron_vl test
opencv-python-headless >= 4.11.0 # required for video test
datamodel_code_generator # required for minicpm3 test
lm-eval[api]==0.4.8 # required for model evaluation test
lm-eval[api] @ git+https://github.com/EleutherAI/lm-evaluation-harness.git@bc811365ef3917a15da6aac9b7feafb4ba0fb42f # required for model evaluation test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Pinning a dependency to a specific commit hash, as done here for lm-eval, can be risky for long-term maintenance and security. It makes it difficult to track updates and can lead to build instability if the commit is ever removed or the repository history is altered. It's generally better to use a versioned release. If this pin is a temporary measure to resolve a specific compatibility issue, it would be very helpful to add a comment here explaining the context, linking to the relevant issue or PR, and outlining the conditions under which this can be updated to a released version.

mteb[bm25s]>=1.38.11, <2 # required for mteb test
transformers==4.53.2
tokenizers==0.21.1
Expand Down
10 changes: 8 additions & 2 deletions requirements/test.txt
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ cachetools==5.5.2
# via
# google-auth
# mlflow-skinny
causal-conv1d==1.5.2
# via -r requirements/test.in
certifi==2024.8.30
# via
# fiona
Expand Down Expand Up @@ -410,7 +412,7 @@ lightning-utilities==0.14.3
# torchmetrics
llvmlite==0.44.0
# via numba
lm-eval==0.4.8
lm-eval @ git+https://github.com/EleutherAI/lm-evaluation-harness.git@bc811365ef3917a15da6aac9b7feafb4ba0fb42f
# via -r requirements/test.in
lxml==5.3.0
# via
Expand Down Expand Up @@ -477,7 +479,9 @@ networkx==3.2.1
# scikit-image
# torch
ninja==1.11.1.3
# via mamba-ssm
# via
# causal-conv1d
# mamba-ssm
nltk==3.9.1
# via rouge-score
num2words==0.5.14
Expand Down Expand Up @@ -618,6 +622,7 @@ packaging==24.2
# via
# accelerate
# black
# causal-conv1d
# datamodel-code-generator
# datasets
# evaluate
Expand Down Expand Up @@ -1079,6 +1084,7 @@ torch==2.7.1+cu128
# -r requirements/test.in
# accelerate
# bitsandbytes
# causal-conv1d
# efficientnet-pytorch
# encodec
# fastsafetensors
Expand Down
3 changes: 0 additions & 3 deletions tests/models/language/generation/test_hybrid.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,6 @@

HYBRID_MODELS = [
"ai21labs/Jamba-tiny-dev",
# NOTE: Running Plamo2 in transformers implementation requires to install
# causal-conv1d package, which is not listed as a test dependency as it's
# not compatible with pip-compile.
"pfnet/plamo-2-1b",
"Zyphra/Zamba2-1.2B-instruct",
"hmellor/tiny-random-BambaForCausalLM",
Expand Down
Loading