Skip to content

[WIP] release-v2.7 CI #1202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+23 −14
Open

[WIP] release-v2.7 CI #1202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6195e56
Attempt libtiff uninstall to deal with vulnerability
trvachov Sep 29, 2025
8d88bd8
Update release notes to v2.7
trvachov Sep 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CODE-REVIEW.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
docs/docs/user-guide/contributing/code-review.md
docs/docs/main/contributing/code-review.md
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
docs/docs/user-guide/contributing/contributing.md
docs/docs/main/contributing/contributing.md
3 changes: 3 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,3 +372,6 @@ RUN chmod 777 -R /workspace/bionemo2/
# FIXME the following results in unstable training curves even if faster.
# See https://github.com/NVIDIA/bionemo-framework/pull/421
# ENV NVTE_FUSED_ATTN=1 NVTE_FLASH_ATTN=0
# Remove libtiff just for v2.7 release due to CVE https://nvd.nist.gov/vuln/detail/CVE-2025-9900
# This has no effect on bionemo since we do not do any image processing.
RUN apt remove -qy libtiff6
Comment on lines +375 to +377
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

🧩 Analysis chain

Verify CVE reference and use apt-get for scripting.

The change removes libtiff6 to mitigate CVE-2025-9900. However:

  1. The apt command is intended for interactive use; scripts should prefer apt-get.
  2. The command will fail if libtiff6 is not installed, breaking idempotency.
  3. No cleanup of apt cache is performed after removal.

Apply this diff to use apt-get and ensure idempotency:

-RUN apt remove -qy libtiff6
+RUN apt-get remove -qy libtiff6 || true && \
+    apt-get autoremove -qy && \
+    rm -rf /var/lib/apt/lists/*

Run the following script to verify the CVE exists and check dependencies:

Additionally, search the web to verify the CVE:

🏁 Script executed:

#!/bin/bash
# Verify CVE-2025-9900 exists and check libtiff6 dependencies

# Check for any references to libtiff in the codebase
echo "=== Checking for libtiff usage in codebase ==="
rg -i "libtiff|tiff" --type py --type dockerfile -C 2

# Check if there are any Python packages that depend on libtiff
echo "=== Checking Python package requirements ==="
fd -e txt -e toml . | xargs cat | rg -i "pillow|opencv|imageio|scikit-image" || echo "No common image processing libraries found in requirements"

Length of output: 173

🌐 Web query:

CVE-2025-9900 libtiff6

💡 Result:

Summary — CVE-2025-9900: a high-severity "write-what-where" memory-corruption bug in LibTIFF (reported 23 Sep 2025). A specially crafted TIFF with an abnormally large image-height can cause arbitrary writes, leading to denial-of-service or remote code execution when an application using libtiff parses the file. CVSSv3 score 8.8 (AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H). [1][2][3]

Affected: upstream LibTIFF (impacts distributions' libtiff packages, e.g. libtiff6 in many Linux distros); vendors (Ubuntu, Debian, Red Hat, SUSE) have published advisories. Check your distro’s security notice for exact affected package version(s). [3][4][5]

Mitigation: update libtiff/libtiff6 to the patched package from your distribution (install the vendor security update) or avoid processing untrusted TIFF files until patched. Vendor advisories/patches linked below. [3][4][5]

References

  • Ubuntu security advisory CVE-2025-9900 (publication & mitigation). [1]
  • NVD entry (description/NVD placeholder). [2]
  • OSV / Debian tracker import of vendor advisory. [3]
  • SUSE advisory summary. [4]
  • Aggregated vulnerability pages (CVE details / security trackers). [5]

🏁 Script executed:

#!/bin/bash
echo "=== Checking for PIL imports in Python code ==="
rg -nP --type=py 'import\s+PIL' -C2 || echo "No 'import PIL' statements found"
rg -nP --type=py 'from\s+PIL' -C2 || echo "No 'from PIL' statements found"
echo "=== Checking for Image.open usage in Python code ==="
rg -nP --type=py 'Image\s*\.\s*open' -C2 || echo "No 'Image.open' usage found"

Length of output: 192

Use apt-get for non-interactive removal and ensure idempotency.

Removal of libtiff6 to mitigate CVE-2025-9900 is safe—no PIL imports or Image.open usage found in the codebase. However, in Docker scripts:

  • Prefer apt-get over apt.
  • Make the remove command idempotent (|| true).
  • Auto-remove dependencies and clear apt lists to reduce image size.

At lines 375–377 in Dockerfile, apply:

-RUN apt remove -qy libtiff6
+RUN apt-get remove -qy libtiff6 || true && \
+    apt-get autoremove -qy && \
+    rm -rf /var/lib/apt/lists/*
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Remove libtiff just for v2.7 release due to CVE https://nvd.nist.gov/vuln/detail/CVE-2025-9900
# This has no effect on bionemo since we do not do any image processing.
RUN apt remove -qy libtiff6
# Remove libtiff just for v2.7 release due to CVE https://nvd.nist.gov/vuln/detail/CVE-2025-9900
# This has no effect on bionemo since we do not do any image processing.
RUN apt-get remove -qy libtiff6 || true && \
apt-get autoremove -qy && \
rm -rf /var/lib/apt/lists/*
🤖 Prompt for AI Agents 
In Dockerfile around lines 375 to 377, replace the non-idempotent apt removal
with an apt-get based, non-interactive, idempotent and cleanup sequence: use
apt-get remove (or apt-get purge) with -y, fall back to ignore errors (e.g., add
"|| true") so the build doesn't fail if package is absent, run apt-get
autoremove -y to remove orphaned dependencies, and finally clear apt lists (rm
-rf /var/lib/apt/lists/*) to reduce image size; ensure these steps are combined
in the same RUN layer to avoid extra image layers.

2 changes: 1 addition & 1 deletion VERSION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.7rc1
2.7
28 changes: 17 additions & 11 deletions docs/docs/main/about/releasenotes-fw.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,33 @@
# Release Notes

## BioNeMo Framework v2.7rc1
## BioNeMo Framework v2.7

### Updates & Improvements

- Adds a header to SCDL archives, providing improved provenance tracking and supporting future releases. Also adds tracking of the AnnData API coverage in SCDL tests.
This header stores metadata about the archive and its composite arrays, including a version, the array lengths and data types, and information about the RowFeatureIndexes. This adds the features necessary to fix https://github.com/NVIDIA/bionemo-framework/issues/999 as well as implement simple bit-packing of the rowptr, colptr, and data arrays. It also should make SCDL more secure, enable strict compatibility checking, and open the door to more performance improvements. https://github.com/NVIDIA/bionemo-framework/pull/1030
- Evo2 model improvements:

- Spike-no-more support for potentially improved training stability: https://github.com/NVIDIA/bionemo-framework/pull/1011
- Context, tensor and data parallelism support in the prediction endpoint as well as support for context lengths over 8192 https://github.com/NVIDIA/bionemo-framework/pull/1123. Fixes https://github.com/NVIDIA/bionemo-framework/issues/910 and https://github.com/NVIDIA/bionemo-framework/issues/1048.

- Improvements in tag masking in Evo2 loss https://github.com/NVIDIA/bionemo-framework/pull/1008
- LoRA fine-tuning by @gabenavarro: https://github.com/NVIDIA/bionemo-framework/pull/980. Note: internal CI coverage of LoRA convergence is still a work in progress; therefore, we cannot guarantee convergence.

- Flash decode support in inference https://github.com/NVIDIA/bionemo-framework/pull/1000
- Fix a 2x memory-usage issue during Evo2 generation: https://github.com/NVIDIA/NeMo/pull/14515

- Evo2 Lora feature by @gabenavarro https://github.com/NVIDIA/bionemo-framework/pull/980
- Add flash-decode support in inference: https://github.com/NVIDIA/bionemo-framework/pull/1000

- Added partial-conv benchmark for Evo 2 finetune (from checkpoint) and Evo 2 LoRA finetuning https://github.com/NVIDIA/bionemo-framework/pull/1028
- Update Rotary Embedding and sequence-length defaults to address incorrect checkpoint conversion: https://github.com/NVIDIA/NeMo/pull/14514

- Improvements to tag masking in the Evo2 loss: https://github.com/NVIDIA/bionemo-framework/pull/1008

- Support for [Spike-no-more](https://arxiv.org/abs/2312.16903) to improve training stability: https://github.com/NVIDIA/bionemo-framework/pull/1011

- Added a header to SCDL archives, providing improved provenance tracking and supporting future releases. It also adds tracking of AnnData API coverage in SCDL tests.
This header stores metadata about the archive and its composite arrays, including a version; the array lengths and data types; and information about the RowFeatureIndexes. This adds the features necessary to fix https://github.com/NVIDIA/bionemo-framework/issues/999 as well as to implement simple bit-packing of the rowptr, colptr, and data arrays. It should also make SCDL more secure, enable strict compatibility checking, and open the door to further performance improvements: https://github.com/NVIDIA/bionemo-framework/pull/1030

- `bionemo-geometric` has been deprecated and removed. The molecular-featurization tooling in this package has moved to [cuik-molmaker](https://github.com/NVIDIA-Digital-Bio/cuik-molmaker).

### Known Issues

- Users have reported issues in the predict script at sequence lengths over 8192 https://github.com/NVIDIA/bionemo-framework/issues/910 and https://github.com/NVIDIA/bionemo-framework/issues/1048 for example.
- PR in progress to update Rotary Embedding and sequence length defaults to support incorrect checkpoint conversion https://github.com/NVIDIA/NeMo/pull/14514
- PR in progress to fix 2x memory usage issue during generation https://github.com/NVIDIA/NeMo/pull/14515
- We have removed `libtiff` from the container due to a known vulnerability, [CVE-2025-9900](https://ubuntu.com/security/CVE-2025-9900). `libtiff` isn't directly used in any BioNeMo code; however, users might face issues with e.g. Pillow or other common image-manipulation libraries inside this container.

## BioNeMo Framework v2.6.3

Expand Down