ADR: Integrate Advanced Image Description Models into NVDA Core

[tianzeshi-study]

ADR status

• Proposed
• Accepted
• Rejected
• Superseded

Impact area

NVDA core image description capability, AI model integration architecture, accessibility features.

Related issues, PRs, discussions, or mailing list threads

#19807
#19451
#19425
#18475

Context and problem statement

An earlier attempt was made to integrate image description functionality directly into NVDA core. However, the related PR was withdrawn, and the feature was temporarily moved to an add-on implementation to allow faster iteration and experimentation.

During subsequent add-on development, the image description architecture was improved. The add-on introduced:

• A more flexible model management system.
• Support for multiple image description models.
• Integration with newer vision-language models, including experimental support for Qwen 3.5 ONNX models.
• A standalone inference runtime based on miniqinference[cli].

The Qwen 3.5 0.8B ONNX model has shown significantly improved description quality compared with previous models. It can generate useful descriptions in approximately one minute and supports multiple languages through localized prompt templates (currently Chinese and English).

The current decision is whether and how to bring this capability back into NVDA core.

Whether the core implementation should only replace the existing default model, or whether it should adopt the more flexible multi-model architecture developed in the add-on.

UX impact via user stories

As a user who relies on NVDA image descriptions, I want higher quality descriptions, so that I can better understand visual content.

As a user who uses different languages, I want image descriptions to respect my language preferences, so that generated descriptions are easier to understand.

As a user with different hardware capabilities, I want appropriate model choices, so that I can balance description quality and performance.

As an NVDA add-on developer, I want a stable image description API, so that additional models or integrations can be developed without modifying core functionality.

Decision drivers

• Improve image description quality.
• Avoid repeating previous limitations of the core-only implementation.
• Balance implementation complexity and long-term maintainability.
• Decide whether multi-model support belongs in NVDA core.
• Maintain compatibility with existing add-ons.
• Minimize disruption to existing user workflows.

Options considered

Option A - Replace only the current default image description model.

The core implementation remains mostly unchanged. The existing model download workflow is updated to use a newer model.

Advantages:

• Smaller code change.
• Lower maintenance cost.
• Faster integration.

Disadvantages:

• Does not support multiple models.
• Future model replacement may require additional core changes.

Option B - Introduce the add-on's model management architecture into NVDA core.

The core provides a model manager and supports multiple image description models.

Advantages:

• Better long-term extensibility.
• Allows users to select models based on quality, speed, and hardware.
• Matches the architecture already validated in the add-on.

Disadvantages:

• Larger scope.
• Requires additional design work for model lifecycle management and UI.

Option C (status quo) - Keep image description functionality in the add-on only.

Advantages:

• No core changes required.
• Allows continued experimentation.

Disadvantages:

• Core users cannot benefit from integrated functionality.
• Duplicate implementations may continue.

Proposed technical design

The proposed design should separate the stable image description capability from individual model implementations.

Core components:

• Image description API:

  • Provides a stable interface for requesting image descriptions.
  • Does not expose model-specific details.

• Model management layer:

  • Manages available models.
  • Handles model metadata, download status, and selection.

• Model providers:

  • Implement specific vision-language models.
  • Allow future models to be added independently.

• Inference runtime:

  • Uses a standalone inference binary.
  • The add-on currently uses miniqinference[cli] for model loading and inference.
  • The core integration should evaluate whether this runtime approach is suitable.

The exact scope of model management integration should be determined separately from the initial image description API integration.

Proposed decision and rationale

Recommended approach: split the work into separate PRs.

PR 1:

• Reintroduce the core image description API and basic model integration.
• Ensure the architecture supports future model providers.

PR 2:

• Introduce multi-model management capabilities if the design is accepted.
• Migrate functionality from the add-on model manager where appropriate.

This approach reduces risk by separating the fundamental capability from the larger model management redesign.

A direct default model replacement is simpler but may repeat the limitations of the previous implementation. A complete model manager integration provides better long-term flexibility but requires broader architectural discussion.

Impact analysis

Positive impacts:

• Higher quality image descriptions.
• Better foundation for future AI model improvements.
• Reduced dependency on add-on-only implementations.
• Potential support for multiple models.

Neutral impacts:

• Existing image description users should continue to work normally.
• Existing add-ons should not require immediate changes.

Negative impacts:

• Introducing model management increases complexity.
• Additional UI and lifecycle decisions are required.
• Larger changes may require longer review cycles.

Performance:

• Qwen 3.5 0.8B ONNX currently requires no more than one minute for inference in experimental testing.
• Performance improvements and hardware acceleration remain future work.

Hardware acceleration:

• DirectML acceleration has been investigated but is not currently included.
• Some model operators appear incompatible with DirectML.

Architecture and code change plan

Planned architecture changes:

Phase 1:

• Restore image description capability in core.
• Define stable interfaces.
• Integrate improved default model support.

Phase 2:

• Evaluate migration of add-on model manager concepts.
• Add multi-model support if approved.

Target modules/files:

_localCaptioner/*

API compatibility strategy:

• Preserve existing image description API concepts.
• Avoid exposing specific model implementations.
• Allow future model providers without breaking API changes.

Integration plan:

1. Review add-on implementation and identify reusable architecture.
2. Define core API boundaries.
3. Decide PR split between basic integration and model management.
4. Implement core image description support.
5. Evaluate multi-model support separately.

Rollback strategy:

• Keep model-specific changes isolated.
• Revert model integration without affecting unrelated accessibility features.

Risks and mitigations

Risk: Core scope becomes too large due to introducing full model management.

Mitigation:

• Separate API/model integration from model management.

Risk: Runtime dependency becomes difficult to maintain.

Mitigation:

• Keep inference runtime isolated and versioned.

Risk: Hardware requirements are too high.

Mitigation:

• Support multiple model sizes and document requirements.

Validation and success criteria

Technical validation:

• Core image description API works independently of model implementation.
• Qwen 3.5 model integration produces valid descriptions.
• Existing workflows remain functional.

UX validation:

• Description quality improves compared with previous models.
• Users can complete image description tasks reliably.

Success criteria:

• Image description functionality can be maintained in core.
• Future model upgrades do not require architectural rewrites.
• The chosen scope is maintainable for NVDA core development.

Open questions

• Should NVDA core initially support only one default model or introduce multi-model support?
• Should model management be part of core or remain add-on responsibility?
• Should the inference runtime remain external or be integrated differently?
• What model download and update experience should core provide?
• Should DirectML acceleration be revisited separately?

[hwf1324]

Just for reference in the discussion, I recently noticed the MNN inference engine, which seems to be friendly to low-performance devices and has minimal dependencies. It also supports various model formats such as ONNX.

The small OpenCV-like image processing module it provides might be helpful when integrating PaddleOCR in the future.

[seanbudd]

Should NVDA core initially support only one default model or introduce multi-model support?

Let's introduce multi-model support, but start with just a single model as you suggested.
I am a bit concerned that 2 PRs won't split up this work enough. How big is the add-on roughly in Lines of Code?

Should model management be part of core or remain add-on responsibility?

Part of core

Should the inference runtime remain external or be integrated differently?

What is the size of the inference runtime? How will it be packaged with NVDA?
Does it take time to initialize? When initializing or after initialization, does it slow down using NVDA?

What model download and update experience should core provide?

I think we should have an interface similar to the add-on store, but a bit more simplified with filtering. This is somewhat covered in #19337

Should DirectML acceleration be revisited separately?

That's GPU/NPU acceleration right? We can consider that in a follow up