Caption system overhaul V2

[CalamitousFelicitousness]

Description

War & Peace in the description but at this point might as well go full hog.

Overhaul the caption/interrogation subsystem: rename modules/interrogate/ to modules/caption/, extract the caption API into a self-contained module (modules/api/caption.py), and improve VLM model support with better offloading, inference fixes, and fixes to the model-aware prompt filtering (for the preset tasks). Full support for WaifuDiffusion and DeepBooru (DeepDanbooru) taggers was added as well.

The goal is to make captioning a subsystem with a clean API layer, consistent naming, and robust handling, up to date with our code conventions (as much as I can manage anyway). Also got the docs for the API available under /docs enpoint, and fixed an array of breaking bugs, less breaking bugs, and things that just managed to miff me.

Notes

Module rename: modules/interrogate/ → modules/caption/

All caption-related modules are renamed from interrogate to caption. This is a breaking change for any external code importing from modules.interrogate. All internal imports, settings keys, and UI references are updated:

• Settings keys: interrogate_* → caption_* (e.g. interrogate_vlm_model → caption_vlm_model, interrogate_offload → caption_offload)
• Function names: vqa.interrogate() → vqa.caption()
• Existing user config.json files with old interrogate_* keys will fall back to defaults - as discussed with @vladmandic, no migration to save on code for a non-critical function.
• Breaking also has a benefit (or a curse) of being a smoketest for any weirdness in the old-growth code that somehow still interacted with interrogate. There was an instance or two of things injecting itself halfway into the function for some reason.

Caption API extraction (modules/api/caption.py)

The caption API endpoints were previously scattered across modules/api/endpoints.py and modules/api/models.py. They are now consolidated into a single self-contained module with:

• Direct endpoints: POST /sdapi/v1/openclip, POST /sdapi/v1/tagger, POST /sdapi/v1/vqa; each with backend-specific Pydantic request/response models
• Unified dispatch: POST /sdapi/v1/caption; routes to any backend via a backend discriminator field
• Discovery endpoints: GET /sdapi/v1/openclip, GET /sdapi/v1/vqa/models, GET /sdapi/v1/vqa/prompts, GET /sdapi/v1/tagger/models
• Shared internal _do_openclip, _do_tagger, _do_vqa functions avoid duplication between direct and dispatch handlers
• Old endpoint/model definitions removed from endpoints.py and models.py

VLM improvements

• Model-aware prompt filtering fixed up and expanded: get_prompts_for_model() returns only the prompts supported by the selected model (e.g. Florence gets detection prompts, PromptGen gets analysis prompts and tagging prompts which previously were available to entire Florence2 family, Moondream gets point/detect/gaze prompts)
• Florence-2 detection parsing: Object detection, phrase grounding, OCR, and region proposal results are parsed into structured bounding-box data with optional annotated image overlay (via vqa_detection.py)
• SDPA bypass: New devices.bypass_sdpa_hijacks() context manager to temporarily restore original SDPA for models incompatible with SageAttention or other attention hijacks
• Offload support: Caption models respect caption_offload setting, moving models to CPU when not in use, many of them did not or used deprecated methods
• Moondream fixes: Fixed infinite recursion in Moondream3, disabled flex_attention to avoid torch.compile hang, added cache_dir for downloads - I went a bit far with implementing the functionality, but it's now feature complete, due to caching support you can run multiple tasks on the same image (multiple detections, multiple different tasks) in miliseconds.
• BLIP model loading: New _load_blip_model() helper pre-loads BLIP models with explicit cache_dir and sets them on the Interrogator config directly, avoiding redundant downloads to the default HF cache
• DeepBooru endpoint migration: DeepBooru is now exclusively accessed via the /sdapi/v1/tagger endpoint (or the despatcher), removed from the openclip/interrogate endpoint for full consistency

UI changes

• Default caption task changed from "Short Caption" to "Normal Caption"
• Task dropdown dynamically updates available prompts when VLM model selection changes - this has become wonky since the first time it was introduced, but it got retightened
• Placeholder hints update per-task to guide user input - not new, but again, revisited and confirmed it all works well
• Tooltips and hints added to Caption tab controls
• Localization strings updated in locale_en.json - added hints where still missing, and moved them into their own subsection for convenvience.
• Removed settings for Interrogate/Caption from Settings menu entirely, one function (offloading) got moved to the offloading section. Everything user ever needs to set is right there in the snazzy UI.

Other

• CLI tools updated: cli/api-interrogate.py renamed to cli/api-caption.py with updated endpoints
• Test suite: cli/test-caption-api.py (3400 lines); comprehensive test coverage for ALL caption API endpoints, parameter validation, and edge cases, error handling, deliberate induction of errors. Depending on how many models are available, but that is 150+ tests, all live, veryfing outputs.
• modules/caption/caption.py: Thin facade module providing a unified entry point for the caption subsystem, saves on pointing to different internal functions directly

Things to watch

• The interrogate_* → caption_* settings rename means existing user configs will lose their caption customizations on upgrade (they'll get defaults). No issues noted, new settings in Caption tab now save to config by default.
• The modules/interrogate/ directory still exists after the rename (the old interrogate.py is deleted but the directory may linger if other files are present); verify clean state
• cli/test-caption-api.py is large (3400 lines); it's a standalone test script, not a pytest suite

Environment and Testing

• OS: Linux (WSL2, Ubuntu), Fedora, Windows 11, Android for mobile UI testing
• GPU: NVIDIA RTX 3090 (CUDA 12.9), RTX 6000 Ada
• Python: 3.12, 3.13
• Tested: All three caption backends (OpenCLIP/BLIP, WaifuDiffusion/DeepBooru tagger, VLM) via both UI and API endpoints
• Linting: ruff check and pylint on all modified files - I promise

[vladmandic]

lgtm