Add TwelveLabs Pegasus video understanding agent (opt-in)#12
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Add TwelveLabs Pegasus video understanding agent (opt-in)#12mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
Add an opt-in PegasusVideoUnderstanding role that answers questions about videos using TwelveLabs Pegasus. Unlike the transcript-based Q&A agent, it understands video natively (visuals + audio), so it needs no local Whisper transcription and can reason about on-screen actions, scenes, and objects. Accepts a local file, a directory of videos, or a public URL (local files are uploaded as TwelveLabs assets). Wired through a new pegasus() helper in environment/config/llm.py and registered in registry.json. Activates only when twelvelabs_api_key is set in config.yml, so existing behavior is unchanged. Adds twelvelabs dependency, README docs, and tests (a live test gated on TWELVELABS_API_KEY plus no-network unit tests).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
This PR adds an opt-in TwelveLabs Pegasus video-understanding agent to VideoAgent's multi-agent toolkit.
What it adds
PegasusVideoUnderstandingrole (environment/roles/vid_qa/pegasus_understanding.py) that answers questions about videos using TwelveLabs Pegasus. It accepts a local video file, a directory of videos, or a public video URL (local files are uploaded as TwelveLabs assets).pegasus()/twelvelabs_client()helper inenvironment/config/llm.py, mirroring the existinggpt/claude/geminiprovider helpers.environment/config/registry.jsonso the framework auto-discovers it like every other role.twelvelabs_api_key,twelvelabs_base_url) inconfig.yml, thetwelvelabsdependency inpyproject.toml, README docs, and tests.Why it helps this project
The existing
VideoContentQAagent answers questions from a Whisper transcript — it only sees speech and needs a local GPU model. Pegasus understands the video natively (visuals and audio), so it can reason about on-screen actions, scenes, and objects with no local transcription step. It's a complementary video-understanding backend in the agentic loop.Opt-in / non-breaking
The agent only activates when
twelvelabs_api_keyis set inconfig.yml. ThetwelvelabsSDK is imported lazily, defaults are untouched, and the existing transcript-based flow is unchanged.How it was tested
execute()happy path (mocked Pegasus), and the missing-path error path.TWELVELABS_API_KEYthat confirms the SDK + credentials reach the backend via a Marengo text embedding (verified 512-dim).analyzerequest wiring verified against the live API (parameters accepted server-side); a full end-to-end Pegasus run on a long sample video is server-side and slow, so that path is wiring-verified.isort --profile blackrun on changed files; all files compile;pytest tests/test_pegasus_understanding.py→ 4 passed.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.