Skip to content

Conversation

D1m7asis
Copy link

From #1073

This PR adds first-class support for AI/ML API as a new unified model provider to VLMEvalKit.
AI/ML API provides access to 300+ models, including GPT-4o, Claude 3.5, Gemini 1.5, DeepSeek, LLaMA3, Mistral, and many others — via a single OpenAI-compatible interface.


🔧 What was added

  • ✅ New model wrapper: vlmeval/api/aimlapi.py
  • ✅ Registration in vlmeval/api/__init__.py
  • ✅ Multiple preconfigured model entries in vlmeval/config.py via ai_ml_api_models
  • ✅ Extended api_models with support for AIMLAPI
  • ✅ English and Chinese documentation in docs/en/Quickstart.md and docs/zh-CN/Quickstart.md
  • ✅ Ready-to-use AIML_API_KEY environment variable support

🚀 Usage Example

AIML_API_KEY=your_key_here python run.py --data CCBench --model aimlapi_gpt_4o

✅ Checklist

  • Code follows repo structure
  • Docs (EN + ZH) updated
  • Tested on: MMBench, CCBench, MME
  • Uses OpenAI-compatible API structure

Let me know if you'd like to adjust or split this PR into parts.

@kennymckormick
Copy link
Member

Hi, @D1m7asis

Would you please share me a temporary account on the AI / ML API platform for testing (w. $10 credit, e.g.)? My email is [email protected].

@D1m7asis
Copy link
Author

@kennymckormick hi! Can i get a review here pls? Thanks!

@OctavianTheI
Copy link

Hi @kennymckormick - any chance we could get a review here?
We'll implement changes within a day if anything needs to be adjusted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants