The Fastest Way to Benchmark Diffusion Models.
Built for AI researchers, labs, and developers. Automates prompts, seeds, and metrics so benchmarks that take weeks now run reproducibly in hours.
⭐ Star to Get Early-Supporter Perks ⭐
Product Vision: AI Research
DreamLayer AI is an open source platform for benchmarking and evaluating diffusion models. It currently supports image generation, with video and audio model benchmarking coming soon. It automates the full workflow from prompts and seeds to metrics and logging, making experiments reproducible by default.
No custom scripts, no manual logging, no wasted compute. A streamlined workflow for:
- AI researchers benchmarking models, datasets, and samplers
- Labs and teams running reproducible evaluations across multiple seeds and configs
- Developers integrating custom metrics and evaluation pipelines
Status: ✨ Now live: Beta V1
⭐ Star the repo for updates & to get early-supporter perks
Easiest way to run DreamLayer 😃 Best for non-technical users
- Download this repo
- Open the folder in Cursor (an AI-native code editor)
- Type
run itor press the "Run" button — then follow the guided steps
Cursor will:
- Walk you through each setup step
- Install Python and Node dependencies
- Create a virtual environment
- Start the backend and frontend
- Output a localhost:8080 link you can open in your browser
⏱️ Takes about 5-10 minutes. No terminal needed. Just click, run, and you’re in. 🚀
On macOS, PyTorch setup may take a few retries. Just keep pressing Run when prompted. Cursor will guide you through it.
linux:
./install_linux_dependencies.shmacOS:
./install_mac_dependencies.shWindows:
install_windows_dependencies.ps1linux:
./start_dream_layer.shmacOS:
./start_dream_layer.shWindows:
start_dream_layer.batinstall_dependencies_linux DLVENV_PATH // preferred path to python virtual env. default is /tmp/dlvenv
start_dream_layer DREAMLAYER_COMFYUI_CPU_MODE // if no nvidia drivers available run using CPU only. default is false
- Frontend: http://localhost:8080
- ComfyUI: http://localhost:8188
DreamLayer ships without weights to keep the download small. You have two ways to add models:
DreamLayer can also call external APIs (OpenAI DALL·E, Flux, Ideogram).
To enable them:
Edit your .env file at dream_layer/.env:
OPENAI_API_KEY=sk-...
BFL_API_KEY=flux-...
IDEOGRAM_API_KEY=id-...
STABILITY_API_KEY=sk-...Once a key is present, the model becomes visible in the dropdown. No key = feature stays hidden.
Step 1: Download .safetensors or .ckpt files from:
- Hugging Face
- Civitai
- Your own training runs
Step 2: Place the models in the appropriate folders (auto-created on first run):
- Checkpoints/ → # full checkpoints (.safetensors)
- Lora/ → # LoRA & LoCon files
- ControlNet/ → # ControlNet models
- VAE/ → # optional VAEs
Step 3: Click Settings ▸ Refresh Model List in the UI — the models appear in dropdowns.
Tip: Use symbolic links if your checkpoints live on another drive.
The installation scripts will automatically install all dependencies and set up the environment.
| 🔍 Feature | 🚀 How it’s better |
|---|---|
| Automated Benchmarking | Turn a 1–2 week manual benchmarking task into 3–5 hours per model |
| Reproducibility | Every run is logged with prompts, seeds, configs, and metrics for exact replay |
| Metrics Built In | CLIP Score, FID, Precision, Recall, F1 with support for custom metrics |
| Multi Modal Ready | Benchmark image, video, and audio models with one pipeline |
| Researcher Friendly | Runs locally or on your GPUs with CSV exports, reports, and leaderboard ready outputs |
- Python 3.8+
- Node.js 16+
- 8GB+ RAM recommended
Starring helps us trend on GitHub which brings more contributors and faster features.
Early stargazers get perks:
- GitHub Hall of Fame: Your handle listed forever in the README under Founding Supporter
- Early Builds: Download private binaries before everyone else
- Community first hiring: We prioritize contributors and stargazers for all freelance, full-time, and AI artist or engineering roles.
- Closed Beta Invites: Give feedback that shapes 1.0
- Discord badge: Exclusive Founding Supporter role
⭐ Hit the star button right now and join us at the ground floor
☺️
- Star this repository.
- Watch releases for the July code drop.
- Join the Discord (link coming soon) and say hi.
- Open issues for ideas or feedback & Submit PRs once the code is live
- Share the screenshot on X ⁄ Twitter with
#DreamLayerAIto spread the word.
All contributions code, docs, art, tutorials—are welcome!
- Create a PR and follow the evidence requirements in the template.
- See CHANGELOG Guidelines for detailed contribution process.
Full docs will ship with the first code release.
DreamLayer AI will ship under the GPL-3.0 license when the code is released.
All trademarks and closed-source models referenced belong to their respective owners.
### Made with ❤️ by builders, for builders • See you in July 2025!
DreamLayer AI includes a comprehensive test suite covering all functionality including ClipScore integration, database operations, and API endpoints.
# Install test dependencies
pip install -r tests/requirements.txt
# Run all tests
python tests/run_all_tests.py
# Run specific test categories
python tests/run_all_tests.py unit # Unit tests only
python tests/run_all_tests.py integration # Integration tests only
python tests/run_all_tests.py api # API endpoint tests
python tests/run_all_tests.py clipscore # ClipScore functionality tests
# Run with verbose output
python tests/run_all_tests.py all -v| Test File | Coverage | Description |
|---|---|---|
test_txt2img_server.py |
Text-to-Image API | Tests txt2img generation and database integration |
test_img2img_server.py |
Image-to-Image API | Tests img2img generation and database integration |
test_run_registry.py |
Run Registry API | Tests database-first API with ClipScore retrieval |
test_report_bundle.py |
Report Generation | Tests Mac-compatible report bundle creation |
test_clip_score.py |
ClipScore Integration | Tests CLIP model calculation and database storage |
test_database_integration.py |
Database Operations | Tests 3-table schema and database operations |
- ✅ Unit Tests - Individual component testing
- ✅ Integration Tests - End-to-end workflow testing
- ✅ API Tests - HTTP endpoint testing with Flask test client
- ✅ Database Tests - SQLite operations with temporary test databases
- ✅ Mock Testing - External dependency mocking (ComfyUI, CLIP model)
- ✅ Error Handling - Edge cases and error condition testing
- ✅ Mac Compatibility - ZIP file generation testing
# Run specific test file
python -m pytest tests/test_clip_score.py -v
# Run specific test method
python -m pytest tests/test_clip_score.py::TestClipScore::test_clip_score_calculation_with_mock -v
# Run with coverage report
python -m pytest tests/ --cov=dream_layer_backend --cov-report=htmlThe test suite requires these additional dependencies:
pytest- Test frameworkpytest-cov- Coverage reportingpytest-mock- Mocking utilitiesrequests-mock- HTTP request mocking
Install with: pip install -r tests/requirements.txt