DynEval: Holistic Evaluation of Text-to-Image Generative Models in the Wild (ECCV-26)

Shyam Marjit, Dheeraj Baiju, Anuj Shikarkhane, Akhil Sakthieswaran, Sayak Paul, and Anirban Chakraborty

Model Checkpoints: DynEval-2B & DynEval-4B · Dataset: DynEval-1K, GenDB, DynEvalInstruct (with teacher model responses)

Main Contributions

(i) We construct two large-scale datasets, GenDB and DynEvalInstruct with well-balanced prompt coverage and image generations from 36 diverse T2I models for evaluator training. We construct GenDB, a large-scale prompt–image dataset with well-balanced prompt coverage and generations from 36 diverse T2I models, and derive DynEvalInstruct from GenDB for evaluator training.

(ii) Unlike static QA methods, we propose DynEval, a dynamic evaluator that jointly evaluates prompt-generated image alignment as well as builds scene graphs from generated images to compose structured, image-specific questions for fine-grained image quality assessment.

(iii) To obtain a robust evaluator, we introduce tier-based prompt categorization with tier-specific T2I model generation to cover T2I models failure modes across varying prompt complexities and model capabilities To train a robust evaluator, we introduce tier-matched prompt–model generation, pairing prompts of varying complexity with T2I models of corresponding capability to capture informative failure modes across the model spectrum.

(iv) Across multiple established benchmarks, DynEval achieves superior correlation with human judgments than prior works. Our extensive analysis over 36 T2I models reveals multiple sub-categories to identify attributes that remain challenging for current SOTA T2I models, offering insights for improving next-generation models.

Overview of GenDB and DynEvalInstruct construction

Overview of DynEvalInstruct construction and the DynEval evaluation framework.

Installation

Clone the repository and install dependencies using the provided requirements.txt. A CUDA-capable GPU is required.

The dependencies follow the same requirements as the official Qwen3-VL installation.

git clone https://github.com/vcl-iisc/dyneval_code
pip install -r requirements.txt

Dataset Construction and Training

Step 1 — Filter Diverse Prompts

Place the DiffusionDB prompt file at data/diffusiondb-prompts.txt, or provide its location with --input, then run:

python extract_diverse_prompts.py --input path/to/diffusiondb-prompts.txt

Keep jacordian-overlap-removel.py and score_prompts_qwen3_6_27b_one_dimension.py in the same directory as the pipeline script.

The script:

removes exact duplicates, prompts shorter than 30 characters, and near-duplicates;
scores prompt complexity with Qwen/Qwen3.6-27B using nine factors;
sorts prompts by score and builds a corpus of up to 500,000 prompts;
assigns prompts to Tier 1 (score >= 200), Tier 2 (100 <= score < 200), or Tier 3 (score < 100);
assigns semantic categories within each tier.

Outputs are written under diverse-prompt-results/ in the preprocess, scores, tiers, and categories directories. Thresholds, corpus size, model settings, processing limits, resume behavior, and individual stages can be controlled through command-line arguments; run python extract_diverse_prompts.py --help for details.

The resulting prompt metadata is also available in the Hugging Face dataset.

Step 2 — Generate Images

Using the prompt-to-tier assignments from Step 1, generate images by matching prompt tiers to model tiers (i.e., Tier 1 prompts are assigned to Tier 1 models only). Generation scripts for each model are available in the image_gen_scripts folder.

The output of this step is the GenDB dataset, containing ⟨image, text prompt⟩ pairs.

All used (prompt, image) pairs are present in the Hugging Face dataset.

Step 3 — Distill Annotations with a Teacher VLM

Pass each ⟨image, text prompt⟩ pair to the teacher VLM (Qwen/Qwen3-VL-235B-A22B-Instruct) to generate T2IA (Text-to-Image Alignment) and IQA (Image Quality Assessment) annotations.

Use the scripts in Distill Annotations/:

Run the T2IA and IQA teacher workflows to generate questions and answers.
Run build_dynevalinstruct.py to convert those outputs into DynEvalInstruct JSON for fine-tuning.

The teacher model uses natural-language prompts only. The student DynEval model is trained with task tokens <\|T2IA\|>, <\|IQA\|>, and <\|EVALUATION\|> (IDs 151669–151671), which are inserted during the build step—not sent to the teacher VLM.

Step 4 — Training

Train DynEval-2B or DynEval-4B for T2IA and IQA with:

training/train_t2ia_qwen3vl.py
training/train_iqa_qwen3vl.py

The trainer expects a prepared SFT data directory:

data/sft/<dataset_name>/train.jsonl
data/sft/<dataset_name>/val.jsonl

Pass this directory with:

--data-dir data/sft/<dataset_name>

4a — Task Tokens

The tokenizer is initialized with three task tokens:

Token	Used for
`<T2IA>`	Text-to-image alignment elements and questions
`<IQA>`	Image-quality assessment questions
`<EVALUATION>`	Image-based scoring from 1 to 5

4b — SFT Annotation Format

Training data is stored as JSONL. Each line is one sample. The messages field is what the trainer reads. The task token appears in the user message, and the assistant message is the target output.

<T2IA> element extraction

This task is prompt-only. It trains the model to output the important image-generation elements as a JSON list.

{
  "id": "sample_id_elements",
  "prompt_id": "sample_id",
  "prompt": "a photo of a carrot",
  "elements": ["carrot (food)"],
  "task": "T2IA_element_extraction",
  "messages": [
    {"role": "system", "content": [{"type": "text", "text": "Given an aigc prompt, extract the elements that are important for generating images."}]},
    {"role": "user", "content": [{"type": "text", "text": "<T2IA>\nPrompt: a photo of a carrot\nElements:"}]},
    {"role": "assistant", "content": [{"type": "text", "text": "[\"carrot (food)\"]"}]}
  ]
}

<T2IA> single-question generation

This task is also prompt-only. It trains the model to generate one yes/no verification question for one extracted element.

{
  "id": "sample_id_single_question_000",
  "prompt_id": "sample_id",
  "prompt": "a photo of a carrot",
  "element": "carrot (food)",
  "question": "Is there a carrot in the photo?",
  "task": "T2IA_single_question_generation",
  "messages": [
    {"role": "system", "content": [{"type": "text", "text": "Given a prompt for image generation and one of its related elements, generate one easy Yes/No question to verify whether the element is represented in the image generated by the prompt."}]},
    {"role": "user", "content": [{"type": "text", "text": "<T2IA>\nDescription: a photo of a carrot\nElement: carrot (food)\nReturn JSON:\n{\"question\": \"...\", \"answer\": \"yes/no\"}"}]},
    {"role": "assistant", "content": [{"type": "text", "text": "{\"question\": \"Is there a carrot in the photo?\", \"answer\": \"yes\"}"}]}
  ]
}

<IQA> three-step image-quality assessment

The <IQA> task is trained by a separate script, training/train_iqa_qwen3vl.py, because it uses a three-step, image-conditioned pipeline (see Inference). Each image produces up to three rows, all prefixed with the <IQA> token, and each assistant target matches exactly what the inference script asks for at that step.

Scene graph (task: "IQA_scene_graph") — image + prompt → {"nodes": [...], "edges": [...]}.

{
  "id": "sample_id_iqa_scene_graph",
  "task": "IQA_scene_graph",
  "image_path": "/path/to/image.png",
  "messages": [
    {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "<IQA>\nHere is an image generated for this prompt \"a photo of a bench\". ..."}]},
    {"role": "assistant", "content": [{"type": "text", "text": "{\"nodes\": [{\"id\": \"object_1\", \"label\": \"bench\", \"attributes\": [\"wooden\"]}], \"edges\": [{\"source\": \"object_1\", \"relation\": \"on\", \"target\": \"object_2\"}]}"}]}
  ]
}

Node-grounded questions (task: "IQA_question_generation") — image + prompt + scene graph JSON → {"questions": [{node_id, question, target_answer}]}.
Per-question scoring (task: "IQA_evaluation") — image + prompt + scene graph + question JSON → {"questions": [{question, target_answer, answer, score, reasoning}]}, where each score is 1–5.

<EVALUATION> image scoring

This task uses the image. It trains the model to score each generated question from 1 to 5.

{
  "id": "sample_id_model_evaluation",
  "prompt_id": "sample_id",
  "prompt": "a photo of a carrot",
  "image_ref": "model_name/image.png",
  "image_path": "/path/to/image.png",
  "questions": ["Is there a carrot in the photo?"],
  "answers": [{"question": "Is there a carrot in the photo?", "score": 5}],
  "task": "EVALUATION_scoring",
  "messages": [
    {"role": "system", "content": [{"type": "text", "text": "You are a strict visual evidence evaluator."}]},
    {
      "role": "user",
      "content": [
        {"type": "image"},
        {"type": "text", "text": "<EVALUATION>\nQuestions to score:\n1. Question: Is there a carrot in the photo?"}
      ]
    },
    {"role": "assistant", "content": [{"type": "text", "text": "[{\"question\": \"Is there a carrot in the photo?\", \"score\": 5}]"}]}
  ]
}

Do not put task tokens in assistant responses.

4c — Build SFT Data

T2IA and EVALUATION. Use data/build_t2ia_sft.py to convert teacher-generated question/answer files into SFT JSONL for the <T2IA> and <EVALUATION> tasks.

Expected inputs:

--questions-dir: JSON files containing prompt-level elements and T2IA questions
--answers-dir: JSON files containing image paths and question scores
--images-root: root folder used to resolve image paths

Example:

python data/build_t2ia_sft.py \
  --questions-dir /path/to/questions \
  --answers-dir /path/to/answers \
  --images-root /path/to/images \
  --output-dir data/sft/my_dyneval_sft_data \
  --val-ratio 0.05 \
  --seed 42

The output directory will contain:

data/sft/my_dyneval_sft_data/train.jsonl
data/sft/my_dyneval_sft_data/val.jsonl
data/sft/my_dyneval_sft_data/manifest.json

This produces rows for:

<T2IA> element extraction
<T2IA> single-question generation
<EVALUATION> image scoring

IQA. The <IQA> task uses a separate builder, data/build_iqa_sft.py, which reads the teacher IQA outputs (from Distill Annotations/IQA/):

--scene-graph-dir: teacher scene-graph/question JSON files (iqa_outputs/)
--answers-dir: teacher answer JSON files (iqa_answers/); optional, enables the scoring rows
--images-root: root folder used to resolve image paths

python data/build_iqa_sft.py \
  --scene-graph-dir /path/to/iqa_outputs \
  --answers-dir /path/to/iqa_answers \
  --images-root /path/to/images \
  --output-dir data/sft/my_iqa_sft_data \
  --val-ratio 0.05 \
  --seed 42

This produces rows for:

<IQA> scene graph generation
<IQA> node-grounded question generation
<IQA> per-question scoring

The same builder is also available inline via training/train_iqa_qwen3vl.py --prepare-only.

4d — Run Training

Use --finetune-mode full for full-parameter training. train_t2ia_qwen3vl.py trains the <T2IA> and <EVALUATION> tasks; the <IQA> task is trained separately in 4f.

DynEval-2B:

CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 training/train_t2ia_qwen3vl.py \
  --train-only \
  --finetune-mode full \
  --model-path /path/to/qwen3vl-2b-checkpoint \
  --data-dir data/sft/my_dyneval_sft_data \
  --output-dir checkpoints/final/qwen3vl-2b-dyneval-new-v1 \
  --device-map none \
  --gradient-checkpointing \
  --ddp-find-unused-parameters \
  --per-device-train-batch-size 1 \
  --gradient-accumulation-steps 8 \
  --epochs 1 \
  --lr 1e-7 \
  --lr-scheduler-type cosine \
  --warmup-ratio 0.03 \
  --optim adafactor \
  --eval-steps 500 \
  --save-steps 1000000 \
  --logging-steps 20 \
  --dataloader-num-workers 4 \
  --max-length 4096 \
  --report-to none

DynEval-4B:

CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 training/train_t2ia_qwen3vl.py \
  --train-only \
  --finetune-mode full \
  --model-path /path/to/qwen3vl-4b-checkpoint \
  --data-dir data/sft/my_dyneval_sft_data \
  --output-dir checkpoints/final/qwen3vl-4b-dyneval-new-v1 \
  --device-map none \
  --gradient-checkpointing \
  --ddp-find-unused-parameters \
  --per-device-train-batch-size 1 \
  --gradient-accumulation-steps 8 \
  --epochs 1 \
  --lr 1e-7 \
  --lr-scheduler-type cosine \
  --warmup-ratio 0.03 \
  --optim adafactor \
  --eval-steps 500 \
  --save-steps 1000000 \
  --logging-steps 20 \
  --dataloader-num-workers 4 \
  --max-length 4096 \
  --report-to none

4f — Train the IQA Task

The <IQA> task is trained with training/train_iqa_qwen3vl.py. It uses the same trainer machinery and task tokens, but its data covers the three image-conditioned IQA steps (scene graph, node-grounded questions, per-question scoring).

Start from a checkpoint that already has <T2IA> and <EVALUATION> trained, so the resulting model supports all three tasks.

DynEval-2B:

CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 training/train_iqa_qwen3vl.py \
  --train-only \
  --finetune-mode full \
  --model-path checkpoints/final/qwen3vl-2b-dyneval-new-v1 \
  --data-dir data/sft/my_iqa_sft_data \
  --output-dir checkpoints/final/qwen3vl-2b-dyneval-iqa-v1 \
  --device-map none \
  --gradient-checkpointing \
  --ddp-find-unused-parameters \
  --per-device-train-batch-size 1 \
  --gradient-accumulation-steps 8 \
  --epochs 1 \
  --lr 1e-7 \
  --lr-scheduler-type cosine \
  --warmup-ratio 0.03 \
  --optim adafactor \
  --eval-steps 500 \
  --save-steps 1000000 \
  --logging-steps 20 \
  --dataloader-num-workers 4 \
  --max-length 4096 \
  --report-to none

DynEval-4B:

CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 training/train_iqa_qwen3vl.py \
  --train-only \
  --finetune-mode full \
  --model-path checkpoints/final/qwen3vl-4b-dyneval-new-v1 \
  --data-dir data/sft/my_iqa_sft_data \
  --output-dir checkpoints/final/qwen3vl-4b-dyneval-iqa-v1 \
  --device-map none \
  --gradient-checkpointing \
  --ddp-find-unused-parameters \
  --per-device-train-batch-size 1 \
  --gradient-accumulation-steps 8 \
  --epochs 1 \
  --lr 1e-7 \
  --lr-scheduler-type cosine \
  --warmup-ratio 0.03 \
  --optim adafactor \
  --eval-steps 500 \
  --save-steps 1000000 \
  --logging-steps 20 \
  --dataloader-num-workers 4 \
  --max-length 4096 \
  --report-to none

Per-question 1–5 scores are derived from the teacher's judgement with --score-mode: correct (default) maps correct/incorrect answers to 5/1, while overall uses the teacher's single overall score for every question.

Inference

Use inference/run-inference.py for single-image inference with DynEval-2B or DynEval-4B from vcl-iisc/DynEval-Evaluator.

By default, the script loads DynEval-4B from Hugging Face and computes both scores:

T2IA score: text-to-image alignment score on a 1–5 scale.
IQA score: image-quality assessment score on a 1–5 scale.

The script can also compute only one score with --score-type t2ia or --score-type iqa.

T2IA pipeline (3 steps)

Element extraction — the model reads the text prompt and outputs a JSON list of important image-generation elements (objects, activities, attributes, etc.).
Question generation — for each element, the model generates one yes/no verification question with a target answer.
Visual scoring — the model looks at the image and scores each question from 1 (definitely no) to 5 (definitely yes). The T2IA score is the mean over all questions.

IQA pipeline (3 steps)

Scene graph generation — the model looks at the image (using the text prompt as a reference) and outputs a scene graph: a list of visible nodes (objects with attributes) and edges (spatial/relational links between them).
Node-grounded question generation — using the scene graph nodes and edges, the model generates yes/no quality questions per node, targeting shape, texture, distortions, 3D spatial consistency, and other perceptual objectives.
Visual scoring — the model answers and scores each question against the image from 1 to 5. The IQA score is the mean over all questions.

Run DynEval-4B from Hugging Face

DynEval-4B is the default, so --model-size 4b is optional.

CUDA_VISIBLE_DEVICES=0 python inference/run-inference.py \
  --prompt "a photo of a carrot" \
  --image example.jpg \
  --output-file output_4b.json

Run DynEval-2B from Hugging Face

CUDA_VISIBLE_DEVICES=0 python inference/run-inference.py \
  --model-size 2b \
  --prompt "a photo of a carrot" \
  --image example.jpg \
  --output-file output_2b.json

Choose Score Type

Compute both scores (default):

--score-type both

Compute only text-to-image alignment:

--score-type t2ia

Compute only image quality assessment:

--score-type iqa

Example:

CUDA_VISIBLE_DEVICES=0 python inference/run-inference.py \
  --model-size 4b \
  --score-type iqa \
  --prompt "a photo of a bench" \
  --image example.jpg \
  --output-file output_iqa.json

When --output-file is provided, the script also saves the structured JSON result.

Command-Line Arguments

--model-size {2b,4b} or --variant {2b,4b}: Hugging Face model variant; defaults to 4b.
--score-type {t2ia,iqa,both}: score to compute; defaults to both.
--repo-id REPO_ID: Hugging Face repository; defaults to vcl-iisc/DynEval-Evaluator.
--checkpoint PATH: optional local checkpoint path; overrides Hugging Face loading.
--prompt TEXT: text-to-image prompt corresponding to the image (required).
--image PATH: image to evaluate (required).
--output-file PATH: optional path at which to save the JSON result.
--dtype {bfloat16,float16,float32,auto}: model precision; defaults to bfloat16.
--device-map DEVICE_MAP: model device placement; defaults to auto.
--max-new-tokens-elements N: T2IA element-extraction generation limit; defaults to 256.
--max-new-tokens-questions N: T2IA per-element question-generation limit; defaults to 256.
--max-new-tokens-answers N: T2IA visual scoring generation limit; defaults to 768.
--max-new-tokens-iqa-scene-graph N: IQA scene-graph generation limit; defaults to 512.
--max-new-tokens-iqa-questions N: IQA node-grounded question-generation limit; defaults to 768. (also accepted as --max-new-tokens-iqa-decomposition)
--max-new-tokens-iqa-final N: IQA visual scoring generation limit; defaults to 512.
--hide-elements: hide extracted T2IA elements from terminal and saved JSON.
--include-raw: include raw model responses in the saved JSON. Useful for debugging when scene graph nodes or questions appear empty.

Quantitative Results

Zero-shot cross-dataset evaluation across diverse benchmarks, comparing existing scoring methods with EvalMuse and DynEval variants.

More recent Zero-shot cross-dataset evaluation across diverse benchmarks with newer T2I evaluators.

Qualitative Results

Evaluation on the GenEval dataset. Inputs consist of image–text prompt pairs from a mix of real and generated images, shown alongside human ratings, the mean human rating, and the DynEval score (scale: 1–5). Although DynEval is trained on synthetic images, the fine-tuned model demonstrates the ability to generalize to real images.

Evaluation on the AGIKA-3K dataset. Inputs consist of image–text prompt pairs shown alongside human ratings, the mean human rating, and the DynEval score (scale: 1–5).

Evaluation on the GenAI-Bench dataset. Inputs consist of image–text prompt pairs shown alongside human ratings, the mean human rating, and the DynEval score (scale: 1–5).

Alignment scores across prompt sub-categories in DynEval-1K evaluation dataset, grouped by model tier. The 42 sub-categories span nine semantic dimensions, and scores represent the average DynEval alignment score. Models are grouped into three tiers based on overall alignment performance, with bars showing the tier-averaged score for each sub-category. Tier-1 models consistently achieve stronger alignment across most sub-categories, with the largest performance gaps appearing in challenging categories such as counting, text rendering, and high-complexity prompts.

Citation

@misc{marjit2026dynevalholisticevaluationst2i,
      title={DynEval: Holistic Evaluations of T2I Generative Models in the Wild},
      author={Shyam Marjit and Dheeraj Baiju and Anuj Shikarkhane and Akhil Sakthieswaran and Sayak Paul and Anirban Chakraborty},
      year={2026},
      eprint={2607.11199},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.11199},
}

Name		Name	Last commit message	Last commit date
Distill Annotations		Distill Annotations
assets		assets
data		data
evaluation_methods		evaluation_methods
image_generation_scripts		image_generation_scripts
training		training
.gitattributes		.gitattributes
.gitignore		.gitignore
README.md		README.md
extract_diverse_prompts.py		extract_diverse_prompts.py
requirements.txt		requirements.txt
run_inference.py		run_inference.py

Navigation Menu

Uh oh!

Repository files navigation

DynEval: Holistic Evaluation of Text-to-Image Generative Models in the Wild (ECCV-26)

Table of Contents

Main Contributions

Installation

Dataset Construction and Training

Step 1 — Filter Diverse Prompts

Step 2 — Generate Images

All used (prompt, image) pairs are present in the Hugging Face dataset.

Step 3 — Distill Annotations with a Teacher VLM

Step 4 — Training

4a — Task Tokens

4b — SFT Annotation Format

4c — Build SFT Data

4d — Run Training

4f — Train the IQA Task

Inference

T2IA pipeline (3 steps)

IQA pipeline (3 steps)

Run DynEval-4B from Hugging Face

Run DynEval-2B from Hugging Face

Choose Score Type

Command-Line Arguments

Quantitative Results

Qualitative Results

Citation

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

DynEval: Holistic Evaluation of Text-to-Image Generative Models in the Wild (ECCV-26)

Table of Contents

Main Contributions

Installation

Dataset Construction and Training

Step 1 — Filter Diverse Prompts

Step 2 — Generate Images

All used (prompt, image) pairs are present in the Hugging Face dataset.

Step 3 — Distill Annotations with a Teacher VLM

Step 4 — Training

4a — Task Tokens

4b — SFT Annotation Format

4c — Build SFT Data

4d — Run Training

4f — Train the IQA Task

Inference

T2IA pipeline (3 steps)

IQA pipeline (3 steps)

Run DynEval-4B from Hugging Face

Run DynEval-2B from Hugging Face

Choose Score Type

Command-Line Arguments

Quantitative Results

Qualitative Results

Citation

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages