NeurIPS 2026 · Evaluations & Datasets Track

EditJudge-Bench

Auditing VLM Image-Edit Judges with Synthetic Ground Truth

Jens Parslov1 Marco Schouten1 2 Dim P. Papadopoulos1 2

1 Technical University of Denmark 2 Pioneer Center for AI

Paper arXiv Code Dataset
Auditing VLM image-edit judges. We procedurally alter 3D scenes and render controlled before/after image pairs spanning nine edit types, then use these pairs to audit VLM image-edit judges — exposing systematic reliability gaps in current VLM-as-a-judge protocols.

Abstract

Can VLM judges actually see the edits they score?

Vision–language models (VLMs) are increasingly used as automated judges for image-editing benchmarks, yet their reliability is poorly understood. Auditing these judges is difficult because real edited images lack controlled ground truth and entangle the requested change with model artifacts, unintended side effects, and ambiguous degrees of success. We introduce EditJudge-Bench, a diagnostic benchmark of 1,382 synthetic before/after image pairs rendered in Blender, with edits parameterized across nine categories derived from real user requests and exact ground truth from known 3D scene parameters. Because every edit is generated by a known intervention on a controlled scene, EditJudge-Bench evaluates VLM judges as binary verifiers of whether an instruction was correctly executed, isolating judge perception from editor quality.

Across 26 judge configurations, we find substantial reliability gaps: (a) the widely adopted VIEScore template reaches only 69.0% AUROC with GPT-4o; (b) fine-tuned judges score 51.7–66.3%, and spatial edits such as rotation and shape remain near chance; (c) prompt-formatting choices like image–text ordering shift performance by up to 13.9%. Despite using only synthetic data, judge rankings on EditJudge-Bench correlate strongly with two real-image human-preference benchmarks (ρ = 0.93 and ρ = 0.77), indicating that controlled audits surface failure modes relevant to real evaluation.

The benchmark

Edits

Every edit is a deterministic change to a known Blender scene parameter, so its magnitude and direction are known exactly — and everything outside the edit is pixel-identical.

Counterfactual distractors

The same image pair is paired with a wrong instruction. A judge is scored by its ability to discern right from wrong.

DIRECTIONAL “Move the books down”

The core parameter is inverted — left ↔ right, larger ↔ smaller, open ↔ closed. Tests perception of direction.

WRONG OBJECT “Move the table right”

Same edit, different visible object. Tests object grounding.

CROSS-EDIT-TYPE “Make the sofa bigger”

A visually confusable edit type (e.g. zoom vs. scale). Tests what kind of change occurred.

NO EDIT “Don’t change anything”

Tests open-ended visual differencing between the two images.

Citation

BibTeX

@inproceedings{parslov2026editjudge,
  title     = {EditJudge-Bench: Auditing VLM Image-Edit Judges with Synthetic Ground Truth},
  author    = {Parslov, Jens and Schouten, Marco and Papadopoulos, Dim P.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS),
               Track on Evaluations and Datasets},
  year      = {2026}
}