0 cumulative citations
View corpus contextA 100-task benchmark shows image-editing models pass 34–83% of marketing/design edits per attempt, but factoring in retries and human review raises the effective cost per successful edit to $0.66–$1.42; cheaper per-image models can be pricier in practice once failures and review time are counted.
HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models
HYPE-EDIT-1, a 100-task benchmark for reference-based marketing/design image edits, finds per-attempt pass rates of 34–83% and effective cost-per-success (including retries and human review) of $0.66–$1.42, showing that models with lower per-image prices can become more expensive once retries and reviewer time are accounted for.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per successful edit that combines model price with human review time. We release 50 public tasks and maintain a 50-task held-out private split for server-side evaluation, plus a standardized JSON schema and tooling for VLM and human-based judging. Across the evaluated models, per-attempt pass rates span 34-83 percent and effective cost per success spans USD 0.66-1.42. Models that have low per-image pricing are more expensive when you consider the total effective cost of retries and human reviews.
Summary
Main Finding
HYPE-EDIT-1 is a reliability-focused benchmark for reference-based marketing/design image edits that quantifies both per-attempt success probability and the “effective cost” of obtaining a usable edit once retries and human review are included. Evaluating seven contemporary image-edit models on 100 curated tasks (50 public / 50 private) with 10 independent attempts per task shows wide differences in reliability: per-attempt pass rates span ~34–83%, and effective cost per successful edit (model cost + human review × expected retries, with a retry cap) ranges ≈ $0.66–$1.42. Low per-image list prices can therefore be misleading: cheaper models may become more expensive in practice once retries and review time are accounted for.
Key Points
- Benchmark scope
- 100 realistic, marketing/design editing tasks (50 public, 50 held-out private).
- Task types: change (50), remove (21), restructure (17), enhance (12).
- Mostly single-image tasks (89), some multi-image (11). Resolutions long-edge 2048–5504 px.
- Repeated-trial protocol
- For each model-task pair generate K = 10 independent outputs.
- Human judging: 5 raters per output vote PASS/FAIL; majority determines label.
- A VLM judge (Gemini 3 Flash) is included as an auxiliary deterministic check; it agrees with human majority ~80% of the time and tends to be stricter.
- Reliability metrics
- P@1 (first-attempt pass rate), P@10 (at least one success in 10 attempts), expected attempts under a retry cap A (default A = 4), and Hype Gap = P@10 − P@1 (measures best-of sampling uplift).
- Cost model (practical effective cost)
- Per-attempt cost = model generation cost + human review cost.
- Default review cost: $50/hr and 20 s inspection per image → Creview ≈ $0.278 per candidate.
- Per-candidate model costs used in the paper (USD): gemini-3-pro-preview $0.134; seedream-4.5 $0.04; seedream-4.0 $0.03; gpt-image-1.5 $0.17; flux-2-max $0.10; riverflow-2-b1 $0.15; qwen-image-edit-2511 $0.03.
- Effective cost per successful edit combines expected attempts under cap A = 4 with per-attempt cost and aggregated success probability (reported as a single-dollar figure per model).
- Reported results (combined split, human-majority labels)
- Pass Rate (P@1): riverflow-2-b1 82.7%; gemini-3-pro-preview 63.8%; gpt-image-1.5 61.2%; flux-2-max 45.7%; qwen-image-edit-2511 45.4%; seedream-4.0 35.6%; seedream-4.5 34.4%.
- Pass@4 (%) and Expected Attempts (avg over tasks) range: P@4 57–90.5%; expected attempts 1.40–2.64.
- Effective cost per successful edit (paper’s aggregation): riverflow-2-b1 $0.66 (lowest), gemini-3-pro-preview $0.95, gpt-image-1.5 $1.30, flux-2-max $1.41, qwen-image-edit-2511 $1.33, seedream-4.0 $1.42, seedream-4.5 $1.39.
- Key observation: a model priced at $0.03/image can reach ≈ $1.42 per successful edit after retries and review, while a higher-priced model can be cheaper per success.
- Reproducibility & tooling
- Public repository with 50 public tasks, JSON schema, Gemini-3-Flash judge example, and a human-judge web UI: https://www.github.com/sourceful-official/hype-edit-1-benchmark
- Private 50-task split held for server-side evaluation to reduce contamination.
Data & Methods
- Data
- 100 hand-curated, real-world marketing/design editing tasks; reference images hosted on CDN; assets licensed CC BY 4.0.
- Task JSON includes id, instruction, task type, input images, target dims.
- Evaluation protocol
- Generate K = 10 independent samples per model-task.
- Human panel of five blind raters per output votes PASS/FAIL; majority label used for counts.
- VLM judge (Gemini 3 Flash) used as deterministic check; not primary labeler.
- Reliability estimation
- Compute per-task empirical pass probability pt from 10 repeats.
- Under retry cap A = 4, per-task success probability St = 1 − (1 − pt)^A and expected attempts Et = [1 − (1 − pt)^A] / pt (with special-case handling for pt = 0).
- Aggregate across T tasks to report p@4 (mean St) and E (mean Et).
- Effective-cost calculation
- Per-attempt cost Cattempt = Cmodel + Creview (Creview default ≈ $0.278).
- Effective cost per success Ceff = E · Cattempt / p@4 (paper reports aggregated dollar figures).
- Models evaluated
- Seven current/preview image-editing systems (Gemini 3 Pro preview, Seedream 4.5 & 4.0, GPT-Image-1.5, Flux-2-max, Riverflow-2-b1, Qwen-image-edit-2511) with stated per-candidate costs used in analysis.
- Limitations reported by authors
- Task set focused on marketing/design; not comprehensive across all editing scenarios.
- Assumes independence across attempts.
- Human-VLM judge mismatch and judge biases possible; VLM tends to be more conservative.
- Benchmark small relative to some suites but designed for repeated-trial, reliability-focused evaluation.
Implications for AI Economics
- Price-per-image is an incomplete cost signal
- Buyers who compare models by per-image price alone risk underestimating total cost: higher variance / lower reliability models incur more retries and review labor, substantially raising cost per usable output.
- Example (paper): sub-$0.05/image model → ≈$1.3–1.4 effective cost per usable edit after retries and review; a higher priced but more reliable model can yield a lower effective cost.
- Procurement and ROI
- Procurement decisions should incorporate reliability metrics (P@1, P@k, expected attempts under a retry budget) and labor costs for QA/review.
- SLAs or pricing that report “effective cost per success” would be more informative to enterprise buyers than nominal per-image unit costs.
- Product & workflow design
- Firms should set retry caps, measure expected attempts empirically for target task distributions, and factor human review time into unit economics.
- Investing in post-generation filtering, automated reranking, or light-touch human-in-the-loop systems may reduce expected attempts and thus lower effective cost even if model-generation cost is higher.
- Market effects & vendor incentives
- Vendors might compete on reliability as a monetizable feature (e.g., guaranteed success rate, premium-priced “high-reliability” offering, or lower variance sampling modes).
- Transparency on reliability (pass rates on representative tasks) could shift competition away from headline per-image price.
- Labor and substitution dynamics
- Human review time is a non-trivial component of cost (paper uses 20 s @ $50/hr → $0.278 per candidate). As models improve, savings accrue in reduced review time; conversely, low-quality models could sustain review labor demand.
- Policy and procurement recommendations
- Enterprises should request benchmarked reliability on task-representative datasets and include expected-attempt costs in TCO analyses.
- Benchmarking and disclosure (like HYPE-EDIT-1) could be incorporated into vendor evaluations to avoid lock-in to apparently cheap but operationally expensive services.
- Research and market opportunities
- Tools that reduce retries (better conditioning, deterministic modes, improved VLM rerankers) have direct economic value; so do human-in-the-loop UI/UX innovations that lower review time per candidate.
- There is room for standardized reliability reporting (e.g., P@1/P@k, expected attempts under standard retry caps) to be adopted industry-wide.
Limitations to apply cautiously: HYPE-EDIT-1 is focused on marketing/design edits and uses particular assumptions (20 s review time, $50/hr, retry cap A = 4). Effective-cost estimates are sensitive to those parameters; buyers should recompute costs for their own labor rates, review times, retry policies, and task mixes.
If you want, I can: - Recompute effective-costs under different review times, labor rates, or retry caps, or - Produce a short checklist procurement teams can use to evaluate image-editing model vendors using HYPE-EDIT-1 style metrics.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic, empirical measurements (100 curated tasks, 10 independent outputs per task, held-out 50-task split) and reports operational metrics (per-attempt pass rate, pass@10, expected attempts, and an effective cost-per-success combining model price and human review). However, the evidence is limited to a specific task domain (reference-based marketing/design edits), a moderate-sized task sample, binary pass/fail judgments, and relies on cost and reviewer-time assumptions that could vary in real deployments.
Methods Rigormedium — The methodology is transparent and reproducible (public 50-task split, private 50-task holdout for server-side eval, standardized JSON schema, tooling, multiple independent outputs, and human/VLM judging). Rigor is reduced by potential selection bias in task design, limited task diversity, lack of detailed reporting on inter-rater reliability and reviewer instructions, simplification to binary pass/fail outcomes, and sensitivity of cost estimates to pricing and human-time assumptions.
SampleA 100-task benchmark of reference-based marketing/design image-editing tasks; for each task the authors generate 10 independent model outputs to estimate per-attempt pass rates, pass@10, and expected attempts; 50 tasks are publicly released and 50 are held out for private/server-side evaluation; evaluation uses binary pass/fail judgments from human and/or VLM-based reviewers; reported per-attempt pass rates across evaluated models span 34–83% and effective cost per successful edit spans USD 0.66–1.42.
Themesproductivity human_ai_collab
GeneralizabilityLimited to reference-based marketing/design image-editing tasks; results may not generalize to other image-editing domains (medical, scientific, creative art, etc.)., Moderate task count (100) with curated content may not capture full real-world variability or worst-case inputs., Binary pass/fail judgment simplifies quality and may miss gradations important to workflow outcomes., Cost model depends on assumed human review time and model pricing which vary across organizations and regions., Evaluated models and configurations may not represent all available or future VLMs and deployment settings., Public vs private task split may not fully remove selection or demonstration bias in task design.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. Other | negative | bias of public demos relative to real workflow costs (retries and review time) |
Reading fidelity
medium
Study strength
speculative
|
not reported
|
| We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. Other | positive | benchmark size and judging protocol (binary pass/fail) |
Reading fidelity
high
Study strength
high
|
n=100
|
| For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per successful edit that combines model price with human review time. Output Quality | positive | per-attempt pass rate; pass@10; expected attempts under retry cap; effective cost per successful edit |
Reading fidelity
high
Study strength
high
|
n=1000
|
| We release 50 public tasks and maintain a 50-task held-out private split for server-side evaluation. Other | positive | benchmark public/private split |
Reading fidelity
high
Study strength
high
|
n=100
|
| We release a standardized JSON schema and tooling for VLM and human-based judging. Other | positive | availability of standardized schema and tooling |
Reading fidelity
high
Study strength
high
|
not reported
|
| Across the evaluated models, per-attempt pass rates span 34-83 percent. Output Quality | mixed | per-attempt pass rate (binary pass/fail per generated output) |
Reading fidelity
high
Study strength
medium
|
34-83 percent
|
| Effective cost per success across evaluated models spans USD 0.66-1.42. Organizational Efficiency | mixed | effective cost per successful edit (USD) |
Reading fidelity
high
Study strength
medium
|
USD 0.66-1.42
|
| Models that have low per-image pricing are more expensive when you consider the total effective cost of retries and human reviews. Organizational Efficiency | negative | comparison of nominal per-image price versus total effective cost per successful edit (including retries and human review) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The benchmark uses binary pass/fail judging for each edit task. Other | positive | evaluation criterion (binary pass/fail) |
Reading fidelity
high
Study strength
high
|
n=100
|