0 cumulative citations
View corpus contextSmall, realistic image edits systematically sway vision-language agents' choices, exposing reproducible visual biases and potential safety vulnerabilities; the method provides a scalable auditing tool to surface and interpret what VLMs prefer.
Visual Persuasion: What Influences Decisions of Vision-Language Models?
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
Optimized, visually plausible edits to images reliably shift vision-language models' selection probabilities in controlled choice tasks and reveal consistent visual themes that drive those preferences.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent's decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents.
Summary
Main Finding
Optimizing naturalistic image edits (composition, lighting, background, props) substantially and systematically shifts the choices of state-of-the-art vision–language models (VLMs) in realistic decision tasks. These shifts are measurable via pairwise comparisons, repeatable across tasks and models, partially interpretable by automated pipelines, and only partially mitigated by a normalization step — implying practical vulnerabilities in image-based automated decision systems.
Key Points
-
Goal and framing
- Treats a VLM’s decision function as a latent visual utility that can be probed by revealing preferences over systematically edited versions of the same image.
- Focuses on perceptually plausible, identity-preserving edits (not pixel-level adversarial noise).
-
Methods introduced
- Visual prompt optimization: iterative editing via text prompts to an image-editing model.
- Three concrete algorithms:
- VisualTextGrad (VTG): adapts TextGrad to update editable prompt text using LLM-produced feedback.
- VisualFeedbackDescent (VFD): proposer–evaluator loop (proposal, apply edit, evaluate, accept/reject).
- Competitive Visual Prompt Optimization (CVPO): novel competitive loop where challengers and champions are pitted with judge panels; losers are refined.
- Identity-maintenance constraint ensures the same underlying object/scene is preserved.
-
Empirical results
- Large-scale experiments across 4 agentic tasks: product purchasing, house searching, candidate hiring, hotel scouting.
- Datasets: 100 images per task (products from ABO, houses from price dataset, synthetic candidates from StyleGAN-Human, hotels from prior aesthetic datasets).
- Evaluated 9 frontier VLMs (judge model used for optimization: Gemini 3 Flash; edit model: Nano Banana / Gemini 2.5 Flash Image).
- Optimized edits moved head-to-head choice probabilities substantially versus originals and versus zero-shot edits.
- Even zero-shot edits from an image-to-image model often shift choices relative to original images.
- Human study (N = 154) shows many of the same edited images also shift human preferences, though not identically.
- Visual normalization (aligning contextual features before decision) reduces but does not eliminate model sensitivities.
-
Interpretability & robustness
- Developed an auto-interpretability pipeline (LLM-based) to surface recurring visual themes (e.g., richer backgrounds, warmer lighting, added luxury cues) that drive model choices.
- Optimization often exploits consistent, human-plausible cues (not purely inscrutable adversarial noise).
- Stopping rules use patience or equilibrium (vote split near 50%) given noisy judgments.
-
Limitations noted by authors
- Dependence on current, powerful image-edit models (editing fidelity matters).
- Identity predicates and semantic preservation are approximate.
- Benchmarks limited to selected datasets and tasks; judge and edit model choices may shape discovered preferences.
Data & Methods
-
Data
- Four task datasets; 100 seed images per task:
- Products: Amazon Berkeley Objects (ABO), 20 categories.
- Houses: house-price estimation dataset (price-comparable sampling).
- Candidate portraits: synthetic StyleGAN-Human images.
- Hotels: rooms and lobbies drawn from a hospitality-image aesthetic dataset.
- Human participant study: N = 154.
- Four task datasets; 100 seed images per task:
-
Editing & judge models
- Image-edit model: Nano Banana (Gemini 2.5 Flash Image).
- Judge for optimization and comparisons: Gemini 3 Flash (used in optimization pipelines).
- Default model temperatures used for VLM evaluators.
-
Optimization procedure
- Start from base prior prompt p0 (e.g., “keep the same product and make image more appealing”).
- Iterate propose → edit → judge via pairwise comparisons; accept edits that consistently beat incumbent under order-randomized trials and consistency checks (k = 3).
- Identity-maintenance enforced via instructions and post-hoc checks (or similarity thresholds); early stopping via patience or equilibrium threshold.
-
Evaluation & statistical view
- Pairwise two-alternative forced choice with order randomization; inconsistent order-dependent judgments are flagged.
- Probabilistic framing: Bradley–Terry/Luce logistic linking of latent utilities; increases in win probability correspond to increases in latent utility gaps.
- Algorithms adapted from text prompt optimization frameworks (TextGrad, Feedback Descent) to operate on editable prompt space that parameterizes image edits.
Implications for AI Economics
-
Market design and platform outcomes
- Platforms that rely on VLMs (for recommendations, ranking, automated selection) can have outcomes systematically shifted by presentation/visual design, not just intrinsic product quality. This introduces potential for strategic image optimization to skew market shares, click-through rates, and conversion.
- Sellers and advertisers can gain an outsized advantage by optimizing imagery for VLM preferences, amplifying winner-take-all effects and reducing the informational content of listings.
-
Manipulation risk & strategic behavior
- Visual prompt optimization provides a practical method for actors (including adversaries or competitive firms) to discover presentation edits that increase selection probability. This raises risks of algorithmic manipulation at scale with limited detectability, especially if platforms rely on automated agents rather than humans.
- Picture-based signaling (luxury cues, lighting, staging) can substitute for or distort underlying value signals (price, quality), affecting pricing dynamics and welfare.
-
Labor markets & selection processes
- Automated résumé/candidate screening using images (portraits, headshots) could be biased by superficial visual changes. Small, identity-preserving edits could affect hiring or shortlisting outcomes, amplifying inequities and undermining fairness.
-
Real-estate and travel markets
- House and hotel rankings or candidate shortlists relying on VLM-based aesthetic judgments can be gamed via staged photos, shifting demand and possibly prices for properties or accommodation that merely look better in optimized images.
-
Policy, regulation, and auditing
- Need for proactive behavioral audits of VLM-driven decision systems (not only accuracy benchmarks). Instruments like the authors’ visual prompt optimization pipeline can be used by regulators, researchers, and platforms to discover likely manipulations.
- Policies could require provenance, standardized image submissions, mandated image normalization for automated decisions, or disclosure when automated agents drove decisions.
- Mitigation pathways: normalization of images prior to judging (partially effective), human-in-the-loop checks for high-stakes decisions, training models on standardized or counterfactually augmented images, and defenses that penalize decisions overly sensitive to contextual presentation.
-
Measurement and empirical research opportunities
- The paper’s framework gives economists a tractable way to estimate implicit visual utility functions embedded in automated agents — useful for counterfactual policy simulation (e.g., how would enforced image standards change market shares), causal inference on presentation effects, and welfare analyses of automated decision pipelines.
- Can be used to quantify externalities from platform-level incentives to prioritize click-through over substantive quality.
-
Recommended responses for practitioners and policy-makers
- Integrate behavioral stress tests (like CVPO) into model audits before deploying VLM-based decision agents.
- Consider rules for image standardization or provenance (metadata, timestamped originals) to reduce manipulable variance.
- Combine normalization with other robustness measures (training diversity, calibration checks) and mandate human oversight for high-stakes automated choices.
Summary takeaway: VLMs’ decisions are meaningfully influenced by realistic, identity-preserving image edits. For economics, this translates into nontrivial risks of manipulation, altered market outcomes, and distributional effects unless platforms and regulators adopt behavioral audits and mitigation strategies.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The paper provides clear experimental manipulation and shows consistent shifts in model choice probabilities across large-scale controlled edits and multiple frontier VLMs, supporting internal validity for the claim that visual edits affect model decisions; however, it does not measure downstream economic outcomes (e.g., purchases, clicks in real markets), may be sensitive to artifacting from synthetic edits or specific model versions, and external validity across other image domains and real-world agent pipelines is limited.
Methods Rigormedium — The approach adapts a principled revealed-preference framework, uses iterative optimization and an automatic interpretability pipeline, and reports large-scale experiments on multiple VLMs; nevertheless, methods rely on synthetic image generation (possible confounds/artifacts), the paper does not appear to fully document pre-registration, robustness checks across diverse model families, sensitivity to generation seeds, or the sample sizes and statistical adjustment procedures in detail.
SampleControlled experiments using 'common' images (notably product photos) that are systematically edited via an image-generation/composition model to create counterfactual variants; evaluations run at scale across multiple frontier vision-language models in head-to-head pairwise choice tasks, with automated interpretability analyses to summarize recurring visual themes that increase selection probability (exact N and model names are not specified in the abstract).
Themesgovernance human_ai_collab
IdentificationTreats the VLM decision function as a latent utility and identifies causal effects by systematically perturbing input images (composition, lighting, background) via an image-generation model and measuring changes in head-to-head choice probabilities; uses iterative visual-prompt optimization to propose counterfactual edits and revealed-preference comparisons to infer which visual features drive selection.
GeneralizabilityFindings may not generalize to non-product-image domains (e.g., scenes, faces, medical images)., Synthetic edits from the image-generation model may introduce artifacts not present in natural variation, biasing results., Results on current 'frontier' VLMs may not hold for other architectures, model versions, or fine-tuned systems used in production., Lab-style head-to-head choice tasks may not map directly to real-world agent decisions or economic outcomes (clicks, purchases, recommendations)., Interaction effects when images are embedded in broader multimodal contexts (text, user history) are not addressed.
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| We introduce a framework for studying VLMs by placing them in controlled image-based choice tasks and inferring a latent visual utility through revealed preference (choices between systematically edited images). Decision Quality | positive | ability to infer a latent visual utility from observed choices |
Reading fidelity
high
Study strength
low
|
not reported
|
| We propose methods for visual prompt optimization by adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (e.g., composition, lighting, background). Decision Quality | positive | effectiveness of visual prompt optimization methods at producing plausible image edits |
Reading fidelity
high
Study strength
low
|
not reported
|
| We evaluate which edits increase selection probability. Decision Quality | positive | selection probability (choice probability) after specific image edits |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Through large-scale experiments on frontier VLMs, optimized edits significantly shift choice probabilities in head-to-head comparisons. Decision Quality | positive | change in choice/selection probability between edited images and original images |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. Ai Safety And Ethics | positive | identification of consistent visual themes driving model selection |
Reading fidelity
high
Study strength
medium
|
not reported
|
| This approach offers a practical and efficient way to surface visual vulnerabilities and safety concerns, supporting more proactive auditing and governance of image-based AI agents. Governance And Regulation | positive | ability to surface visual vulnerabilities and support auditing/governance |
Reading fidelity
high
Study strength
speculative
|
not reported
|