Survey layout, not model creed: pinning options to fixed answer slots generated a robust ‘market‑rationalist’ cluster across LLMs, but a balanced randomized-order replication shows the apparent ideology collapses—answer‑slot bias, not underlying model beliefs, explains most of the effect.

Main Finding

Fixed-order forced-choice instruments can manufacture apparent political “ideologies” in large language models. In a 400-question governance benchmark administered to 15 open-weight models, a conventional fixed-option-order analysis produced a neat two-cluster ideological map (a small “market-rationalist” cluster vs. an “institutionalist” majority). That structure almost entirely disappeared when option order was randomized in a fully balanced way: the measured ideology correlated essentially zero with the de-confounded (content-only) profile and instead tracked a strong, model-specific position bias (especially first-slot preference).

Key Points

• Experiment scale and outcome
  • 400 governance scenarios, 4-option forced-choice each, administered to 15 open-weight models.
  • Fixed-order run: 640,000 responses (multiple temperatures and repeats). Randomized-order replication: ≈48k parsed responses (balanced permutations).
• Spurious ideology under fixed-order
  • Fixed-order analysis yielded two stable clusters (market-rationalist minority of 3 models vs. institutionalist majority of 12), with split‑half reliability ρ ≈ 0.91.
  • Cluster differences: utility +23.3, metrics +20.4, incentives +15.0 (market-rationalists higher).
• Position bias drives the apparent structure
  • Balanced randomized-order design (each content occupies each display slot equally within every question) produces:
  • De-confounded map reliable (split-half ρ ≈ 0.87) but uncorrelated with the fixed-order map (ρ = 0.07).
  • The market-rationalist signature correlates r = 0.91 with a model’s first-slot bias.
  • Position bias is large and extremely stable across runs (slot bias split-half r ≈ 0.998 for pooled slot means).
• Practical impact on recommendations
  • Changing only option order (holding model, question, and content fixed) alters the top recommendation on 71% of model–question pairs vs. a 33% run-to-run noise floor.
  • Self-reported confidence barely changes despite frequent reversals.
  • Reversals concentrate on low-signal questions, consistent with satisficing/primacy effects.
• Multiverse sensitivity
  • Because randomized data sampled all content-slot pairings, authors re-ran the benchmark under all 13,824 alternative content-to-slot assignments.
  • The originally chosen arrangement (which listed market-favored options first) was the single most misleading assignment.
  • Nine of 15 models would be labeled “market‑rationalist” under at least one slot arrangement; the model most often so labeled (qwen3:8b) was not among the three from the fixed-order run.
  • About a quarter of slot assignments would produce no coherent cluster at all.
• What survives de-confounding
  • A state-vs-market index (constructed from responsibility and instrument frames) is relatively robust (ρ ≈ 0.79), but even it can be materially shifted for many models by slot assignment.
• Other diagnostics
  • Temperature had negligible effect on the fixed-order map.
  • Confidence and rationale text provide only weak, inconclusive signals about the position-confound.
• Boundaries and caveats
  • All evidence is from reproducible open-weight models in the 1–20B parameter range; results are not tested on closed frontier commercial models.
  • The forced-spread (≥30-point spread) scoring instruction may inflate magnitude of bias but cannot explain directional, model-specific slot habits.

Data & Methods

• Stimuli and framing
  • 400 scenarios across four frame families (100 each): responsibility locus (who), policy instrument (how), ethical principle (why), evidence stance (on what basis).
  • Each family had four consistent options (canonical order used in fixed-order run).
• Elicitation protocol
  • Prompt: score each of four labeled options 0–100, name top option, report confidence 0–100, and provide a short justification. Prompt requested at least a 30-point spread to discourage ties.
• Fixed-order run
  • Canonical mapping: each option content fixed to one display slot across all scenarios.
  • Crossed with two temperatures (T = 0.2, 0.8) and repeated runs to assess reliability.
• Randomized replication (de-confounding)
  • For each question, 8 permutations of the 4 contents across slots were administered such that each content occupied each slot exactly twice (content × slot orthogonal within every question).
  • This yields a clean estimate of position bias (mean centered score by display slot) and a de-confounded content-driven fingerprint (mean centered score by canonical content).
• Analysis
  • Scores row-centered within each response to remove scale differences.
  • Distances between model fingerprints are the main inference target.
  • Dimensionality reduction and grouping: Multiple Factor Analysis (MFA), hierarchical clustering, archetypal analysis.
  • Reliability: split-half correlations reported for fixed-order (≈0.91) and randomized (≈0.87) fingerprints; position bias shown extremely stable (≈0.998).
  • Multiverse: exhaustive re-computation over all 13,824 content-to-slot assignments enabled by balanced randomized data.
• Robustness checks
  • Specification curves, alternative encodings, leave-one-out perturbations, temperature variation.
  • Permutation tests to assess model identity vs. temperature effects.

Implications for AI Economics

• Measurement validity
  • Forced-choice, fixed-order instruments can produce coherent-seeming ideological structure that primarily reflects display position, not substantive model preference. Studies that infer machine political leanings from single, fixed-order multiple-choice batteries risk conflating position bias with content preference.
• Audit and certification practice
  • Model audits, regulatory assessments, or certifications that use forced-choice instruments should:
  • Randomize option order and use balanced placement so each content appears in each slot sufficiently often (or report why not).
  • Report position-bias diagnostics (mean centered scores by display slot) alongside substantive results.
  • Include multiverse or sensitivity analyses over content-to-slot assignments where possible.
• Research design recommendations
  • Prefer elicitation formats less vulnerable to simple positional primacy (e.g., balanced randomization, many repeated permutations, pairwise comparisons, or well-designed Likert scales that control acquiescence); when forced-choice is used, ensure content-slot orthogonality.
  • Report split-half reliabilities for de-confounded fingerprints and assess how much variation is explained by slot vs. content.
  • Where publication or policy claims rely on a single fixed-order administration, explicitly acknowledge the slot-assignment degree of freedom and quantify its possible effects.
• Theoretical and policy consequences
  • Treating language models as “revealed-preference” agents in political-economy settings requires careful instrument design: apparent “governance philosophies” may be joint properties of models plus arbitrary measurement choices rather than intrinsic model commitments.
  • Because cheap, widely reused LLMs can act as low-cost advisors, measurement artifacts could mislead policy makers and the public about the distribution of machine advice—potentially affecting delegation, regulatory focus, or model-selection decisions.
• Limits and future work
  • Need to test whether these position effects and magnitudes generalize to larger closed proprietary models and to other elicitation families.
  • Explore corrective analytics (e.g., statistical adjustment for position bias when balanced randomization is infeasible) and alternative elicitation protocols that minimize satisficing or primacy in models.

Overall takeaway: do not trust fixed-order forced-choice ideological maps of LLMs without balanced randomization and explicit diagnostics for position bias.