0 cumulative citations
View corpus contextRevealing a model’s identity shifts peer-judge scores: a pilot study finds identity-aware inflation on geopolitical and certain reasoning/preference items while factual scores remain stable; the authors propose a blockchain commit–reveal protocol to cryptographically enforce blind, auditable benchmarking.
Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
In a paired pilot study, revealing the identity of candidate LLMs produced content-dependent score shifts—negligible for factual items, marginal for stress-reasoning, and statistically significant increases (or decreases) on geopolitically sensitive and preference items—motivating a blockchain commit-reveal protocol to make blind benchmarking auditable.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
LLM benchmarks can build an organization's reputation and attract customers, but only when results are transparent and verifiable. Unverified claims that DeepSeek R1 outperformed OpenAI's o1 contributed to market panic on January 27, 2025, when Nvidia lost USD589 billion in market value. Yet vendor benchmarks often depend on an honor system. Academic reassessments and independent leaderboards have found undisclosed changes to proprietary models, contaminated training data, and selective reporting. LLM-as-a-judge methods scale evaluation by reducing human review. Studies, however, suggest that judges may show identity-aware bias, scoring an answer according to its source model rather than its quality. This bias has not been fully measured or corrected across politically sensitive, reasoning-intensive, and preference-based tasks. We examine this problem using seven verifier models: GPT-OSS 120B, Llama 3.3 70B, GLM 5.1, Qwen3 32B, DeepSeek V4 Pro, Mistral Large3, and Sarvam M. They score anonymous and identity-disclosed responses from three primary models on 58 factual, reasoning, political, and preference-based questions. Identity disclosure slightly raises scores for factual questions, moderately affects stress-reasoning tasks, and causes large changes for geopolitically sensitive topics. Notable results include GLM5.1 (+7.00 points, p = 0.0249) and Llama 3.3 70B (+1.56 points, p = 0.00). We also introduce a blockchain-based commit-reveal protocol using Autonomous Economic Agents on an Ethereum-compatible ledger. In Phase 1, each judge records a one-way hash of its score and a secret salt before candidate identities are revealed. In Phase 2, the identity and raw score are disclosed and verified on-chain. This creates a tamper-evident audit trail that separates blind evaluation from post-hoc claims and reduces the verification burden on independent researchers and leaderboard operators.
Summary
Main Finding
Identity disclosure of candidate LLMs systematically shifts peer-evaluation scores in a content-dependent way: negligible on general factual items, moderate on stress/reasoning tasks, and significant on geopolitically sensitive and preference questions. The authors demonstrate these Identity-Aware Bias effects empirically and propose a blockchain-based commit–reveal protocol (implemented with Autonomous Economic Agents on an Ethereum-compatible ledger) to cryptographically enforce blind scoring and provide an auditable trail.
Key Points
- Measured "Identity Gap" (IG) = score(anonymized) − score(revealed) using paired evaluations to isolate identity effects while keeping the candidate response fixed.
- Experimental scope: 58 questions (29 factual, 29 subjective split into political and preference), three candidate models (DeepSeek V4 Pro, GPT OSS 120B, Sarvam M), seven verifier/judge models (GPT OSS 120B, Llama 3.3 70B, GLM 5.1, Qwen3 32B, DeepSeek V4 Pro, Mistral Large 3, Sarvam M).
- Primary empirical results (examples reported):
- Geopolitical sensitivity: GLM 5.1 awarded Sarvam M +7.00 points when identity was revealed (p = 0.0249). Qwen3 gave Sarvam M +5.20 (p = 0.0426).
- Political category (merged): Llama 3.3 70B awarded GPT OSS +1.56 points (n = 9, p = 0.0033). Qwen3 had a similar but marginal effect (gap +6.86, p = 0.0895).
- Stress/reasoning factual items: Qwen3 gave GPT OSS +2.40 points upon revelation (n = 10, p = 0.0552; marginal).
- Preference items: GPT OSS boosted DeepSeek by +4.27 points (n = 15, p = 0.0433).
- Overall: identity effects are minimal for straightforward factual items, increase for hard reasoning, and are strongest for politically sensitive content and preference-based evaluations.
- Proposed technical mitigation: a two-phase commit–reveal protocol where judge agents commit a hash(score + salt) on-chain before the candidate identity is disclosed; after reveal the raw score and salt are submitted and verified on-chain. Implementation layers: Agent (AEA with DID and wallet), Communication (uAgents P2P), Blockchain (Ethereum-compatible smart contract, tested on Anvil).
Data & Methods
- Question bank: 58 items divided into factual (geography, science, mathematics, stress-reasoning, technology) and subjective (political, preference) categories.
- Candidate responses collected and stored in PostgreSQL to keep the response constant across paired blind/transparent evaluations.
- Judges: seven distinct LLMs from different ecosystems; judges were prevented from scoring their own outputs to limit self-bias.
- Evaluation conditions: each judge scored each candidate response twice — first anonymously, then after identity reveal — with sessions started in fresh contexts to avoid prior memory.
- Control measures: same response used across both conditions to isolate identity effects; reasoning/temperature settings controlled (paper reports differing settings for political vs factual subtests).
- Statistical analysis: paired t-tests on paired blind vs revealed scores; reporting per-subcategory and aggregated political-category results. Many effects are statistically significant or marginal; authors flag sample-size and multiple-comparison limitations.
- Limitations noted: pilot scale, manual web-playground session management (vs API), potential for authorship inference via stylometry, limited judge diversity and per-cell sample sizes.
Implications for AI Economics
- Market impact of benchmark claims: unverified or vendor-led benchmark announcements can move capital markets dramatically (paper cites the January 27, 2025 episode linked to DeepSeek R1 claims and large market losses). Reliable verification matters economically.
- Information asymmetry & credibility: cryptographic, auditable blind-evaluation infrastructure can reduce asymmetric information between model vendors and investors/customers by making benchmarking claims verifiable.
- Verification costs & ecosystem effects: a commit–reveal blockchain protocol could lower the burden on independent researchers and third-party leaderboards to re-evaluate every claim, reducing redundant verification costs and accelerating trust formation.
- Incentive alignment and new institutions: AEAs with on-chain identifiers and verifiable scoring could enable reputation, staking, or penalty mechanisms that discourage dishonest reporting or post-hoc claim manipulation — changing competitive dynamics among model providers.
- Adoption and transaction costs: real-world benefit requires broad industry adoption, standardization of protocols, and handling of operational costs (on-chain gas/throughput, agent orchestration). Practical rollout also needs protections against collusion among judges, leakage of salts, and inference attacks (stylometry).
- Residual risks: commit–reveal prevents post-hoc score edits but does not eliminate all manipulation vectors (e.g., training-data contamination, secret model alterations, stylometric identification, strategically chosen judge pools). Complementary measures (TEEs, zero-knowledge proofs of model provenance, standardized APIs and audits) will be necessary.
- Policy relevance: regulators, index providers, and institutional investors can treat verifiable benchmarking as a market-stabilizing infrastructure; standards and certifications for cryptographically verifiable evaluations would help reduce market volatility from unverified claims.
- Next steps recommended by authors (economic research agenda): scale up experiments, expand judge diversity, integrate API-based evaluation, explore zk-proofs or trusted execution for provenance, and study how verifiable benchmarking changes strategic behavior of model developers and investor responses.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The study uses a sensible within-judge paired design and multiple independent verifier models, which supports causal claims that identity revelation shifts scores. However, it is a small pilot with limited per-cell sample sizes (reported n per comparison often < 20), manual web-playground session management, incomplete control against stylometric authorship inference, multiple comparisons without clear correction, and some marginal p-values; these limit the strength and robustness of the empirical claims.
Methods Rigormedium — Strengths: paired design with identical stored responses, a factual control group, multiple verifier and candidate models, and basic inferential testing (paired t-tests). Weaknesses: manual session handling (web playgrounds) adds variance; unclear randomization/preregistration; limited sample sizes per subcategory; no correction for multiple hypothesis tests is reported; potential for judges to infer authorship through stylometry or artifacts; scoring rubric details (scale, inter-annotator reliability) are not fully specified.
SamplePilot experiment using a 58-question benchmark (29 factual, 29 subjective: political and preference) where three primary models (DeepSeek V4 Pro, GPT OSS 120B, Sarvam M) produced baseline responses; seven verifier models (GPT OSS 120B, Llama 3.3 70B, GLM 5.1, Qwen3 32B, DeepSeek V4 Pro, Mistral Large 3, Sarvam M) scored the same responses in two conditions (anonymous and identity-revealed). Scores and responses were stored in a PostgreSQL database; per-comparison sample sizes vary (examples reported n = 9 to 17 for subcategories). Sessions were run via web playgrounds with manual session clearing.
Themesgovernance adoption
IdentificationWithin-judge paired design: identical model-generated responses are scored twice by the same verifier models under anonymous and identity-revealed conditions; the difference in scores (anonymous minus revealed) is the estimated identity-aware bias and is tested with paired t-tests (control group of factual items used to check baseline). Self-scoring was prevented (judges were not allowed to score their own outputs). Scores were committed to a database to keep responses constant across the two conditions.
GeneralizabilitySmall pilot sample and limited per-cell n restricts statistical power and replication certainty, Only a small set of models (three primary, seven verifiers) — findings may not hold across other model families or future model versions, Web-playground/manual session management introduces variability versus API-driven, production benchmarking workflows, Possible stylometric or latent metadata signals could allow author inference even when identities are hidden; experiment does not fully rule this out, Question bank is limited (58 items) and focused on particular content types; results may be content-dependent and time-specific, Scoring is LLM-as-judge (no human adjudicators), so implications for human evaluation contexts are indirect
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Identity revelation had a negligible effect on scores for objective factual questions. Decision Quality | null_result | Difference in verifier-assigned scores between anonymous and transparent evaluation conditions on factual questions |
Reading fidelity
high
Study strength
low
|
n=58
|
| GLM 5.1 awarded Sarvam M responses scores 7.00 points higher after the model identity was revealed for geopolitically sensitive chip-war questions. Decision Quality | positive | Change in verifier-assigned evaluation score after candidate identity disclosure |
Reading fidelity
high
Study strength
medium
|
n=58
+7.00 points, p = 0.0249
|
| Qwen3 32B awarded Sarvam M responses scores 5.20 points higher after identity revelation for geopolitically sensitive chip-war questions. Decision Quality | positive | Change in verifier-assigned evaluation score after candidate identity disclosure |
Reading fidelity
high
Study strength
medium
|
n=58
+5.20 points, p = 0.0426
|
| Mistral produced a lower score for Sarvam M responses during transparent evaluation of politically sensitive questions. Decision Quality | negative | Change in verifier-assigned score after candidate identity disclosure |
Reading fidelity
high
Study strength
low
|
n=58
Gap −1.50, p = 0.0756
|
| Llama 3.3 70B awarded GPT OSS 120B responses 1.56 points more after identity revelation in the merged political-question category. Decision Quality | positive | Change in political-question evaluation score following candidate identity disclosure |
Reading fidelity
high
Study strength
medium
|
n=9
1.56 points higher, p = 0.0033
|
| Qwen3 32B awarded GPT OSS 120B responses 2.40 points more after identity revelation on stress-reasoning questions. Decision Quality | positive | Change in reasoning-question evaluation score following candidate identity disclosure |
Reading fidelity
high
Study strength
low
|
n=10
2.40 points higher, p = 0.0552
|
| GPT OSS gave DeepSeek a higher score after identity revelation on preference-based questions. Decision Quality | positive | Change in preference-question evaluation score after candidate identity disclosure |
Reading fidelity
high
Study strength
medium
|
n=15
Gap = 4.27, p = 0.0433
|
| The magnitude of identity effects increased with content sensitivity: negligible for objective factual questions, marginal for stress-reasoning questions, and significant for geopolitical content. Decision Quality | mixed | Identity-induced change in LLM evaluator scores across question categories |
Reading fidelity
high
Study strength
low
|
n=58
|
| The proposed blockchain commit-reveal protocol records a judge's hashed score before candidate identity disclosure and verifies the raw score and salt on-chain after disclosure. Governance And Regulation | positive | Ability to cryptographically verify blind-evaluation commitments and create an auditable evaluation record |
Reading fidelity
high
Study strength
low
|
not reported
|
| The authors propose that the blockchain-based blind-evaluation method can reduce the workload currently borne by independent researchers and third-party leaderboards when verifying benchmark claims. Organizational Efficiency | positive | Need for independent and third-party verification of LLM benchmark claims |
Reading fidelity
high
Study strength
speculative
|
not reported
|