Consensus can mask fragility: organizations that suppress divergent reports create the illusion of agreement and miss accumulating tail risks until abrupt correction. Simple governance diagnostics—recorded dissent patterns, voting-anonymous gaps, and method/pipeline diversity—can distinguish healthy integration from dangerous exclusion.

Main Finding

Organizations that exhibit internal cohesion can hide two very different processes: (1) healthy information integration, or (2) exclusionary variance depletion (conformity, gatekeeping, intolerance of dissent). Using a formal signal-aggregation model, the paper shows that exclusionary cohesion produces fragile, state-contingent biases — small in normal times but large in displaced/tail regimes — along with "illusory precision", effective concentration of inputs, and dynamic lock‑in (delayed detection of regime shifts and abrupt correction). Low observed disagreement and unanimity can therefore signal latent fragility rather than robustness.

Key Points

• Cohesion is observationally ambiguous:
  • Integration: heterogeneous inputs debated and synthesized, preserving variance.
  • Exclusion: disagreement removed via conformity pressure, report shrinkage toward an anchor, and discarding outlying reports (tolerance rule).
• Model mechanisms:
  • Anchor belief maintained by organization; two exclusion channels formalized:
    • Report shrinkage: individual reports are pulled toward the anchor before aggregation.
    • Tolerance threshold: reports beyond a deviation threshold are discarded.
• Main model consequences (relative to full inclusion benchmark):
  • State-contingent bias: bias is small in normal regimes but grows sharply when the environment displaces from typical regimes.
  • Illusory precision: observed disagreement falls even as true estimation error in tails rises (apparent confidence increases while accuracy declines).
  • Effective concentration: nominal participant counts overstate the independent information actually influencing decisions (variance depletion creates fewer effective inputs).
  • Dynamic lock-in: if the anchor updates from filtered aggregates, the system delays recognizing regime shifts and then corrects abruptly (overshoot/jerkiness).
  • External inputs that bypass internal filtering reduce recognition delays and break lock-in.
• Testable governance diagnostics proposed:
  • Patterns in recorded dissent (frequency, amplitude) and differences between anonymous vs formal voting records.
  • Diversity of scenario sets and scenario-generation methods.
  • Concentration in pipelines and methods (e.g., few dominant teams/models).
  • Anchor lag: delay between true regime changes and organizational anchor update.
  • Metrics linking latent fragility to observable disagreement statistics, discarded-report rates, and effective participant counts.

Data & Methods

• Formal model: a signal-aggregation framework where each participant/report supplies a noisy signal about the true state; the organization maintains an anchor and applies two filtering rules (shrinkage and tolerance/discarding) before aggregation.
• Benchmark: full-inclusion aggregation (no shrinkage or discarding) used to compare bias, variance, and responsiveness.
• Analytical results characterize how filtering transforms the distribution of aggregated estimates, produces state-dependent bias, and reduces observed disagreement while increasing tail estimation error.
• Numerical examples / simulations (implied) illustrate dynamic behavior: delayed regime recognition, abrupt corrections, and how bypassing filters shortens delays.
• Output: model-derived diagnostic statistics and proposed empirical metrics to detect exclusionary cohesion in real organizations (logs of dissent, variance of scenario sets, voting/anonymous-report gaps, pipeline concentration measures).

(Note: the supplied summary does not list specific datasets; empirical application would require collection of internal reports, vote records, scenario libraries, and metadata on pipelines/method diversity.)

Implications for AI Economics

• For AI governance and risk assessment:
  • Unanimity or low internal conflict is not a reliable indicator of robust decision-making; it can be a leading indicator of hidden fragility.
  • Safety assessments, deployment approvals, and internal risk reviews should treat low disagreement with suspicion and test whether heterogeneity is being integrated or excluded.
• Practical governance recommendations:
  • Preserve and surface raw/unfiltered inputs (anonymized if necessary) to measure true disagreement and avoid report shrinkage.
  • Require parallel, independent evaluation pipelines (reducing method/pipeline concentration) and track their correlations to estimate effective participant count.
  • Implement mechanisms for external, bypassing inputs (third‑party audits, red teams with independent reporting paths) to shorten regime recognition delays.
  • Monitor diagnostics: recorded dissent rates, anonymous vs formal voting gaps, fraction of discarded/outlier reports, scenario-set diversity, and anchor lag statistics.
  • Avoid aggregation rules that unduly shrink reports toward prevailing anchors; consider blind aggregation and aggregation rules that preserve variance.
• For economic modeling of AI-related systemic risk:
  • Incorporate endogenous information-filtering and conformity mechanisms into models of organizational decision-making and technology deployment.
  • Evaluate how homogeneity across firms (e.g., common model architectures, shared training data, similar evaluation pipelines) can create systemic illusory precision and amplify tail risks.
  • Use the proposed diagnostics to empirically identify organizations or sectors at elevated tail-risk exposure and to design policy interventions (mandatory disclosure of dissent logs, diversity requirements for safety evaluations, independent oversight).
• Empirical research directions:
  • Collect and analyze governance artifacts (internal reports, voting records, scenario libraries) to validate model predictions.
  • Measure effective input concentration (e.g., via inter-evaluator correlation, Herfindahl-style indices over pipelines/methods).
  • Test for illusory precision by relating observed disagreement to out-of-sample tail errors during realized regime shifts.