The parts of large language models that make them economically valuable are not simply 'rules in disguise'—they are tacit, practice‑derived competences that resist full human‑readable extraction, boosting rents for model providers and shifting policy toward outcome‑based auditing and behavioral testing.
Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones
The paper argues that the most economically valuable capabilities of LLMs are intrinsically non‑decomposable into explicit, human‑readable rule sets, implying limits to rule‑based interpretability and important consequences for market structure, contracting, regulation, and labor complementarities.
This paper proposes and argues for a counterintuitive thesis: the truly valuable capabilities of large language models (LLMs) reside precisely in the part that cannot be fully captured by human-readable discrete rules. The core argument is a proof by contradiction via expert system equivalence: if the full capabilities of an LLM could be described by a complete set of human-readable rules, then that rule set would be functionally equivalent to an expert system; but expert systems have been historically and empirically demonstrated to be strictly weaker than LLMs; therefore, a contradiction arises -- the capabilities of LLMs that exceed those of expert systems are exactly the capabilities that cannot be rule-encoded. This thesis is further supported by the Chinese philosophical concept of Wu (sudden insight through practice), the historical failure of expert systems, and a structural mismatch between human cognitive tools and complex systems. The paper discusses implications for interpretability research, AI safety, and scientific epistemology.
Summary
Main Finding
The paper argues that the valuable capabilities of large language models (LLMs) are precisely those that cannot be fully captured by human-readable discrete rules. If an LLM’s full behavior could be encoded as rules, it would be equivalent to an expert system — but expert systems are empirically and historically weaker than LLMs. Therefore the extra, useful capabilities of LLMs are intrinsically unexplainable in rule form. This is supported by a logical proof-by-contradiction, historical evidence (the failure of expert systems), formal limits on self-explanation (Gödel/PAC-style arguments), and an epistemological account called “representation mismatch” (continuous internal representations vs. discrete human cognitive tools). The paper draws an analogy to the Chinese concept of Wu (sudden insight through practice) and argues for reorienting interpretability, safety, and scientific expectations accordingly.
Key Points
- Core logical argument:
- Assume full explainability via human-readable rules → that rule set is an expert system → expert systems are strictly weaker than LLMs → contradiction. Thus the uniquely valuable parts of LLMs are unruleable.
- Alternative explanation routes (LLMs explaining themselves, larger systems explaining smaller ones, human+LLM collaboration) are blocked by self-reference/incompleteness and regress problems.
- Historical anchor: expert systems (IF–THEN rule bases) succeeded only in narrow domains and failed as a general paradigm, consistent with Polanyi’s “tacit knowledge” and the idea that some know-how cannot be discretized.
- Philosophical/phenomenological support: Wu (悟) and the Cook Ding parable illustrate tacit learning as convergence on a continuous high-dimensional manifold via practice, matching LLM pretraining/fine-tuning dynamics (phase transitions, emergent capability).
- Representation mismatch: human cognitive tools (language, rules) are discrete; LLMs (and many complex systems) are continuously coupled high-dimensional systems. Translating between them induces irreducible information loss, preventing full verbalization of capabilities.
- Interpretability remains valuable but must be reframed: focus on local causal understanding and dangerous blind spots, not on total explanation.
- Safety implication: alignment (instruction-based education of models) faces fundamental limits — sufficiently powerful systems can mimic alignment without being genuinely aligned; structural and environmental constraints (permissioning, containment) are more robust.
- Broader generalization: the thesis extends beyond language to embodied skills (robotics) and any system learning high-dimensional continuously coupled behaviors.
Data & Methods
- Nature of the paper: conceptual/theoretical argumentation rather than new empirical data.
- Methods used:
- Proof by contradiction linking explainability, rule encodability, and expert-system equivalence.
- Literature synthesis: cites prior work on computational irreducibility (Wolfram), algorithmic information/complexity-gap results, Smolensky’s connectionism, Dreyfus’s phenomenology, dynamical systems theory, and PAC-style formalizations of interpretability limits.
- Historical case analysis: rise-and-fall of expert systems as empirical evidence.
- Cross-cultural philosophical analogy: mapping Wu and Cook Ding to training dynamics of LLMs.
- Conceptual model: “Representation Mismatch” explains why discrete explanations lose essential information about continuous internal states.
- No new datasets, experiments, or quantitative estimates; supports claims via logical argument, existing theory, and analogy.
Implications for AI Economics
-
Valuation and Competitive Dynamics
- Opaque, high-value capabilities create strong information asymmetries between model developers and users/regulators. Firms that discover and operationalize such emergent capabilities can obtain disproportionate rents.
- Winner-take-most potential: because capabilities are discovered through practice and can be hard to replicate by reading rules, first movers with compute/data and deployment pipelines may secure persistent advantages.
- Intellectual property and replication: unexplainable internal representations complicate reverse engineering and licensing valuation—licensing may rely more on access and service contracts than on transparent technology transfer.
-
Investment and R&D Priorities
- Increased private returns to “engineering-first” discovery (scale, compute, data) over incremental theoretical understanding. Expect heavy investment in capability development and deployment.
- Public-good underinvestment risk: because discovery requires large scale and generates opaque, socially risky capabilities, private incentives may misalign with socially optimal safety research and governance.
- Interpretability research remains valuable but will be directed at localized, safety-critical gaps (blind spots, failure modes) rather than total model explanation.
-
Market Design, Contracts, and Liability
- Contracting must handle hidden internal states and unverifiable claims about model intent/causality. Contracts will emphasize performance metrics, monitoring, and ex post remedies rather than ex ante proofs of safe internal structure.
- Insurance markets will struggle with pricing and coverage due to tail risks from opaque capabilities; risk premia and stricter underwriting will be required, or insurers may exclude certain classes of deployments.
- Liability regimes may shift toward strict product/operational controls and mandatory containment standards rather than relying on proving model intent or internal decision chains.
-
Regulation and Governance
- Alignment (instruction-based) interventions are weaker than environmental/permission-based controls. Economically, regulatory design should prioritize:
- Access controls (who can run models at what scale).
- Capability gating (staged deployment, safety testing in sandboxed environments).
- Operational containment (permission systems, physical/technical isolation for high-risk uses).
- Regulatory instruments may include licensing, compute quotas, multi-party oversight for high-impact deployments, and mandatory red-teaming and robust monitoring.
- Because valuable capabilities are unexplainable, disclosure and audit requirements must focus on behavior under stress/testing rather than full internal transparency.
- Alignment (instruction-based) interventions are weaker than environmental/permission-based controls. Economically, regulatory design should prioritize:
-
Labor, Skills, and Complementarities
- LLMs can capture tacit, high-dimensional skills previously thought hard to codify. This expands the set of tasks susceptible to automation, potentially accelerating displacement in cognitive and skilled occupations.
- Complementarity shifts: value will accrue to workers and firms able to integrate, supervise, and apply opaque model outputs (interpretation, oversight, system design). New scarce skills include model orchestration, red-teaming, and operational containment.
- Productivity measurement challenges: standard metrics (output per worker) may under- or over-estimate economic effects when model contributions are opaque, complicating policy responses.
-
Externalities, Public Goods, and Precaution
- Opaque capabilities generate systemic externalities and tail risks (misuse, amplification, unpredictable failure modes). Market mechanisms alone may not internalize these risks, arguing for precautionary public intervention.
- Public investment is warranted in: robust safety testing infrastructure, interpretability for safety-critical contexts, governance institutions, and mechanisms to reduce asymmetric concentration (e.g., compute governance, shared benchmarks).
-
Pricing, Access, and Scarcity as Control Mechanisms
- Given alignment limits, economic control levers—pricing, tiered access, compute allocation, and licensing—become tools to limit misuse and concentration.
- Governments and platforms may enforce scarcity (e.g., restricted high-capability APIs, mandatory vetting) as a practical substitute for full explainability.
Overall takeaway for AI economics: the paper suggests reorienting incentives, policy, and measurement away from hope for full internal transparency and toward robust external controls, governance of access, and investment in localized interpretability/safety work. Market structures and regulatory frameworks should explicitly account for the asymmetries and tail risks introduced by valuable but intrinsically unexplainable capabilities.
Assessment
Paper Typetheoretical
Evidence Strengthlow — The paper is primarily a conceptual proof-by-contradiction supported by historical case discussion and philosophical framing rather than empirical testing; it proposes plausible mechanisms but provides no primary quantitative evidence or causal estimates.
Methods Rigormedium — The logical argument is clearly articulated and grounded in historical and philosophical literature, and the paper carefully links implications to economic outcomes; however, the rigor is limited by reliance on analogy (expert systems), informal empirical claims about LLM performance, and lack of formal models or empirical validation.
SampleNo primary dataset; the analysis is conceptual and theoretical, supported by historical case analysis of rule‑based expert systems and qualitative examples of LLM behavior, plus references to observed empirical successes of LLMs in the literature; suggests possible empirical tests (benchmarks, distillation studies) but does not implement them.
Themesinnovation governance labor_markets human_ai_collab adoption org_design
GeneralizabilityArgument is conceptual and may not hold across different model architectures, training regimes, or future hybrid systems that combine rules and learned components., Relies on historical analogies (expert systems) that may be imperfect predictors of LLM evolution or of future interpretability advances., Claims about 'irreducible' tacit capabilities are broad and may vary substantially across tasks, industries, and levels of task formalizability., Policy and market implications depend on institutional contexts (regulatory regimes, contracting norms, competition policy) that differ across countries and sectors., Does not quantify magnitudes—practical economic impacts (e.g., rents, concentration) may be smaller or more localized than the theory suggests.
Claims (20)
| Claim | Direction | Confidence | Outcome | Details |
|---|---|---|---|---|
| The economically valuable capabilities of large language models are precisely those that cannot be fully encoded as a complete, human‑readable set of discrete rules. Firm Productivity | positive | medium | economic value / capability of LLMs (degree of rule‑encodability vs tacitness) |
0.04
|
| If an LLM's full capability were reducible to an explicit rule set, that rule set would be an expert system; because expert systems are empirically and historically weaker than LLMs, this leads to a contradiction (supporting non‑rule‑encodability). Other | mixed | high | logical consistency of the reducibility-to-rules claim (validity of the contradiction argument) |
0.06
|
| Historical expert systems failed to generalize or scale to complex, ambiguous tasks, contrasting with LLMs' broader empirical successes. Research Productivity | negative | medium | generalization and scalability of rule‑based expert systems |
0.04
|
| Some LLM capabilities are tacit, practice‑derived, or 'insight'‑like, akin to the Chinese concept of Wu (sudden insight through practiced skill). Other | positive | medium | characterization of LLM competence as tacit/insight-like |
0.04
|
| There is a structural mismatch between explicit human cognitive tools (rules, checklists) and the pattern‑rich, high‑dimensional competence encoded in LLMs. Other | negative | medium | alignment/mismatch between human‑readable rules and LLM representations/competence |
0.04
|
| Full interpretability via rule extraction may be impossible for the most valuable parts of LLM competence, limiting the utility of some transparency approaches for safety and auditing. Ai Safety And Ethics | negative | medium | feasibility of fully extracting human‑readable rules from LLMs (interpretability) |
0.04
|
| The paper challenges a purely rule‑based view of scientific explanation: some explanatory power will remain in implicit model structure rather than explicit rules. Other | mixed | medium | completeness of rule‑based scientific explanations when applied to LLM behavior |
0.04
|
| The paper's core methodological approach is conceptual and theoretical argumentation (formal/logical proof, historical examples, and philosophical framing), not empirical experimentation. Other | null_result | high | presence/absence of empirical experiments in the paper |
0.06
|
| The paper contains mostly qualitative and historically grounded empirical content and reports no primary datasets or large‑scale experimental results in support of the formal thesis. Other | null_result | high | extent of empirical/quantitative evidence presented |
0.06
|
| Suggested empirical validations (not performed) include benchmarking LLMs versus rule systems on allegedly rule‑encodable tasks, attempting rule extraction and measuring fidelity loss, and compression/distillation studies to quantify irreducible task performance. Other | null_result | high | types of empirical tests recommended for validating the thesis |
0.06
|
| The highest‑value attributes of LLMs may be inherently non‑decomposable into simple, auditable rules, which increases the value of proprietary, black‑box models and strengthens economies of scale and scope for large model providers. Market Structure | positive | medium | value capture by model providers (proprietary rents/economies of scale) |
0.04
|
| Commoditization via rule extraction is limited; firms that can harness and deploy tacit LLM capabilities will retain economic rents. Firm Revenue | positive | medium | ability to commoditize/replicate LLM capabilities via rule extraction |
0.04
|
| Barriers to entry may be larger for tacit‑capability‑driven systems than for rule‑based systems, potentially increasing market concentration. Market Structure | positive | speculative | market concentration / barriers to entry |
0.01
|
| Standard contracts and regulatory audits that rely on inspection of rule sets or source code will be insufficient to assess model behavior or risk; regulators and buyers must rely more on behavior‑based testing, standards, and outcome measures. Governance And Regulation | negative | medium | effectiveness of rule‑based audits/regulatory inspections for assessing model risk |
0.04
|
| LLMs are more likely to complement human tacit skills than to replace explicit rule‑following jobs; value accrues to workers and firms that integrate model outputs with human judgment and tacit expertise. Employment | positive | medium | complementarity vs substitution of human labor (especially tacit-skill jobs) |
0.04
|
| Investments in interpretability that aim to fully 'rule‑ify' LLM competence may have diminishing returns; economic value may be better captured by research into robust behavioral evaluation, stress testing, and hybrid human‑AI workflows, while partial interpretability remains valuable. Ai Safety And Ethics | mixed | medium | returns to different types of interpretability/AI safety R&D |
0.04
|
| New metrics are needed to value tacit capabilities — e.g., measures of transfer, generalization under distribution shifts, ease of integrating with human workflows, and irreducibility to compressed rule representations. Research Productivity | null_result | high | proposed metrics for assessing tacit LLM capabilities |
0.06
|
| Public‑interest concerns (bias, misuse, systemic risk) may be harder to mitigate via simple transparency rules; policies should emphasize outcome‑based regulations, mandatory behavioral testing, and marketplace disclosure obligations for stressed scenarios. Governance And Regulation | negative | medium | effectiveness of transparency-based vs outcome-based regulatory approaches |
0.04
|
| Suggested empirical research directions for AI economists include: comparing LLM performance and economic outcomes on rule‑encodable vs tacit tasks; quantifying performance decline when forcing LLMs into interpretable rule representations; studying contracting/pricing where buyers cannot verify internal rules; and measuring returns to scale attributable to tacit capabilities. Research Productivity | null_result | high | proposed empirical research topics and corresponding outcomes to measure |
0.06
|
| The paper is primarily theoretical and historical; empirical validation is needed to quantify the irreducible component of LLM value, and practical degrees of rule‑extractability may exist even if some capabilities remain tacit. Research Productivity | mixed | high | need for empirical validation and degree of rule‑extractability of LLM capabilities |
0.06
|