Counterfactual explanations—popular tools for explaining and reversing AI decisions—depend heavily on upstream design choices; how an organization measures features, labels outcomes, validates models and defines success can change the recommended recourse as much as switching the explanation method itself, meaning firms must disclose and justify these upstream choices alongside any counterfactuals.

Main Finding

Counterfactual explanations (CEs) are sensitive to contestable, upstream choices in the ML pipeline (e.g., how features and labels are measured, business requirements, validation criteria, and success metrics). These upstream interventions can change generated counterfactuals as much as—or more than—changing the CE algorithm itself. As a result, presenting CEs as standalone justifications or as reliable recourse recommendations is normatively fragile: CEs routinely omit the organizational decisions that produced both the model and the candidate counterfactuals, and therefore cannot by themselves answer important “why” questions about whether a decision-maker ought to have acted differently.

Key Points

• Role of CEs: CEs answer “why” by proposing minimal changes to inputs that would flip a model’s output; they are widely used for debugging, local explanation, justification, and algorithmic recourse.
• Four desiderata for CEs commonly evaluated: validity (cross the decision boundary), minimality/sparsity, actionability (avoid immutable or sensitive features), and robustness (stable under small model/data changes).
• Rashomon effect and multiplicity: multiple plausible CEs typically exist for the same decision; different CE methods often disagree substantially.
• Upstream sensitivity: choices made upstream—measurement models for features and labels, labeling mechanisms, business requirements, model validation protocols, and the metric used to operationalize model fitness—are meaningful, contestable, and materially alter the CEs produced downstream.
• Normative implication: when CEs are used for justification or recourse, they implicitly assume that upstream design and governance choices are fixed and unproblematic. This masks organizational responsibility and can mislead subjects seeking recourse.
• Modal robustness: recourse should be valuable across a range of possible circumstances (including model updates, label changes, and varying business rules). CEs that ignore upstream variation may fail this standard.
• Practical consequence: ensuring technical robustness of CEs is necessary but not sufficient; broader transparency about pipeline assumptions and mechanisms of choice is required.

Data & Methods

• Conceptual framing: the authors model the ML pipeline as a structural causal model (SCM) and treat “upstream choices” (e.g., observation/measurement models, labeling rules, choice of evaluation metric, business constraints) as interventions on that SCM. This formalization links managerial/organizational choices to downstream model behavior and generated explanations.
• Empirical approach: they run four empirical experiments that intervene at different upstream stages of the pipeline and measure effects on generated counterfactual explanations. A running example is resume screening / candidate fitness scoring (motivated by real-world hiring automation practices).
• Counterfactual generation: CEs are operationalized as standard optimization problems balancing change in model output and distance from the original instance (arg min L(F(x),F(x')) + λ d(x,x')), subject to actionability and validity constraints. The paper uses this standard formulation to generate and compare CEs under different upstream interventions.
• Comparative analysis: the authors compare variation in CEs induced by upstream interventions to variation induced by changing CE generation methods, and quantify how upstream changes can produce differences of comparable magnitude to method changes.
• Limitations (as reported): the precise experimental datasets and low-level implementation details are tailored to the paper’s illustrative resume-screening setup; the main goal is to reveal sensitivity patterns rather than to advocate a single prescriptive technical fix.

Implications for AI Economics

• Firm incentives and strategic choice: organizations can (intentionally or not) steer counterfactuals by selecting measurement models, labels, evaluation metrics, or business objectives. This creates an informational asymmetry: firms can shape the apparent reasons for decisions in ways that favor their objectives, with potential market power or reputational consequences.
• Regulatory design and compliance costs: laws that require “meaningful information about the logic involved” (e.g., GDPR, AI Act) cannot be satisfied by surface-level CEs alone. Regulators and firms will need to specify requirements for disclosure of upstream choices, increasing compliance, auditing, and documentation costs that should be internalized in economic models of AI adoption.
• Recourse as an economic good: reliable recourse supports planning, investment in human capital, and labor-market mobility. If CEs are unstable due to upstream variability, individuals face higher uncertainty and reduced ability to act on recourse recommendations—this can reduce welfare and distort behavior (e.g., suboptimal investments to chase unstable signals).
• Market for “explainability services”: demand will rise for third-party audits, provenance tools, and governance frameworks that reveal upstream choices. This can create new industries (and rents) around certification, audits, and transparent measurement design.
• Competition and adverse selection: firms that choose measurement and labeling rules strategically may attract or deter certain applicants/customers; opaque upstream choices increase adverse selection risk for affected users and for platforms relying on those users.
• Accountability and liability: economics of liability shift. If CEs cannot, by themselves, justify decisions, then legal and reputational liability will hinge on organizations’ upstream decisions (measurement, labels, metrics). Insurers, courts, and regulators will require evidence about these choices—affecting insurance premiums, expected litigation costs, and firm behavior.
• Policy prescriptions with economic effects:
  • Mandate disclosure of upstream design choices relevant to high-stakes models (increasing transparency but raising compliance costs).
  • Require audits that evaluate whether recourse is modally robust across plausible upstream variations.
  • Incentivize use of interpretable models where feasible (reduces need for complex post-hoc explanations).
  • Support standards for actionable, stable recourse (reduces welfare loss from unreliable recommendations).
• Broader modeling implication: economic analyses of AI-driven decision systems should incorporate governance, measurement choice, and model lifecycle risk into cost–benefit calculations. Modeling “explainability” as a single attribute is insufficient; it should be decomposed into algorithmic explainability plus upstream transparency and contestability.

Summary takeaway: CEs are a useful technical tool, but their normative and economic reliability for justification and recourse depends crucially on upstream, organizational choices. Policymakers, economists, and firms should treat explainability not only as a technical deliverable but as a socio-technical bundle that requires disclosure, governance, and accountability upstream of the explanation itself.