Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge

Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants. General purpose benchmarks do not measure whether a model reasons safely and correctly about aviation specific operational knowledge, and the high stakes, regulated nature of the domain makes that gap consequential. We present Pre-Flight, an open source benchmark of 300 multiple choice questions drawn from international standards and airport ground operations material, covering international airport ground operations, ICAO and US FAA regulations, aviation general knowledge and complex operational scenarios. Questions were authored and reviewed by practitioners with experience in air traffic management, ground operations and commercial flying. We evaluate a range of contemporary commercial and open weight models using the Inspect evaluation framework, scoring by accuracy under a standard multiple choice protocol, and we maintain the leaderboard on a rolling basis as new models are released. Against an informal expert reference of around 95%, obtained from a low sample quiz of aviation professionals at a conference, even the strongest model evaluated (released in 2026) reaches 82.7%, having improved only gradually from roughly 75% in early 2025. A substantial and persistent gap below expert level reliability therefore remains. We release the dataset, the evaluation harness and the results, and the benchmark is available within the community evaluations package distributed with inspect_evals. We argue that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.

Summary

Main Finding

Pre-Flight, a 300-item multiple-choice benchmark of aviation operational knowledge, shows that state-of-the-art LLMs (snapshot 29 June 2026) reach a top accuracy of 82.7% (GPT-5.5) versus an informal expert reference ≈95%. Progress since early 2024 has been gradual, a persistent ~10–15 percentage point gap to expert-level reliability remains, and open-weight models are now competitive with commercial APIs at the frontier. The authors argue domain-specific evaluation like Pre-Flight is a necessary precondition for responsible deployment in non‑safety‑critical aviation operations.

Key Points

Benchmark scope and purpose
- Targets non-safety-critical aviation business/operational knowledge: international airport ground ops, ICAO and FAA regulations, dispatch rules, and operational scenarios.
- Designed to evaluate whether LLMs reason safely and correctly about aviation operational knowledge rather than just produce plausible text.
Dataset composition
- 300 multiple-choice questions (4–5 options, single correct answer).
- Category breakdown: International airport ground operations 152 (50.7%), ICAO rules 85 (28.3%), FAA rules 51 (17.0%), aviation trivia 8 (2.7%), complex ground scenarios 4 (1.3%).
- Items derived from authoritative sources (Ground Operations Safety Manuals, ICAO Annexes, 14 CFR), authored/reviewed by experienced practitioners.
- Public release (easy tier) on Hugging Face (MIT license); a harder withheld tier is being held back to avoid contamination.
Evaluation protocol
- Runs in the Inspect framework (inspect_evals), zero-shot multiple-choice prompting, deterministic exact-match scoring, single-pass evaluation (n=300).
- Rolling public leaderboard; snapshot on 29 June 2026 evaluates 44 models (13 earlier models shown at original scores).
Aggregate results and trends
- Top models (snapshot): GPT-5.5 82.7%, GPT-5 80.3%, Claude Opus 4.8 79.0%, Gemini 2.5 Pro 79.0%, Qwen3.5 quantised local 77.3%.
- Frontier rose from mid-60s (early 2024) to ~83% (mid-2026) — steady but slow improvement.
- Open-weight models (quantised local runs) are competitive at the top ranks.
- Notable outlier: ALLaM 2 7B scored ~25% (near chance).
Item-level and failure patterns
- Micro-averaged failure rate across items ≈28.4%.
- US (FAA) regulation items are the hardest (mean failure ≈53%), ICAO and international ops easier (≈22–24% failure).
- 60.3% of items require reasoning (vs recall); reasoning items show higher failure (30.1% vs 25.9%).
- Some benchmark keys contained errors; consensus model disagreement helped surface these.
Qualitative failure modes
- Pattern‑matching over reasoning (models reproduce familiar solutions rather than reason from problem statement).
- Violations of physical/temporal constraints in scheduling tasks.
- Complex visual reasoning and multi-step diagram interpretation are weak.
- Overconfidence/calibration gaps — models may be confident while incorrect.

Data & Methods

Dataset
- 300 MCQs in JSONL with unique IDs, stems, 4–5 options, single ground-truth option grounded in source docs.
- Authored by aviation practitioners; partial manual validation by two lead reviewers (~25 years’ experience each), subset SME review; no formal inter-annotator agreement statistic.
- Public easy-tier release (AirsideLabs/pre-flight-06), harder tier withheld.
Experimental setup
- Inspect framework paired with the standard multiple_choice solver and choice scorer; deterministic exact-match accuracy is reported.
- Zero-shot prompts using default templates; models evaluated once (epochs=1). Standard error for n=300 ≈ 0.025 (binomial SE).
- Models: 44 entries in leaderboard (commercial APIs and locally-run open-weight models on DGX Spark). Snapshot uses release dates (not evaluation dates).
Key empirical numbers (snapshot 29 June 2026)
- Top accuracy: GPT-5.5 82.7% (SE 0.022).
- Top 6: GPT-5.5 82.7; GPT-5 80.3; Claude Opus 4.8 79.0; Gemini 2.5 Pro 79.0; Qwen3.5 quantised local 77.3; GPT-5.1 77.0.
- Item difficulty: 9 items missed by every model; 76% of items were missed by fewer than half the models.
Limitations noted by authors
- Partial validation of keys, small low-n informal expert reference (≈95%), withheld hard tier (to avoid contamination), and dataset currently limited to text-only MCQs (visual probes are illustrative only).

Implications for AI Economics

Deployment economics and value propositions
- Current performance suggests LLMs can augment non-safety-critical aviation operations (document drafting, checklists, training support) but are not yet substitutes for human experts; the persistent 10–15pp accuracy gap implies continued human oversight and verification costs.
- Cost savings from automation must be weighed against error remediation, compliance, and reputational risk; models’ calibration issues increase verification burden.
- Open-weight models being competitive can lower model licensing/runtime costs, shifting economic emphasis to evaluation, fine-tuning, retrieval/grounding and operational integration overheads.
Market and investment signals
- Demand for domain-tailored models, retrieval-augmented systems, tool-augmented agents, and robust evaluation services is likely to grow.
- Vendors and integrators who provide domain-grounding, retrieval pipelines, and rigorous benchmarking/certification capabilities may capture higher value than generic LLM providers.
Regulation, liability, and adoption pace
- Aviation’s regulated environment and high-cost errors mean stricter predeployment evaluation and monitoring will be required; this raises compliance costs and slows adoption relative to less regulated sectors.
- Liability and certification frameworks will materially affect ROI calculations — firms will pay for assurance, audit trails, and conservative deployment boundaries (non-safety-critical tasks first).
Research and productivity tradeoffs
- Because reasoning items and US-regulation detail are the hardest, investments in retrieval-augmentation, fine-grained domain data curation, and hybrid human–AI workflows are economically justified to improve effective reliability.
- The benchmark shows diminishing returns from general-model scaling alone in domain competence; targeted data and architectural investments may be more cost-effective.
Policy and procurement recommendations
- Procurement and cost-benefit models should require domain-specific benchmark performance (e.g., Pre-Flight) as a gating criterion for deployment in aviation operations.
- Regulators and industry bodies could standardise domain benchmarks and metrics to reduce asymmetric information and procurement risk.
- Subsidising creation of contamination-resistant test tiers and third-party evaluation services could improve market confidence and reduce systemic risk.
Externalities and competition
- Public benchmarks accelerate transparency and competition, but contamination risk and benchmark-gaming create externalities; withheld harder tiers and private testbeds may be economically efficient to preserve discriminative power.
- Standardised aviation benchmarks may shift R&D investment from general-purpose scaling to domain-specialisation and evaluation tooling.

Suggested economist actions based on the paper - Include domain-benchmark performance and uncertainty estimates in cost-benefit and deployment models for aviation AI projects. - Value investments in retrieval, grounding, and human-in-the-loop verification explicitly; do not assume parametric LLM improvements alone will close the reliability gap. - Consider procurement rules that prioritise audited domain evaluation results and fund third‑party evaluation infrastructure to reduce search and verification costs.

References to the primary artifact - Pre-Flight dataset and code: AirsideLabs/pre-flight-06 on Hugging Face; included in inspect_evals community package. Snapshot and leaderboard reported in the paper (29 June 2026).

Assessment

Paper Typedescriptive Evidence Strengthn/a — This is a descriptive benchmark and model evaluation rather than a causal inference study; it does not attempt to identify causal effects on economic outcomes. Methods Rigormedium — The benchmark is carefully constructed (300 multiple-choice questions drawn from ICAO/FAA standards and operations material) and questions were authored and reviewed by domain practitioners, and multiple contemporary models were evaluated with a consistent protocol; however, the expert reference is informal and small-sample, the item set is limited in size and scope, potential selection and framing biases in question design are not fully documented, and there is limited information on inter-rater reliability or adversarial robustness testing. SampleA 300-question multiple-choice benchmark (Pre-Flight) derived from international airport ground operations material, ICAO and US FAA regulations, general aviation knowledge, and complex operational scenarios; questions authored and reviewed by practitioners in air traffic management, ground operations and commercial flying; evaluated across a range of commercial and open-weight LLMs (models released 2025–2026) using an Inspect evaluation framework and standard multiple-choice scoring; an informal expert reference score (~95%) comes from a low-sample quiz of aviation professionals at a conference. Themesgovernance adoption human_ai_collab GeneralizabilityLimited to airport ground operations, ICAO/FAA regulations and related scenarios—may not generalize to other aviation subdomains (e.g., in-flight procedures, air traffic control at scale), Multiple-choice format may not reflect real-world interactive, multi-step, or context-rich operational tasks, Question selection and authorship by a limited set of practitioners may introduce topical or regional bias, Expert reference is informal and small-sample, limiting confidence in the exact expert ceiling, Results are time-dependent and model-version dependent; leaderboard outcomes may change rapidly as models update, Language and cultural/regulatory emphasis unclear (possible English/ICAO/US bias)

Claims (11)

Claim	Direction	Outcome	Confidence & Evidence	Details
Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants. Adoption Rate	positive	proposals/applications of LLMs for aviation business operations (qualitative prevalence)	Reading fidelity high Study strength speculative	not reported 0.03
General purpose benchmarks do not measure whether a model reasons safely and correctly about aviation specific operational knowledge. Ai Safety And Ethics	negative	coverage of aviation-specific operational reasoning by general-purpose benchmarks	Reading fidelity high Study strength speculative	not reported 0.03
The high stakes, regulated nature of the aviation domain makes the gap (in domain-specific evaluation) consequential. Governance And Regulation	negative	consequential risk from lack of domain-specific evaluation	Reading fidelity high Study strength speculative	not reported 0.03
We present Pre-Flight, an open source benchmark of 300 multiple choice questions drawn from international standards and airport ground operations material, covering international airport ground operations, ICAO and US FAA regulations, aviation general knowledge and complex operational scenarios. Research Productivity	positive	creation of a domain-specific benchmark (300 multiple choice questions)	Reading fidelity high Study strength high	n=300 0.3
Questions were authored and reviewed by practitioners with experience in air traffic management, ground operations and commercial flying. Research Productivity	positive	authorship/review provenance of benchmark questions (practitioner involvement)	Reading fidelity high Study strength medium	not reported 0.18
We evaluate a range of contemporary commercial and open weight models using the Inspect evaluation framework, scoring by accuracy under a standard multiple choice protocol, and we maintain the leaderboard on a rolling basis as new models are released. Research Productivity	positive	model accuracy on Pre-Flight under a standard multiple choice protocol; availability of rolling leaderboard	Reading fidelity high Study strength medium	not reported 0.18
Against an informal expert reference of around 95%, obtained from a low sample quiz of aviation professionals at a conference, even the strongest model evaluated (released in 2026) reaches 82.7%. Decision Quality	negative	accuracy on Pre-Flight benchmark (multiple choice)	Reading fidelity high Study strength medium	n=300 82.7% accuracy (model); ~95% (informal expert reference) 0.18
The strongest model's performance improved only gradually from roughly 75% in early 2025 to 82.7% (released in 2026). Decision Quality	positive	change in model accuracy on Pre-Flight over time	Reading fidelity high Study strength medium	n=300 improved from roughly 75% to 82.7% 0.18
A substantial and persistent gap below expert level reliability therefore remains. Decision Quality	negative	gap between model accuracy and expert accuracy on domain-specific tasks	Reading fidelity high Study strength medium	n=300 0.18
We release the dataset, the evaluation harness and the results, and the benchmark is available within the community evaluations package distributed with inspect_evals. Research Productivity	positive	public release and availability of dataset and evaluation tools	Reading fidelity high Study strength high	n=300 0.3
Domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations. Governance And Regulation	positive	policy/practice requirement for responsible deployment	Reading fidelity high Study strength speculative	not reported 0.03

LLMs remain short of expert reliability in airport operations: the best models score 82.7% on a 300-question aviation benchmark versus roughly 95% for practitioners, highlighting a persistent, safety-relevant performance gap and the need for domain-specific evaluation before deployment.