Large language models are getting better at functional programming but still struggle: Haskell and OCaml see far higher error rates and non‑idiomatic, imperative-style outputs than Scala or Java, and static analysis feedback helps only partially, raising concerns about long-term maintainability.

Main Finding

LLMs have meaningfully improved at generating functionally correct code in functional programming (FP) languages as model scale advances, but substantial gaps remain versus imperative languages. Advanced models (GPT-5) increase pass@1 roughly threefold over GPT-3.5, yet produce more non-idiomatic, lower-maintainability FP code — a trade-off that can increase technical debt. Static-analysis feedback plus hand-crafted instructions can partially enable LLM self-repair for both correctness and quality.

Key Points

• Benchmark and scope
  • FPBench: 721 algorithmic tasks from LeetCode (184 easy, 346 medium, 191 hard).
  • Languages: Haskell, OCaml, Scala (FP languages) and Java (imperative baseline).
• LLMs evaluated (zero-shot, one sample per task): GPT-3.5, GPT-4o, GPT-5.
• Functional correctness (pass@1) improves with model capability:
  • GPT-3.5: Haskell 14.15%, OCaml 9.43%, Scala 19.28%, Java 22.19%.
  • GPT-4o: Haskell 27.18%, OCaml 36.20%, Scala 38.83%, Java 43.69%.
  • GPT-5: Haskell 42.34%, OCaml 52.16%, Scala 58.36%, Java 61.14%.
• Compilation errors and failures
  • Higher compilation/type-error rates in FP languages vs Java; e.g., GPT-5 compilation errors: Haskell 24.28%, OCaml 13.73%, Scala 9.27%, Java 4.62%.
  • Test failures remain a meaningful share even in best models.
• Code style and maintainability
  • “Clean” (no static-analysis violations) proportion among functionally-correct outputs drops for GPT-5 vs GPT-4o in many cases:
    • Haskell: GPT-4o ~46% clean, GPT-5 ~45%.
    • OCaml: GPT-3.5/4o ~61–63% clean, GPT-5 ~44% clean.
    • Scala: GPT-4o ~77% clean → GPT-5 ~57% clean.
    • Java: GPT-4o ~63% → GPT-5 ~51%.
  • Common issues: non-idiomatic imperative patterns (e.g., return, mutable vars), unnecessary lambdas/eta-expansions, redundant parens/semicolons, unused variables, non-exhaustive patterns.
• Imperative bias and “reward hacking”
  • Models often favor imperative-style solutions in FP languages, apparently optimizing for passing tests at the expense of idiomatic functional style.
  • Higher correctness sometimes correlates with worse non-functional quality.
• Self-repair
  • Providing static-analysis feedback and targeted instructions enables partial automatic repair of correctness and style issues.

Data & Methods

• Data collection
  • Tasks sourced from public LeetCode content (2021–2025), excluding premium problems.
  • Starter templates for Haskell and OCaml were generated by a transpiler that converts Python templates into typed FP templates, inferring types and normalizing I/O semantics.
  • Private test cases synthesized (via GPT-4o) to augment public tests and cover edge/boundary cases.
• Evaluation pipeline
  • Execution: Docker-based isolated runtimes and language-specific test harnesses to compile/run FP code.
  • Functional correctness metric: pass@1 (single generated candidate must pass full test suite).
  • Failure taxonomy: Compilation Errors, Test Failures, Timeouts.
  • Style/maintainability: static analysis tools — HLint & GHC for Haskell, ocamlformat & dune for OCaml, Scalastyle for Scala, Checkstyle/PMD for Java.
  • “Clean” code defined as no static-analysis violations for a given solution.
• LLM configuration
  • Zero-shot prompting with natural-language task description + language starter template.
  • GPT-3.5/GPT-4o temperature = 0.7; GPT-5 used default (1.0) with reasoning disabled; max_tokens = 2048; single-shot generation (one candidate).
• Analysis
  • Quantitative: aggregate pass rates, error-type distributions, clean-code proportions.
  • Qualitative: categorization of recurring style violations and imperative patterns via static-analysis logs and pattern matching.

Implications for AI Economics

• Productivity vs. technical debt trade-offs
  • Short-term productivity gains (higher pass@1) from advanced LLMs may be offset by longer-term maintenance costs because generated FP code can be non-idiomatic and harder to maintain. Firms should weigh immediate development speed against future debugging and refactoring costs.
• Labor and skill composition
  • LLMs can lower entry barriers to FP by helping novices achieve correct solutions more quickly, potentially expanding FP adoption. However, reliance on LLM outputs that embody imperative bias may reduce incentives to develop idiomatic FP expertise among engineers, altering skill premium dynamics.
• Incentives for LLM developers and buyers
  • Current evaluation incentives (test-passing) may induce “reward hacking” where models prioritize passing tests over generating maintainable code. Economic actors (platforms, enterprises) should invest in training/evaluating models with multi-objective reward signals that include style/maintainability metrics to reduce downstream costs.
• Market for domain-specific models and tools
  • There is likely demand (and economic rent) for FP-specialized models or fine-tuning services that reduce compilation/type errors and produce idiomatic FP code. Similarly, integrated toolchains that combine LLM generation with static-analysis-guided repair workflows will have commercial value.
• Data and model investment priorities
  • The persistent performance gap in FP languages suggests underrepresentation of idiomatic FP code in model training corpora. Economic decisions about data acquisition and curation (e.g., licensing high-quality FP repositories, incorporating textbooks/idiomatic examples) can yield higher returns for FP code generation capabilities.
• Risk, security, and externalities
  • Non-idiomatic FP code (e.g., unnecessary mutability) can introduce reliability or security vulnerabilities in systems expected to be deterministic or formally verifiable. This imposes negative externalities that organizations and regulators may need to account for when endorsing LLM-assisted development.
• Cost-benefit for organizations adopting LLM-assisted FP development
  • Small teams and startups may benefit most from immediate correctness gains, whereas large or safety-critical organizations should invest in tooling and human oversight to enforce idiomatic style and avoid accumulating technical debt, impacting procurement and process design.
• Evaluation and procurement metrics
  • Procurement and benchmarking practices should include multi-dimensional metrics (functional correctness, compilation robustness, static-analysis adherence, maintainability) rather than only test-passing rates to align vendor incentives with buyer welfare.

Limitations to consider when applying these findings - FPBench is LeetCode-derived and algorithmic; results may differ on real-world codebases or domain-specific FP code. - Zero-shot, single-candidate setup understates potential gains from few-shot prompting, multiple samples, or fine-tuning. - GPT-5 configuration differed (temperature and reasoning disabled) which may affect direct comparability. - Static-analysis “clean” thresholds are tool-dependent and may not capture all aspects of maintainability.

Takeaway LLMs are materially better at producing correct FP solutions than earlier generations, but economic choices around adoption should account for increased risk of non-idiomatic, lower-quality outputs. Investing in specialized data, multi-objective training/evaluation, and integrated repair tooling can improve long-term value and reduce maintenance externalities.