Large language models drastically reduce false alarms in enterprise static-analysis: hybrid LLM+SAT methods remove 94–98% of false positives in Tencent’s advertising codebase and cut per-alarm review from 10–20 minutes to seconds at negligible cost. Results are promising for developer productivity but are drawn from one company, one customized tool and three bug types, so broader applicability remains to be tested.

Main Finding

Large language models (LLMs), especially when combined with static-analysis-derived information (hybrid approaches), can dramatically reduce false positives from enterprise static analysis: in a Tencent production dataset of 433 alarms, hybrid LLM+static methods eliminated roughly 94–98% of false positives while maintaining high recall, at very low per-alarm time and monetary cost (2.1–109.5 seconds and $0.0011–$0.12 per alarm), producing orders-of-magnitude savings over manual review.

Key Points

• Dataset and prevalence
  • Real-world enterprise dataset from Tencent’s Advertising & Marketing Services: 433 alarms (328 false positives, 105 true positives) across three bug types.
  • Bug-type breakdown: NPD (Null Pointer Deref) 136 total (107 FP / 29 TP), DBZ (Divide-by-Zero) 158 (139 FP / 19 TP), OOB (Out-of-Bounds) 139 (82 FP / 57 TP).
  • Overall false-positive rate ≈ 76%; manual inspection averages 10–20 minutes per alarm.
• Techniques evaluated
  • Wide range: baseline supervised learning models, vanilla LLMs, advanced prompting (few-shot, chain-of-thought, etc.), and hybrid approaches that feed static-analysis outputs/path constraints into LLM reasoning (e.g., LLMPFA-like methods).
• Performance highlights
  • Hybrid LLM+static-analysis methods eliminated 94–98% of false positives across backbone LLMs while maintaining high recall (i.e., few missed true bugs).
  • LLM-based techniques outperform traditional supervised learning approaches in this industrial setting (which often lack large labeled corpora).
  • Effectiveness varies by bug type: best results on DBZ, worst on NPD.
• Costs
  • Time per alarm for LLM-based processing: 2.1–109.5 seconds (depending on method and model).
  • Monetary cost per alarm: $0.0011–$0.12 (depending on model / provider and prompt size).
  • These costs are tiny compared to manual-review labor (10–20 minutes), implying large per-alarm and aggregate savings.
• Limitations observed
  • LLMs struggle with very long code contexts, complex cascaded constraints, and deep semantic reasoning; these cause remaining false positives/false negatives.
  • Dataset confidentiality prevented public release; replicability is supported via provided methods/replication package.

Data & Methods

• Data collection
  • Source: Tencent’s in-house static analyzer BkCheck applied to AMS codebases (Sept 2024 – Jun 2025).
  • Workflow: automated scanning → two-round manual review (developer review + reviewer validation) → extraction and filtering of bug-report forms.
  • Filtering removed deprecated code, incomplete contexts, unresolved decisions, and duplicates.
• Dataset composition
  • 433 validated instances: 328 false positives, 105 true positives; three prioritized categories (NPD, OOB, DBZ) covering 86% of warnings.
• Evaluated methods
  • Supervised learning baselines (CNN, Transformer classifiers trained on warning+code features).
  • LLM approaches: direct classification prompts, few-shot and chain-of-thought prompts, retrieval/RAG-style augmentation, and hybrid methods that encode static-analysis path constraints or summarized contexts into LLM prompts.
• Metrics and evaluation
  • Primary metrics: false-positive elimination rate (reduction), recall on true positives, per-alarm latency and monetary cost.
  • Qualitative case analysis to identify failure modes (long context, chained constraints, semantic gaps).

Implications for AI Economics

• Labor cost reduction and ROI
  • Replacing or pre-filtering manual review (10–20 minutes per alarm) with LLM-based triage yields large labor savings. Example: at $50/hr developer cost, 10 minutes ≈ $8.3 saved vs. LLM cost ≤ $0.12 → ~69x cost ratio; with senior rates the multiple is larger. Even conservative estimates show substantial per-alarm ROI.
• Scalability & marginal costs
  • Very low per-alarm monetary and time costs imply scalable deployment across large codebases and frequent CI scans; marginal cost of handling extra alarms is minimal relative to manual review.
• Allocation of human capital
  • Developers/reviewers can be reallocated from routine alarm triage to higher-value tasks (complex reviews, fixes, architectural work), boosting productivity and potentially accelerating delivery cycles.
• Investment trade-offs
  • Short-term: integrate LLM prompts/hybrid pipelines into CI to realize quick savings.
  • Medium-term: consider private/on-prem LLMs for data privacy and predictable costs—this entails infrastructure, fine-tuning, and maintenance expenses that must be weighed against recurring API costs and privacy risk.
• Risk and governance costs
  • False negatives (missed real bugs) have asymmetric downstream costs; enterprises must preserve high recall and invest in monitoring, human-in-the-loop fail-safes, and coverage testing.
  • Legal/privacy/regulatory considerations when sending proprietary code to third-party LLM providers may necessitate on-premises models, changing the cost calculus.
• Research and product priorities
  • Economics favors hybrid solutions (static-analysis + LLM) that both improve precision and limit model hallucination risks.
  • Prioritize research/product work on handling long contexts and complex constraint reasoning to capture remaining economic upside from harder cases.

Suggested practical next steps for firms considering adoption: pilot hybrid LLM triage on a high-volume subsystem, track false-negative incidents closely, estimate developer-hour savings to compute payback period, and evaluate whether on-prem or vendor LLMs best meet privacy and total-cost objectives.