An expectation-aware summarizer, ExpSum, substantially improves code-documentation quality for an industrial OS: on HarmonyOS it boosts BLEU-4 by up to 26.7% and ROUGE-L by 20.1% versus prior models, and human and LLM-based evaluations show its summaries align better with developers' documentation needs.

Main Finding

Existing state-of-the-art code summarization methods (including GPT-4.5 and research SOTA) frequently produce summaries that industrial developers reject—not mainly because of incorrect functionality descriptions but because generated summaries violate practical documentation expectations (57.4% rejection rate in a HarmonyOS expert study). The paper proposes ExpSum, an expectation-aware pipeline (function modeling, metadata checking, context-aware cascaded domain-term retrieval, and constraint-driven prompting) that substantially improves alignment with industrial documentation expectations and achieves large gains on HarmonyOS benchmarks (HMSum-12/13) and community datasets.

Key Points

• Empirical gap: In a questionnaire study with four HarmonyOS documentation experts, 57.4% of automatically generated summaries were rejected. Fleiss’ Kappa for expert annotations was high (≈0.86–0.87), indicating strong agreement.
• Developer expectations identified (three primary expectations):
  • Exp-1: Use appropriate domain terms (project-specific names/operations), sensitive to package/path context.
  • Exp-2: Explicitly indicate function category (e.g., field vs. procedural functions).
  • Exp-3: Avoid redundant, fine-grained implementation details that are noisy or implied by context.
• Limitations of prior systems: Global term embeddings (ignore path context), weak guidance for function-category inference, and overexposure of implementation details in prompts.
• ExpSum components:
• Function modeling: abstracts code into structured metadata to remove unnecessary implementation detail and focus on essential features (addresses Exp-3).
• Information checking: filters out empty/uninformative metadata to avoid redundant summary content (further supports Exp-3).
• Context-aware cascaded knowledge retrieval: retrieves domain terms from a term KB using dual-similarity (path/context + semantic) matching to disambiguate terms with identical lexemes in different contexts (addresses Exp-1).
• Constraint-driven prompting: guides LLMs to infer function categories and generate summaries that follow a category-specific constraint/schema (addresses Exp-2).
• Empirical gains:
  • On HarmonyOS (HMSum datasets), ExpSum outperforms baselines with >26.71% improvement in BLEU-4 and >20.10% in ROUGE-L versus official summaries.
  • Ablation studies show each component contributes positively.
  • Generalizability: ExpSum also improves performance and LLM-based alignment evaluations on community benchmarks (CodeSearchNet and a C/C++ benchmark).
• Resources: Two industrial benchmarks released—HMSum-12 (22,138 functions) and HMSum-13 (23,003 functions)—with officially reviewed summaries reflecting industrial expectations; code and artifacts are made available.

Data & Methods

• Datasets:
  • HMSum-12: 22,138 functions from HarmonyOS v12 with developer-reviewed summaries.
  • HMSum-13: 23,003 functions from HarmonyOS v13 with developer-reviewed summaries.
  • Additional community benchmarks: CodeSearchNet and a C/C++ dataset for cross-project evaluation.
• Human study:
  • Questionnaire with four HarmonyOS documentation experts.
  • Random sampling: one function per package from HarmonyOS v12.
  • Compared GPT-4.5 and a SOTA research baseline (PRIME); collected 532 expert feedback entries; high inter-annotator agreement (Fleiss’ Kappa ≈0.86–0.87).
• Method (ExpSum):
  • Metadata extraction: parse function into structured fields capturing core semantics (e.g., API calls, parameter roles, return semantics) while suppressing low-level details.
  • Metadata checking/filtering: remove empty/uninformative fields to reduce noise.
  • Cascaded retrieval from a term KB: dual-similarity matching that considers both path/package context and semantic embedding similarity to choose correct domain terms and forms.
  • Constraint-driven LLM prompting: feed metadata and retrieved terms plus a constraint schema that requires explicit function-category labeling and enforces summary patterns; includes self-refinement prompting.
• Evaluation:
  • Automatic metrics: BLEU-4, ROUGE-L comparisons to official, developer-reviewed summaries.
  • LLM-based evaluation: alignment with developer expectations on community projects.
  • Ablations: component-wise removal to measure contribution.
  • Baselines: GPT-4.5 and PRIME (and other SOTA models on community benchmarks).

Implications for AI Economics

• Productivity and cost impact:
  • Reducing documentation rejection rates (from >50%) can materially cut reviewer/editor time and rework in large industrial codebases, raising developer productivity and lowering maintenance costs.
  • Tools like ExpSum that improve practical acceptability increase the realized ROI of LLM-based code assistants in enterprise settings.
• Market and productization:
  • Demand for domain-aware, expectation-aligned LLM integrations (e.g., documentation automation tailored to firm-specific terminology and styles) is likely high for large software vendors and regulated industries.
  • Firms may pay premiums for systems that include domain KB construction and path/context-aware retrieval, not just generic LLM access.
• Data and infrastructure economics:
  • Industrial-quality datasets (like HMSum-12/13) are costly to curate but unlock higher-value automation; this motivates investment in proprietary labeled corpora and knowledge bases.
  • Maintaining a domain term KB and path-aware indices introduces ongoing data-maintenance costs but yields better disambiguation and lower error rates than purely model-based solutions.
• Evaluation and incentives:
  • Standard community metrics (BLEU/ROUGE) and benchmarks can misrepresent utility for industry; there is an economic incentive to adopt evaluation criteria that reflect acceptance by end-users (e.g., human-review acceptance rates, domain-term correctness).
  • Firms buying/using summarization tools should request domain-specific validation (human-in-the-loop acceptance metrics) to avoid overpaying for models that perform well on open benchmarks but fail in production.
• Labor and organizational effects:
  • Improved automation of high-quality summaries may shift reviewer work from writing to higher-level auditing, potentially reducing headcount for routine documentation but increasing demand for specialists who curate domain KBs and set standards.
  • Productivity gains could accelerate development velocity, affecting software release cycles and competition dynamics.
• Risks and compliance:
  • Misuse of domain terms or incorrect functional categorization in automated summaries can create legal/compliance risks (liability for incorrect documentation in regulated contexts); firms will value systems that minimize such errors.
  • Investment in guardrails (constraint schemas, metadata checks) is economically justified where documentation errors have outsized costs.
• Wider ecosystem effects:
  • A demonstrated improvement from expectation-aware systems encourages a market for complementary services: KB construction, path/context indexing, and documentation-review workflows that integrate LLM outputs.
  • Opens opportunities for specialized vendors offering turnkey industrial documentation LLM stacks with measurable acceptance-rate SLAs.

If you want, I can: - Extract example summaries (good vs. rejected) from the paper and annotate how ExpSum would change them. - Produce a short checklist for product teams evaluating code-summary tools for industrial adoption.