Coding frameworks that follow familiar conventions are markedly easier for AI code assistants to use than novel declarative designs; in a controlled benchmark Agno scored 0.55 on a new AI-assistability metric while the highly declarative DSPy scored 0.07, and structural alignment predicts functional correctness (r=0.576).

Main Finding

Declarative design alone does not guarantee that AI coding assistants can generate correct, framework-specific code. Instead, "convention alignment" — how closely a framework’s API, patterns, and documentation match conventions present in LLM training data — is the primary driver of what the authors call AI-assistability. Using a novel benchmark (DDL2PropBank) and identical agent logic implemented across 10 multi-agent frameworks, the paper shows large variation in both human-coded complexity and LLM-assisted implementability. Agno scores highest on the composite AI-assistability metric (AI = 0.55, 72% pass@1), while DSPy—designed to be highly declarative—scores lowest (AI = 0.07). Structural alignment correlates moderately with runtime correctness (r = 0.576 between average structural alignment ¯σ and pass@1), but convention alignment explains failures where novel abstractions are under-represented in model training data.

Key Points

• AI-assistability defined: a composite metric combining structural alignment (how closely AI-generated code matches idiomatic human reference implementations) and functional correctness (pass@1 — whether generated code executes end-to-end and produces valid outputs).
• DDL2PropBank benchmark: novel, controlled task mapping relational DB schemas (DDL) to PropBank rolesets (event frames). Designed to be outside common LLM training data to avoid memorization confounds.
• Architecture used for evaluation: Agent-as-a-Tool pattern with three agent types—Orchestrator, Coordinator, TableMapper—interacting with standardized Model Context Protocol (MCP) servers (PropBank and filesystem).
• Evaluation scope: identical agent logic implemented in 10 frameworks (Claude SDK, OpenAI Agents, Google ADK, Pydantic AI, Agno, DSPy, LangChain, Microsoft Agent Framework, Smolagents, AgentScope).
• Static code complexity: measured via logical lines of code (LLOC), cyclomatic complexity (CCN), and import surface. Observed 1.7x variation in LLOC and 3.3x variation in import count across frameworks.
  • Low-complexity: Pydantic AI, Agno (least boilerplate and control flow).
  • Mid-complexity: Smolagents, LangChain, Google ADK, OpenAI Agents, Claude SDK, Microsoft.
  • High-complexity: AgentScope, DSPy (largest overhead; DSPy highest CCN).
• AI-assistability evaluation: three AI coding assistants (GitHub Copilot, Claude Code, Cursor) were asked to reimplement the same agent for each framework using only public docs/APIs. Generated code was checked for (a) structural alignment with human reference and (b) runtime functional validity.
• Main quantitative outcomes:
  • Agno: highest AI score (0.55), high structural alignment and 72% pass@1.
  • Pydantic AI: low structural alignment but still robust pass@1 (~61%), showing non-idiomatic code can still execute.
  • DSPy: lowest AI score (0.07) because its abstractions are novel and under-represented in training data.
  • Correlation between structural alignment and pass@1: r = 0.576.
• Core insight: frameworks that follow canonical, well-represented conventions in documentation and API design are easier for LLMs to produce correct scaffolding for; highly declarative or novel abstractions hurt LLM-assisted code generation unless those abstractions are covered in model training data.

Data & Methods

• Task: DDL2PropBank — map each table Ti in a DDL schema S to one or more PropBank rolesets (sense_id) with argument→column grounding and confidence scores κ ∈ [0,1].
• Novelty controls: PropBank access provided via a custom MCP server and the task is not present in public corpora, minimizing memorization. All frameworks use the same MCP servers and identical prompts/system messages to isolate framework effects.
• Agent design:
  • Orchestrator: top-level tool that invokes Coordinator and dispatches TableMapper agents in parallel.
  • Coordinator: checks filesystem state (via MCP) to determine pending tables; supports idempotent re-runs.
  • TableMapper: extracts column/constraint context, obtains candidate lemmas via a single LLM-tool (GetActionVerbs), queries PropBank (search_by_lemma / search_by_sense_id), grounds arguments to columns, estimates κ, writes mappings.
• MCP servers:
  • PropBank MCP server (StreamableHTTP): programmatic access to rolesets (search_by_lemma, search_by_sense_id).
  • Filesystem MCP server: list/read/write operations for persistent outputs and incremental progress.
• Implementations: human-authored reference implementations in 10 frameworks; static analysis on database_mapper.py per framework (LLOC, CCN, # functions, imports).
• AI-assistability experiment:
  • Three coding assistants used to reimplement the agent from documentation only.
  • Generated implementations were evaluated on structural alignment with the reference (¯σ) and executed against test schemas to measure pass@1 (functional validity).
  • Composite AI metric combines ¯σ and pass@1 (details/formula in paper).
• Artifacts: benchmark, MCP server code, and all framework implementations are available on the authors’ GitHub (link in paper).

Implications for AI Economics

• Procurement and TCO: When organizations choose multi-agent frameworks, they should evaluate not only human-development effort (manual LLOC/CCN) but also AI-assistability. Frameworks with higher AI-assistability reduce labor/time-to-delivery when using LLM-based coding assistants—lowering total cost of ownership and accelerating deployment.
• Market dynamics & adoption: Frameworks that align with widely used conventions (APIs, canonical patterns, plentiful documentation, example corpora) will have faster adoption because off-the-shelf LLM assistants can generate reliable code. This favors frameworks that re-use well-known abstractions over those that introduce novel declarative constructs unless accompanied by substantial example corpora.
• Competitive advantage & lock-in: Vendors that design for AI-assistability (canonical patterns, MCP-like standards, llms.txt-style corpora) can gain an adoption advantage, potentially increasing market concentration. Conversely, vendors introducing unique abstractions risk higher switching costs for customers who rely on LLMs without custom fine-tuning.
• Investment priorities: Framework developers and platform owners should invest in:
  • Standardized connectors (e.g., MCP) and canonical examples to reduce friction for AI assistants.
  • Publishing clear, canonical examples and "llms.txt"-style corpora to ensure their patterns appear in model training/fine-tuning data.
  • Tooling to measure AI-assistability as part of the procurement and development lifecycle.
• Internal models vs external assistants: Organizations using bespoke or highly declarative frameworks might need to invest in private model fine-tuning or in-house assistants to achieve parity in productivity, increasing fixed costs but potentially lowering long-run variable costs.
• Policy and standards: Open standards (like MCP) that decouple agent logic from provider infra can reduce vendor lock-in and improve cross-framework interoperability, which is economically desirable for competition and innovation.
• Limitations & economic caution: The benchmark is a single controlled task; LLMs and frameworks evolve rapidly. Economic conclusions should be tempered by the fact that AI-assistability may shift as model training corpora and framework documentation change. Procurement decisions should weigh both static complexity and current AI ecosystem fit.

If you want, I can: - Extract the full table of per-framework LLOC/CCN/import counts and AI-assistability scores from the paper and present them as a concise comparison table. - Draft a short checklist (quantitative and qualitative) organizations can use when evaluating MAFs for AI-assisted development.