Two agentic systems achieve comparable verified task completion but differ widely in operational cost: OpenClaw and NanoBot show no statistically significant full-completion gap (31% vs 25%), yet OpenClaw commonly consumed nearly three times the wall time and roughly 19× peak memory, raising capability–cost trade-offs for deployment.
Resource Constraints and Performance in Agentic AI Systems
A paired benchmark finds no statistically established full-task-completion advantage between OpenClaw (31%) and NanoBot (25%, 95% CI for difference −3 to 15 percentage points), while NanoBot achieves similar or better partial-completion coverage using far less wall time and peak memory.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Progress toward more autonomous AI increasingly depends on agentic systems that combine a language model with tools, memory, state management, and multi-step execution. These mechanisms shape both task capability and operational burden. We compare OpenClaw and NanoBot as complete agentic systems using a paired primary benchmark and a more detailed instrumented subset of paired prompts. In the primary benchmark, the rate of full task completion was 31% for OpenClaw and 25% for NanoBot, a six-percentage-point difference with a 95% task-bootstrap interval from -3 to 15 percentage points, providing no statistically established full-completion advantage for either system. In the instrumented layer, both systems achieved 26% full completion, while NanoBot reached at least partial completion on 43% of prompts compared with 26% for OpenClaw. OpenClaw took longer on 83% of prompts and had a higher recorded peak-memory value on every prompt, with geometric mean ratios of 2.98 for wall time and 19.44 for peak memory. Among the ten detailed-layer prompts on which at least one system achieved partial or full completion, NanoBot weakly dominated on eight; across all 23 prompts, however, ten of its eighteen dominance cases were cheaper joint failures. Outcome labels differ across the two evidence layers, showing why agent-system evaluation should connect capability and resource measurements to attempt-level execution and scoring provenance. These findings show that progress toward more autonomous AI should be evaluated through verified task completion, observed resource use and records linking each result to the execution that produced it.
Summary
Main Finding
- When treated as complete agentic systems, OpenClaw and NanoBot show similar verified full-task capability but materially different operational footprints.
- Primary benchmark: OpenClaw full completion 31% vs NanoBot 25% (difference 6 percentage points; 95% bootstrap interval −3 to 15 pp → not statistically significant).
- Instrumented subset: both systems achieved 26% full completion, but NanoBot reached at least partial completion on 43% of prompts vs 26% for OpenClaw.
- Resource costs strongly favor the lightweight harness (NanoBot). OpenClaw took longer on 83% of prompts and had higher peak memory on every prompt; geometric-mean ratios were 2.98× for wall time and 19.44× for peak memory (OpenClaw / NanoBot).
- Outcome labels and conclusions differ across the two evidence layers, highlighting the need for attempt-level provenance linking each scored outcome to the exact execution trace.
Key Points
- Unit of analysis: whole agentic systems (harness + model + tools + runtime), not language models alone. Differences can arise from orchestration, memory, verification, and retry policies.
- Two complementary evidence layers:
- Primary paired benchmark: 100 prompts, one run per system per prompt, ordinal outcomes and basic telemetry.
- Instrumented detailed layer: purposive subset of 23 prompts (7 short, 8 medium, 8 long) with full container versions, wall-clock duration, CPU/memory traces, termination reasons, manual outcome assessments, and richer run metadata.
- Performance vs cost trade-off:
- Capability (full completion) was similar in aggregate, but NanoBot achieved more partial successes and often dominated per-prompt when accounting for resource use.
- Many NanoBot “wins” were cheaper joint failures (both systems failed, but NanoBot failed using fewer resources).
- Provenance matters: lack of attempt-level linkage in the primary layer produced mismatches in outcome labelling between layers and complicated attribution of successes/failures to specific runs.
- Contributions claimed: a paired capability–cost comparison, cross-layer concordance/run-lineage analysis, and suggested design requirements for reproducible agent-system evaluation.
Data & Methods
- Systems compared:
- OpenClaw: service-oriented harness, broader orchestration surface (openclaw-container-v3 in instrumented runs).
- NanoBot: lightweight harness with a smaller runtime surface (nanobot-container-v3 in instrumented runs).
- Evidence layers:
- Primary benchmark: 100 prompts, each prompt evaluated once by each system; recorded ordinal outcome, category, startup latency, CPU, memory, trace complexity. Product-level identifiers only (no full run lineage).
- Detailed instrumented layer: 23 horizon-balanced prompts; full run metadata captured (container versions, wall time, avg/peak CPU, peak memory, traces, retries, termination reason, failure type) plus manual outcome adjudication. Prompts linked across layers by identifier but primary layer lacks attempt-level lineage.
- Analytical approach:
- Paired comparison design (each prompt evaluated by both systems).
- Outcome comparison (full completion rates, partial completion rates, ordinal outcomes).
- Resource comparisons using geometric-mean ratios for skewed metrics (wall time, peak memory).
- Prompt-level dominance analysis (considering capability and resource use) and resource-bounded completion curves.
- Bootstrap interval for difference in full-completion rates to assess statistical uncertainty.
- Limitations noted by authors:
- Primary layer lacks attempt-level run provenance, limiting traceability.
- Observations depend on the joint configuration (harness, model, tools, permissions, runtime settings); results are not a decomposition into model vs harness causal effects.
- Study does not claim to measure AGI; it evaluates relevant system-level trade-offs for autonomous action.
Implications for AI Economics
- Cost-performance trade-offs are central for deployment decisions:
- Two systems with similar verified capability can differ enormously in operational cost (here ~3× time and ~19× peak memory), with direct implications for infrastructure spending, latency-sensitive applications, and scaling plans.
- Procurement and cost–benefit analysis should use whole-system metrics:
- Buyers and platform teams should evaluate verified task completion, end-state verification procedures, and measured resource consumption (wall time, peak memory, retries), not model-only benchmarks or provider claims.
- Infrastructure and capacity planning:
- Large memory and time multipliers materially increase compute bills and limit parallelism; lightweight harness choices can yield substantial cost savings, especially at scale or for high-throughput use cases.
- Reliability and labor substitution:
- Partial completion and cheaper failures matter economically: a cheaper failure may be preferable for rapid human-in-the-loop recovery, while costly failures amplify operational burden.
- Time-horizon and reliability estimates for task substitution require repeated-attempt measurements under pinned conditions and task-duration baselines—single-shot benchmarks are insufficient for ROI or labor-substitution forecasts.
- Evaluation and regulatory/audit needs:
- Reproducibility and run-level provenance are essential for audits, compliance, and risk accounting. Economic valuation of agentic systems should factor in traceability costs (instrumentation, logging, storage) and their role in trustworthy assessment.
- Design choices and market differentiation:
- Lightweight harnesses that reduce orchestration overhead can be economically superior in many contexts, even if they offer fewer advanced orchestration features; conversely, richer harnesses may be justified where complex verification, delegation, or long-lived state are required.
- Recommendations for economic actors (practitioners, procurement, policymakers):
- Require verified end-state metrics and resource accounting in benchmarks used for procurement.
- Insist on attempt-level provenance to link outcomes to execution traces for reproducible evaluation and liability assessment.
- Use horizon-balanced, resource-bounded benchmarks to estimate operating costs and reliability under realistic workloads.
- Incorporate dominance analyses (capability + cost) rather than capability-only pass rates when making deployment decisions.
Summary takeaway: Evaluating agentic AI economically requires combining verified task-completion metrics with detailed resource accounting and run-level provenance. Harness design choices can dramatically change operational cost without proportionate capability gains, so whole-system, reproducible evaluation is essential for informed economic decisions.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The study provides systematic, paired measurements and an instrumented subset that jointly report task outcomes and resource metrics, but the sample size is modest, the detailed subset is purposive (not random), the primary layer lacks attempt-level linkage and full provenance, and only two system versions are compared, limiting external validity and causal attribution.
Methods Rigormedium — Strengths: paired design, two evidence layers, explicit resource metrics (wall time, CPU, peak memory), outcome verification, and a run-lineage audit; Weaknesses: purposive subset selection, absence of attempt-level linkage in the primary layer, relatively small N for some analyses (23 prompts), potential selection bias in prompt set, limited reporting on statistical power and control for confounders (e.g., differing model settings or tool availability).
SamplePrimary layer: 100 distinct prompts, each evaluated once by OpenClaw (product label) and once by NanoBot (product label), recording ordinal outcomes and resource indicators; Detailed layer: purposive subset of 23 prompts (7 short, 8 medium, 8 long) evaluated by openclaw-container-v3 and nanobot-container-v3 in agent mode with richer traces (prompt content, wall-clock duration, average/peak CPU, peak memory, retries, termination reason, manually assessed ordinal outcomes). The empirical unit is the complete agentic system; the dataset links prompts across layers but the primary layer lacks attempt-level run linkage.
Themesproductivity adoption org_design
IdentificationPaired within-prompt system comparison: each prompt evaluated once by OpenClaw and once by NanoBot (100-prompt primary paired benchmark) with a purposive 23-prompt instrumented subset for detailed execution traces; no randomized assignment of broader populations and no attempt-level linkage for the primary layer, so differences are descriptive comparisons of whole-system performance rather than causal estimates attributable to particular components.
GeneralizabilityFindings limited to the two evaluated system versions (openclaw-container-v3 and nanobot-container-v3) and chosen prompt set; newer versions or different harness configurations may differ., Purposive selection of the 23-prompt detailed subset reduces representativeness of the instrumented results., Benchmarks focus on specific task horizons and prompt types; results may not generalize to other tasks, domains, or production workloads., Hardware, deployment environment, toolsets, model settings, and permission boundaries differ across real deployments, constraining external validity., Primary-layer lack of attempt-level provenance impedes replication and detailed failure-mode attribution.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| In the primary 100-prompt paired benchmark, OpenClaw achieved full task completion on 31% of prompts, compared with 25% for NanoBot. Organizational Efficiency | positive | Rate of full task completion |
Reading fidelity
high
Study strength
medium
|
n=100
31% for OpenClaw versus 25% for NanoBot; six-percentage-point difference
|
| The primary benchmark did not establish a statistically significant full-completion advantage for either OpenClaw or NanoBot. Organizational Efficiency | null_result | Difference in full task-completion rates |
Reading fidelity
high
Study strength
high
|
n=100
95% task-bootstrap interval: −3 to 15 percentage points
|
| In the 23-prompt instrumented subset, OpenClaw and NanoBot each achieved full completion on 26% of prompts. Organizational Efficiency | null_result | Rate of full task completion in the instrumented subset |
Reading fidelity
high
Study strength
medium
|
n=23
26% for each system
|
| NanoBot reached at least partial completion on a larger share of instrumented prompts than OpenClaw: 43% versus 26%. Organizational Efficiency | positive | Rate of at-least-partial task completion |
Reading fidelity
high
Study strength
medium
|
n=23
43% versus 26%; 17-percentage-point difference
|
| OpenClaw took longer than NanoBot on 83% of the instrumented prompts. Task Completion Time | negative | Relative wall-clock execution time |
Reading fidelity
high
Study strength
medium
|
n=23
OpenClaw took longer on 83% of prompts
|
| OpenClaw had a higher recorded peak-memory value than NanoBot on every instrumented prompt. Organizational Efficiency | negative | Peak memory consumption |
Reading fidelity
high
Study strength
medium
|
n=23
Higher on every prompt; geometric mean ratio 19.44
|
| OpenClaw's wall-clock time was substantially higher than NanoBot's in geometric-mean terms, with a ratio of 2.98. Task Completion Time | negative | Wall-clock execution time |
Reading fidelity
high
Study strength
medium
|
n=23
Geometric mean ratio of 2.98
|
| Among the 10 detailed-layer prompts on which at least one system achieved partial or full completion, NanoBot weakly dominated on 8 prompts. Organizational Efficiency | positive | Prompt-level joint capability-and-resource dominance |
Reading fidelity
high
Study strength
low
|
n=10
NanoBot dominated on 8 of 10 prompts
|
| Across all 23 detailed-layer prompts, 10 of NanoBot's 18 dominance cases were cheaper joint failures. Organizational Efficiency | mixed | Interpretation of resource-cost dominance under joint task failure |
Reading fidelity
high
Study strength
low
|
n=23
10 of 18 dominance cases
|
| The study argues that agent-system evaluation should link verified task completion and observed resource use to the execution records that produced each result. Governance And Regulation | positive | Evaluation reproducibility and provenance of task outcomes |
Reading fidelity
high
Study strength
low
|
n=123
|