A document-oriented agent runtime (String) lets language-model agents use apps as Markdown, matching curated-skill success on an 87-task benchmark while cutting tokens by about one-third; staged, partial disclosure improves action-selection accuracy and reduces wrong actions from ~28% to ~2%.
String: An Agentic OS Where Every App Is a Markdown File
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
String, a document-based runtime (SFMD) and agentic OS, matches or slightly improves task success versus curated skills on an 87-task benchmark while cutting token use among completed episodes by ~33% and dramatically improving action-dispatch accuracy via staged partial views.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
LLM agents have become a new class of software user, but every surface they work through was designed for someone else. Pages are built for human eyes, which can skim and ignore; tool schemas for programs, which pay nothing to carry definitions they never call. An agent has neither luxury: it re-reads, and pays again for, everything it is shown on every turn. We present String, an open-source runtime that gives this user an interface of its own and treats the job as an operating-systems problem. Tool knowledge moves out of the agent's context and into a common layer that renders it back one view at a time as Markdown. A single SFMD (String-Flavored Markdown) document declares an application's views, typed actions, navigation, and credentials, and the runtime handles discovery, validation, execution, state, and secrets behind two core verbs: /open to see and /act to do. Web and app turn out to be two renderings of one architecture: an SFMD site serves styled HTML to browsers and the raw document to agents, so one grammar reaches apps, files, shells, and the web, even legacy HTML, with no per-site integration. Views stay partial by design, and the staging is causal: disclosing one tier of detail a single turn too early costs up to 23 accuracy points, while proper staging drops wrong-action selection from 28% to 2%. Privilege follows provenance: a remote page may call HTTP but never the shell, and caller-supplied text never expands a stored secret. On an 87-task benchmark that pairs each task with curated skills, operationalizing those procedures as on-demand String apps yields comparable aggregate success across six models from frontier to small (+1.3pp) while using 33.5% fewer tokens among completed episodes, and the resident interface stays a constant 53 tokens at any catalog size. We report the design, the evaluation, and what three months of production use taught us.
Summary
Main Finding
String (and its SFMD format) recasts agent–tool integration as an OS-like runtime and document format so every app is a single, addressable Markdown file. By pushing tool schemas and procedural code out of the agent’s context and into a staged, partial-rendering runtime, String preserves a tiny constant resident interface (53 tokens), reduces per-task token consumption, recovers action-dispatch accuracy through staged disclosure, and yields equal-or-better aggregate task success while substantially lowering token costs.
Key Points
- Architecture and design principles
- SFMD (String-Flavored Markdown): a strict superset of CommonMark that declares views, navigation, typed actions, credentials; an app is literally a Markdown file (P3).
- Four design principles:
- P1 Partial exposure: intentionally render views incomplete; everything held back is addressable.
- P2 Uniform surface (location transparency): same SFMD object runs as local app, file, or web page; agents use the same verbs (/open, /act).
- P3 Documents as programs: installation is copying a file; program errors are format errors.
- P4 Recursive rendering: action outputs re-render as SFMD and compose with the same interface.
- Runtime, state, and trust
- Topics scope state and privileges; provenance enforces capability (remote SFMD can invoke HTTP but not shell).
- Secrets are stored per-app and never expanded from caller-supplied inputs; webhook events are append-only and must be acknowledged.
- The runtime provides OS-like services (addressable shells, event inbox, package manager) behind a single MCP-like tool.
- Partial-rendering discipline
- URL hiding, schema hiding (schemas revealed on demand with –help), address-based partial reads (doc#block), bounded lists/diffs.
- Staged disclosure is causal: revealing too much or too early harms accuracy.
- Implementation
- Open-source TypeScript codebase (~17.8k LOC), daemon bound to loopback; v0.1 single-user, loopback-only, unauthenticated.
- Evaluation highlights (see Data & Methods for details)
- Comparable or slightly better task success versus curated skill libraries (+1.3 percentage points average) across six models.
- Mean token reduction among completed episodes: 33.5% (range 21.7–44.3% across models).
- Resident interface cost: 53 tokens vs full JSON schema contracts costing ~103,518 tokens.
- Staging effects: wrong-action selection fell from 28% to 2%; moving detail one turn early reduced accuracy by up to 23.3 points on smaller models.
- Limitations & threats
- Internal measurements (replicable with provided artifacts); v0.1 lacks multi-user/auth/audit; some effects reflect combined packaging + rendering, not isolated.
Data & Methods
- Benchmark and task set
- SkillsBench v1.1: 87 practitioner tasks across 8 domains, each paired with curated skills in the original benchmark.
- Conditions compared
- No skills, curated skills (benchmark baseline), and String apps (operationalized skills packaged as SFMD actions).
- Models tested (6)
- GPT-5.5, DeepSeek V4 Pro, Kimi K2.6, Claude Opus 4.8, Claude Sonnet 4.6, Claude Haiku 4.5.
- Experimental setup
- For each model × condition there were 3 runs per task (3 runs × 87 tasks per cell), run on the OpenHands harness.
- Key metrics: task success rate, tokens consumed among completed episodes, dispatch/wrong-action selection rates.
- Main quantitative results (selected)
- Average success: No skills 33.3% → Skills 50.5% → String 51.8% (String +1.3pp vs Skills).
- Token cut among completed episodes (String vs Skills): mean 33.5% reduction (median ~29.7%).
- Resident context token sizes: full OpenAPI JSON schemas ~103,518 tokens; minimal one-line indices ~3,291 tokens; String fixed stub 53 tokens.
- Dispatch accuracy gains from three-tier staging: +10 to +37 points across model×catalog-size cells; wrong-action selection drop 28% → 2%.
- Staging timing sensitivity: early disclosure of tier-2 detail reduced accuracy by 11.6 points (Sonnet) to 23.3 points (Haiku), statistically significant.
- Additional experiments
- Large-scale residency simulation: converted 100 public OpenAPI services into String apps to measure token residency slope (~1,035 tokens per app for full schemas).
- Fine-tuning open-weight models on production trajectories: improved success on 3/4 models and reduced generated tokens by 51.9–78.9% on held-out tasks (co-design evidence).
- Reproducibility
- Authors release runtime, SFMD, and apps (GitHub) and used public benchmark/harness; ports are auditable in appendices.
Implications for AI Economics
- Lower inference cost per task
- Token-based pricing (and compute costs tied to context processing) means String’s ~33.5% token reduction among completed episodes can translate directly into lower per-task billing and operational cost.
- Fixed resident context of 53 tokens vs schema-resident O(n) reduces a substantial constant overhead that otherwise scales with integrations.
- Reduced marginal cost of integrating tools
- Traditional approaches pay O(n) context cost per added tool; String’s O(1) resident cost and addressable-on-demand model decouple the number of integrations from the recurring per-turn token overhead, lowering marginal cost of onboarding new APIs/apps.
- This enables broader ecosystems (many small integrations) without proportional increases in run-time costs.
- Commodity pressure on large models and skill-pack vendors
- By converting procedural skill text into invocable apps and compressing interface knowledge into a tiny grammar, String narrows the performance gap between large and smaller models (observed gains even for smaller models). That could reduce premium pricing power for frontier models in operational settings and shrink the market for bespoke, model-specific skill engineering.
- Market opportunity shift toward runtime platforms and app marketplaces
- If apps are single files installable atomically, marketplaces for SFMD apps, runtimes, and discovery tooling become valuable. Revenue models may shift from custom integration services toward platform and marketplace fees.
- Productivity and labor effects
- Faster, cheaper agent automation of multi-step tasks increases substitution pressure on labor for routine or procedural knowledge work. The lower compute and integration costs make automation economically viable for more tasks and firms.
- Risk and compliance economics
- The runtime enforces provenance-based privileges and keeps secrets out of model context, lowering risk of secret exfiltration and unauthorized shell access. This reduces expected compliance and breach-remediation costs relative to ad-hoc integrations, but full multi-user, authenticated designs are still required to realize enterprise-grade risk reduction.
- Distributional and strategic effects
- Lowers barrier to entry for startups using smaller open models (cost-effective automation), potentially increasing competition and commoditization of certain AI services.
- Conversely, demand may increase for high-quality curated apps, signed/verified SFMD packages, and trusted runtimes—creating new captive markets and potential lock-in.
- Caveats and investment needs
- Current v0.1 limitations (single-user, loopback-only, unauthenticated) mean additional investment in security, auditing, and multi-tenant governance is required before enterprise deployment—these are nontrivial costs but are focused engineering investments compared to perpetual token costs.
- Economic benefits depend on workload mix: tasks that are rarely completed or require heavy external computation may see less direct token-cost savings.
- Quick back-of-envelope
- If a deployment’s agent-run workloads are primarily charged by tokens, a sustained 33% token cut can reduce cloud costs by a similar fraction on those episodes; when combined with the removal of large schema-resident context, platform-wide savings can be much larger for systems with many integrations.
Summary: String reframes agent interfaces from context-heavy schemas to a staged, document-based OS problem. That architectural change materially reduces token/compute costs, lowers marginal integration costs, and shifts economic value toward runtimes and app ecosystems—while also reducing some security risks—creating tangible operational savings and strategic market implications across AI tooling and automation.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The paper reports systematic, controlled comparisons on a standard benchmark (SkillsBench) and model families, shows consistent patterns across models, and includes targeted manipulations with statistical intervals supporting claims about staging. However, the evaluation is internal (author-written ports), limited to one benchmark and task type, reports some outcomes conditional on completed episodes, and production findings are anecdotal, which reduces external validity.
Methods Rigormedium — Design uses matched within-task comparisons and multiple models, and includes ablation-style manipulations (staging timing) with CIs; it also measures token costs at scale. Weaknesses: author-created ports may introduce bias, limited transparency on randomization/seed choices and statistical tests beyond CIs, small number of runs (3) per cell, outcome reporting conditional on completed episodes, and single-benchmark focus.
SamplePrimary evaluation: SkillsBench v1.1 (87 practitioner tasks across eight domains) with three experimental conditions (no skills, benchmark curated skills, String apps). Six LLMs were tested (GPT-5.5; DeepSeek V4 Pro; Kimi K2.6; Claude Opus 4.8; Claude Sonnet 4.6; Claude Haiku 4.5), with 3 runs per model–task pair. Additional experiments: scale test converting 100 public OpenAPI services into String apps (measuring resident context tokens) and targeted timing manipulations of staged disclosures; three months of production use reported qualitatively. Implementation is open-source and ports are author-written.
Themesproductivity human_ai_collab skills_training
IdentificationControlled within-task comparisons: each of 87 SkillsBench tasks is run under three end-to-end conditions (no skills, curated skills, and String apps) across six LLMs with 3 runs per model–task cell; token- and success-differences are attributed to the interface/runtime changes. Additional targeted manipulations (e.g., moving tier-2 detail one turn early) test causal effects of staging with reported confidence intervals. Scale experiments (100 OpenAPI→apps) measure resident context token costs. Evaluation is internal and not randomized beyond matched task comparisons.
GeneralizabilitySingle benchmark (SkillsBench) and author-authored skill→app ports may not represent broader task distributions or independently authored apps., Results conditional on completed episodes — token savings may partly reflect selection differences across conditions., Limited number of runs (3) per cell and internal-only evaluation reduce robustness to run-to-run variability and operational heterogeneity., Environment constraints: String v0.1 is single-user, loopback-only, unauthenticated; real-world multi-user, permissioned deployments may behave differently., Models tested span several families but may not reflect future models or different fine-tuning/conditioning regimes.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Operationalizing curated benchmark procedures as executable String apps produced 51.8% aggregate task success, compared with 50.5% for the benchmark's curated skills and 33.3% with no skills. Task Completion Time | positive | Task success rate |
Reading fidelity
high
Study strength
medium
|
n=87
51.8% vs. 50.5% aggregate success (+1.3 percentage points)
|
| String apps reduced token usage by 33.5% on average among completed episodes relative to the curated-skills condition. Organizational Efficiency | positive | Token consumption per completed episode |
Reading fidelity
high
Study strength
medium
|
n=87
33.5% fewer tokens on average
|
| The String condition achieved equal or higher observed success than the curated-skills condition on five of the six evaluated models. Task Completion Time | positive | Observed task success by model |
Reading fidelity
high
Study strength
medium
|
n=6
equal or higher success on 5 of 6 models
|
| String's resident interface remained fixed at 53 tokens when tested with 100 public OpenAPI services, compared with 103,518 tokens for full JSON schemas and 3,291 tokens for a one-line-per-tool index. Organizational Efficiency | positive | Resident tool-context size |
Reading fidelity
high
Study strength
medium
|
n=100
53 tokens vs. 103,518 tokens for full JSON schemas
|
| In execution against the full-schema baseline, String reduced token usage by 93.5% for Sonnet 4.6 and 91.8% for Haiku 4.5. Organizational Efficiency | positive | Token consumption during execution |
Reading fidelity
high
Study strength
medium
|
n=2
93.5% reduction for Sonnet 4.6 and 91.8% reduction for Haiku 4.5
|
| Three-tier staged disclosure reduced wrong-action selection from 28% to 2% and improved dispatch accuracy in all six tested model-by-catalog-scale cells. Error Rate | positive | Wrong-action selection and dispatch accuracy |
Reading fidelity
high
Study strength
medium
|
n=6
wrong-action selection decreased from 28% to 2%
|
| Moving tier-2 detail one turn early reduced dispatch accuracy by 11.6 percentage points for Sonnet 4.6 and 23.3 percentage points for Haiku 4.5, with 95% confidence intervals excluding zero. Decision Quality | negative | Dispatch accuracy |
Reading fidelity
high
Study strength
medium
|
n=2
11.6 points on Sonnet 4.6; 23.3 points on Haiku 4.5
|
| Fine-tuning four open-weight models on production trajectories improved success on three of the four models and reduced generated tokens by 51.9%–78.9% on 20 held-out tasks. Skill Acquisition | positive | Task success and generated-token usage after fine-tuning |
Reading fidelity
high
Study strength
low
|
n=20
51.9%–78.9% fewer generated tokens; success improved for 3 of 4 models
|
| String enforces provenance-based execution privileges: local files or locally installed apps may run CLI actions, whereas remote SFMD documents may invoke HTTP actions but not the shell. Ai Safety And Ethics | positive | Execution privilege isolation by document provenance |
Reading fidelity
high
Study strength
low
|
not reported
|
| Caller-supplied values cannot expand stored secrets: the runtime resolves stored variables only in author-defined template positions and rejects variables appearing in command arguments. Ai Safety And Ethics | positive | Protection of stored credentials from model- or caller-supplied input |
Reading fidelity
high
Study strength
low
|
not reported
|