AgentLogs opens GitHub Copilot’s black box: a 64 million–entry, session‑level dataset capturing how cloud agents explore, edit, run commands, and interact with GitHub across 35,810 popular repositories—enabling fine‑grained study of agent behavior, cost, failure modes and human–agent collaboration.
AgentLogs: A Dataset for Opening the Black Box of GitHub's Cloud Agent
AgentLogs is a large-scale dataset of GitHub Copilot agent activity containing 307,416 tasks, 549,239 sessions, and 64.3 million step-by-step session log entries (prompts, reasoning traces, tool calls, and token usage) across 35,810 popular public repositories collected in July 2026.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Generative AI-based software engineering agents are becoming routine contributors to real-world software projects. On GitHub, developers can assign tasks to the Copilot cloud agent, which autonomously explores the repository, edits code, runs commands, and opens or reviews pull requests, producing a detailed log of every step along the way. While existing datasets capture outcomes of agent contributions, such as agent-authored pull requests, the process by which agents produce these contributions remains largely unexplored. To address this gap, we introduce AgentLogs, a large-scale dataset of agent activity on GitHub. AgentLogs comprises 307,416 agent tasks and 549,239 agent sessions in 35,810 of the 1,812,362 popular public repositories that we scanned, together with 64,255,174 session log entries that record each agent run step by step, including prompts, intermediate reasoning, tool calls (e.g., file edits, git operations, and GitHub interactions), and token usage. By exposing not only what agents contribute but also how they work, AgentLogs enables research on agent behavior, efficiency and cost, task formulation, failure modes, and human-agent collaboration in agentic software engineering.
Summary
Main Finding
AgentLogs is a large, public dataset that opens the black box of GitHub Copilot cloud agents by recording step-by-step execution traces (prompts, intermediate reasoning, tool calls, token usage, model metadata) for real agent tasks and sessions across popular public GitHub repositories. The dataset enables analysis of not only agent outcomes (e.g., PRs) but how agents operate, their costs, failure modes, and human–agent interaction patterns.
Key Points
- Scale and scope
- Scanned 1,812,362 public repositories with >10 stars; 35,810 repos (≈1.98%) had agent tasks.
- 307,416 agent tasks, 549,239 agent sessions, and 64,255,174 session log entries.
- Dataset size ≈56.7 GB (Parquet tables, sharded).
- What is recorded
- Full session logs: prompts, assistant messages, intermediate reasoning text, tool calls (file edits, git operations, GitHub interactions), sub-agent delegation, and token usage (prompt/completion counts).
- Session and task metadata: model identifiers, timestamps, states, branch/PR identifiers, creator IDs/usernames.
- Partial repository metadata (name, license, language, stars, forks).
- Accessibility & licensing
- Public release (version 0.2) on Hugging Face and GitHub (sample & docs). Code MIT; dataset CC BY 4.0.
- Example analysis tools and patterns provided (DuckDB, Polars, streaming).
- Collection method & constraints
- Seeded from an up-to-date list of repos (>10 stars) on 2026-07-10; data collected July 10–17, 2026.
- Used GitHub REST API plus undocumented GitHub Copilot API endpoints to fetch session logs and metadata (authors assert this is allowed for public data).
- Did not fetch extra metadata for referenced PRs/branches/workflows in this release.
- Example content
- Appendix shows detailed session snippet with tool calls that edit files, reply to review comments, and token counts for the session (enabling cost estimation).
Data & Methods
- Data sources and sampling
- Source repos: public GitHub repos with >10 stars (sampling frame from https://seart-ghs.si.usi.ch/).
- Collected task/session identifiers via REST API then pulled fuller metadata and session logs via undocumented Copilot API endpoints.
- Tables and schema
- Parquet tables: repositories, agent_tasks, agent_sessions, agent_session_logs, users.
- Sharded files sized for practical parallel processing (log entries in ~220MB chunks).
- Full schema and Python type definitions provided (PyPI package: risenlab-agentlogs).
- Temporal coverage
- Single snapshot collection window: July 10–17, 2026 (so temporal analyses of growth require combining multiple snapshots or longitudinal data).
- Processing & tools recommended
- Examples provided for DuckDB SQL over Parquet, Polars lazy aggregation, streaming Parquet read, and Hugging Face Datasets streaming.
- Limitations & caveats
- Coverage bias: only public, popular repos (>10 stars); omits private/org-internal agent usage.
- Snapshot nature: single-week crawl; trend inferences need caution.
- No additional fetch of related PR/CI metadata in this release (limits some outcome analyses).
- Reliance on undocumented API endpoints: possible future compatibility issues if platform changes.
- User privacy: only GitHub ids/usernames included from metadata; no deeper user data fetched.
Implications for AI Economics
- Cost modeling and operational economics
- Token usage and model identifiers per session enable granular estimates of compute/token costs per task, per session, and per outcome (e.g., cost-per-PR or cost-per-bug-fix).
- Researchers and firms can estimate marginal cost of agentic assistance and compare pricing/billing schemes (per-token, per-session, per-action, subscription).
- Productivity, ROI, and adoption dynamics
- Trace-level data permits measuring time-to-resolution, number of iterations, and correspondence between agent activity and repository outcomes (subject to augmenting dataset with PR/CI metadata). This supports ROI analysis for adopting agents and quantifying productivity gains versus human effort.
- The 1.98% adoption among popular repos is a baseline for penetration of agent tools in public projects; correlates (language, stars, license, contributors) can reveal which repo characteristics predict adoption and thus markets for agent tooling.
- Model competition and platform effects
- Session-level model metadata (e.g., Claude, Codex variants) supports comparative cost-effectiveness studies: different models’ token usage, session lengths, and success/failure modes inform vendor competition and pricing.
- Evidence of sub-agent delegation, usage of custom skills, and third-party integrations points to emergent agent marketplaces and potential differentiation strategies (value of repo-specific context / custom agents).
- Labor markets and task allocation
- Fine-grained logs allow analysis of what kinds of tasks agents handle autonomously vs require human revision—informing which developer tasks may be displaced, augmented, or complemented, and how wages/productivity might shift across developer roles.
- Patterns of follow-up sessions and reviewer interactions reveal supervision costs and the need for human oversight which should be included in economic models of automation.
- Externalities, infrastructure, and pricing of shared resources
- Agents trigger repo-side actions (CI runs, PRs, comments) that impose infrastructure costs on projects and CI providers; dataset allows quantifying these externalities (e.g., frequency of CI triggers per agent session).
- Findings can inform platform fee structures and cost-sharing arrangements (e.g., should platforms subsidize CI costs if agents increase CI usage?).
- Policy and market design
- Dataset supports auditing agent behavior (failure modes, hallucinations, bad edits), informing regulatory requirements around transparency, billing fairness, and accountability.
- Enables studies on information asymmetries (developers unaware of token/compute costs) and suggests design of billing models that better align incentives (pay-per-successful-outcome vs pay-per-token).
- Research and practical next steps
- Immediate analyses: cost-per-session distributions, token→outcome efficiency, model-level comparisons, adoption correlates among repos, and common failure modes requiring human intervention.
- For fuller outcome/economic impact studies, combine AgentLogs with additional GitHub metadata (PR outcomes, CI results, merge times) and longitudinal snapshots.
If you want, I can: - Propose a short analysis plan (SQL/Polars queries) to estimate cost per accepted PR and cost distribution by model. - Draft concrete hypotheses linking repo characteristics to agent adoption for an econometric test.
Assessment
Paper Typedescriptive
Evidence Strengthn/a — This is a dataset/benchmark paper that does not make causal claims or test hypotheses; it provides raw and processed data for future empirical work rather than evidence for a substantive economic claim.
Methods Rigormedium — The authors scanned a large, well-documented sampling frame (public repos with >10 stars), used GitHub REST plus undocumented Copilot API endpoints to retrieve detailed session logs, parsed logs into structured Parquet tables, and provide schema, code, and streaming-friendly shards; limitations include a one-week snapshot, reliance on undocumented endpoints that may change, minimal enrichment of related PR/branch metadata, and possible selection bias toward popular public repositories.
SampleScanned 1,812,362 public GitHub repositories with >10 stars (snapshot from seart-ghs on 2026-07-10). Collected agent task identifiers and session metadata using GitHub REST API and undocumented GitHub Copilot API endpoints between July 10–17, 2026, producing 307,416 agent tasks, 549,239 agent sessions, and 64,255,174 session log entries across 35,810 repositories; users table contains 33,573 GitHub IDs/usernames. Data stored as sharded Parquet tables and published (Hugging Face v0.2) with schema, example notebooks, and a small sample.
Themeshuman_ai_collab adoption
GeneralizabilityLimited to public repositories with >10 stars — excludes private and less-popular projects, Single short collection window (July 10–17, 2026) — may not reflect temporal trends before/after snapshot, Relies on GitHub Copilot cloud agent logs and platform-specific formats — not representative of other agent platforms or local-only tools, Use of undocumented API endpoints: format/access may change, limiting reproducibility over time, Related resource metadata (full PR/branch/workflow details) not fetched in this version, constraining some analyses, Possible geographic, language, and domain biases in the sampled popular repositories
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| AgentLogs contains 307,416 agent tasks and 549,239 agent sessions from 35,810 of the 1,812,362 popular public GitHub repositories scanned. Other | null_result | Dataset coverage of GitHub cloud-agent activity |
Reading fidelity
high
Study strength
medium
|
n=1812362
|
| The dataset includes 64,255,174 session log entries that record agent runs step by step, including prompts, intermediate reasoning, tool calls, and token usage. Other | null_result | Availability and granularity of agent execution traces |
Reading fidelity
high
Study strength
medium
|
n=64255174
|
| Only 1.98% of the popular public repositories scanned had agent tasks. Adoption Rate | positive | Repository adoption of coding agents |
Reading fidelity
high
Study strength
medium
|
n=1812362
1.98%
|
| The complete AgentLogs dataset contains 66,957,764 records and occupies 56.7 GB. Other | null_result | Dataset scale |
Reading fidelity
high
Study strength
medium
|
n=66957764
56.7 GB
|
| Agent task records include metadata such as task name, request, state, creator, timestamps, and branch or pull-request identifiers. Other | null_result | Information captured in agent-task records |
Reading fidelity
high
Study strength
medium
|
n=307416
|
| Agent session records include the model, prompt, outcome, usage information, and branch or pull-request identifiers. Other | null_result | Information captured in agent-session records |
Reading fidelity
high
Study strength
medium
|
n=549239
|
| Session-log records include messages, usage details, and tool calls for file edits, Git operations, GitHub issues, pull requests, comments, and continuous-integration activity. Other | null_result | Types of agent actions represented in session logs |
Reading fidelity
high
Study strength
medium
|
n=64255174
|
| The authors collected the dataset between July 10 and July 17, 2026. Other | null_result | Dataset collection period |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The AgentLogs dataset is publicly available on Hugging Face, with schema documentation, example analysis notebooks, and a sample available through the associated GitHub repository. Other | positive | Dataset accessibility for research use |
Reading fidelity
high
Study strength
medium
|
not reported
|