Pairit lets researchers declare and run complex live human–AI experiments from a single auditable configuration, enabling randomized team compositions and server-hosted AI agents that both converse and act on shared workspaces; the platform is validated in multiple large-scale deployments and supports reproducible, shareable protocols.
Pairit: A Platform for Live Experiments on Human-AI Collaboration
Pairit is a declarative platform that lets researchers specify, run, and share live human–AI experiments via a single auditable YAML configuration, validated through multiple large-scale deployments capturing fine-grained interaction logs.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Organizational design in the era of artificial intelligence requires experimental methods that can test how human-AI groups coordinate, delegate, and make decisions. Programmable platforms coordinate live human-to-human sessions or real-time human-AI chat, but researchers cannot easily declare experiment protocols in which AI participants both communicate and act on shared work within one auditable configuration. Here we introduce Pairit, an online platform that facilitates the design, testing, and deployment of experiments that test human-AI organizational designs and interventions. Through a single YAML configuration file, researchers declare an executable experiment graph (pages, routing, randomization, matchmaking, chat, shared workspaces, server-hosted agents, surveys, timers, and custom HTML components) and combine any number of humans and AI agents in live sessions. We have validated the feasibility of the platform through multiple live deployments, including peer-reviewed published studies, capturing high-resolution process traces of communication, negotiation, and collaborative work in live human-AI dyads. By representing complex interactive protocols as standardized, auditable configuration files, Pairit provides reusable infrastructure for specifying, deploying, and sharing live human-AI organizational experiments.
Summary
Main Finding
Pairit is a declarative platform that lets researchers specify, run, audit, and share live experiments that mix any number of human participants and server-hosted AI agents in real time. Experiments are declared in a single YAML file as an executable directed graph (pages, routing, randomization, matchmaking, chat, shared workspaces, agents, timers, custom HTML). Pairit treats AI as first‑class organizational actors that can both converse and take protocol-defined actions (e.g., edit shared documents, update session state), and it has been validated in large live deployments (thousands of participants).
Key Points
- Declarative experiment graph: entire protocol (flow, branching, matchmaking, agents, UI components) is one auditable YAML configuration that compiles to an execution graph.
- Built-in primitives: matchmaking pools, real‑time chat, collaborative workspaces (keystroke-level logging), timers, surveys, randomization, media, and custom HTML components.
- Agents as first-class actors: server-hosted LLM agents (connectable to providers like OpenAI/Anthropic) with trigger rules (on_join, every_message, periodic), conditional prompts (interpolating session state), and tools (assign_state, end_chat, write_workspace).
- Component-first architecture: pages compose components; components can conditionally render based on typed session_state.
- High-resolution logging and exports: timestamped chat, workspace revisions, state transitions, and custom telemetry exported as CSV/JSON/JSONL from MongoDB event store.
- Validation: platform used in peer‑reviewed and working‑paper deployments (e.g., 2,234 Prolific participants in a paired advertising task; ~2,500 participants in a negotiation study). Supported published analyses (PNAS paper and others).
- Reproducibility & auditability: Manager CLI lints/compiles against strict JSON schema; configs and export schemas intended to be shared as executable research objects. Public docs and source: docs.pairium.ai/pairit and github.com/pairium/pairit.
- Extensions: in‑silico pretesting by populating the same config with simulated LLM agents; sandboxed custom HTML embeds with pairit client API.
- Operational caveats noted by authors: execution risks include model access, hosting, recruitment, and review constraints; authors are Pairium co‑founders (disclosure).
Data & Methods
- Configuration & runtime
- Authoring: experiments declared in YAML; Manager CLI validates and compiles to a bundle uploaded to Manager Server.
- Runtime: Lab Server manages sessions, matchmaking, and agents; Lab App (browser client) receives server-sent events and issues HTTP actions.
- Persistence: event and session stores in MongoDB; exports to CSV/JSON/JSONL.
- Experimental primitives & mechanics
- Matchmaking pools coordinate synchronous grouping (FIFO, balanced, block); randomization supports balanced-random and block assignment; routing conditional on session_state.
- Chat rooms and live-workspace support split layouts; permissions (editableBy participant/agent/both); agents may read/write workspace.
- Agent triggers and tools: replyCondition filters, dynamic prompts with conditional blocks, tools to change state or edit documents; agents can be scheduled (every N messages) or react to events.
- Custom interfaces: sandboxed HTML with pairit.ready / setState / event / done API; can be marked required to prevent navigation until completion.
- Logging & outputs
- High-frequency traces: timestamped messages, keystroke-level document revisions, onPlay/onPause media events, form and survey events.
- Export formats suitable for sequence analysis, event-history econometrics, and standard cross-sectional analyses.
- Validation deployments
- Large-scale randomized designs demonstrated: human–human vs. human–AI randomization; within human–AI arms, randomized agent personalities via trait prompts. Studies supported analyses of gender gaps, diversity collapse, bargaining assistance, and productivity.
- Platform used to coordinate synchronous dyadic and small-group collaborations at scale.
Implications for AI Economics
- Enables causal tests of organizational and market mechanisms involving AI agents:
- Experimentally vary team composition, delegation rules, facilitation regimes, or assistance timing to identify causal effects on output quality, productivity, bargaining outcomes, surplus division, and distributional impacts.
- Study complementarities vs. substitution between human tasks and AI actions by toggling agent editing rights and decision authority.
- Test matching algorithms, pricing or fee structures, and contract designs in platform markets by embedding agents as suppliers, buyers, or intermediaries.
- Rich process data for mechanism analysis:
- Sequence data (chat timestamps, revision histories, delegation events) lets researchers go beyond outcome-level metrics to identify communication, coordination, and bargaining pathways and compute dynamic treatment effects.
- Keystroke and edit logs facilitate fine-grained measures of division of labor, authorship, and agent intervention intensity.
- Reproducibility and policy transparency:
- Executable configs allow peer reviewers and regulators to inspect prompts, routing, timeouts, and agent tools—useful for auditing human‑AI field experiments that influence policy or platform design.
- Shared configs lower friction for replication studies and multi‑lab comparability.
- Enables pretesting and design iteration:
- In‑silico simulation with LLMs using the same config helps debug and pilot protocols before costly field recruitment.
- Practical applications and examples
- Test how AI-mediated negotiation affects prices and welfare in bilateral or multi‑party markets.
- Evaluate how AI assistants change task assignment and labor demand on gig platforms (e.g., effect on match rates, completion times, earnings).
- Explore discrimination dynamics (gender, race) under different agent prompt designs and matchmaking rules.
- Investigate multi-agent market microstructures (auctions, matching markets) with mixed human/AI participants.
- Limitations and research design cautions
- External validity: lab-style synchronous experiments may not capture long-run adoption, learning, or equilibrium responses in markets.
- Dependence on LLM providers, model versions, and hosting costs: model updates can change agent behavior across replications.
- Data/privacy and IRB considerations: sharing executable configs is powerful but requires careful consent and disclosure design (Pairit supports consent & disclosure in the config).
- Conflict of interest: platform authorship requires transparency when Pairit is used in published work.
- Recommendation for AI economics researchers
- Use Pairit to run randomized, mechanism-centered lab-in-field experiments where process traces matter (coordination, bargaining, delegation).
- Pre-register protocols and publish YAML configs alongside papers to maximize auditability.
- Combine Pairit’s event data with standard econometric techniques (event-study, dynamic panel, mediation analysis, structural estimation) to link process mechanisms to aggregate outcomes.
If you want, I can (a) map a short example YAML design for a simple human–AI bargaining experiment showing key components to randomize and log, or (b) list concrete empirical questions in labor and market design that Pairit is especially well suited to test. Which would you prefer?
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper is primarily a systems/platform description but supports its claims with multiple live deployments (including randomized team composition) totaling thousands of Prolific participants and a peer-reviewed PNAS paper; however, the main contribution is infrastructure, not novel causal findings, and empirical results are summarized rather than reported in full here.
Methods Rigormedium — The platform appears carefully engineered with built-in randomization, matchmaking, logging, and schema validation; the authors report randomized deployments and large sample sizes, but the current text provides limited detail on statistical analysis, robustness checks, or independent replication of empirical claims within this paper.
SampleEvidence for feasibility comes from multiple live deployments on Prolific: a randomized pairing study with 2,234 participants collaborating in synchronized pairs to produce advertising campaigns, and a dyadic negotiation study with ~2,500 participants; authors also cite a PNAS paper and several working papers that used the platform. Data captured include timestamped chat, keystroke-level workspace revisions, session state, and exported CSV/JSON logs.
Themeshuman_ai_collab org_design productivity
GeneralizabilityParticipants recruited from online crowdsourcing (Prolific) — results may not generalize to firm employees or other populations., Findings validate platform feasibility for synchronous, browser-based tasks; applicability to asynchronous or heavily instrumented field settings is untested., Behavior and outcomes may depend on specific LLM models, prompts, and provider access; model updates could affect reproducibility., Operational barriers (hosting, model cost/limits, institutional approvals) may limit adoption across labs., Custom HTML components and complex tasks may require engineering that reduces immediate reproducibility for non-technical users.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Pairit allows researchers to declare live experiments involving arbitrary mixtures of human and AI participants in a single YAML configuration file. Organizational Efficiency | positive | Ability to specify human–AI experimental protocols |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Pairit enables server-hosted AI agents to participate in live sessions by conversing, editing shared documents, and executing protocol-defined actions. Task Allocation | positive | AI agent participation and action capabilities in collaborative tasks |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Pairit captures detailed process traces from live human–AI collaboration, including timestamped chat messages, workspace revisions, and delegation choices. Organizational Efficiency | positive | Granularity and exportability of collaboration process data |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Pairit has been used in live deployments involving 2,234 Prolific participants who were randomized to human–human or human–AI teams and collaborated in synchronized pairs to produce advertising campaigns later tested in a live market. Team Performance | positive | Feasibility of live randomized human–AI team experiments |
Reading fidelity
high
Study strength
medium
|
n=2234
|
| Pairit has been used to randomize AI personality through trait prompts within human–AI teams. Task Allocation | positive | Experimental manipulation of AI personality in live teams |
Reading fidelity
high
Study strength
medium
|
n=2234
|
| Pairit has been used to embed real-time AI assistance in dyadic negotiations involving 2,500 participants. Decision Quality | positive | Feasibility of real-time AI assistance in negotiations |
Reading fidelity
high
Study strength
medium
|
n=2500
|
| Pairit’s declarative architecture can coordinate complex, real-time interactions across diverse organizational settings. Organizational Efficiency | positive | Reliability and generality of live human–AI experiment coordination |
Reading fidelity
high
Study strength
medium
|
n=4734
|
| Pairit’s Manager CLI validates and compiles experiment configurations against a strict schema before deployment. Organizational Efficiency | positive | Configuration validation and deployment reproducibility |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Pairit supports reproducibility by allowing researchers to publish executable configurations that other laboratories can inspect, adapt, and run. Governance And Regulation | positive | Auditability and portability of experimental protocols |
Reading fidelity
high
Study strength
medium
|
not reported
|