A lightweight, coordination-free work-stealing pipeline keeps GPU workers busy and tolerates preemption: on a 24GB A10G it sustains up to 3.4× the throughput of static sharding under heavy skew and recovers all tasks when half the workers are killed. The same pipeline shows flan-t5-base can label SST-2 at 94.7% agreement for about $0.0022 per 1,000 items, but performs poorly on irony (49.6%), underscoring task-dependent label quality.

Main Finding

A simple, coordination-free pipeline (work-stealing ring pool + memory-aware GPU concurrency) makes LLM-teacher labeling at scale practical and measurable: it sustains much higher throughput under skew and preemption than static sharding, is implementable with a single SQLite file (no broker), and—together with a "relabel-gold" benchmark—lets practitioners measure label quality per dollar so they can decide whether an LLM teacher or a human/hybrid loop is worth the cost.

Key Points

• Work-stealing ring pool
  • Each worker owns a queue, tries its own queue first, then walks a fixed ring of successors to steal tasks.
  • Exactly-once claims are implemented as conditional writes (UPDATE in an IMMEDIATE SQLite transaction).
  • No centralized coordinator or broker required; the whole control plane can be a single file.
• Fault tolerance: stale-claim sweeping
  • Tasks with running-state timestamps older than a threshold Δ are reset to pending by a sweep thread and re‑claimed.
  • This tolerates worker preemption (spot instances) with bounded duplicate work.
• Memory-aware concurrency rule
  • Compute per-node concurrency n = clamp(floor((Mtotal − Mreserve) / Mcopy), 1, C), where Mcopy is a conservative per-model budget and C is CPU count.
  • Using total device memory (not instantaneous free memory) avoids over-provisioning and OOMs.
• Relabel-gold benchmarking methodology
  • Have an LLM teacher relabel a public dataset with existing gold labels.
  • Quality = agreement / macro-F1 vs gold. Cost = measured throughput converted to $ per 1,000 items (1000 * P / (3600 * R)).
  • Chunk tasks (B items per task) to amortize conditional-write latency tc; choose B so tc/B ≤ 0.05 * tx.
• Implementation and reproducibility
  • Reference implementation ~500 lines Python, SQLite backend only, experiments reproducible and artifacts released.

Data & Methods

• Coordination protocol
  • Claim via single-row conditional UPDATE inside IMMEDIATE SQLite transaction (compare-and-set primitive).
  • Stale-claim sweep threshold Δ set to a few times expected chunk duration (authors used 60 s).
• Chunk sizing and cost model
  • Per-item coordination overhead ≈ tc / B; choose B (authors used B = 50) so overhead is small (<5% of inference time).
  • Cost per 1,000 items = 1000 * P / (3600 * R) where P is instance hourly price and R is measured aggregate throughput.
• Experiments (single 24 GB NVIDIA A10G instance, 4 vCPUs)
  • Synthetic CPU-bound tasks to isolate scheduler for throughput/fault tests; flan-t5-base for quality/cost.
  • Throughput under skew (W ∈ {2,4,8}, skew fraction of tasks on Q0):
    • At W=8, skew=0.9: work-stealing throughput 1324 items/s vs static sharding 386 items/s → speedup 3.43.
    • At zero skew, stealing matches static sharding (no overhead penalty).
  • Fault-tolerance under kill (W=4, kill 2 workers):
    • Static sharding without sweep: 1,047/2,000 tasks completed (953 lost).
    • Steal + stale-sweep: 2,000/2,000 completed (0 lost).
  • Teacher quality & cost (flan-t5-base, W=4, 24 GB device, 4 copies):
    • SST-2 sentiment (2,000 items): agreement 94.7%, macro-F1 0.947, 125.5 items/s → $0.0022 per 1,000 items.
    • tweet_eval irony (2,000 items): agreement 49.6%, macro-F1 0.331, 93.6 items/s → $0.0030 per 1,000 items.
  • Memory validation
    • Observed ~0.58 GB per fp16 model copy; conservative Mcopy used = 2.0 GB to cover activations/fragments; rule predicted safe concurrency (used 4 copies) with zero OOMs.
• Limitations called out by authors
  • Only the SQLite single-file backend was implemented and tested (multi-node/object-store backend not benchmarked).
  • Skew model is worst-case (single hot queue); real-world skew may be more diffuse.
  • The methodology measures teacher quality vs gold but does not improve it; experiments are for single unsupervised teacher baseline.

Implications for AI Economics

• Direct measurement of label quality per dollar
  • The relabel-gold benchmark gives a concrete, comparable metric: agreement (or macro-F1) versus cost per 1,000 items. This lets teams decide whether a cheap instruction‑tuned teacher is good enough (e.g., SST-2 sentiment at $0.0022/1k) or whether to invest in larger teachers or human-in-the-loop (e.g., irony at ~50% agreement, where teacher is inadequate).
• Reduced wasted compute and lower effective labeling cost
  • Work-stealing + stale-sweep recovers work after preemption and rebalances skew, improving throughput (up to ~3.4× vs naive sharding) and avoiding large losses from node failures. That increases effective utilization of cheap spot capacity and reduces the budget needed to label large corpora.
• Lower infrastructure/operational costs
  • A coordination-free design that uses only conditional writes (SQLite or object stores) avoids a separate scheduler/broker service, reducing operational complexity and cost for medium-scale deployments.
• Portability trade-offs: single-file vs fleet
  • SQLite backend is low-latency (tc ≲ 2 ms), enabling small chunk sizes and fine-grained balancing on a shared filesystem. Porting to object stores removes the single-file constraint and enables larger fleets, but tc rises (tens of ms) and chunk sizes must increase—this increases per-task latency but can be amortized. Economically, this is a scale vs latency trade-off: object-store port permits larger fleets at the cost of coarser granularity and potentially higher aggregated cost unless chunk sizing and concurrency are tuned.
• Resource utilization across device sizes
  • The memory-aware concurrency rule supports running the same labeling code across GPUs of different sizes (e.g., 16 GB → 2-way, 24 GB → 4-way with same Mcopy). This increases deployability on heterogeneous commodity hardware and can improve per-dollar throughput by packing more model copies safely on larger devices.
• Practical decision heuristic for hybrid loops
  • The pipeline lets teams estimate: (a) per-item labeling cost for unsupervised LLMs; (b) how much label quality a human-supervised judge would need to improve to justify the human-evaluation cost. This makes the build-vs-buy (or human-in-loop vs pure-LLM) decision data-driven.

Practical recommendations (from the paper) - Use relabel-gold on representative datasets to map quality vs $/1k and decide whether to deploy an LLM teacher or invest in a human/judge loop. - Choose chunk size B so tc/B ≤ 0.05 * tx (authors used tc ≲ 2 ms, tx 8–11 ms, B=50). - Set stale-claim Δ to a few times expected chunk duration (authors used ~60 s). - For fleet deployments, plan to port the same conditional-write claim protocol to an object-store (higher tc, raise B accordingly).

Caveats - Fleet-scale and object-store performance were not empirically evaluated—claims about multi-node portability are design-level and untested in this paper. - Teacher quality results are single‑teacher, unsupervised baselines; hybrid loops and judge models (Stages A–C) are described but not experimentally evaluated here.