Thumbtack replaced legacy request forms with an LLM-driven, per-occupation catalog reconstruction system that mapped 73% of legacy answers directly and preserved 92.5% of operational preferences, enabling rapid onboarding across 132 service occupations; the approach is observational and production-focused rather than experimentally validated.

Main Finding

The paper presents an AI-native, per-occupation autoresearch loop that reconstructs provider-facing preference catalogs for two-sided service marketplaces (i.e., the tagging layer used for probabilistic matching) from legacy request-form Q&A. Deployed in production across 132 occupations, the system (i) generates occupation-specific tag sets in parallel, (ii) evaluates them with a recalibrated six-rubric LLM-as-judge plus a seven-persona critic panel (weighted penalties, no hard vetoes), and (iii) maps legacy Q&A back to the new catalog via a parity-mapping stage. In a 14-day post-launch cohort (1,840 pros, 9.3M filter evaluations) the reconstruction preserved screening preferences with high fidelity (pooled coverage 92.5%; per-occupation median 100%), exposed concrete operational gaps (catalog-hygiene and canonical-tag seeding gaps), and enabled scalable, human-audited retirement of legacy forms.

Key Points

• Unit of generation: the system reconstructs provider preference tag catalogs per occupation (not a single global occupation hierarchy).
• Iterative loop: a propose–evaluate–keep cycle runs independently per occupation, seeded from a baseline prompt and allowing a small number of targeted prompt edits (default budget 5).
• LLM stack and cross-family design:
  • Generator: GPT-5-4.
  • Judge: GPT-5-4-mini (zero temperature).
  • Critics, editor, parity mapper: Claude Sonnet 4.6 (different family to reduce shared bias).
• Six-rubric judge (per-tag, sum up to 15):
  • Screening vs. Intake (0–3)
  • Tag Legibility (0–3)
  • Preference Variance (0–3)
  • Cross-Category Consistency (0–2)
  • Canonical Coverage (0 or 2)
  • Information Loss / Traceability (0–2)
• Set-level penalty ρ(·) corrects for issues the per-tag rubrics miss: tag proliferation (near-duplicates) and legacy-coverage gaps.
• Seven-persona critic panel provides weighted penalties (and a small bonus channel for reasoning quality); critics include PM, Pro, Taxonomy, Adversarial, Reasoning-Quality, Occupation-Coherence, and Parity. No hard vetoes—penalties adjust composite score.
• Editor agent: proposes one targeted prompt edit per iteration, focusing on the weakest rubric dimension not recently targeted.
• Parity mapping: maps tags ↔ legacy Q&A many-to-many; used to compute coverage, identify unmapped Q&A (intake-only or missing preferences), and produce human-reviewable artifacts (JSON + sheet) before deployment.
• Deployment results (summary):
  • Live across 132 occupations (April 2026).
  • 14-day cohort: 1,840 enrolled pros, 9.3M filter evaluations.
  • Legacy answers mapping: 73.3% map directly to generated tags; 20.8% were intake-only and intentionally excluded; 6.0% were regrettable misses.
  • Measured against preferences that should map: pooled coverage 92.5%; per-occupation median coverage 100%, mean 97.8%.
  • Operational gaps found:
  • Catalog-hygiene gap: deprecated canonical tags still emitted by request-time enrichment.
  • Seeding gap: 40.66% of filter evaluations failed because the required canonical tag was not present on the pro’s profile (deployment-side seeding missing).
• Human sign-off: mandatory human QA before deploying any generated catalog.

Data & Methods

• Data sources:
  • Legacy request-form (RF) Q&A per occupation and category.
  • Recent pro enrollment/select/deselect signals (e.g., top-30 most-deselected answers over 90 days used by parity critic).
  • Production filter-evaluation logs (9.3M evaluations in cohort).
• Method pipeline:
• Seed prompt p(0) + occupation data D_o → Generator G produces candidate tag set T.
• Judge E scores each tag on six rubrics; per-tag sums → set mean s ∈ [0,15].
• Subtract set-level penalty ρ(T) for proliferation and coverage gaps → pre-critic composite â.
• If â > best_so_far, run seven critics in parallel to compute weighted penalties π_k(T); adjusted composite a = â − Σ w_k π_k(T).
• Editor R proposes one prompt edit per iteration (targets weakest rubric not edited recently); accept if new composite improves.
• On loop termination (budget exhausted or plateau), parity mapping P(T*, Q_o) produces many-to-many tag↔Q mappings, unmapped Q and tags, and artifacts for human QA.
• Model families intentionally separated to reduce shared failure modes (proposal/evaluation vs. critique/mutation run on different models).
• Recalibration: judge rubrics were tuned against PM review to account for practical interpretability (e.g., penalizing bare-noun tags).

Implications for AI Economics

• Catalog reconstruction is a foundational public-good for AI-native matching markets:
  • Shared representational primitives (canonical and specialty tags) are necessary for probabilistic matching, pricing models, and cross-category generalization.
  • High-fidelity reconstruction preserves provider screening autonomy while enabling AI inference of consumer intent.
• Market efficiency and liquidity:
  • The reported seeding gap (40.66% failed filter evaluations due to missing canonical tags) illustrates how schema migration risk can translate to lost matches, reduced lead flow, and lower marketplace liquidity unless migration and seeding are managed.
  • Ensuring parity coverage minimizes frictions in redirection from structured forms → probabilistic matchers, helping preserve match rates and allocation efficiency.
• Incentives and strategic behavior:
  • Adversarial critic highlights gaming risks: poorly constructed tags can be misused by providers to surface leads they cannot service, creating negative externalities and search quality degradation. Multi-critic evaluation plus human sign-off mitigates, but does not eliminate, strategic risk.
• Operational economics of scaling:
  • Per-occupation parallel generation enables rapid onboarding (new occupations onboardable within hours), which supports scale and responsiveness to market evolution—but requires monitoring per-occupation hygiene metrics (coverage, proliferation) to avoid heterogeneous quality.
• Model risk and governance:
  • Cross-family evaluation and mandatory human QA are practical governance mechanisms to reduce correlated LLM biases; economic policy should budget for ongoing recalibration, audits, and human-in-the-loop checks to prevent long-term degradation of market outcomes.
• Recommendations for marketplaces:
  • Track and report catalog metrics (parity coverage, unmapped Q&A, tag proliferation) as operational KPIs tied to marketplace health.
  • Prioritize canonical-tag seeding and remediation to prevent large-scale failed evaluations and lost matches.
  • Use autonomous reconstruction but require human sign-off and periodic audits; combine automated coverage enforcement with economic monitoring (match rates, lead conversion) to detect downstream effects.
• Broader research implications:
  • The per-domain autoresearch approach suggests economic value in decomposing global schema problems into many local optimization problems to reduce semantic interference and produce occupation-specific, operationally meaningful constructs.