AI rewriting agents raise model-predicted chances of getting job referrals by 14% for weaker requests while leaving strong requests intact; however, the improvement is measured by a prediction model, not by real-world referrals.

Main Finding

Using LLMs to rewrite strangers’ job-referral requests increases model-predicted likelihood of getting referrals for weaker requests but can reduce it for stronger requests; augmenting the LLM with Retrieval‑Augmented Generation (RAG) (contextual example retrieval + light editorial guidance) prevents harms to strong requests and amplifies gains for weak ones. Overall, LLM+RAG raises predicted success for weaker requests by ~14% without degrading performance on stronger requests. These gains are measured against a fine‑tuned sentence‑transformer reward model (AUROC = 0.681) used as a proxy for real referral outcomes.

Key Points

• Context and outcome
  • Data from Blind’s “Jobs & Referrals” channel (Feb–Nov 2024). Raw collection: 17,542 posts and 27,150 comments.
  • Success label: a post is “successful” if it receives at least one explicit referral offer (e.g., “DM me for Google”).
  • Many posts get no offers: 54% of requests receive no referral offers.
• Evaluation model (reward function)
  • Three candidate encoders: TF‑IDF, a featurized semantic model, and a sentence transformer.
  • Best performing model: sentence transformer (all‑distilroberta‑v1 fine‑tuned with LoRA), AUROC = 0.681, accuracy ≈ 63%. Calibration shows predicted probabilities align reasonably with observed referral rates.
  • Training/test split uses a temporal holdout (train on data before Sept 24, test after Sept 25) to reduce leakage.
• Basic LLM workflow
  • Improver agent: calls gpt‑5‑mini to rewrite title and body under system/user prompts; mask tokens ([ROLE], [LOCATION], etc.) used to hide credentials.
  • Evaluator agent: compares predicted success (reward model) before vs after rewrite.
  • Result: basic LLM revisions raise predicted success for low‑quality requests but can reduce predicted success for high‑quality requests (asymmetric effects).
• RAG workflow
  • Retriever agent: indexes high‑quality historical requests, filters to examples with higher predicted success relative to the target, and returns the closest 5 examples by embedding cosine similarity; one‑time indexing trims extreme outliers and clusters top examples for fast retrieval.
  • Explainer agent: provides sentence‑level and title ratings (strong/moderate/weak) to guide editing; however, ablation shows most gains come from the retriever—ratings modestly helpful.
  • Result: RAG prevents degrading edits on strong requests and amplifies improvements on weak requests; net effect is +14% predicted success for weak requests with no degradation for strong ones.
• Design choices and safeguards
  • Masking of sensitive/inherent attributes to force edits to focus on writing quality rather than exploiting unobservable advantages.
  • Prompting discourages inventing facts; retrieval examples are context‑filtered to avoid irrelevant stylistic transfer.
• Limitations (highlighted by authors)
  • Reward model is a proxy: improved model‑predicted success does not guarantee more real‑world referrals or hires.
  • Observational label (offers seen on platform) cannot verify off‑platform follow‑through (actual referrals or hires).
  • Potential for LLMs to introduce subtle biases or manipulate signals in ways not captured by the proxy.

Data & Methods

• Data
  • Source: Blind (teamblind.com), “Jobs & Referrals” channel.
  • Time window: Feb 29 – Nov 17, 2024.
  • Raw counts: 17,542 posts, 27,150 comments. Modeling sample (after filtering/masking) reported N ≈ 11,358 for some analyses.
  • Mask tokens added for profession, years, seniority, employer, etc., and tokenizers updated to recognize these.
• Outcome definition
  • Binary success = at least one platform comment that is an explicit referral offer (e.g., “DM me”).
• Predictive models
  • TF‑IDF (unigrams + bigrams), featurized semantic model (binary attributes inspired by prior literature + linguistic metrics), and sentence transformer (all‑distilroberta‑v1, 768‑dim embeddings).
  • Classification head: logistic regression with L1 penalty; LoRA used to fine‑tune transformer embeddings.
  • Metrics: AUROC (main), accuracy, precision, recall, F1; bootstrapped CIs reported.
  • Temporal train/test split to assess generalization over time.
• Improvement workflows
  • Basic: LLM rewrite (gpt‑5‑mini) + evaluator (reward model).
  • RAG: retriever selects contextual, high‑quality examples; explainer rates segments; improver uses retrieved examples + ratings in prompts.
  • Retrieval details: index top successful requests (trim extremes), cluster via FAISS; for each query compute p, compute candidate pool with predicted success higher than p by at least half the potential gap (pmax – p)/2, select 5 nearest by cosine similarity.
• Evaluation of rewrites
  • Compare reward‑model predicted probabilities before vs after edit, stratified by baseline predicted quality (weak vs strong).
  • Ablations: remove ratings, remove retriever, etc., to attribute gains.

Implications for AI Economics

• Low‑cost pre‑testing of interventions
  • Using a learned reward model as a proxy enables fast, cheap iteration on interface/agent designs before committing to field experiments. This can reduce costs/time in development of platform interventions or marketplace features.
• Markets for writing/assistance services
  • AI agents that boost the persuasiveness of requests can change access to networked opportunities (referral markets). There is commercial potential for tools that improve outreach to gatekeepers (referrers, hiring managers).
• Distributional effects and inequality
  • The asymmetric effect (largest gains for weaker requests) suggests AI assistance could help less‑connected or less‑effective communicators close some access gaps. But it also creates the possibility of an arms race: if everyone uses assistance, the baseline standard may shift, potentially neutralizing gains and favoring those who can pay for premium agents.
• Equilibrium and platform effects
  • Widespread use of high‑quality automated requests may alter supply/demand for referrals (referrers could become more selective, change norms, or implement stricter filters). Platforms may need to manage volume or authenticity (spam, gaming).
• Measurement and validation recommendations
  • Model‑predicted gains are promising but insufficient. Economists should run randomized controlled trials (A/B tests) on live platforms to measure effects on: (a) referral offers, (b) interview callbacks, (c) hires and retention, and (d) referrer welfare/costs.
  • Track downstream matching and hiring outcomes, not just platform signals; measure externalities (e.g., increased workload for referrers).
• Policy, fairness, and ethics
  • Masking credentials is a useful design to reduce unequal exploitation, but further work is needed to audit for bias (gender/race inference from text, occupation stereotypes).
  • Platforms might regulate agent use (labels, rate limits, verification) to preserve authenticity and avoid crowding out organic referrals.
• Research agenda suggestions
  • Field experiments to validate model predictions and estimate causal impacts on hiring.
  • Study general equilibrium: how referrer behavior adapts when many applicants use agents.
  • Welfare analysis: quantify gains for applicants vs costs on referrers and platforms.
  • Cost–benefit of different RAG designs (retriever quality, number of examples, explainer granularity).

If you want, I can: - Draft an experimental design (RCT) to validate predicted gains on a live platform (power calculations, treatment arms, outcomes), or - Produce a short slide‑style brief summarizing the paper for nontechnical policy audiences.