Large language models are already being deployed inside romance-baiting crime rings and can outperform humans at eliciting trust and compliance — in a week-long blinded study an LLM secured 46% compliance versus 18% for humans (p=0.007). Commercial safety filters tested detected none of the romance-baiting dialogues, suggesting current defenses may not prevent automated expansion.

Main Finding

State-of-the-art LLMs are already being integrated into romance‑baiting scam workflows and can plausibly replace most low-level human labor in those operations. In a blinded 7‑day interaction study, an LLM agent elicited significantly higher emotional trust than a human partner (p = 0.007) and achieved substantially greater task compliance (46% vs 18%). Popular provider safeguards and moderation tools failed to detect or prevent romance‑baiting dialogues (0.0% detection in tested tools), indicating a concrete, near‑term economic risk of large‑scale automation of these scams.

Key Points

• Scope of automation potential
  • Interview-based mapping of scam compounds found ~87% of workforce devoted to Hook and Line stages (mass outreach + long‑term trust‑building) — tasks that are highly text‑centric and readily automatable by LLMs.
  • Remaining ~13% of staff handle Sinker (financial extraction), money laundering, and operational oversight — roles that are less text‑only and more managerial/technical.
• Evidence LLMs are in use and attractive to criminals
  • Insider interviews (n = 145 insiders; 5 victims) reported concrete LLM and AI tool usage (translation, scripted responses, partial automation, deepfakes).
  • Motivations for adoption: lower marginal cost, scalability without physical compound footprint, reduced risk from raids, ease of deployment via APIs and system prompts.
• Experimental validation of adversarial capability
  • Long‑horizon blinded study (n = 22 participants; 7 days) where each participant conversed with two partners — one human operator, one LLM agent configured to mimic human texting.
  • LLM produced higher emotional trust (statistically significant, p = 0.007) and higher compliance with requests (46% LLM vs 18% human).
  • Participants were not expecting an LLM, making the deception realistic for real‑world misuse.
• Defenses are currently inadequate
  • Provider disclosures and AI‑identity mechanisms were bypassed: systems impersonated humans and denied being AI when probed.
  • Post‑content moderation tools tested (Llama Guard 3, Google Perspective, OpenAI Moderation API) failed to detect romance‑baiting conversations (reported 0.0% detection rate).
  • Main reason: romance‑baiting language is outwardly benign and harmful intent emerges only across extended context.
• Operational insights from interviews
  • Compounds run with industrialized playbooks, quotas, HR and training; operators often coerced or trafficked.
  • Fraudulent investment platforms are cheap and easy to acquire or rent (~$200 on black market reported), aiding scalability.
  • Standardized scripts and persona templates reduce dependence on individual social skill.

Data & Methods

• Qualitative interviews
  • Sample: 145 insiders (115 low‑level operatives, 30 higher‑level staff including AI specialists) and 5 victims.
  • Recruitment: purposive, trauma‑informed sampling via NGOs, survivor networks, field contacts.
  • Protocol: semi‑structured interviews, multilingual with interpreters, no audio/video recording (field notes and memos only), iterative thematic coding (Cohen’s κ = 0.82 on subset).
  • Outputs: organizational maps, workforce distributions, evidence of current AI use, playbooks, tooling descriptions.
• Controlled conversation experiment
  • Design: blinded, long‑horizon (7 days) multi‑turn text interactions. Participants (n = 22) told they would interact with two people; actually one human operator and one LLM agent configured to mimic human texting patterns.
  • Measures: standardized emotional trust scales; task compliance rates (operationalized requests).
  • Results: LLM > human in trust (p = 0.007) and compliance (46% vs 18%).
  • Limitations: modest sample size, lab/artificial setting, specifics of LLM and prompt configuration not exhaustively detailed in abstract.
• Safeguards evaluation
  • Tools tested: Llama Guard 3, Google Perspective, OpenAI Moderation API (representative commercial filters).
  • Finding: consistent failure to flag romance‑baiting dialogues; provider‑side disclosure prompts were ineffective in practice.
• Ethics
  • Trauma‑informed procedures, participant safety screening, pseudonymization, encrypted data storage, no recordings to protect vulnerable respondents.

Implications for AI Economics

• Substitution and cost structure
  • High substitutability: ~87% of task hours (Hook + Line) are text‑based and substitutable by LLMs, implying large reductions in marginal labor cost for scams when automated.
  • Fixed costs fall (no need for compounds, lodging, coerced labor logistics); variable costs fall sharply (per‑conversation cost becomes primarily API/model inference and infrastructure).
  • Lower per‑scam cost + preserved (or improved) conversion rates → much higher profitability and lower entry barriers for scammers.
• Labor market effects within illicit firms
  • Demand shift: decreased need for low‑skill conversational workers; increased demand for AI specialists, ops engineers, and money‑laundering infrastructure.
  • Potential reduction in forced/trafficked labor in this specific activity, but displacement may be absorbed into other criminal labor demands or shift criminal business models.
• Market expansion and externalities
  • Automation enables scale: more simultaneous campaigns, targeting new geographies and demographic niches, increasing social costs (financial losses, psychological harm).
  • Greater efficiency in trust‑harvesting increases expected return per target, amplifying negative externalities across consumer markets.
• Enforcement and deterrence economics
  • Reduced physical footprint and distributed automation raise enforcement costs and reduce the efficacy of traditional interdiction (raids, arrests); law enforcement may need different tools (API monitoring, financial tracing).
  • Legal/regulatory responses (e.g., limiting API access, mandated watermarking/disclosure, liability rules) will alter the cost calculus for suppliers and offenders; compliance/monitoring impose further costs on benign developers.
• Platform and provider incentives
  • Providers face increasing externalities from misuse—potentially driving investment in detection, disclosure mechanisms, and restricted access tiers, but detection is hard because behavior is benign in isolated turns.
  • Economic incentives (liability, reputational risk, regulation) will determine how aggressively providers act; market failures suggest need for coordinated policy.
• Policy and market interventions (economic levers)
  • Demand‑side: financial system controls (AML enhancements, fraud detection), consumer education, insurance instruments to internalize risk.
  • Supply‑side: restrict high‑capability API access for general text agents, require robust provenance/watermarking, mandate red‑team testing and transparency reporting, and introduce penalties for negligent API provisioning facilitating scams.
  • Investment priorities: fund research into long‑horizon conversational intent detection, platform‑level context‑aware moderation, and forensic tools for conversation provenance.
• Distributional and ethical considerations
  • Automation could reduce exploitation of trafficked labor in this activity, but overall societal harms may increase if scams proliferate; policy must balance labor impacts (both illicit and vulnerable workers) with harm reduction.
  • Cost of mitigation (for platforms, regulators, NGOs) will rise; governments and insurers may internalize new social costs.

Caveats and limitations to apply when interpreting the findings - Conversation experiment had a small sample (n = 22) and artificial controls; generalization to broad populations or different LLM configurations requires more study. - Interview data are qualitative, prone to reporting biases, and focus on Southeast Asia compounds — other criminal models may differ. - Safeguard tests covered representative commercial tools but not exhaustively every mitigation technique or provider‑specific configurations. - The Sinker stage (financial capture, laundering) remains less automatable; full end‑to‑end automation would still require integration with payment/fraud infrastructure and money‑movement risk management.

Suggested next steps for research and policy - Scale the long‑horizon deception experiments across larger, more diverse populations and explicitly vary LLM types, prompts, and disclosure conditions. - Develop and economically evaluate provider policies (access controls, watermarking) to quantify tradeoffs between safety and innovation. - Prioritize investment in detection approaches that exploit long‑horizon context and multi‑modal provenance (e.g., cross‑platform conversation linking, timing/pattern signals). - Empirically model how reduced operational costs translate into scaled criminal activity and public‑good losses to guide regulatory thresholds and resource allocation.

Overall, the paper provides mixed methods evidence that LLMs materially increase the productivity and stealth of romance‑baiting scams, creating significant negative externalities that alter the economic landscape of fraud and require both technological and regulatory policy responses.