How you ask matters: prompts framed with urgency or pressure lead LLMs to produce less correct and less secure code across multiple open models and benchmarks; conversely, other influence-style framings shift tone, documentation, and hallucination rates but do not uniformly improve correctness.

Main Finding

Prompt framings inspired by psychological influence tactics measurably change LLM code-generation behaviour. In a large-scale study, certain framings — notably Pressure/urgency-style prompts — were associated with lower functional correctness and increased security weaknesses. More broadly, different tactic-based prompts produced systematic qualitative differences (tone, explanation, hallucination rates, commenting) that can affect code quality, maintainability, and reliability.

Key Points

• Research question: Do human-inspired influence tactics (from organizational psychology) — when encoded as prompt templates — alter LLM outputs in software engineering tasks?
• Influence tactics: Eight tactics were operationalized using items from the Influence Behaviour Questionnaire-General (IBQ-G) into reproducible prompt templates (examples include rational persuasion, ingratiation, exchange, pressure).
• Scale: Evaluated across five open-weight LLMs on two benchmarks, producing over ~123,000 generations on LiveCodeBench and ~57,000 on SWE-bench Verified (human-validated subset).
• Evaluation dimensions: Functional correctness, code quality, maintainability, and security.
• Main empirical result: Pressure/urgency framings correlated with reduced correctness and poorer security. Other tactics produced detectable but heterogeneous effects (some increased explanation/detail; others increased hallucinations).
• Theoretical interpretation: Effects likely arise from distributional/linguistic cues in training data (LLMs pick up associations between pragmatic framing and response style), not from the model “understanding” persuasion.
• Deliverables: A taxonomy-aligned set of reproducible prompt templates and a mixed-methods analysis (quantitative test-run metrics + qualitative triangulation coding).

Data & Methods

• Benchmarks
  • LiveCodeBench: 1,055 programming problems (easy/medium/hard).
  • SWE-bench Verified: 500 human-validated GitHub issues.
• Prompt design
  • Eight influence-tactic prompt templates derived from IBQ-G and Yukl & Falbe taxonomy; templates included tactic-specific lexical/pragmatic cues (e.g., urgency/demands for Pressure; personal appeal for Personal Appeal).
  • Prompts run zero-shot (no fine-tuning) and multiple runs per prompt to capture variance.
• Models
  • Five open-weight LLMs (paper reports results across multiple leading open models; names not listed in excerpt).
• Generation and evaluation pipeline
  • Extracted code snippets from model responses.
  • Ran generated code against benchmark test suites to measure functional correctness.
  • Applied automated/heuristic analyses for code quality, maintainability (e.g., comments, modularity indicators), and security (static patterns, known misuses).
  • Quantitative dataset sizes: ~123k generations for LiveCodeBench; ~57k for SWE-bench Verified.
• Qualitative triangulation
  • Random sampling and codebook development to code tonal/structural differences, hallucination types, and explanatory content.
• Statistical approach
  • Compared tactic framings against neutral baselines; reported associations (directional effects such as reduced correctness/security for some tactics). Multiple runs used to estimate variance; exact effect sizes not provided in excerpt.
• Reproducibility
  • Authors provide the prompt templates as a resource for the community.

Implications for AI Economics

• Productivity and output quality
  • Prompt framing affects correctness/security: small linguistic shifts can change developer productivity and rework costs. Economic models of LLM-augmented engineering should treat prompt phrasing as an operational variable that influences output quality and thus downstream labor/time cost.
• Risk and cost of deployment
  • Framings that increase security weaknesses (e.g., urgency/pressure) raise expected mitigation costs, liability exposure, and potential costs of incidents. Risk assessments and expected-value models for deploying code-generation assistants must incorporate prompt-framing sensitivity.
• Incentives and behavior design
  • Product designers and platform owners can (and should) shape user prompts (e.g., via templates, UI affordances or guardrails) to incentivize high-quality outputs and reduce risky framings. These design choices affect realized productivity gains and their distribution across firms and workers.
• Pricing and productization
  • Prompt engineering (templates, UX controls, safety wrappers) becomes a monetizable feature. Firms can charge for curated, high-quality prompt libraries or prompt-optimization services that demonstrably improve correctness/security.
• Measurement and evaluation
  • Benchmarks for economic evaluation of LLM tools should include prompt-framing sensitivity analyses. Simple accuracy metrics under a single prompt are insufficient for cost-benefit modeling; firms should stress-test across pragmatic framings typical of real users.
• Labor-market and task allocation effects
  • If novice users naturally use more urgent/pressured language and this degrades output, the realized substitution of human effort by LLMs may be less than predicted. Training and governance investments (prompt literacy) influence the pace and distribution of automation gains.
• Adversarial and regulatory considerations
  • Linguistic framing can be weaponized: adversarial prompt injections or malicious UX that encourage risky framings could degrade outputs. Regulators and auditors should consider prompt-framing audits as part of safety/compliance checks.
• Research and policy priorities
  • Need for cost-centric experiments: quantify how framing-induced changes translate into hours of developer time, bug-fix costs, security incident probabilities, and firm-level ROI. Policy interventions (standard prompts, required testing) can be evaluated economically.

Suggested practical steps for stakeholders - Firms: Standardize and pre-fill safe, high-quality prompt templates in developer tooling; monitor prompt logs to detect high-risk framings (e.g., pressured/urgent language). - Product managers: Treat prompt templates and prompt-A/B tests as levers for product performance and safety; include framing sensitivity in MVP evaluations. - Economists/researchers: Measure effect sizes in monetary terms (time to fix, defect rates, incident costs) and study distributional impacts across skill levels and sectors.

Limitations (to bear in economic modeling) - Effects reported are associative and benchmark-based; translating these to real-world monetary impacts requires mapping correctness/security deltas to time, cost, and incident probabilities. - Study uses open-weight models and two benchmarks; effects may differ on closed models, multi-turn agentic workflows, or domain-specific corpora. - Operationalization of influence tactics is one plausible set of templates; alternative phrasings could produce different magnitudes/directions.

Bottom line Prompt framing is an economically consequential design choice: language that humans use to persuade or pressurize also shifts LLM-generated code quality and security. Incorporating prompt-framing sensitivity into cost, risk, and product design models will lead to more accurate assessments of LLMs’ value and required safeguards.