A modular multimodal coaching system, SETU, produces explainable, persona-aware feedback for multilingual sales pitches and—on an internal 18-recording pilot—raised coach agreement (ρ=0.78) and cut coach review time by about 41%, though results are preliminary and limited to a small, internal dataset.
SETU: An Agentic Ecosystem for Multilingual, Persona-Aware Communication Coaching
SETU is an agentic, multimodal, persona-aware coaching ecosystem for multilingual sales pitches that produces explainable modality-attributed feedback and, in an 18-video pilot, improved coach agreement and reduced coach review time.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Corporate training teams need scalable and explainable tools to improve workforce communication in multilingual settings. Existing systems often score text, audio, or video in isolation, or produce black-box outputs that are difficult to audit for coaching use. This paper presents SETU, an agentic ecosystem for corporate communication coaching aimed at recruiters, frontline sales professionals and training units who prepare for audience specific conversations. SETU is designed for two scoped scenarios: (i) recruiter-candidate eligibility-and-interest calls with persona context and (ii) sales pitches with target-audience adaptation; owing to limited evaluation resources, this paper reports results on scenario (ii) only. The ecosystem decomposes analysis into specialized video, audio-speech, text-relevance, scoring, notification and reporting agents coordinated through trust-aware orchestration. It generates modality-attributed coaching reports for formative training, with human reviewers retaining final judgment. The name SETU (bridge in several Indic languages) reflects the goal of bridging communication gaps across regional languages and audience expectations.
Summary
Main Finding
SETU is an agentic, multimodal ecosystem that produces explainable, persona-aware coaching for multilingual sales pitches by decomposing evaluation into specialized agents (video, audio/speech, text/relevance, relevance, scoring, notification, report) coordinated via a trust-aware orchestrator and human-in-the-loop review. In a pilot sales-pitch study (18 clips, multilingual/code-mixed), SETU yielded higher coach agreement and better explainability than text-only and non-agentic baselines while reducing coach review time.
Key Points
-
Architecture
- Agentic decomposition: separate agents for visual behavior, acoustic/speech analysis, transcript/text scoring, and persona-target relevance; extended agents for sync, orchestration, scoring and reporting.
- Trust-aware orchestration: agent trust = α·accuracy + β·consistency + γ·runtime reliability; trust weights modulate aggregation so noisy modalities are down-weighted.
- Human-in-the-loop: coaches validate and finalize reports; SETU is positioned as formative coaching (not autonomous hiring/selection).
-
Persona & Multilingual Focus
- Persona-aware scoring: relevance agent judges pitch against explicit buyer persona and prompt (e.g., SME CEO expects ROI, concise delivery).
- Multilingual/code-mixed support: STT via Sarvam (saaras:v3 codemix), language detection with fallback heuristics; code-switching allowed but flagged when excessive.
-
Explainability & Outputs
- Modality-attributed scores and traceable evidence for each recommendation; coaches can inspect which agent produced which deduction.
- Coaching reports include visual, audio, text, relevance scores, language profile, trace links, and recommended rewrites.
-
Pilot results (sales-pitch only)
- Dataset: 18 webcam-recorded pitches (45–120 s; mean 78 s). Languages: English, Hindi, Telugu, and English–regional code-mix variants.
- Representative metrics: coach agreement Spearman ρ = 0.78; explainability rating 4.3/5; language macro-F1 = 0.86.
- Latency: batch mean 38.6 s per clip; live alert refresh 2–4 s.
- Operational impacts: coach review time reduced from 22 → 13 minutes/session (≈40.9% reduction); average practice cycles to reach Fair/Good fell from 4.1 → 2.6.
- Baselines: SETU outperformed text-only LLM feedback (ρ = 0.61) and non-agentic multimodal prompt (ρ = 0.69).
-
Formalization & Implementation
- System modeled as M = (A, T, S, E, O, C, Mem); agents xi represented as (πi, ϕi, ψi) for policy, feature extraction, and structured output.
- Orchestration maps subtasks to agents O(T,S,A,Mem) → {(xi, tj)} and optimizes reliability vs coordination cost.
- Feature toolchain: MediaPipe landmarks (video), Librosa acoustic features (RMS, pitch, silence ratio, SNR, WPM, filler density), Sarvam STT, LLM rubric scoring for text/relevance.
-
Limitations reported
- Small, internal pilot (18 clips) and scoped to sales-pitch scenario; recruiter scenario described but not evaluated.
- Dependence on audio/video quality and ASR robustness for code-mixed speech; cultural/appearance biases (eye contact, posture) need careful governance.
- Need for larger benchmarks, fairness audits, longitudinal evaluation.
Data & Methods
-
Data
- Pilot corpus: 18 pitch videos, 720p webcam, mono office-mic audio; products: HR–payroll SaaS (8), EdTech (5), Insurance (5).
- Target personas: SME CEO, School Teacher, Mid-level Manager, DINK in IT, Retired Government Officer.
- Language mix: English (4), Hindi (3), Telugu (2), English–Hindi (5), English–Telugu (4); includes English–Bengali example in case study.
-
Preprocessing & Features
- Video: MediaPipe Holistic landmarks → posture stability, eye-contact approximation, facial expressiveness, head movement; temporal smoothing.
- Audio: STT (Sarvam codemix), Librosa features (RMS energy, pitch variation, silence ratio, SNR, WPM, filler density); language detection using API confidence + script heuristics.
- Text: transcript fed to LLM rubric scoring for grammar, fluency, professionalism, semantic completeness and persona alignment.
-
Agents & Orchestration
- Agents output modality-specific scores (0–10) and structured evidence; relevance agent scores persona-task fit on a 0–50 scale then normalizes to 0–10.
- Trust-weighted aggregation to produce final report; orchestrator implemented as LangGraph workflow with shared AgentState.
-
Evaluation protocol
- Comparisons against two baselines: B1 (text-only LLM feedback) and B2 (non-agentic multimodal summary).
- Measured metrics: modality sub-scores, overall score, Spearman ρ vs human coaches, explainability rating (1–5), language macro-F1, latency, coaching-time reduction.
Implications for AI Economics
-
Productivity & Training ROI
- Quantified coach-time savings (≈41% per session) and faster upskilling (fewer practice cycles) imply direct cost savings for corporate training programs and faster time-to-competence for sellers; these can improve marginal productivity of training spend.
- Multimodal, persona-aware coaching supports higher-quality human capital development where communication skill is economically valuable (sales, recruiting, client-facing roles).
-
Market and Productization Opportunities
- Modular agentic design enables componentized markets: specialized vendors for video analytics, multilingual ASR, persona-relevance LLMs, and orchestrators—lowering integration costs and allowing incremental upgrades.
- Demand for localized, code-mixed ASR and persona-tuned LLM scoring in emerging markets (e.g., Indic languages) opens niche markets and differential pricing opportunities.
-
Measurement & Procurement
- Explainable, modality-attributed scoring improves auditability for HR procurement and contracts—making it easier to specify SLAs (accuracy, latency, fairness) and to measure ROI of coaching platforms.
- Trust-weighting and human-in-loop governance reduce regulatory and legal risk (bias/ fairness concerns), potentially lowering compliance costs and accelerating procurement in regulated industries.
-
Labor Economics & Augmentation vs Displacement
- SETU is augmentation: reduces routine coach workload and shifts coach effort toward higher-value validation/mentoring rather than scoring. May increase coaching capacity and change the skills demand for trainers (more emphasis on rubric design, governance).
- There is potential for partial displacement of entry-level evaluative tasks, but human oversight remains central for high-stakes judgments—moderating immediate displacement risks.
-
Adoption Frictions & Externalities
- Upfront costs: data collection, ASR tuning for local code-mix, fairness audits, and rubric calibration—these are non-trivial and may delay adoption, especially for smaller firms.
- Externalities: if deployed widely without fairness safeguards, automated scoring could amplify accent, gender, or appearance biases; economic benefits depend on transparent governance and validated benchmarks.
-
Research & Public-Good Needs
- Broader, expert-labeled benchmarks for multilingual/code-mixed multimodal evaluation will be valuable public goods to reduce asymmetric information between vendors and buyers and to enable credible cost–benefit assessments.
- Standardized fairness and ROI evaluation frameworks would help organizations compare offerings and measure economic outcomes.
Caveat: The economic implications above are conditional on the pilot findings being robust at scale. The paper’s evaluation is small and internal; scaling, external validation, fairness audits and improved ASR for low-resource/code-mixed languages are necessary before generalizing projected economic impacts.
Assessment
Paper Typedescriptive
Evidence Strengthlow — The paper reports a small internal pilot (N=18 videos) without randomized assignment, external benchmark datasets, or out-of-sample validation; quantitative metrics (e.g., Spearman ρ, latency, coach-time reduction) are reported but on limited data and with potential selection and rubric subjectivity.
Methods Rigormedium — The authors provide a clear systems design, mathematical formulation, modality-specific agents, and an evaluation protocol with baselines and quantitative metrics; however, the empirical evaluation is limited in scale, lacks third-party benchmark validation, and depends on subjective coach rubrics and an internal dataset.
SampleInternal pilot dataset of 18 recorded sales-pitch videos (45–120s, mean 78s) captured at 720p with mono office-microphone audio; products: HR–payroll SaaS (8), EdTech (5), insurance (5); target personas include SME CEO, School Teacher, Mid-level Manager, DINK, Retired Government Officer; languages include English, Hindi, Telugu and code-mixed English–Hindi/English–Telugu/English–Bengali; each clip labeled Good/Fair/Poor by sales coaches using a rubric.
Themesskills_training human_ai_collab
GeneralizabilityVery small sample (N=18) and internal dataset limits external validity, Limited set of buyer personas and product categories; may not generalize across industries, Evaluated languages/code-mix combinations are a subset of Indic and global languages; ASR and relevance scoring may degrade on other languages or dialects, Controlled webcam/audio capture and office noise settings may not reflect field conditions (phone calls, poor cameras, heavy background noise), Coach rubric subjectivity and small number of human raters can bias score labels and agreement metrics, System-level performance hinges on ASR and LLM components whose behavior may change with upstream model updates, No longitudinal outcome measures of real sales performance or hiring outcomes; only immediate coach-time and practice-cycle proxies
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| On the 18-session pilot dataset, SETU achieved coach agreement of Spearman ρ = 0.78. Decision Quality | positive | Agreement between SETU pitch evaluations and coach judgments |
Reading fidelity
high
Study strength
low
|
n=18
ρ = 0.78
|
| SETU outperformed the text-only LLM baseline and the non-agentic multimodal baseline on coach agreement and explainability in the pilot set. Decision Quality | positive | Coach agreement and perceived explainability of pitch feedback |
Reading fidelity
high
Study strength
low
|
n=18
Coach agreement: ρ = 0.78 vs. 0.61 and 0.69; explainability: 4.3/5 vs. 2.9/5 and 3.4/5
|
| Buyer-persona-conditioned relevance scoring improved agreement with coaches by 0.11 Spearman-ρ compared with non-persona prompting. Decision Quality | positive | Coach agreement for pitch relevance judgments |
Reading fidelity
high
Study strength
low
|
n=18
0.11 in ρ
|
| SETU reduced coach review time from 22 minutes to 13 minutes per session, a 40.9% reduction. Organizational Efficiency | positive | Human coach time required to review a sales pitch session |
Reading fidelity
high
Study strength
low
|
n=18
40.9% reduction
|
| The average number of practice cycles needed to reach a Fair or Good rating decreased from 4.1 to 2.6 when using SETU. Training Effectiveness | positive | Number of practice cycles required to reach a Fair/Good pitch rating |
Reading fidelity
high
Study strength
low
|
n=18
decreased from 4.1 to 2.6 practice cycles
|
| SETU achieved a mean batch-processing latency of 38.6 seconds per clip and a live alert refresh time of 2–4 seconds. Organizational Efficiency | positive | System processing latency and live alert refresh time |
Reading fidelity
high
Study strength
low
|
n=18
38.6 s per clip; 2–4 s live alert refresh
|
| SETU achieved a language macro-F1 of 0.86 for the evaluated multilingual sales-pitch language handling. Output Quality | positive | Multilingual language detection or profiling performance |
Reading fidelity
high
Study strength
low
|
n=18
language macro-F1 0.86
|
| SETU received an average explainability rating of 4.3 out of 5 from coaches in the pilot. Ai Safety And Ethics | positive | Coach-perceived explainability of the generated coaching feedback |
Reading fidelity
high
Study strength
low
|
n=18
4.3/5
|
| In the representative weak payroll-SaaS pitch to an SME CEO, SETU assigned visual, audio, text, and relevance scores of 3.8/10, 4.1/10, 4.6/10, and 5.4/10, respectively. Output Quality | negative | Multimodal sales-pitch performance scores |
Reading fidelity
high
Study strength
speculative
|
n=1
Visual 3.8/10; audio 4.1/10; text 4.6/10; relevance 5.4/10
|
| The pilot dataset consisted of 18 recorded sales-pitch videos covering three product types, five buyer-persona groups, five pitch stages, and English, Hindi, Telugu, and code-mixed language conditions. Other | mixed | Coverage and composition of the evaluated sales-pitch dataset |
Reading fidelity
high
Study strength
low
|
n=18
18 videos
|
| The authors caution that the pilot results should be validated on larger expert-labeled benchmarks before deployment at scale. Governance And Regulation | negative | External validity and readiness for large-scale deployment |
Reading fidelity
high
Study strength
high
|
n=18
|