A Bengali-capable legal AI, MINA, matches or beats average human performance on Bangladesh Bar-exam tasks and drafts while costing a fraction (≈0.1–0.6%) of traditional legal services, suggesting big potential to expand low-cost access to justice—though real-world client outcomes and deployment risks remain untested.

Main Finding

MINA is a multilingual, LLM-powered legal assistant adapted to the Bangladeshi legal context that, using a two-stage RAG pipeline plus agentic orchestration and jurisdictional tooling, attains human-comparable performance on high-stakes national certification exams (Bangladesh Bar Council 2022–2023). In evaluations, MINA consistently scored ~75–80% across MCQ, written, and viva-voce stages, matching or exceeding average human performance while operating at roughly 0.1–0.6% of typical human lawyer cost (implying a ~167–1000x cost reduction).

Key Points

• Purpose and gap: Targets access-to-justice deficits in Bangladesh by providing low-cost, bilingual (Bengali/English) legal assistance tailored to local statutes, archaic/Farsi-influenced terminology, and low-resource constraints.
• System architecture:
  • Orchestrator Agent decides routing (direct answer vs. retrieval).
  • RAG Agent executes a two-stage retrieval (Act-level then Section-level) over separate vector DBs to avoid conflation of unrelated provisions.
  • Tools include keyword generation, web search & parsers (used selectively), file reader (.pdf/.docx/.pptx), a Legal Dictionary for archaic terms, Chat/Question relevance analyzers, and a socio-economic simulation module.
• Multilingual retrieval and grounding:
  • Cohere embed-multilingual-light-v3.0 for bilingual embeddings.
  • Two separate vector DBs: Act summaries (compact searchable representations) and chunked Section entries with Act metadata for precise filtering.
• LLMs and prompting:
  • Evaluated many models (proprietary and open-source, small→large). Prompt engineering used chat history, uploaded docs, and RAG state.
• Evaluation:
  • Benchmarked on real Bangladesh Bar Council exams (MCQ, Written, Viva Voce) for 2022 and 2023.
  • MCQs: automated marking (5 runs per setup to reduce variance).
  • Written & Viva: assessed by law faculty; written answers scored by at least two judges and averaged; viva simulated in chat with evaluators giving granular scores.
  • Best-performing setups (two-stage RAG + tools) achieved roughly 75–82% depending on model and year; the paper reports MINA overall at ~75–80% across stages.
• Practical outcomes:
  • Provides context-aware legal drafts, statute citations, plain-language Bengali explanations, and document generation via an interactive chat UI.
  • Emphasis on low-resource efficiency and selective external tool usage to reduce cost and complexity.
• Claims on cost-efficiency:
  • Operating cost reported at ~0.1–0.6% of human lawyer costs (paper frames this as enabling scalable, affordable legal assistance).
• Adaptation for jurisdictional specifics:
  • Legal Dictionary for colonial-era/Farsi-influenced terms and explicit handling of bilingual/mixed-language inputs.
• Validation & robustness:
  • Evaluation spanned modalities and years; authors report error analysis and deployment insights (details summarized; full analyses in paper appendices).

Data & Methods

• Legal corpus:
  • As of Apr 2025: 595 Acts, 18,023 Sections (avg ~24 sections/Act).
  • Average lengths: Act title ~50.3 chars; Section content ~737 chars.
• Retrieval design:
  • Two-stage RAG: retrieve top-n Acts (default n=5) using LLM-generated semantic keywords over Act summaries; filter Section DB within retrieved Acts to get top-n Sections (default n=10).
  • Act DB entries include LLM-generated summaries + metadata; Section DB contains chunked sections with Act linkage to avoid cross-Act conflation.
• Embeddings & indexing:
  • Cohere embed-multilingual-light-v3.0 used for bilingual semantic search; Chroma vector stores used for indexing.
• Agents and orchestration:
  • LangGraph-style state machine enabling persistent memory across multi-turn interactions.
  • Orchestrator Agent decides whether to use RAG, reuse chat/doc context, or call tools.
• Tools:
  • Keyword Generator (LLM-based, with regex fallback), Web Search (DuckDuckGo), Web Page Parser (BeautifulSoup), File Content Reader, Question Relevance Analyzer, Legal Dictionary, Socio-Economic Simulation.
• LLMs evaluated:
  • Proprietary: GPT-4o, Gemini-2.0/2.5-Flash, etc.
  • Open-source: gemma/gemma-3 variants, Llama3 variants (incl. 70B), Qwen 30B, Cohere command-a series.
  • Retrieval/agent setups compared: no-RAG, naive RAG, 2-step RAG, and 2-step RAG + tools.
• Evaluation protocol:
  • MCQ: automatic marking; 5-run averages per model/setup.
  • Written: 13-question structure; scoring by at least two qualified judges; 5-evaluator averages reported in tables.
  • Viva: simulated chat viva with evaluators; scored out of 100 and averaged.
• Reported metrics:
  • Scores reported as percentages. Tables in paper show performance by model and setup; best results generally with 2-step RAG + tools and larger models (commercial and large open-source).
• Limitations noted by authors:
  • Bengali NLP resource scarcity, tokenization/NER challenges, potential for hallucination (mitigated via RAG and metadata), and social deployment issues (digital literacy, gendered access, offline availability). Full error analyses and appendices are referenced for more detail.

Implications for AI Economics

• Large potential for cost-effective scaling of legal services:
  • Reported operating cost of 0.1–0.6% of human lawyers implies an AI-to-human cost ratio of ~1:1,000 to ~1:167. If accurate and generalizable, this suggests massive reductions in marginal cost for routine legal assistance.
• Demand-side effects:
  • Lower transaction costs and simplified legal language could unlock latent demand from previously unserved low-income populations, increasing total market size for basic legal services (document drafting, statutory interpretation, triage).
• Labor and task reallocation:
  • Likely substitute effect for routine drafting/research tasks; complementary effects for higher-value lawyer activities (strategy, courtroom advocacy, negotiation). This could shift legal labor demand toward supervision, quality assurance, and complex litigation.
• Welfare and justice externalities:
  • Faster access and improved legal literacy may reduce reliance on informal mechanisms, potentially lowering social costs from unresolved disputes and improving enforcement of property/contract rights—outcomes with broad economic productivity implications.
• Public provision and subsidy considerations:
  • Low operational costs make public-service AI deployments (legal aid, pro bono scaling) financially plausible, but sustained impact requires investment in localization, governance, monitoring, and offline/low-bandwidth access.
• Quality, liability, and regulation:
  • Economic gains are contingent on reliable grounding, rigorous validation, and liability frameworks. Mis-advice in legal contexts has outsized social costs; thus, regulation, certification, and hybrid human-AI workflows will be critical to capture benefits while limiting harms.
• Investment priorities:
  • High social return expected from investments in multilingual embeddings, localized legal corpora digitization, and domain-specific prompt/tooling rather than solely scaling model size.
• Distributional impacts:
  • Potential to reduce inequality in legal access, but benefits depend on digital inclusion policies (literacy, connectivity, gender equity). There is risk that private deployment without oversight could create new asymmetries (e.g., differential access to premium model versions).
• Research and monitoring needs:
  • Economic evaluations should quantify real-world demand elasticity, labor-market adjustments in the legal profession, impacts on litigation rates and court backlogs, and the value of improved legal outcomes (e.g., faster dispute resolution, reduced enforcement costs).
• Caution on cost claims:
  • The paper’s cost percentages are compelling but require transparency on what is included (compute, data labeling, human-in-the-loop review, deployment overhead). Policymakers and funders should require end-to-end costing and robustness checks before scaling.

If you’d like, I can: - Extract the specific score tables for the top-performing model/setups and compute exact cost-reduction multipliers; or - Draft a short policy brief with recommended regulatory and pilot-deployment steps for public-sector adoption in Bangladesh.