MOSAIC lets untrusted GPUs perform the heavy lifting of transformer inference without learning model or inputs, cutting client overhead to optimal levels and producing near–full-precision accuracy on 70B models while relying on standard post-quantum assumptions.

Main Finding

MOSAIC introduces a practical, cryptographically grounded protocol for outsourcing the dominant linear computations of large Transformer-style models (matrix multiplications) from a trusted but weak client to untrusted accelerators while keeping both model weights and client inputs confidential. By allowing a small, controlled approximation error (noisy unmasking) and using low-rank masking plus a small Gaussian noise term, MOSAIC attains optimal client asymptotic overhead O((m + n) l), concrete runtimes orders of magnitude faster than prior secure-outsourcing work, and model accuracy close to full-precision inference (e.g., on 70B Transformers). Security reduces to decisional LWE and LPN.

Key Points

• Core protocol
  • Masking: W' = W + L M + E_w and X' = X + N R + E_x, where L,M,N,R are low-rank uniform matrices and E_w, E_x are small-norm Gaussian noise matrices.
  • Untrusted server computes Y' = W' X' and returns it; client removes low-rank correction terms with precomputed material in O((m + n) l) work, recovering W X up to a small approximation noise.
  • Security reductions to decisional LWE (for Gaussian noise masking) and LPN (for recursive/transparent low-rank masking layers).
• Complexity and scalability
  • Client online complexity per outsourced multiplication is optimal up to the security-determined inner masking rank r: O(r (m + n) l) ≈ O((m + n) l) for fixed r.
  • Server (accelerator) work remains the full matmul cost O(m n l).
  • Inner rank r depends only on security parameter (e.g., for 140-bit security they use r ≈ 1536), so efficiency improves as model dimensions grow (typical modern n ≫ r).
• Approximation & accuracy
  • MOSAIC intentionally relaxes exact correctness: small Gaussian noise is introduced and propagated; error accumulation across many layers is mitigated with random Hadamard rotations (a cheap pre/post linear transform that scales error benignly).
  • Fixed-point quantization (16-bit) is used for cryptographic arithmetic; MOSAIC’s error + quantization yields accuracy comparable to or better than common quantization baselines.
  • Empirical results: for LLaMA-3-70B on WikiText-2, MOSAIC perplexity ≈ 1.8342 vs BF16 reference 1.8273 and NF4 ≈ 2.247. HumanEval performance matches BF16 in experiments reported.
• Performance (concrete runtimes)
  • Reported single secure matrix-vector (matvec) runtimes are small (server plus client). Example entries:
    • 36864 × 36864: MOSAIC total ≈ 8.86 ms (server 8.57 ms, client 0.29 ms)
    • 16384 × 16384: total ≈ 2.20 ms (server 2.04, client 0.16)
    • MOSAIC runtimes are orders of magnitude faster than the closest prior secure-outsourcing implementation reported in the literature (e.g., [6]).
  • End-to-end decoding/prefill runtimes for a 70B model: MOSAIC incurs a 3× (decode) to 5–11× (prefill) slowdown relative to non-confidential baseline on current hardware in their emulation — largely attributable to lack of native 32-bit integer tensor-core support (they emulate 32-bit arithmetic on 8-bit tensor cores, causing 10× underlying matmul launches).
• Deployment model and use-cases
  • Layer-by-layer outsourcing: client performs all non-linear ops (RMSNorm, softmax, SiLU, attention score computation) and outsources the heavy linear projections (WQ, WK, WV, WO, gate, up, down).
  • Primary use case: data-center confidential inference where a small trusted computing base (e.g., a client-side TEE) orchestrates work across an untrusted accelerator pool connected by low-latency interconnect (RDMA-like). This leverages existing multilayer/phase distributed inference patterns.
  • Secondary use case: remote decision inference over the Internet; for single forward-pass applications they report <5s for a 70B model.
• Practical considerations and limitations
  • Protocol operates over integer rings (fixed-point) and needs 32-bit integer accumulation; current fastest accelerator paths are optimized for FP/BF16/INT8, so MOSAIC presently emulates 32-bit integer matmuls on 8-bit tensor cores with overhead.
  • Approximate output (noise) is small but nonzero — acceptable in many LLM tasks but may be problematic for precision-critical applications.
  • Protocol requires per-layer communication and client-side non-linear computation; latency and round-trip counts matter, but modern data-center interconnects mitigate this for the primary use-case.
  • Security depends on standard post-quantum assumptions (LWE, LPN); obfuscation-style heuristics used in some prior works are avoided.

Data & Methods

• Cryptographic construction
  • Simple core masking (described above) and a full protocol that recursively nests masking (replacing dense low-rank terms with smaller-rank LPN instances) to keep client overhead optimal and security provable.
  • Recovery: the returned product expands to true W X plus low-rank correction terms and small-noise terms; the client removes low-rank terms using precomputed projections (cost O((m + n) l)), leaving only bounded Gaussian noise.
  • Error control: random Hadamard rotations applied to weight and activation matrices before quantization reduce error accumulation across layers by spreading/reshaping quantization and protocol noise.
• Fixed-point and quantization pipeline
  • Convert BF16 activations/weights to 16-bit fixed-point integers for protocol arithmetic; compute in integer ring (32-bit accumulation); after unmasking, cast back to BF16 and add biases.
• Security assumptions and parameters
  • Security reduction to decisional LWE for the Gaussian-masked construction; LPN used in recursive masking steps.
  • Example security parameter: 140-bit security with inner rank r ≈ 1536 (paper gives parameter examples).
• Implementations and experiments
  • End-to-end implementation published (github: https://github.com/jachiang/mosaic).
  • Benchmarking includes: microbenchmarks of secure matvec at large matrix sizes, layer-level end-to-end forward-pass experiments on LLaMA-3-70B (perplexity / HumanEval), and end-to-end latency studies for data-center and Internet scenarios.
  • Comparisons made to prior work categories: FHE (very slow), MPC/FSS (large overheads), trapdoor-matrix based outsourcing [6] (higher client cost O(m n ε l) and slower in reported CPU runtimes), and heuristic obfuscation (insecure in general).

Implications for AI Economics

• Lowers cost barrier for confidential inference at scale
  • MOSAIC enables outsourcing heavy linear compute to commodity (untrusted) accelerators without exposing model weights or client inputs, potentially lowering capital and operating costs for confidential-AI services compared with fully trusted hardware pools or heavy cryptographic options (FHE/MPC).
  • Because client work scales optimally with model dimensions, MOSAIC becomes relatively cheaper as models grow (n ≫ r), aligning with current industry trends toward larger models and more distributed inference.
• New product and monetization opportunities
  • Cloud providers and model vendors could offer confidential inference-as-a-service that preserves model secrecy while leveraging large accelerator pools, enabling monetization of proprietary weights with lower trust and attestation overhead.
  • Enterprise customers that must keep inputs private (e.g., health, finance) can outsource compute without moving trust to an entire accelerator fleet or building bespoke secure hardware.
• Infrastructure and hardware incentives
  • Current hardware limitations (no native 32-bit integer tensor cores) impose non-negligible emulation overhead. MOSAIC creates a clear incentive for accelerator vendors to add efficient 32-bit integer/mixed integer support to lower confidential inference cost and latency.
  • Network/interconnect investments (ultra-low-latency RDMA-style links) become more valuable since MOSAIC requires layer-by-layer communication but otherwise uses existing distributed inference patterns.
• Competitive and regulatory effects
  • MOSAIC could shift competitive dynamics: smaller providers can safely run proprietary models on shared accelerator pools without exposing weights to cloud operators, reducing bargaining power of large incumbent CSPs that currently control secure hardware.
  • From a regulatory standpoint, MOSAIC reduces attack surface for data exfiltration of client inputs and model IP — this may ease compliance burdens and insurance costs for deploying large models in regulated domains.
• Limitations that affect economics
  • Residual approximation error means MOSAIC may not be a drop-in replacement for all workloads (safety- or correctness-critical tasks may still require exact computation); some verticals may need additional validation/guardrails.
  • For low-latency, highly interactive workloads or highly fragmented (many tiny requests) remote-internet settings, round-trip overhead may increase apparent cost; data-center deployments with low-latency interconnect are the best economic fit initially.
  • Security rests on LWE/LPN (post-quantum) assumptions — broadly accepted but still an assumption; risk pricing may be needed for high-assurance contexts.

Overall, MOSAIC presents a practical, well-benchmarked path to confidential outsourcing of the computationally dominant linear layers in modern large models. By carefully trading exactness for efficiency and grounding security in standard post-quantum assumptions, it materially changes the cost–feasibility calculus for confidential AI at datacenter scale and creates clear hardware and product incentives for adoption.