On-Device AI on Phones: The 4 tok/s Reality

Your phone can run a large language model inside a browser tab. No server, no API key, no text leaving the device.

Here is what that looks like in September 2026: 4 to 9 tokens per second on flagship Android phones. A 76-second wait for the first token on a long prompt. Tabs that reload mid-generation with no catchable error. And feature detection that reports full WebGPU capability on devices that complete zero inference runs.

This article is the full mobile study behind the one-section summary in Browser LLMs + WebMCP + MCP UI: The Coming Frontend Stack. Same thesis, all the receipts. You get a devices × models × prompt-lengths benchmark matrix, the five ways phones fail, and the warmup-gating pattern that turns “does WebGPU exist?” into “can this phone actually do it?”.

TL;DR

• Mobile browser inference works within narrow bounds. 3B-class models decode at ~4–5 tok/s on the Pixel 7 (January 2024) and Pixel 8 Pro (January 2026). A 1B model reaches ~9–10 tok/s on the Pixel 8a (June 2026), and 34 tok/s on the iPhone 17 Pro Max with a purpose-built engine (March 2026).
• The generation gap dominates everything. An iPhone 11 Pro Max (2019) completes zero inference runs; an iPhone 17 Pro Max (2025) flies. Chip age matters more than OS version.
• Feature detection lies on phones. An iPhone reporting webgpu: true with f16 support, a Pixel in an in-app browser reporting a full 4 GB ceiling, and Adreno GPUs passing every check still fail real runs. Benchmark — never just detect.
• Long context is the mobile killer, not decode speed. Prefill runs at ~5–15 tok/s on phones, so time-to-first-token explodes past ~200 input tokens: 76–77 seconds at 1,200 tokens versus ~3.8 seconds at 52.
• The shipping pattern is a warmup gate. Load the smallest probe model, run a tiny timed inference, and route to full local, lite local (short prompts only), or server fallback. Measured TTFT decides, not the user agent.
• Mobile WebGPU reach as of September 2026: Android 12+ on Chrome 121+ (January 2024) across Qualcomm, ARM, Intel, and Imagination GPUs, with Samsung Xclipse expected around Chrome 154. Safari 26 is default-on (September 2025). Firefox Android is still behind a flag.

What You Will Learn Here

• The full mobile benchmark matrix: which devices, models, and prompt lengths actually work in a phone browser tab.
• What “mobile WebGPU” covers in September 2026 — and where the gaps are.
• The five failure modes that are specific to phones, with symptoms and mechanisms.
• The warmup-gating pattern: decision flow, thresholds, and a runnable snippet.
• How to run mobile evals correctly: matrix design, thermal discipline, and metered-download etiquette.
• Which models fit in a phone tab, and what to build, prototype, or wait on.

The Benchmark Matrix: Read This First

Every number below is a measured in-browser run on a real phone, with its date and source. Methodologies differ — WebLLM usage stats versus wall-clock timing, cold versus warm cache, Q4 versus Q8 quantization — so treat these as order-of-magnitude truth, then measure your own matrix.

Two rows deserve a footnote before we go further. The iPhone 17 Pro Max numbers come from real-device testing via LambdaTest with a purpose-built 18-shader WGSL engine and Q8 quantization. Impressive, but a single source with a custom stack — not a stock-engine result. The desktop Firefox row (~1 tok/s versus ~52 on Chrome for the same model) is there to make a point this article returns to. On every platform, the browser and engine you measure with matter as much as the silicon.

What “Mobile WebGPU” Means in September 2026

WebGPU is the only path to GPU-accelerated LLM inference in a phone browser — WebLLM is WebGPU-only, and the fast paths in Transformers.js and wllama need it too. Here is the per-platform truth, from primary sources:

Two ecosystem notes round out the picture. First, Transformers.js v4 (February 9, 2026) rewrote its WebGPU runtime in C++ with 200+ supported architectures. But its own coverage notes call mobile “still rough” and recommend WASM fallback as the realistic path for broad audiences. The official WebGPU guide still warns that non-Chromium mobile behavior is experimental. Second, a community project called web-ai-bench is building the crowdsourced WebNN-vs-WebGPU-vs-WASM compatibility matrix the ecosystem lacks — worth watching, and worth contributing your phone’s results to.

My take: the API surface is basically shipped on the two engines that matter (Chromium Android, WebKit iOS). Everything hard now lives above the API: drivers, memory budgets, thermal envelopes, and hosts that report capability they cannot deliver.

Reading the Matrix: Five Lessons

1. The generation gap beats the OS version. The iPhone 11 Pro Max runs Safari 26 and reports WebGPU with f16 — and completes nothing. The iPhone 17 Pro Max runs the same OS family and decodes a 1B model at 34 tok/s. On Android the story is flatter but consistent: 3B-class decode sat in the 4–5 tok/s band from the Pixel 7 (January 2024) to the Pixel 8 Pro (January 2026). Progress came from smaller models and better engines, not from phone GPUs getting dramatically faster at this workload. Segment your mobile strategy by chip generation, not OS number.

2. Prefill is the wall; decode is the floor. The Pixel 8 Pro versus PC comparison in web-llm #759 is the clearest illustration. Prefill collapses ~20× (111 → 5.4 tok/s). Decode drops only ~3× (18 → 5 tok/s). That asymmetry is why TTFT explodes with prompt length while streaming, once started, feels merely slow. Design rule: budget your prompt tokens like money and stream every token you generate.

3. Engine choice is a multiplier, not a detail. Ludion’s cross-engine runs found the same model on the same hardware varying several-fold by engine. web-llm #524 reports Gemma 2 2B and Phi 3 Mini crashing the tab under WebLLM while the same-class .gguf weights run — “crawl” is the reporter’s word — under wllama. And the custom webgpu-gemma engine hits 34 tok/s where stock engines manage ~9 on comparable Apple silicon. If your numbers disappoint, swap the engine before you blame the phone.

4. Browser choice dominates more than benchmark posts admit. Firefox at ~1 tok/s versus Chrome at ~52 on the same M3 (Llamas on the Web, May 2026; interactive results at the benchmark site) is the desktop version of the LINE in-app browser stalling. Same story on phones: Chrome 149 on the Pixel 8a completes runs that stall in its in-app sibling. “Works in the browser” is not a statement about the web platform — it is a statement about one browser build on one GPU driver.

5. Compare quantizations explicitly or don’t compare. The eye-catching iPhone numbers are Q8_0; most WebLLM phone numbers are Q4 variants. Different quantization means different memory traffic, different quality, and different download sizes (~1 GB for the 1B Q8 versus ~700 MB-class for 1–1.5B Q4). Every benchmark table you publish — including the one in this article — should carry the quant column, or readers will draw lines between incomparable points.

Why Phones Fail the Way They Do

Desktop inference fails with error messages. Phone inference fails with reloads, stalls, and “Aw Snap” pages. Five failure modes, each with its symptom and mechanism:

1. OOM tab kill. The classic. Android’s low-memory killer terminates Chrome mid-inference (the adb logcat in web-llm #209 shows lmkd killing the renderer to free ~5 GB RSS), or the GPU device is lost mid-run. WebLLM 0.2.36 learned to catch createBuffer() OOMs (PR #402, May 2024). But error scopes and device.lost proved unreliable before the crash: Chrome keeps asking for memory until the tab dies. Treat process death as a normal outcome and design recovery, not just handling.

2. Silent reload (iOS). On the iPhone 11 Pro Max, Llama-3.2-1B reached generation on the WebGPU backend and then the page reloaded mid-generation — no JavaScript-visible exception, no catchable OOM. The smaller Qwen2.5-0.5B via wllama reloaded at init. Zero runs completed across every engine and model. Your error telemetry will show nothing; only run-completion accounting catches this.

3. In-app-browser stall. The Pixel 8a in LINE’s in-app browser reported webgpu: true, an Arm Valhall adapter with f16, and the full 4 GB buffer ceiling. Those limits are identical to the Chrome run on the same phone that completed. It stalled every run. Nothing in the adapter limits distinguished viable from non-viable. This is the single strongest argument for the warmup gate below.

4. Driver failure. On Qualcomm Adreno GPUs — the dominant Android GPU family — WebLLM 0.2.84 fails engine init with VK_ERROR_DEVICE_LOST even for 0.5B–1B models (web-llm #836). The reporter notes 0.2.79 worked while 0.2.80+ fails (0.2.80 moved CPU work to GPU kernels), which makes this version-sensitive, not just hardware-sensitive. Pin your engine version and re-test on every bump.

5. Standby worker death. A phone going to sleep can terminate the Web Worker holding your loaded model. The frontend still thinks a model is loaded, so the next inference fails with “a valid external Instance reference no longer exists” (web-llm #524). WebLLM fixed this for service workers (PR #471) and then web workers (PR #533, npm 0.2.56) by reloading on backend/frontend state mismatch. Design for reload anyway: re-verify loaded state after visibilitychange.

One mitigation is worth naming. ggml-webgpu PR #21533 (April 2026) throttles in-flight command buffers on iOS, where the WebGPU backend otherwise crashes. A matching WebKit bug tracks whether the limit is expected behavior. Engine authors are tuning per-platform; your job is to stay current and re-measure.

The Warmup Gate: Measure, Don’t Assume

The pattern falls out of everything above. Your availability check must be run a tiny inference and time it, not read an adapter limit. Ludion ships this idea as a routing policy with six rules derived from 54 archived runs across five device configurations (policy). In-app browsers, WebGPU-less environments, and iOS WebKit route to server. Desktop is allowed broadly. Android Chromium is allowed only for short prompts (≤200 tokens) with streaming, and unknown territory defaults to server. The snippet below — built on WebLLM’s CreateMLCEngine — is the same idea as a per-session probe:

flowchart TD
    Start["User reaches AI feature"] --> GPU{"navigator.gpu + requestAdapter?"}
    GPU -- "no" --> Server["tier: server — cloud LLM"]
    GPU -- "yes" --> Load["Load smallest probe model"]
    Load --> Warm["Timed warmup: tiny prompt, 30s cap"]
    Warm -- "fail / timeout" --> Server
    Warm -- "TTFT under 8s" --> Full["tier: full — local inference"]
    Warm -- "TTFT over 8s" --> Lite["tier: lite — short prompts only"]
    Full --> Re["Re-probe on crash, standby, or update"]
    Lite --> Re
    Server --> Re

// Warmup gate for WebLLM — measure TTFT on a tiny prompt before enabling features.
// Run once per session (ideally after a user gesture); cache the verdict.
import { CreateMLCEngine } from "@mlc-ai/web-llm";

const WARMUP_MODEL = "SmolLM2-360M-Instruct-q0f16-MLC"; // smallest viable probe; confirm in prebuiltAppConfig for your version
const WARMUP_TIMEOUT_MS = 30000;
const FULL_TIER_TTFT_MS = 8000;

export async function probeLocalTier() {
  if (!navigator.gpu) return { tier: "server", reason: "no-webgpu" };
  const adapter = await navigator.gpu.requestAdapter().catch(() => null);
  if (!adapter) return { tier: "server", reason: "no-adapter" };
  // Adapter limits are advisory only: phones reporting full ceilings have
  // failed every inference run (Ludion, June 2026). The timed warmup is the check.
  const engine = await CreateMLCEngine(WARMUP_MODEL, { logLevel: "WARN" });
  const t0 = performance.now();
  const timeout = setTimeout(() => engine.interruptGenerate?.(), WARMUP_TIMEOUT_MS);
  try {
    const stream = await engine.chat.completions.create({
      messages: [{ role: "user", content: "Reply with exactly: OK" }],
      max_tokens: 8,
      stream: true,
    });
    for await (const chunk of stream) {
      const ttft = performance.now() - t0; // first chunk received = TTFT
      clearTimeout(timeout);
      await engine.unload?.();
      if (ttft > WARMUP_TIMEOUT_MS) return { tier: "server", reason: "warmup-timeout" };
      return ttft < FULL_TIER_TTFT_MS
        ? { tier: "full", ttftMs: Math.round(ttft) }
        : { tier: "lite", ttftMs: Math.round(ttft), maxPromptTokens: 200 };
    }
  } catch (err) {
    return { tier: "server", reason: `warmup-failed:${err?.name ?? "error"}` };
  }
  return { tier: "server", reason: "warmup-empty" };
}

Three design notes. First, the 8-second full-tier threshold and 200-token lite cap are starting points from the measured data (Pixel 8a: ~3.8s TTFT at 52 tokens, unusable at 1,200). Calibrate against your task’s patience budget, not mine. Second, the probe doubles as shader warmup. First inference after load takes 2–3× longer while WebGPU compiles shaders. A gate that only checks navigator.gpu would both over-admit (broken devices) and mis-time (cold shaders). Third, re-probe on the events that change the answer: tab crash, return from standby, engine version bump, and model swap.

Measuring Right: Mobile Eval Methodology

If the warmup gate is the runtime pattern, this is its build-time twin: the matrix that tells you which tiers to offer. The parent article’s eval series covers golden-task sets, rubric scoring, and LLM-as-judge — read that for correctness method. This section is about performance method on phones, where everything that is a footnote on desktop becomes the headline.

Run a devices × models × prompt-lengths matrix. Minimum viable is three devices (a mid-range Android, a flagship Android, one iPhone on iOS 26+) × two models (0.5B and 1B; add 3B only for flagships) × two prompt lengths (~50 and ~1,200 tokens). Add a second engine if you can (WebLLM plus wllama or Transformers.js). Ludion’s 54 runs across five environment configurations with cold cache at both prompt lengths is the template to copy (repo).

Record TTFT at both lengths — it is the number that predicts viability. Time to first streamed chunk at ~50 tokens tells you whether the feature feels alive. TTFT at ~1,200 tells you whether long context is even offerable. This snippet is the whole instrument:

// Record TTFT at 50 and 1200 input tokens — the two numbers that predict viability.
// engine: a loaded WebLLM MLCEngine; prompts: your own short/long fixtures.
async function measureTtftCurve(engine, shortPrompt, longPrompt) {
  const run = async (prompt) => {
    const t0 = performance.now();
    const stream = await engine.chat.completions.create({
      messages: [{ role: "user", content: prompt }],
      max_tokens: 32,
      stream: true,
    });
    for await (const chunk of stream) return performance.now() - t0; // first chunk
  };
  const ttftShort = await run(shortPrompt); // ~50 tokens
  await new Promise((r) => setTimeout(r, 5000)); // cool-down: thermals skew back-to-back runs
  const ttftLong = await run(longPrompt); // ~1200 tokens
  return { ttftShortMs: Math.round(ttftShort), ttftLongMs: Math.round(ttftLong) };
}

Treat crash and reload as first-class results, not missing data. A run that reloads the tab is a zero, not an N/A — the iPhone 11 Pro Max’s “0 successful rows” is the most informative cell in Ludion’s dataset. Log completion rate alongside tok/s or your averages will lie by survivorship.

Apply thermal discipline. Phones throttle under sustained GPU load. Cool down between runs (the SitePoint WebGPU-vs-WASM methodology mandates cooldown between suites). Randomize run order, and record it. A 3B model that scores 5 tok/s cold and 3 tok/s hot is telling you about the chassis, not the weights.

Respect metered connections. A 1.5B model costs a ~728 MB download (Ludion’s Qwen2.5-1.5B figure); the 1B Q8 costs ~1 GB with ~1.5 GB GPU memory (gemma-webgpu). Cache aggressively (WebLLM and wllama both cache weights locally), warn before first download, offer Wi-Fi-only toggles, and study Range-request layer streaming. It fetches ~44 MB weight shards and frees JS memory as you upload to GPU. It keeps peak JS memory near 50 MB on iPhones.

Debug on-device, record the environment. Android: chrome://inspect with USB debugging. iOS: Safari Web Inspector over cable. Per device, record the chrome://gpu (or equivalent) dump plus a webgpureport adapter report. Driver version and reported limits are the only way to make sense of the numbers six months later.

Which Models Fit in a Phone Tab

The memory math decides before benchmarks do. Approximate budgets, with the honest attribution each row deserves:

Rows marked “estimate” are my synthesis from reported crashes and successes, not vendor specs. WebLLM exposes per-model vram_required_MB and buffer_size_required_bytes metadata — read those at runtime instead of trusting any table. Two structural notes. First: WebLLM’s -1k model variants exist because 128 MB maxStorageBufferBindingSize devices (Galaxy S23-class, web-llm #209; gate implemented in PR #256) cannot address larger per-buffer allocations. Context length is a memory-layout question before it is a quality question. And KV cache grows with context, so every extra 1k of prompt is paid twice: once in prefill time, once in memory that can tip a flagship into a reload.

The model to watch is Gemma 3n (developer guide, June 26, 2025). It is a mobile-first open-weights family with MatFormer nested submodels (E4B contains E2B; up to 2× faster inference on the submodel). Per-layer embeddings keep only ~2B/4B parameters in accelerator memory (2 GB/3 GB footprint), and KV-cache sharing brings 2× prefill gains. The guide names transformers.js among its supported tools — but as of September 2026 there are no published in-browser phone numbers for it. File under “most likely to change this table,” not under “ship today.”

Build, Prototype, or Wait: The Mobile Edition

Build now: warmup-gated summarization, extraction-to-JSON, and classification on short inputs. Add a server-fallback architecture where the phone tab is the fast path, not the only path. Add WASM fallback for embeddings and small tasks on devices that fail the gate. And ship honest download UX (progress, size warnings, Wi-Fi-only option). Register any page tools through WebMCP once — the parent article’s mobile inversion (tools first, model second) still holds.

Prototype: full local chat on the flagship tier, Range-streamed weight loading, and cross-engine A/B (WebLLM versus wllama versus Transformers.js v4) on your actual target devices. Add per-generation tuning: what flies on a 2025 iPhone crawls on a 2019 one, so ship different defaults.

Wait: built-in AI on mobile (desktop-only; Android prototyped with no ship date — see the parent article’s table). Also wait on Firefox Android WebGPU (flagged) and Samsung Xclipse support (~Chrome 154). And on any plan that depends on in-app browsers behaving like full browsers.

Production Checklist

• Gate on measured TTFT, never on UA string or adapter limits. (iOS 26 freezes the UA at iPhone OS 18_6 — detection-by-identity is broken by design.)
• Count crashes and reloads as results; alert on completion-rate drops per device class.
• Re-probe after standby resume, engine upgrade, model swap, and browser update.
• Warn before gigabyte downloads; cache-first; consider Range streaming past ~500 MB.
• Cool down between eval runs; record run order, driver versions, and adapter dumps.
• Ship a no-AI path: every feature must work (slower, server-side, or manually) with local inference unavailable. The accessibility argument from the eval series applies doubly on phones, where the failure rate is highest.
• Instrument tier outcomes (full / lite / server) in product analytics — your warmup gate is a measurement instrument wearing a feature flag’s clothes.

The Real Shift

The parent article’s closing line was “reach for the tab first.” Phones add the qualifier: reach for the tab first after the warmup passes.

The phone tab is a real inference runtime with a 4 tok/s floor and a ~200-token attention span. Design inside that envelope — short prompts, streamed output, measured gating, server fallback — and it works offline, privately, and free. Assume desktop, and the tab reloads under you without even an exception to log.

Sources

• Browser inference field notes — Ludion report (June 2026 cross-device test: Pixel 8a TTFT curve, iPhone 11 zero-completion, LINE IAB stall; 728 MB Qwen fetch in policy v0 rationale)
• Ludion-ai/Ludion (54 archived runs, 5 configs, R1–R6 routing policy, desktop contrast numbers)
• Chat demo does not work on Android because of maxStorageBufferBindingSize — web-llm #209 (Nov 2023–May 2024: Pixel 7 4.4 tok/s, 128MB gate, -1k models, lmkd kills)
• Regarding the slow performance issue of Android Chrome — web-llm #759 (Jan 3, 2026: Pixel 8 Pro 5.1 tok/s, phone-vs-PC prefill collapse)
• Gemma 2 2B crashes on mobile phone — web-llm #524 (WebLLM-crash vs wllama-crawl, standby worker death, PR #471/#533 fixes)
• WebLLM Engine Initialization Fails on Qualcomm Adreno GPUs — web-llm #836 (Adreno VK_ERROR_DEVICE_LOST on 0.5–1B models, 0.2.80+ version sensitivity)
• Disable most models on Android phone — web-llm PR #256 (Jan 2024: the original buffer-size hard-gate)
• Catch WebGPU OOM error — web-llm PR #402 (May 2024: createBuffer() OOM catching, npm 0.2.36)
• svenflow/webgpu-gemma (Mar 13, 2026: iPhone 17 Pro Max 34.4 / 101.1 tok/s via LambdaTest)
• gemma-webgpu — npm (Range-request streaming, ~50 MB peak JS memory, 18 WGSL shaders)
• Llamas on the Web — paper (May 2026: 16-GPU survey; low cluster is iPhones + Adreno/Mali/PowerVR at 4–17 tok/s decode, only the 4 smallest models fit; Firefox ~1 vs Chrome ~52 on M3)
• Llamas on the Web — benchmark (warmup-then-measure protocol, iPhone tab-crash warning, wllama upstream WebGPU)
• ggml-webgpu: parameterize submission size and add iOS specific limits — llama.cpp PR #21533 (Apr 3, 2026: iOS command-buffer throttling, WebKit bug 311598)
• Implementation Status — gpuweb wiki (per-platform WebGPU ship status: Android vendors, Safari 26, Firefox flags)
• What’s New in WebGPU (Chrome 121) (Android 12+, Qualcomm/ARM launch scope)
• Chrome 121 for Android Update (Jan 23, 2024 release date)
• WebKit Features in Safari 26.0 (Sep 15, 2025: WebGPU default-on, frozen iOS UA string, Transformers.js named)
• Intent to Ship: WebGPU on Android — blink-dev (Adreno/Mali Android 12+ initial scope)
• Running models on WebGPU — Transformers.js docs (device: "webgpu" usage, non-Chromium experimental warning)
• Transformers.js v4 Ships WebGPU Runtime (Feb 9, 2026: C++ runtime, 200+ archs, “mobile still rough”)
• WebGPU vs WebASM: Browser Inference Benchmarks — SitePoint (cooldown/hardware-tier methodology, WASM fallback guidance)
• Introducing Gemma 3n: The developer guide (Jun 26, 2025: MatFormer, PLE, 2–3 GB footprint, KV-cache sharing)
• Build an offline AI chatbot with WebLLM and WebGPU — Appwrite (model VRAM table: SmolLM2 380 MB, Phi-3.5 3.7 GB, Llama-3.1-8B 5 GB)
• abhid1234/web-ai-bench (crowdsourced WebNN/WebGPU/WASM matrix project)
• WebLLM documentation (engine API: CreateMLCEngine, OpenAI-compatible chat, workers)
• WebLLM — repository (in-browser LLM engine; WebGPU-only, OpenAI-compatible API)
• Fix service worker reload mismatch — web-llm PR #471 (standby state-mismatch reload for service workers)
• Move worker reload fix to web worker — web-llm PR #533 (standby state-mismatch reload for web workers, npm 0.2.56)
• WebKit bug 311598 — iOS WebGPU command-buffer limits (tracks whether the iOS inflight-submission limit is expected behavior)
• Browser LLMs + WebMCP + MCP UI: The Coming Frontend Stack (parent article: stack overview, desktop numbers, eval harness)
• Evaluating AI Agents with LangWatch: From Vibes to Scores (golden-task and rubric methodology)
• LLM-as-a-Judge: Agent App Evals, Biases, Fixes (judge-based scoring for prose tasks)