WebLLM

WebLLM is a TypeScript library that runs large language model inference directly inside a web browser, using WebGPU for hardware acceleration. No server-side processing happens: the model downloads, caches, and runs entirely on the client. It targets web developers who want to add LLM-powered chat, JSON generation, or assistant features to a browser app without standing up an inference backend or sending user data to a third-party API.

The engine exposes an interface compatible with the OpenAI chat completions API, so code written against OpenAI's SDK patterns (streaming, JSON mode, message roles) largely carries over. This makes it straightforward to swap a cloud LLM call for a fully local, in-browser one, or to build apps that work offline after the initial model download.

Key features

• In-browser inference: runs model computation in the browser via WebGPU, with no server-side component involved.
• OpenAI API compatibility: uses the same chat.completions.create interface, including streaming, JSON mode, logit-level control, and seeding.
• Structured JSON generation: generates JSON output against a custom schema, implemented in WebAssembly for speed.
• Extensive model support: ships with Llama 3, Phi 3, Gemma, Mistral, and Qwen model families, with the full list available via prebuiltAppConfig.model_list.
• Custom model integration: lets you compile and deploy your own models in MLC format when the built-in list doesn't cover your needs.
• Plug-and-play packaging: installable via npm, yarn, pnpm, or directly from a CDN (jsdelivr) for quick prototyping on platforms like JSFiddle or Codepen.
• Streaming responses: supports real-time token streaming for chat UIs.
• Worker thread support: can offload inference to a Web Worker or Service Worker to keep the UI responsive and avoid reloading the model on every page visit.
• Chrome extension support: includes examples for building browser extensions, including ones that keep the model alive via service workers.
• Configurable cache backends: supports Cache API, IndexedDB, OPFS, and an experimental cross-origin storage extension for caching downloaded model weights.

Ideal use cases

WebLLM fits projects that need LLM features without a backend: privacy-sensitive chat assistants, offline-capable tools, browser extensions, demo apps, or prototypes where you don't want to manage API keys or server infrastructure. It's a good match if your users have WebGPU-capable browsers and you can tolerate a model download on first load (cached afterward).

It's not a good fit if you need guaranteed low latency on underpowered or older hardware, since inference speed depends entirely on the client's GPU. It's also not suited for workloads requiring very large models beyond what a browser and local GPU memory can handle, or for server-side batch processing where a traditional LLM server or API would be more efficient. If your users' browsers lack WebGPU support, the engine won't run as intended.

Installation

Install the package with your preferred package manager:

# npm
npm install @mlc-ai/web-llm
# yarn
yarn add @mlc-ai/web-llm
# or pnpm
pnpm install @mlc-ai/web-llm

Import it in your code:

import * as webllm from "@mlc-ai/web-llm";
// or
import { CreateMLCEngine } from "@mlc-ai/web-llm";

Or load it directly from a CDN, useful for quick prototyping in tools like JSFiddle or Codepen:

import * as webllm from "https://esm.run/@mlc-ai/web-llm";

Create an engine and load a model:

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const initProgressCallback = (initProgress) => {
  console.log(initProgress);
};
const selectedModel = "Llama-3.1-8B-Instruct-q4f32_1-MLC";

const engine = await CreateMLCEngine(selectedModel, {
  initProgressCallback: initProgressCallback,
});

Then run a chat completion:

const messages = [
  { role: "system", content: "You are a helpful AI assistant." },
  { role: "user", content: "Hello!" },
];

const reply = await engine.chat.completions.create({ messages });
console.log(reply.choices[0].message);

For heavier apps, move the engine into a Web Worker or Service Worker using CreateWebWorkerMLCEngine or CreateServiceWorkerMLCEngine to keep the UI responsive and avoid reloading the model across page visits.