What platforms does oMLX support?

Only Apple Silicon Macs running macOS 15.0 (Sequoia) or later, with Python 3.11-3.13. There is no Linux or Windows support since it relies on the MLX framework built for Apple Silicon.

Is oMLX free and open source?

Yes, it's licensed under Apache-2.0 and hosted on GitHub. The maintainer accepts optional donations via Buy Me a Coffee.

Can I use oMLX with existing OpenAI or Anthropic client code?

Yes, it exposes OpenAI-compatible endpoints like `/v1/chat/completions` and `/v1/embeddings`, plus an Anthropic-compatible `/v1/messages` endpoint, so existing clients can point at it with minimal changes.

Why do I need to build native custom kernels separately?

A plain pip install doesn't compile them, and model families like GLM-5.2 fall back to a much slower generic path (about 30x slower in the README's benchmark). Building them requires the full Xcode Metal toolchain; the official DMG ships them precompiled.

Does oMLX support running multiple models at once?

Yes, it can load LLMs, VLMs, OCR models, embeddings, and rerankers in the same server, managing memory via LRU eviction, manual load/unload, pinning, and per-model TTLs.

How does the multi-Mac inference feature work, and is it production ready?

It's explicitly experimental. Source builds can split one model across multiple Macs with unequal memory using MLX pipeline ranks over Ring or Thunderbolt RDMA, but the README flags security boundaries and current limitations that should be reviewed before use.

omlx: Local LLM inference server for Apple

oMLX is an LLM inference server built specifically for Apple Silicon Macs. It wraps mlx-lm and mlx-vlm with continuous batching, a two-tier KV cache (RAM plus SSD), and multi-model management, all exposed through OpenAI- and Anthropic-compatible APIs. It's aimed at developers running local LLMs, VLMs, embedding models, or rerankers on M1 through M5 Macs who want more control than a bare CLI tool but don't want to run a full server stack by hand.

The project ships as a native Swift/SwiftUI menu bar app, a Homebrew formula, or an installable Python package. Under the hood it handles model discovery from a directory of MLX-format models, per-model settings (aliases, TTLs, sampling parameters, chat template overrides), and an admin dashboard for monitoring, chat, benchmarking, and model downloads.

Key features

Tiered KV cache: block-based KV cache management with prefix sharing and copy-on-write, backed by a hot RAM tier and a cold SSD tier (safetensors format) so cached context survives even server restarts.
Continuous batching: concurrent request handling via mlx-lm's BatchGenerator, with a configurable max-concurrency limit.
Multi-model serving: LLMs, VLMs, OCR models, embeddings, and rerankers loaded in the same server, with LRU eviction, manual load/unload, model pinning, and per-model idle TTLs.
API compatibility: drop-in support for /v1/chat/completions, /v1/completions, /v1/messages (Anthropic), /v1/embeddings, /v1/rerank, and /v1/models, including streaming usage stats and vision inputs.
Tool calling: auto-detects function-calling formats across model families (Llama/Qwen JSON, Qwen3.5 XML, Gemma, GLM, MiniMax, Mistral, Kimi K2, and others), plus MCP tool integration.
Admin dashboard: web UI for monitoring, per-model settings, built-in chat, one-click benchmarking, and a HuggingFace model downloader, translated into eight languages and fully vendored for offline use.
Menu bar app: native Swift app to start/stop/monitor the server, with usage history, crash auto-restart, and auto-update.
Experimental multi-Mac inference: splits a model across memory-unequal Macs using MLX pipeline ranks over Ring or Thunderbolt RDMA, from source builds only.

Ideal use cases

oMLX fits developers who want a persistent, always-available local inference server on a Mac, particularly for coding assistants like Claude Code, OpenCode, or Codex, where context-heavy sessions benefit from SSD-backed KV cache reuse across requests and restarts. It also suits anyone juggling multiple models (chat, VLM, OCR, embeddings, rerankers) who wants automatic memory management instead of manually starting and stopping processes.

It's a good match if you're running quantized MLX models locally and want an OpenAI-compatible endpoint that other tools can point at without custom glue code. The admin dashboard and menu bar app also make it approachable for people who don't want to live in a terminal.

It is not a fit if you need Linux or Windows support, since it requires Apple Silicon and macOS 15+. It's also not the right choice if you need a distributed, production-grade multi-node cluster: the multi-Mac inference feature is explicitly experimental and requires source builds with security caveats. If you just need to run a single model once without caching or multi-model orchestration, a plain mlx-lm invocation may be simpler.

Installation

Three install paths are available. The macOS app is the simplest:

Download the .dmg from GitHub Releases, drag it to Applications, and launch. It includes auto-update and installs a CLI shim at ~/.omlx/bin/omlx.

Homebrew:

brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx
omlx start

Optional MCP support after a Homebrew install:

/opt/homebrew/opt/omlx/libexec/bin/pip install mcp

From source:

git clone https://github.com/jundot/omlx.git
cd omlx
pip install -e .          # Core only
pip install -e ".[mcp]"   # With MCP support

Some model families (GLM-5.2, MiniMax M3, Qwen3.5) benefit heavily from native custom kernels, which require the full Xcode Metal toolchain to build:

OMLX_WITH_CUSTOM_KERNEL=1 pip install -e .

Requirements: macOS 15.0+ (Sequoia), Python 3.11-3.13, and an Apple Silicon Mac (M1 through M5). After install, run the server with omlx serve --model-dir ~/models or manage it as a background service with omlx start / omlx stop / omlx restart.