What languages and dependencies does it require?

The engine is written in pure C as a single file plus small headers. It has no BLAS dependency and no Python runtime requirement; a GPU is optional, not required.

Can Colibrì run models larger than my available RAM/VRAM?

Yes. It streams routed experts from disk on demand while keeping the dense part of the model resident, so limited fast memory only slows inference; it does not block the model from running or change its precision.

Does Colibrì support running across multiple machines?

Yes, through a local cluster mode where a coordinator keeps token generation and routing local while separate worker machines execute routed feed-forward layers over TCP.

Is Colibrì meant for production use or research?

Both, but the project frames itself explicitly as an open research platform with no speed SLA; it guarantees semantic correctness (precision and routing never silently change) rather than guaranteed throughput.

Colibri: Run 744B-2.8T parameter MoE models

Q: Which models does Colibrì support?

Eight MoE families as of this README: GLM-5.2/5.3, GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6, and OLMoE, all through the same coli chat/serve/web front end.

Colibrì is an inference engine for Mixture-of-Experts (MoE) language models, written as a single C file with no external dependencies. It targets developers and researchers who want to run models in the 125B to 2.8T parameter range on hardware they already own, rather than renting GPU clusters or relying on hosted APIs.

The core idea is that a MoE model only activates a small fraction of its parameters per token. Colibrì keeps the dense part of the model (attention, shared experts, embeddings) resident in RAM at int4, while the much larger set of routed experts stays on disk and streams in on demand. A per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier decide which experts live where. The project treats VRAM, RAM, and NVMe storage as one placement hierarchy instead of a hard memory requirement: if fast memory is scarce, inference gets slower, but the model's precision and routing semantics never silently change.

Eight model families run through the same front end today: GLM-5.2/5.3, GLM-5.3-Flash (with vision), Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6, and OLMoE.

Key features

Expert streaming: routed experts load from disk on demand instead of requiring the full model to fit in RAM or VRAM.
Zero-dependency C engine: the runtime is a single C file plus small headers, no BLAS, no Python at runtime, no GPU required.
Heterogeneous backends: CPU, CUDA, Metal, and NUMA-aware memory placement share one runtime and can be combined per machine.
Learning cache: usage history (.coli_usage) records which experts a workload routes to and pins the hottest ones automatically over repeated runs.
Dual-SSD striping: a second SSD can hold a full or partial mirror of the model so expert reads split across both drives for higher aggregate bandwidth.
Local cluster mode: a coordinator keeps generation and routing local while disk-backed worker machines execute routed feed-forward layers over TCP.
Speculative decoding: native multi-token prediction and grammar-forced drafts are measured end to end and disabled automatically when they don't pay off.
Web dashboard and visualizations: coli web exposes live token metrics, a per-turn time breakdown, and pages showing routing heat across thousands of experts.

Ideal use cases

Colibrì fits situations where you want private, on-premises access to a frontier-scale MoE model without hyperscaler hardware. A workstation with a modest GPU, plenty of RAM, and fast NVMe storage can run a 744B-parameter model at usable speed by streaming cold experts and keeping hot ones resident. It also suits researchers who want to experiment with inference-side systems ideas such as weight placement, prefetch policies, quantization, and CPU/GPU overlap, since the project explicitly frames itself as a research platform with open, unresolved performance hypotheses.

It is not the right tool if you need guaranteed low latency or a fixed throughput SLA. The project states there's no speed guarantee, only a guarantee that precision and routing semantics stay fixed regardless of available memory. It's also not aimed at training; it's an inference engine. If your model isn't one of the currently supported MoE families, or you need dense (non-MoE) model support, this project won't help directly.

Installation

Colibrì ships as a single C source file (c/colibri.c) with small header dependencies, no BLAS and no Python runtime requirement, so a standard C toolchain is enough to build it. Once built, the coli binary exposes coli chat, coli serve, and coli web as the main entry points for a given converted model directory.

To run a model, point the engine at a converted model directory:

./coli chat

For dual-SSD streaming, set the primary and mirror model paths as environment variables:

COLI_MODEL=/fast/glm52_i4 COLI_MODEL_MIRROR=/second/glm52_i4 ./coli chat

To run a partial mirror planned from expert usage history:

./c/coli mirror plan  --model /fast/glm52_i4 --mirror /second/glm52_i4 --budget-gib 200 --reserve-gib 20
./c/coli mirror stage --model /fast/glm52_i4 --mirror /second/glm52_i4 --budget-gib 200 --reserve-gib 20
./c/coli mirror verify --model /fast/glm52_i4 --mirror /second/glm52_i4

For a multi-machine cluster, start a coordinator and one or more workers:

./coli cluster coordinator --host 0.0.0.0 --port 8765
./coli cluster worker --model /nvme/glm52_i4 --port 9100 --coordinator http://COORDINATOR:8765 --advertise-host WORKER_IP
./coli serve --model /nvme/glm52_i4 --cluster-coordinator http://127.0.0.1:8765

Check the project's CONTRIBUTING.md and benchmark protocol docs for the exact build commands, model conversion steps, and hardware detection flags (CUDA_EXPERT_GB, PIN_GB, COLI_NUMA) referenced throughout the README.