Colibri runs huge mixture-of-experts models by streaming experts from disk
A C inference engine keeps an MoE model's dense part in RAM, reads routed experts from disk on demand and puts the hottest ones on a GPU if present.
What it is
colibri is an open-source inference engine, written in C and released under the Apache-2.0 license, for running very large mixture-of-experts (MoE) models on machines people already own. The repository description sums up the design as hot experts on the GPU (CUDA, Vulkan), warm ones in RAM and cold ones streamed from NVMe. The README stresses that a GPU is optional: the engine is pure C, with one file per model family.
The project was created on 2026-07-01. At the time of the snapshot the repository had 46,874 stars, 675 of them gained in the last day according to GitHub trending.
How the placement works
The README’s argument starts from how MoE models behave: such a model is huge on disk and small per token. Its example is GLM-5.2, which has 744B parameters, uses about 40B for each token, and of those only about 11 GB change from one token to the next, namely the routed experts.
So the model does not have to fit in fast memory; it has to be placed. The dense part (attention, shared experts, embeddings) stays in RAM. The routed experts stay on the disk and are read when the router asks for them, through a cache that learns which experts a given workload uses. When there is a GPU, it holds the hottest experts and the dense layers.
The maintainers describe colibri as an open research platform with a rule attached: an optimisation earns its place with a reproducible end-to-end measurement, and the default policy never silently changes model precision or router semantics. In the README’s words, where a weight sits “changes how fast the answer comes, not which weights or which router decisions produce it.”
Models and hardware
The README lists thirteen engines. Ten are for language models, among them GLM-5.2/5.3, Kimi K3, DeepSeek V4 Flash, MiMo-V2.6 Flash, Qwen3.6 and OLMoE. One, Qwen-Image-2.1, draws pictures, and two answer decisions. The language models also run from already published GGUF files, read as they are, on the CPU, on any GPU through Vulkan and on NVIDIA cards through CUDA.
The stated minimum is 8 GB of RAM (16 GB or more is better) and 22 GB free on the disk for the smallest model. The speeds in the README vary widely by model and machine. For Qwen3.6-35B-A3B it lists 6.0 tok/s on the CPU and 30.0 with CUDA on another machine. For GLM-5.2, the reference model, it lists 0.05-0.1 tok/s cold on a 25 GB laptop, 1.83 on a 128 GB Ryzen AI Max+ 395 and 9.0-9.2 on 6x RTX 5090. For Kimi K3 (2.8T), the largest, the figure is about 9.4 s per token, with experts read at 6.3 GB/s. These are the project’s own measurements, and for the large models the README notes that the disk sets the speed and a fast NVMe drive helps most.
Getting started
The README describes a one-step setup. On Windows it is a double-click on START-HERE.bat; on Linux (Ubuntu and Debian) it is:
sudo apt install git python3 build-essential
git clone https://github.com/JustVugg/colibri
cd colibri
./start-here.sh
The setup looks at RAM, free disk, CPU and GPUs, recommends a model that fits, builds the engine when a compiler is there (otherwise it downloads a prebuilt one), downloads the model with progress and resume, and opens a dashboard in the browser. It also prints an OpenAI base URL and an Anthropic base URL on port 8000 for other apps. ./start-here.sh --list shows every model against the current machine and why one does not fit. For AI coding assistants, coli mcp offers an MCP server with tools to detect the hardware, recommend a model, install, start, stop and check it.
Source: JustVugg/colibri on GitHub