Self-Hosting
-
REPOColibri runs huge mixture-of-experts models by streaming experts from disk
A C inference engine keeps an MoE model's dense part in RAM, reads routed experts from disk on demand and puts the hottest ones on a GPU if present.
A C inference engine keeps an MoE model's dense part in RAM, reads routed experts from disk on demand and puts the hottest ones on a GPU if present.