Inference
-
REPOColibri runs huge mixture-of-experts models by streaming experts from disk
A C inference engine keeps an MoE model's dense part in RAM, reads routed experts from disk on demand and puts the hottest ones on a GPU if present.
-
RELvLLM v0.33.0rc3 drops CUDA 12.9, adds PyTorch 2.14, new models and speculative decoding gains
vLLM v0.33.0rc3 upgrades to PyTorch 2.14, moves to CUDA 13.0-only artifacts, adds 11 new models, and delivers speculative decoding and decode-speed improvements across NVIDIA and AMD hardware.