Inference on skulldchttps://skulldc.net/tags/inference/Recent content in Inference on skulldcHugoen-usTue, 03 Nov 2026 10:27:42 +0000Colibri runs huge mixture-of-experts models by streaming experts from diskhttps://skulldc.net/2026/11/colibri-runs-huge-mixture-of-experts-models-by-streaming-experts-from/Tue, 03 Nov 2026 10:27:42 +0000https://skulldc.net/2026/11/colibri-runs-huge-mixture-of-experts-models-by-streaming-experts-from/A C inference engine keeps an MoE model’s dense part in RAM, reads routed experts from disk on demand and puts the hottest ones on a GPU if present.vLLM v0.33.0rc3 drops CUDA 12.9, adds PyTorch 2.14, new models and speculative decoding gainshttps://skulldc.net/2026/11/vllm-v0-33-0rc3-drops-cuda-12-9-adds-pytorch-2-14-new-models-and-specu/Tue, 03 Nov 2026 00:14:14 +0000https://skulldc.net/2026/11/vllm-v0-33-0rc3-drops-cuda-12-9-adds-pytorch-2-14-new-models-and-specu/vLLM v0.33.0rc3 upgrades to PyTorch 2.14, moves to CUDA 13.0-only artifacts, adds 11 new models, and delivers speculative decoding and decode-speed improvements across NVIDIA and AMD hardware.