Topic
Inference & deployment
Inference engines, deployment shapes and cost optimisation for large models.
Follow this topic
On this site · ElsewhereTopic graph
16 nodes · 64 links- Entries in this topic
- Neighbours outside the topic
- Builds on · Successor · Extends
- Prerequisite · Uses
- Explains · Supports
- Related
Skills and knowledge entries are written in Chinese, as are signals from Chinese sources.
Published signals
31 signalsAnyLanguageModel 1.0: One Year of Local Models on Apple Platforms
AnyLanguageModel 1.0: One Year of Local Models on Apple Platforms
MPS in PyTorch 2.15
MPS in PyTorch 2.15
TorchNeuron: running HF models on AWS Trainium is easier than ever
TorchNeuron: running HF models on AWS Trainium is easier than ever
v0.32.0
v0.32.0 Highlights This release features 814 commits from 363 contributors (121 new)! Structured decisions and generic parsers: the new POST /v1/systemone endpoint answers choice, noul and score questions over a structured state (#59299), is served by default…
DeepSeek-V4 的周期性盲点：信息换个位置，检索准确率差 40 个百分点
字节跳动 Seed 发现，采用分块 KV 缓存压缩的模型，检索准确率会随信息相对压缩窗口边界的位置周期性起伏。128K 上下文里，DeepSeek-V4-Flash-Base 在不同位置之间最多差 40.2 个百分点，后训练过的 V4.1-Flash 仍差 6.1 个百分点。起伏的周期正好等于压缩步长：V4 是 4 个 token，V4.1 是 2 个。平均分会把这种弱点藏起来。
Ollama v0.35.1: Clef decision models
Clef decision models Ollama now supports Clef and Clef Flash, Cloudflare's new open-source decision models, through /v1/systemone. Clef (27B) and Clef Flash (9B) are multimodal: requests can now include images alongside the text state, shared by all questions…
DeepSeek 开源昇腾版 DeepGEMM 等组件：接口与 NVIDIA 版相同
DeepSeek 9 月 30 日开源了一组跑在华为昇腾上的基础组件：算子编译工具 TileLang 的昇腾支持、矩阵乘法库 DeepGEMM-Ascend、专家并行通信库 DeepEP-Ascend，以及 TileKernels、FlashMLA、DeepSelect 等算子。DeepGEMM-Ascend 按 MIT 许可发布，与原版 DeepGEMM 接口完全兼容，在昇腾 950DT 上实测 BF16 达到 431 TFLOPS（硬件利用率 99.8%），FP4 达到 1701 TFLOPS。目前只在 950 系列上开发和验证，没有提到 910B / 910C。
NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction
NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction
TPU 巨核：同样是 16 块芯片，Kimi K3 快 57%
vLLM 原班人马创办的 Inferact 开源了一组 TPU 推理内核：把 Kimi K3 的 92 层计算写进同一个程序，趁当前层还在计算时预取下一层的权重。16 块 TPU v7 配合推测解码每秒输出 709 个 token，基线是 vLLM 在 16 块 GB200 上的 452 个。两种芯片的内存带宽其实是 GB200 略高，差距来自软件怎样用满它。这是面向单个用户、小批量解码的成绩。
Colibrì：25GB 内存跑 744B 模型，权重按需从硬盘读
开源推理引擎 Colibrì 把显存、内存和固态硬盘当成同一套存储层级：744B 的 GLM-5.2 每个 token 只激活约 40B 参数，于是常驻内存的只有约 9.9GB 的稠密部分，19,456 个路由专家留在硬盘上按需读取。它不保证速度，只保证不悄悄改动模型：25GB 的开发机冷缓存时每秒 0.05 到 0.1 个 token，128GB 的纯 CPU 台式机约 1.8 个。
Accelerating vision-language models with LFM2.5-VL-DSpark
Accelerating vision-language models with LFM2.5-VL-DSpark
Pinterest 数百亿向量检索：量化与 SSD 的取舍明细
Pinterest 的搜索平台 Manas 在 80 个集群上服务数百亿嵌入向量。在 1 亿条向量上的离线基准里，标量量化把 HNSW 索引从 121GB 压到 50GB、召回几乎不掉；乘积量化压到 32GB，召回从 93.7% 掉到 77.3%。上线后服务成本降低 20% 到 30%。搬到 SSD 时，SPANN 加乘积量化的吞吐是 DiskANN 的 3 倍、延迟三分之一，召回低 5%。选哪种，取决于你的召回下限，不取决于压缩率。
v0.30.0
v0.30.0 Highlights This release features 762 commits from 315 contributors (104 new)! New models : DeepSeek-V4.1-Flash ( #56214 , #56228 , #56208 ) with the whole KV stored in MXFP8 through the FlashMLA V4.1 record on SM100 ( #56893 ), DeepGEMM Mega-mHC ( #56…
AlphaGenome Atlas: A predictive map of every possible DNA letter change in the human genome
A platform containing predictions for the effects of 9 billion single-nucleotide variants — every single-letter change possible — in the human genome.
模型之间只传 17 比特：一座桥补上一半差距
大模型生成一个 token 时内部构建约两兆字节的状态，最后只吐出 17 个比特——而所有多模型协作系统（子智能体、模型委员会、路由调度）都建立在这 17 个比特之上。Mostik 让 753B 的 GLM-5.2 把隐藏状态通过一座小桥直接交给 4B 的 Qwen-3.5，全程不产生文本、两边权重完全冻结、桥是唯一被训练的部分。结果：4B 补上与 753B 约 50% 的差距、自身准确率 +25%、难题子集上 2 倍，而大模型侧推理成本降到约二十分之一。技术细节尚未公开，团队以比赛未结束为由拒绝披露。
FreeToken：消费级机器不是缩水的服务器，是异构资源
伯克利与 MIT 开源 FreeToken，一个面向消费级硬件的 MoE 推理引擎，共同作者包括 Databricks 联合创始人 Matei Zaharia 与 Ion Stoica，以及 Song Han、Kurt Keutzer。它的前提是换一个看法：个人电脑不是资源受限的数据中心节点，而是可弹性调度的异构资源——缓存未命中时不让 GPU 停下来等，而是按实时互连吞吐在 CPU 与 GPU 之间分配 token 计算。论文报告在 8GB 显存的 RTX 4060 笔记本上跑 Qwen3.6-35B 约每秒 39 个 token，同批 MoE 模型上解码快 3 到 4 倍、预填充快 6 到 30 倍。而社区正在争论的，恰好是这个基线是不是经过手工调优。
Muse Glimmer 开源：30B 压进消费级显卡，报错不停下
Meta 发布 Muse Glimmer：300 亿参数、Apache 2.0 开放权重，专为常驻本地的智能体工作流设计。4 位动态量化把 55GB 以上的显存需求压到 17 至 20GB，配合 DFlash 推测解码在 M5 Max 与 RTX 5090 上把吞吐提到最多 3.1 倍。真正值得注意的不是尺寸，而是它把「工具调用失败之后怎么办」写进了训练目标：API 或终端命令报错时，模型诊断故障并另找路径，而不是终止执行。
Same Cluster, 33 Points More Utilization: What Changed Was the Order
A constraint-aware GPU allocator benchmarked against FIFO across seven scenarios: utilization up to 33 percentage points higher, priority-weighted output up in every one.
vLLM v0.27.0
vLLM v0.27.0 ships 561 commits from 242 contributors, landing full-stack Kimi K3 support in a single release, upgrading to PyTorch 2.13, and extending Model Runner V2 to non-generative workloads.
SGLang v0.5.17
SGLang v0.5.17: 582 PRs from 194 contributors, with day-0 support for Kimi K3 and MiniMax-H3.
Deploy local agents everywhere with LFM2.5-2.6B
LFM2.5-2.6B is built to power capable agents entirely on-device. It supports tool calling and multi-step workflows while staying small and fast enough for everyday hardware, from laptops to phones.
LFM2.5-Encoders for Fast Long-Context Inference on CPU
Two new encoder models, LFM2.5-Encoder-230M and 350M: they match larger encoders on GLUE, SuperGLUE and multilingual tasks, handle 8,192-token context, and run about 3.7x faster than ModernBERT-base on CPU at long context.
Ollama v0.32.4
Ollama v0.32.4 adds Laguna support on Apple GPUs via the MLX engine, quantizes speculative-decoding draft output heads at the requested type, and fixes Qwen3 MoE decoding for differently-quantized experts.
vLLM v0.26.0
vLLM v0.26.0 ships 411 commits from 212 contributors, headlined by a full support stack for the new Inkling model family.
Introducing Gemini 3.6 Flash, 3.5 Flash-Lite, and 3.5 Flash Cyber
We're introducing new Gemini models, including Gemini 3.6 Flash, 3.5 Flash-Lite and 3.5 Flash Cyber. Flash Cyber is a lightweight cybersecurity model to find and patch vulnerabilities.
Welcome Inkling by Thinking Machines
Thinking Machines releases Inkling, an open-weight ~1T-parameter (41B active) Mixture-of-Experts multimodal model with a 1M context window, trained on 45T tokens of text, images, audio and video, with day-one support across Transformers, vLLM, SGLang, llama.cpp and Unsloth.
Ollama v0.32.0
Ollama v0.32.0 turns the bare ollama command into an interactive agent for chatting, coding, web search, and delegating work — the local runtime is repositioning itself as an agent workbench entry point.
vLLM v0.25.0
vLLM v0.25.0 ships 558 commits from 232 contributors. Model Runner V2 becomes the default for all dense models, building on the quantized-model support introduced in the previous release.
Native-speed vLLM transformers modeling backend
The transformers modeling backend in vLLM now meets or beats vLLM's hand-written native implementations across dense and MoE Qwen3 models. Model authors get ultra-fast vLLM inference from their transformers implementation with a single --model-impl transformers flag.
Ollama v0.31.2
Ollama, the local LLM runtime for running open models on personal and edge hardware, shipped stable release v0.31.2.
vLLM v0.24.0
vLLM, the open-source high-throughput LLM inference and serving engine, shipped stable release v0.24.0 while the v0.25 release-candidate series is already in flight.
Related skills
6 skillsBackground knowledge
5 entriesKV 缓存与长上下文推理
生成每个新 token 都要回看前面所有 token；KV 缓存把这份回看要用的中间结果存下来，于是上下文越长，显存和带宽越紧——读懂推理引擎更新和长上下文定价的前提。