Topic
Benchmarks & evaluation
How benchmarks are designed and how far to trust them, third-party evaluations and how they are scored.
Follow this topic
On this site · ElsewhereTopic graph
16 nodes · 76 links- Entries in this topic
- Neighbours outside the topic
- Builds on · Successor · Extends
- Prerequisite · Uses
- Explains · Supports
- Related
Skills and knowledge entries are written in Chinese, as are signals from Chinese sources.
Published signals
16 signalsMiniMax M3.1 Flash：成本降七成，Agent 国模第一
MiniMax 于 10 月 19 日发布 M3.1 Flash，输入每百万 token 0.15 美元、输出 0.60 美元、缓存输入 0.015 美元，官方称 Agent 任务使用成本较上一代降低 70%。模型在 OSWorld V2 得 61.7、Agents' Last Exam 得 29.6，文章称后者位列中国模型第一。此前匿名模型 Space Bunny 被确认是该模型的早期版本。
Google 开源 Harness²：不改模型，递归改智能体的外壳
Google Research 开源智能体框架 Harness²，采用 Apache 2.0 许可，支持 OpenCode 与 Codex，默认使用 OpenCode。它把系统提示词、护栏、记忆、工具指导、子智能体、Skills、脚本和插件拆成 8 类可改组件，通过任务级与 Harnessing 级两层递归在不改模型权重的前提下优化表现。团队称仅一步优化后 WorkBuddy-Bench 最高涨 11.5 分。
华为 Memento 3：把规则写成代码，打满 ARC-AGI-3
华为诺亚方舟实验室联合 UCL 等团队的智能体 Memento 3 打通了 ARC-AGI-3 公开集全部 25 款游戏、183 个关卡，RHAE 满分 100，总操作 7518 次，比人类的 17135 次少约 56%。底模 Claude Opus 5 不更新权重、不做游戏微调，靠「代码即模型」（一份规则手册加一个可执行的世界模型）递归地自我改进。
智能体用三周重构老游戏代码，全程靠逐帧回放验收
CodeScene 发布案例研究：一个编码智能体在三周内重构了《街头霸王 III：三度冲击》开源反编译项目约 30 万行 C 代码，消耗 token 约 4000 美元，覆盖 726 个文件、2903 次提交、252055 行改动，Code Health 从 5.6 升至 10.0。保障机制包括 CodeHealth MCP Server 的确定性评分，以及逐帧比对回滚状态哈希的回放轨迹测试框架；智能体还自行整理出含 22 个方案、82 条辅助说明的重构手册。
何恺明团队 VISTA：让模型回看原始画面，打满 ARC-AGI-3
MIT 何恺明团队的 VISTA 是一套视觉原生的智能体外壳：环境返回的每一帧都原样存档，模型推理时可以按需调回、放大、并排对比，基础模型不用重新训练。配上 Claude Opus 5.0，ARC-AGI-3 的 25 个公开游戏全部通关，相对人类动作效率得满分 100，比人类首次游玩的基线少用 57.4% 的动作。
DeepSeek-V4 的周期性盲点：信息换个位置，检索准确率差 40 个百分点
字节跳动 Seed 发现，采用分块 KV 缓存压缩的模型，检索准确率会随信息相对压缩窗口边界的位置周期性起伏。128K 上下文里，DeepSeek-V4-Flash-Base 在不同位置之间最多差 40.2 个百分点，后训练过的 V4.1-Flash 仍差 6.1 个百分点。起伏的周期正好等于压缩步长：V4 是 4 个 token，V4.1 是 2 个。平均分会把这种弱点藏起来。
Agent Lightning v1.0: A 3,500-Line Lightweight Agentic RL Framework for Training Agents with Real Harnesses
Training AI agents with reinforcement learning can be challenging because their tools, context, and decision-making are managed by complex frameworks. Agent Lightning connects existing agents to RL training, making it easier to improve them without rebuilding…
One Model Family, Two Gold-Level Results: Fine-Tuning Nemotron for IOI and IMO
One Model Family, Two Gold-Level Results: Fine-Tuning Nemotron for IOI and IMO
The Agent Said It Was Done. The Database Disagreed.
The Agent Said It Was Done. The Database Disagreed.
How UK AISI and EvalEval Are Making Benchmark Results Reproducible
How UK AISI and EvalEval Are Making Benchmark Results Reproducible
Your Agent Aced the Task. Will It Do It Again?
IBM Research measures the consistency gap between Mean@k and Pass^k for LLM agents and halves it with guidelines distilled from flip-prone decision points.
BenchMIRT: What are LLM benchmarks actually measuring?
A new method for auditing LLM benchmarks at the level of individual prompts, using multidimensional item response theory.
Piloting the world's first double-blind AI evaluations
Building trust in proprietary model benchmarks using cryptographically secure environments.
Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things
Hands-on evaluation of Qwen 3.8 27B on a 128GB M5 Max and an NVIDIA DGX Spark: the shipped xhigh reasoning default burns tokens on trivial prompts, vision bounding boxes are strong, and throughput of 15–30 tokens per second is the real catch.
TutorMoments: Do AI tutors know when to help and when to hold back?
A replay-based evaluation built off real one-on-one math tutoring sessions, measuring whether LLMs can balance when to step in and when to hold back.
Introducing Real World VoiceEQ: Measuring the human quality of voice AI
Hume AI launches Real World VoiceEQ, a benchmark evaluating the human quality of voice interaction across 40+ models, 15+ dimensions and 60+ metrics, grounded in over 1 million human ratings.