For the past several years, my good friend and fellow Medalist Timothy G…

大模型训练最怕的不是 loss 降得慢，而是它在一切正常时突然尖峰、发散、NaN。本文从 GLM-130B、OPT-175B、PaLM 的真实训…

I’ve been hearing this question a lot lately, and when I do, it reminds …

It occurred to me recently that the mathematical blog medium may be a go…

大模型训练里最贵的不只是算力，还是高质量数据。本文把数据工程拆开讲：语料从哪里来，为什么去重、过滤、混配、污染控制都属于“模型能力工程”，C4、…

Someone : Did you hear that Facebook/Google uses a giant monorepo? WTF! …

I’d like to begin today by welcoming Timothy Gowers to the mathematics b…

Kaplan（2020）第一次把 loss 随参数、数据、算力的幂律关系钉成经验规律，却因为训练配方设计的一个隐藏 bug 把配比钉歪了；Chi…

We spent one day 1 building a system that immediately found a mid 7 figu…

My colleague Ricardo Pérez-Marco showed me a very cute proof of Pythagor…