Tim Dettmers 做量化、多卡和智能体时，习惯先把硬件账和算法账摊开。把「Sparse Networks from Scratch: Faster Training without Losing Performance」写成能落地的笔记，核心不是复述论文摘要，而是：显存、精度、通信和评测会在哪一层先爆。下面用中文重排，去掉原站广告和分享条。

Why Sparse Learning?

This blog post is about my work, Sparse Networks from Scratch: Faster Training without Losing Performance , with Luke Zettlemoyer on fast training of neural networks which we keep sparse throughout training. We show that by developing an algorithm, sparse momentum, we can initialize a neural network with sparse random weights and train it to dense performance levels — all while doing just a single training run. Furthermore, If we use optimized sparse convo

A significant driver of progress in deep learning has been advances in computational resources. From 2010 to 2018 we saw an increase of 9700% in computational GPU performance. However, we can expect increases of just little more than 80% GPU performance in the next 5-8 years due to reaching the physical limits of semiconductor technology. What does a research world look like where we cannot make further improvements in computational power?

Sparse Momentum: An Efficient Way to Train Sparse Networks

A glimpse of this comes from the natural language processing (NLP) community where pretrained language models like ELMO, GPT, BERT, GPT-2, Grover, and XL-Net dominate the entire field by outperforming other methods on most NLP tasks. These models are often rather simple: You train them on lots of documents, and the task is mainly to predict a word given a sequence of other words — a bit like doing a fill-in-the-blank puzzle. The catch? These models are so

One way to achieve this is to look at the human brain for inspiration. The human brain consumes 1/10th of the energy of a GPU but is 10^9 times more powerful. What makes the brain so computational efficient? There are many reasons , but one reason is sparsity .

What is the Main Quality of Good Sparse Learning Algorithms?

It has been found that the more neurons a primate brain has the fewer connections does the average neuron make with all other neurons (Herculano-Houzel et al., 2010). This is very much contrary to how we design deep neural networks, which is to connect every new neuron in a layer with all neurons in the previous layer. We already understand how to compress a fully trained dense network to a sparse network (Han et al., 2015), but there has been little work

This section explains the sparse momentum algorithm from intutiton up to the full algorithm.

Momentum: Finding Weights that Reduce the Error Consistently

Figure 1: Sparse Momentum determines where to grow new weights in a sparse network by looking at the weighted average of recent gradients (momentum) to find weights and layers which reduce the error consistently. (1) We determine the importance of each layer according to the mean momentum magnitude. (2) For each layer, we remove the 50% of the smallest weights. (3) We then redistribute the weights across layers according to layer importance. Within a layer

In sparse learning, the most important thing is to use every single weight in a neural network as effectively as possible. If you define “effectiveness” as “reducing the error,” then we have an obvious perspective on how we can proceed. We need to find a measure which describes how effective a weight is at reducing the error and remove all weights which do not. Once we removed weights, we want to regrow new weights in locations which we think are promising

Redistributing Weights: The Mean Momentum Magnitude of a Layer

If we look at the gradient of the error with respect to the weight, we actually have precisely such a measure. However, if we look at successive gradients, we find that gradients can wildly oscillate. For example if you have a neural network which classifies handwritten digits 0 to 9 then a weight might be good at detecting a straight line at the top and it might help to reduce the error for the numbers 5, 7 but then it might not help or even be detrimenta

If you take the north pole to be a local minimum and a compass needle the gradient towards the local minimum, then you can simulate stochastic gradient descent updates by shaking the compass wildly to spin the compass needle. With every time the needle passes the north pole it will slow down and line-up more and more with the north pole, however, due to the spin it will still “overshoot” that direction. So it might be unclear where the north pole is from t

Which Weights Should be Removed? Where to Regrow them?

This is the main idea behind the momentum optimization technique: We average successive gradients to get a better estimate of the direction of the local minimum. Similarly to the compass needle, which gets more and more accurate over time as it slows down, we want to weight more recent gradient directions in stochastic gradient descent more highly. One way to do this is to assign a weighted average where we assign a much larger weight to the current gradie

From here we make the first important observation for our sparse momentum algorithm: If the momentum of a weight indicates how much it reduces the error consistently, then the mean momentum magnitude of all the weights in a layer should indicate how much each layer is reducing the error on average. We take the magnitude because two different weights might consistently go into a negative direction and a positive direction. By taking the mean momentum magnit

Results

The next two problems are more straightforward: Which are the most useless weights? Where do we grow weights within a layer? The first problem is a common problem in neural network compression research, where one often prunes the weights with the smallest magnitude. Why does this make sense? If we assume all weights receive on average inputs of similar magnitude — a reasonable assumption if one uses batch normalization — then weights with small magnitudes

Once we removed weights and redistributed them to weight-effective layers as measured by the mean momentum magnitude of a layer, we need to decide where exactly to grow them within a layer. One possible solution becomes apparent if we ask: “Which two unconnected neurons would reduce the error consistently if we connect them?” The answer to this question would again point to the momentum magnitude. This time, however, we want to look at the momentum magnitu

落地时建议先做的 5 件事

1. 用自己的模型和序列长度算显存账，不要只看官网 TFLOPS。
2. 量化前后都要有同一套评测：平均分和长尾失败都记。
3. 多卡训练先分清数据并行和模型并行，通信量写进预算。
4. 智能体工作流要有可复现实验，而不是一次演示通过。
5. 硬件选型写清接口、功耗和两年后还能不能买到。

和智能体产品怎么接

龙虾PRO做 OpenClaw 落地时，量化和多卡经验会直接变成推理成本和技能能否本地跑。数字员工和网关应共用同一套 GPU/精度预算，而不是每条链路各买各的卡。

本文侧重全链路风控方法论。落地时请用自身业务单据做回放验证，不要把示例阈值直接当生产策略。 相关：风控体检 · 方案资源