Dan Luu 写系统问题时，习惯先测量、再对照、最后才下结论。把「How often is the build broken?」放到智能体、评测和线上系统里，真正要问的是：默认做法会不会系统性失败。下面用中文整理成可执行的工程笔记，去掉原站导航和无关链接。

问题怎么暴露

I've noticed that builds are broken and tests fail a lot more often on open source projects than on “work” projects. I wasn't sure how much of that was my perception vs. reality, so I grabbed the Travis CI data for a few popular categories on GitHub 1 .

For reference, at every place I've worked, two 9s of reliability (99% uptime) on the build would be considered bad. That would mean that the build is failing for over three and a half days a year, or seven hours per month. Even three 9s (99.9% uptime) is about forty-five minutes of downtime a month. That's kinda ok if there isn't a hard system in place to prevent people from checking in bad code, but it's quite bad for a place that's serious about having w

测量时容易踩的坑

By contrast, 2 9s of reliability is way above average for the projects I pulled data for 2 — only 8 of 40 projects are that reliable. Almost twice as many projects — 15 of 40 — don't even achieve one 9 of uptime. And my sample is heavily biased towards reliable projects. There are projects that were well-known enough to be “featured” in a hand curated list by GitHub. That's already biases the data right there. And then I only grabbed data from the proje

To make sure I wasn't grabbing bad samples, I removed any initial set of failing tests (there are often a lot of fails as people try to set up Travis and have it misconfigured) and projects that that use another system for tracking builds that only have Travis as an afterthought (like Rust) 4 .

对照之后能下的结论

Why doesn't the build fail all the time at work? Engineers don't like waiting for someone else to unbreak the build and managers can do the back of the envelope calculation which says that N idle engineers * X hours of build breakage = $Y of wasted money.

But that same logic applies to open source projects! Instead of wasting dollars, contributor's time is wasted.

落到生产里的动作

Web programmers are hyper-aware of how 100ms of extra latency on a web page load has a noticeable effect on conversion rate. Well, what's the effect on conversion rate when a potential contributor to your project spends 20 minutes installing dependencies and an hour building your project only to find the build is broken?

I used to dig through these kinds of failures to find the bug, usually assuming that it must be some configuration issue specific to my machine. But having spent years debugging failures I run into with make check on a clean build, I've found that it's often just that someone checked in bad code. Nowadays, if I'm thinking about contributing to a project or trying to fix a bug and the build doesn't work, I move on to another project.

值得单独记下的观察

• Categories determined from GitHub's featured projects lists, which seem to be hand curated. [return]

落地时建议先做的 5 件事

1. 用自己的真实负载测，而不是只用公开榜或厂商数字。
2. 把评测设计成能抓到失败模式：平均分好看但尾部崩溃，仍然算失败。
3. 智能体默认不会好好用测试；要写进流程，而不是写在口头规范里。
4. 性能和正确性都要有基线，改模型或改语言前后必须能对比。
5. 结论写成可回滚的决策：哪一版配置、哪一版评测集、谁签字。

和智能体产品怎么接

龙虾PRO做 OpenClaw 落地时，同样吃「先测量再扩面」这条纪律：技能、数字员工和网关都要有可复现评测，而不是只看一次演示通过。

本文侧重全链路风控方法论。落地时请用自身业务单据做回放验证，不要把示例阈值直接当生产策略。 相关：风控体检 · 方案资源