Dan Luu 写系统问题时，习惯先测量、再对照、最后才下结论。把「Given that we spend little effort on testing, how should we test software?」放到智能体、评测和线上系统里，真正要问的是：默认做法会不会系统性失败。下面用中文整理成可执行的工程笔记，去掉原站导航和无关链接。

Manual Test Generation

I've been reading a lot about software testing, lately. Coming from a hardware background (CPUs and hardware accelerators), it's interesting how different software testing is. Bugs in software are much easier to fix, so it makes sense to spend a lot less effort spent on testing. Because less effort is spent on testing, methodologies differ; software testing is biased away from methods with high fixed costs, towards methods with high variable costs. But tha

I don't really know anything about software testing, but here are some notes from what I've seen at Google, on a few open source projects, and in a handful of papers and demos. Since I'm looking at software, I'm going to avoid talking about how hardware testing isn't optimal, but I find that interesting, too.

Random Test Generation

From what I've seen, most test effort on most software projects comes from handwritten tests. On the hardware projects I know of, writing tests by hand consumed somewhere between 1% and 25% of the test effort and was responsible for a much smaller percentage of the actual bugs found. Manual testing is considered ok for sanity checking, and sometimes ok for really dirty corner cases, but it's not scalable and too inefficient to rely on.

It's true that there's some software that's difficult to do automated testing on, but the software projects I've worked on have relied pretty much totally on manual testing despite being in areas that are among the easiest to test with automated testing. As far as I can tell, that's not because someone did a calculation of the tradeoffs and decided that manual testing was the way to go, it's because it didn't occur to people that there were alternatives to

Random Test Generation, Framework

At the hardware company I worked for, we called what programmers call tests, "hand tests" or "hand jobs" because they were written by hand (the latter isn't innuedo, it's becasue we launched tests into "jobs" in our job system). What people in the software world call "fuzzing", "property based testing", "randomized testing", etc., we just called testing because that was the default. How else would you test? Sure, you might have 1% of test writing time go t

The good news is that random testing is easy to implement. You can spend an hour implementing a random test generator and find tens of bugs , or you can spend more time and find thousands of bugs .

Random Test Generation, Coverage Based

You can start with something that's almost totally random and generates incredibly dumb tests. As you spend more time on it, you can add constraints and generate smarter random tests that find more complex bugs. Some good examples of this are jsfunfuzz , which started out relatively simple and gained smarts as time went out, and Jepsen, which originally checked some relatively simple constraints and can now check linearizability.

While you can generate random tests pretty easily, it still takes some time to write a powerful framework or collection of functions. Luckily, this space is well covered by existing frameworks.

Test Generation, Other Smarts

[2026 update: I'm writing this long after I originally wrote this post. Since writing this post, I've sat down with a few people and wrote a fuzzer with them. This has worked very well every time I've tried it and people carried this skill away with them and use it all the time after learning how to do it. The trick is, it's extremely easy to do and I wasn't really providing any knowledge at all, other than the fact that it can be done. If you take a piece

I'll also add that I'm less positive about frameworks, etc., than I used to be. I've tried a number of them out at this point and, while I see the advantages they have, the value add is much less than I would've expected back when I wrote this post and didn't have almost any software experience. There are a variety of reasons for this that are fairly long and should probably be their own post, but 一个稳妥判断是 the two top ones are that almost every test framewo

Why Not Coverage-Based Unit Testing?

Looking back on this post, I consider it a fairly bad failure in that I have a 100% success rate a converting people to being quite good at testing by sitting down with them for an hour or less and the blog post had an epsilon success rate at doing the same thing. Due to the scale of readership a blog post gets, it surely helped more people learn how to test well than I've personally helped, but given how easy it is for me to do this in person, I definitel

Here's an example of how simple it is to write a JavaScript tests using Scott Feeney's gentest , taken from the gentest readme.

值得单独记下的观察

• Among other things, it uses nested if statements instead of && because go's coverage tool doesn't create separate coverage points for && and || . [return]
• Ok, you're slightly worse off due to the overhead of generating and looking at coverage stats, but that's pretty small for most non-trivial programs. [return]

落地时建议先做的 5 件事

1. 用自己的真实负载测，而不是只用公开榜或厂商数字。
2. 把评测设计成能抓到失败模式：平均分好看但尾部崩溃，仍然算失败。
3. 智能体默认不会好好用测试；要写进流程，而不是写在口头规范里。
4. 性能和正确性都要有基线，改模型或改语言前后必须能对比。
5. 结论写成可回滚的决策：哪一版配置、哪一版评测集、谁签字。

和智能体产品怎么接

龙虾PRO做 OpenClaw 落地时，同样吃「先测量再扩面」这条纪律：技能、数字员工和网关都要有可复现评测，而不是只看一次演示通过。

本文侧重全链路风控方法论。落地时请用自身业务单据做回放验证，不要把示例阈值直接当生产策略。 相关：风控体检 · 方案资源