Dan Luu 写系统问题时，习惯先测量、再对照、最后才下结论。把「Notes on concurrency bugs」放到智能体、评测和线上系统里，真正要问的是：默认做法会不会系统性失败。下面用中文整理成可执行的工程笔记，去掉原站导航和无关链接。

Fonseca et al. DSN '10

Do concurrency bugs matter? From the literature, we know that most reported bugs in distributed systems have really simple causes and can be caught by trivial tests, even when we only look at bugs that cause really bad failures, like loss of a cluster or data corruption. The filesystem literature echos this result — a simple checker that looks for totally unimplemented error handling can find hundreds of serious data corruption bugs . Most bugs are simple

Just from personal experience, I've spent more time debugging complex non-deterministic failures than all other types of bugs combined. In fact, I've spent more time debugging some individual non-deterministic bugs (weeks or months) than on all other bug types combined. Non-deterministic bugs are rare, but they can be extremely hard to debug and they're a productivity killer. Bad non-deterministic bugs take so long to debug that relatively large investment

Lu et al. ASPLOS '08

Let's see what the academic literature has to say on non-deterministic bugs. There's a lot of literature out there, so let's narrow things down by looking at one relatively well studied area: concurrency bugs. We'll start with the literature on single-machine concurrency bugs and then look at distributed concurrency bugs.

They studied MySQL concurrency bugs from 2003 to 2009 and found the following:

Tools

Note that these numbers sum to more than 100% because some bugs are tagged with multiple causes. This is roughly in line with the Lu et al. ASPLOS '08 paper (which we'll look at later), which found that 30% of the bugs they examined were deadlock bugs.

The paper defines a semantic failure as one "where the application provides the user with a result that violates the intended semantics of the application". The authors also find that "the vast majority of semantic bugs (92%) generated subtle violations of application semantics". By their nature, these failures are likely to be undercounted — it's pretty hard to miss a deadlock, but it's easy to miss subtle data corruption.

Tools you can actually easily use

The paper defines latent as bugs that "do not become immediately visible to users.". Unsurprisingly, the paper finds that latent failures are closely related to semantic failures; 92% of latent failures are semantic and vice versa. The 92% number makes this finding sound more precise than it really is — it's just that 11 out of the 12 semantic failures are latent and vice versa. That could have easily been 11 out of 11 (100%) or 10 out of 12 (83%).

That's interesting, but it's hard to tell from that if the results generalize to projects that aren't databases, or even projects that aren't MySQL.

Other tools

They looked at concurrency bugs in MySQL, Firefox, OpenOffice, and Apache. Some of their findings are:

Of the 74 non-deadlock bugs studied, 51 were atomicity bugs, 24 were ordering bugs, and 2 were categorized as "other".

Distributed systems

An example of an atomicity violation is this bug from MySQL:

if (thd->proc_info) fputs(thd->proc_info, …) Thread 2:

值得单独记下的观察

• Mutual exclusion: at least one resource must be held in a non-shareable mode. Only one process can use the resource at any given instant of time.
• Hold and wait or resource holding: a process is currently holding at least one resource and requesting additional resources which are being held by other processes.
• No preemption: a resource can be released only voluntarily by the process holding it.
• Circular wait: a process must be waiting for a resource which is being held by another process, which in turn is waiting for the first process to release the resource.
• One hop is faster than two hops
• Zero hops are faster than one hop
• Atomic blocks can't be broken
• I get the incentives that lead to this, and I don't begrudge researchers for pursuing career success by responding to those incentives, but as a lowly practitioner , it sure would be nice if the incentives were different

落地时建议先做的 5 件事

1. 用自己的真实负载测，而不是只用公开榜或厂商数字。
2. 把评测设计成能抓到失败模式：平均分好看但尾部崩溃，仍然算失败。
3. 智能体默认不会好好用测试；要写进流程，而不是写在口头规范里。
4. 性能和正确性都要有基线，改模型或改语言前后必须能对比。
5. 结论写成可回滚的决策：哪一版配置、哪一版评测集、谁签字。

和智能体产品怎么接

龙虾PRO做 OpenClaw 落地时，同样吃「先测量再扩面」这条纪律：技能、数字员工和网关都要有可复现评测，而不是只看一次演示通过。

本文侧重全链路风控方法论。落地时请用自身业务单据做回放验证，不要把示例阈值直接当生产策略。 相关：风控体检 · 方案资源