how many independent tests before you trust a vendor. my number is three.
Genuine methodology question rather than a vendor post, so flairing it Discussion. My rule has been three independent results, from at least two different members, across at least two different batches. Below that I treat a vendor as unknown rather than good. Three is arbitrary. I picked it because it is the smallest…
Four for me, and specifically four across at least a year, because I want to see whether the process drifts.
The BCH record is the example that set my number. Twelve documents in the audit, all complete, results clustered — that is a process. Compare that with a vendor whose six results run 98.4, 97.9, 96.2, 94.8, 98.1, 93.7. The mean of that second set is not a useful number about anything. It is two vendors wearing one name.
"two vendors wearing one name" is exactly the thing i was reaching for. a spread that wide makes the mean actively misleading.
which is why vendor-level averages should always ship with a range. i publish both and people quote the average anyway.
people have always done this. the point estimate is the headline, the interval is the story, and the interval loses every single time.
put the range in the headline then. "98.5 (93.7–99.4)" is not hard to type.
trying that formatting next update. if it changes how people quote it i will report back.