Five Samples Said “Not Proven”. Nine Said 2.6x.

Our AI photo tools run on rented GPUs that only exist while a job is running. The open question was how long a customer waits when no machine is already warm. We measured it, wrote down "not proven", and were wrong — not because the measurements were bad, but because there were not enough of them to see the shape of what we were measuring.

The setup. Serverless GPU platforms offer a feature that ought to remove most of the cold-start cost: snapshot the container's memory after the model weights are loaded, then restore from that snapshot instead of starting from nothing. Modal documents this as memory snapshots. The question was simply whether it helps enough to matter.

Five readings told us nothing. The numbers were scattered. Some restores were dramatically faster than a cold boot, some were barely faster at all. The spread swamped the difference, so the honest conclusion from five samples was that we could not demonstrate a benefit. We wrote that down and moved on.

Nine readings told us 2.6x. Going back with more samples, the scatter resolved into two distinct clusters rather than one noisy blob. The distribution was bimodal. Averaging across it produced a meaningless middle number that described no actual run.

Why we think it is bimodal. Our reading — an inference from the shape of the data, not something the platform documents — is that a snapshot is tied to the kind of worker that produced it. Restore onto the same hardware generation and it is fast; land on a different one and the snapshot cannot be used as directly, so you pay much closer to a full cold boot. Which you get depends on what capacity happens to be free. We saw no middle case in nine runs, which is why averaging the two clusters produces a number that never actually occurred — though nine runs is also not enough to rule a middle case out, and it would be poor form to claim otherwise in this particular article.

With five samples you might draw four from one cluster and one from the other, or three and two. Every split looks like noise. You need enough draws for two groups to become visibly separate before you can say anything true about either.

The measurement trap underneath. There is a second sharp edge here. Memory snapshots are disabled for ephemeral apps — the throwaway deployments you would naturally use for benchmarking. So the convenient way to measure the feature is structurally incapable of measuring it. You can get your cold baseline that way, but the restore number only exists after a real deployment. If we had not noticed, we would have benchmarked the feature switched off and reported that it does not work.

What we took from it. Three things worth carrying to any latency measurement:

• Plot the samples before you average them. A mean is only meaningful for a distribution with one hump, and you cannot tell how many humps you have without looking.
• "Not proven" is a statement about your sample size at least as often as about the thing you measured. It is an honest thing to write down, but it is not a finding, and it should be revisited rather than filed.
• Check that your measurement environment actually has the feature switched on. A benchmark of a disabled feature looks exactly like a benchmark of a useless one.

For what it is worth, the practical answer for people using our AI tools is that the first run after a quiet period is slower than the ones after it, and we tell you a run is queued rather than leaving the page silent. The free tools have none of this problem — they run in your browser, so there is no machine to start.