Picking an Image Format From a Preview Tile Gets It Wrong — Here Is What the Fix Costs

Any tool offering to choose the best format for you has a speed problem. Encoding one photograph three ways to see which wins is slow. The tempting shortcut is to test a small crop and extrapolate. We did that, found cases where it picked the wrong format, and stopped trusting it — here is what those cases looked like.

Why a probe tile lies. The formats do not scale the same way. Our own measurements: AVIF's modelling advantage is largest on a smalltile, while mozjpeg's entropy coding keeps improving as the frame grows. So a small probe tends to flatter AVIF and penalise JPEG, and the two cases we have measured both got worse as the image got bigger — though two cases are not a trend.

The sampling itself adds error on top. We take a centre crop at native resolution — centre because the edges of an image are often blank chrome on a screenshot or sky on a photograph, and neither represents the picture anyone cares about. Even so, a 512-pixel tile is about 15% of a 1600×1067 photograph, and you may land on the one busy, detailed region. On one noisy crop JPEG came out more than twice as large as AVIF and was dropped from the shortlist — on the full frame it won.

What we do instead. The probe still runs — it is cheap and it is good enough to produce a shortlist. But the winner is then confirmed at full size against one challenger, under rules that keep the comparison honest:

• The challenger must match the primary's lossiness. A lossless candidate never competes against a lossy one on size, because that comparison has a foregone conclusion and the wrong winner.
• It runs at full effort, not the fast preset. A format judged at low effort and then shipped at high effort was judged as a different format.
• Only the cheapest viable challenger runs. One second opinion, not a tournament.
• Outputs under 32 KB skip the pass entirely. Below that, the saving cannot repay the time.

The measurements. Two representative cases, encoded in a real browser:

• A photograph the probe wanted to ship as AVIF at 99.6 KB. Verified at full size, JPEG produced 80.9 KB — 19% smaller than the probe's pick.
• A second image where AVIF gave 195 KB and the challenger produced 137.8 KB — 29% smaller.

In both cases the verified choice matched the best explicit format — the one you would have picked by hand if you had encoded everything and compared. Two images are two images, not a rate: we are not claiming you will always save a fifth to a third. What they establish is that the probe's answer and the full-size answer can differ by that much, which is enough to make the shortcut unsafe.

What it costs. One extra full-frame encode of a single challenger. Our measured per-format costs on a 1600×1067 photograph are JPEG 1.7s, WebP 1.7s and AVIF 5.8s, and the cheapest viable challenger is the one that runs — so in practice the verify pass adds something on the order of the JPEG or WebP figure, not the AVIF one. It is real time, and worth it against being wrong by a fifth of the file size. Below 32 KB of output we skip the pass entirely, because there the saving cannot repay the wait.

The reason this is worth writing down is that the shortcut looks so reasonable: the probe agrees with the full-size result often enough that a small test set will show no difference at all, and you will ship it.

The general lesson. If you sample a subset to make a decision about a whole, check how often the subset disagrees with the whole before you trust it — and check it on real inputs, not on the fixtures you built to exercise the code path. Synthetic test images are uniform in a way real photographs never are, which is exactly the property that makes a probe tile look reliable.

For background on how the formats themselves compare, the WebP compression study published by Google is the standard reference — though note what it actually measures: it generates WebP images at the same SSIM as the JPEG originals and compares the resulting file sizes, concluding WebP runs 25–34% smaller at equivalent quality. That is a corpus average, which is a different question from "which format wins onthis image". That question only has an answer per image, which is the whole reason automatic selection exists.

You can watch it choose on the compressor — set the format to AUTO and the result tells you which format it picked and what it saved. It runs in your browser; nothing is uploaded.