How much does the human-reply rate vary from one cold email campaign to the next?

If one cold email campaign gets replies that are mostly genuine, and another campaign next month gets replies that are mostly autoresponders, is that a meaningful difference worth investigating, or just noise? This study's data suggests it is closer to normal variation than anyone running a single campaign would expect.

Across the 74 campaigns in this dataset, spanning ten sending workspaces with an average of 32.68 replies per campaign, the share of replies classified as genuine ranged from a low of 8.8% to a high of 54.3%. The median across campaigns was 28.1%, with 19 campaigns counted in this comparison. That is a wide range: a campaign at the low end would see fewer than one in ten of its replies come from an actual person, while a campaign at the high end would see more than half.

The other two classes move around by similar or larger margins. Out_of_office ranges from 18.2% to 82% across campaigns (median 47.1%), and auto_reply_other ranges from 8.2% to 44.1% (median 25%). In every class, the spread between the lowest and highest campaign is larger than the overall average for that class across the whole dataset, which is the signature of a metric that is driven heavily by which list and which moment in time a given campaign happened to hit.

Why this spread exists. A campaign's genuine-reply share is downstream of at least three things this study did not separately control for: the quality and freshness of the contact list (a stale list produces more bounces and departure notices, pulling genuine share down and auto_reply_other up), the time of year it was sent (more people on leave means a higher out_of_office share, for reasons unrelated to anything about the email itself), and the targeting and relevance of the message to the specific list it was sent to. This dataset cannot tell you which of these three is driving any single campaign's result, only that the result varies enough that all three are plausible contributors.

Why this matters practically. It means a single campaign is a poor unit for judging whether an email program is working. A campaign that happens to land in the bottom quartile of genuine-reply share is not necessarily badly written or badly targeted. It might just be an unlucky list or an unlucky week. Reversed, a campaign at the top of the range is not proof of a winning formula; it might simply be a cleaner list. The size of the swing from 8.8% to 54.3% is large enough that one good or bad campaign should not change anyone's mind about a sending strategy on its own.

It also means that when comparing two campaigns, or two time periods, or two copy variants, averaging over many campaigns matters far more than it might seem. One campaign's genuine share is a noisy, somewhat unreliable estimate of the underlying quality of a list or message. The 74 campaigns pooled here, and the 2,418 replies across them, are what makes the overall figures in this study (genuine at 34.6% of all replies) meaningfully more trustworthy than any single campaign's number would be on its own.