Only about a third of cold email replies are written by a person

When a cold email campaign reports a reply rate, it is natural to assume that every one of those replies is a person reacting to the message, interested or not. That assumption is wrong more often than it is right.

This study labelled every inbound reply in BusySeed's warehouse, 2,418 of 2,419 rows after excluding 1 that could not be classified, across a measurement window running from 2025-12-01 to 2026-10-19. Each reply was independently classified by Claude Haiku 4.5, a large language model, against a written rubric defining three categories: out_of_office (an automated absence notice, such as someone on vacation), auto_reply_other (a bounce message, a support-ticket receipt, a departure notice saying the recipient no longer works there), and genuine (an actual person wrote this reply).

The result: only 34.6% of all replies, 837 of them, were genuine. The remaining replies split between out_of_office, the single largest class at 43.3% (1,046 replies), and auto_reply_other at 22.1% (535 replies). Put together, automated replies of some kind outnumber human replies by roughly two to one.

Why this happens mechanically. Cold email is sent at volume to lists that are, inevitably, somewhat stale. Some fraction of recipients are traveling and have an autoresponder on. Some have left the company and their old address now bounces to a ticketing system or forwards to an assistant who sends a canned departure notice. None of that is a judgment on the email's quality. It is simply what happens when you mail a few thousand addresses: a predictable slice will generate an automated response regardless of what the email says or how well it is targeted.

Why this matters for how you read your own numbers. If a sales team tracks "reply rate" as a single top-line metric, it is quietly combining a signal (a person responded) with noise (an autoresponder fired). A campaign that looks like it is performing well because replies are up may simply be hitting a list with more stale addresses or more people currently out of office. The only way to tell the difference is to break replies down by class before reporting on them, which is exactly what this study's rubric does.

What this number cannot tell you. The classification was performed by a language model, not a human reviewer, and the model's agreement with human-labelled examples was 87.1% on a calibration set of 2,418 labelled replies, with stored precision of 84.43% and stored recall of 86.04% on the genuine class. That means roughly one in eight or nine replies may be mis-classified. It is also worth noting that genuine does not mean interested. A genuine reply includes polite declines, requests to be removed, and outright hostility, alongside real interest. This finding only tells you what fraction of replies came from a human at all, not what that human said.

The genuine share also is not stable across campaigns. Across the 74 campaigns in this dataset, the genuine share ranges from 8.8% at the low end to 54.3% at the high end, with a median of 28.1%. A single campaign's result, in other words, tells you little about what to expect from the next one. What to do Monday morning: before trusting any reply-rate figure, ask whether it has been split into automated versus human responses. If it has not, the number is probably overstating how many people actually read and responded to the email.