Only about a third of cold email replies are written by a person

Why this question matters

A sales or growth team watching a cold email campaign usually tracks one number: reply rate. That number is treated as a proxy for interest. But a reply is not the same thing as a person responding. An inbox can generate a reply automatically, because someone is on leave, because a mail server bounced the message, or because a ticketing system fired an acknowledgment. If a dashboard counts all of these as 'engagement', the team is measuring its mail infrastructure, not its prospects.

This matters most to the people who size pipeline off reply counts: SDR managers setting quota, growth teams reporting campaign performance upward, and anyone comparing two campaigns' reply rates and concluding one messaging angle outperformed another. If the mix of automated versus human replies differs between those two campaigns, for reasons unrelated to the message itself, the comparison is void before it starts.

The finding here is blunt: 34.6% of replies in this dataset were written by a person. The rest, 43.3% plus 22.1%, were not. That means a raw reply count overstates human engagement by roughly a factor of three, on average across this population. Whether that factor holds for any single team's campaigns is a separate question this study also addresses, because the ratio is not constant.

The practical consequence is a question, not a verdict: does your reporting pipeline currently separate out_of_office and auto_reply_other from genuine replies before anyone looks at a rate? If not, the number currently called 'replies' or 'engagement' in a weekly report is a blend of three different events with three different meanings, and the blend ratio can shift for reasons that have nothing to do with whether the campaign is working.

How the measurement works

What counts as one unit. One unit is one inbound reply to a cold email, stored as a row in BusySeed's warehouse. The study pulled 2,419 such rows from the measurement window 2025-12-01 to 2026-10-19 and excluded 1 row that could not be classified (for example, empty body text). That leaves 2,418 labelled units, drawn from 74 distinct campaigns run across ten sending workspaces.

Why a reply, not a thread. The unit of analysis is the individual reply message, not the email thread or the recipient. A single recipient could in principle generate more than one reply (an out-of-office followed later by a genuine one), and each would be labelled separately. This is the right unit for the question asked, because the question is about what a reply event looks like when it lands, not about any one person's eventual disposition.

The three classes. Every unit was sorted into exactly one of three classes, defined by a written rubric:

Who did the labelling. Classification was performed by claude-haiku-4-5, a language model, applied independently to each unit against the written rubric, rather than by a rules-based filter keyed on subject lines or sender headers. This choice trades the perfect consistency of a keyword filter for the ability to read a message's actual content and intent, at the cost of needing a calibration check, covered later in this study.

Why label at all, rather than count replies as replies. Because the entire point of the study is that 'reply' is not a single event type. Collapsing the three classes back into one count is exactly the measurement failure being diagnosed.

The headline result, and its limits

Across 2,418 labelled replies, the breakdown is:

• out_of_office: 1,046 replies, 43.3%
• genuine: 837 replies, 34.6%
• auto_reply_other: 535 replies, 22.1%

The headline, that only about a third of replies are written by a person, is the genuine figure of 34.6% rounded to a plain-language fraction. It is accurate as a description of this pooled dataset over this window.

What it does not mean. It does not mean that a third of people who receive a cold email reply to it. The denominator here is replies, not sends or recipients; this study says nothing about overall response rate, only about the composition of the replies that do arrive. It also does not mean that any individual campaign should expect a one-in-three split. As the next section shows, that split varies a great deal from campaign to campaign, so applying the pooled 34.6% figure to a specific list or sender is an assumption, not a certainty, until checked.

It also does not mean two-thirds of replies are worthless. An out_of_office message confirms a mailbox is live and a recipient exists, which has some value for list hygiene even though it carries no sentiment. auto_reply_other, which bundles bounces, ticket-system acknowledgments, and departure notices, is the closest thing to true noise in this scheme, and at 22.1% it is smaller than either of the other two classes but not negligible.

Finally, the classification came from a language model against a rubric, not from a ground truth audit of every message by a human. The calibration section later in this study gives the figures needed to judge how much the headline percentages should be trusted versus treated as a close approximation.

The shape of the result: it depends on what the recipient already told you

The pooled 34.6% genuine figure hides a pattern that matters more than the headline itself: the mix of reply classes is almost entirely explained by how the recipient had previously been labelled in the CRM, before this reply arrived.

When a recipient had already been tagged interested or not interested, almost every reply that followed was genuine: 91.8% and 97.1% respectively, with zero out_of_office replies in either segment. That makes sense mechanically, those tags are themselves usually produced by reading a prior genuine reply, so the segment is partly downstream of the same signal.

The neutral segment flips almost entirely the other way: 94.8% out_of_office, only 1.8% genuine. The unsubscribe or hostile segment, a small group of 43 units, is dominated by auto_reply_other at 93%, consistent with automated unsubscribe-confirmation messages being swept into that class.

The unlabelled segment, the largest at 638 units, sits closest to the pooled average, which is unsurprising since it is the least pre-filtered group. The one-third-genuine headline is really a statement about an unsorted inbox, not a fixed property of cold email replies in general.

Spread across campaigns: a population property, not one group's artifact

A separate question from segment is campaign: does the genuine share come from a few unusual campaigns, or is it a stable feature across most of them? This was checked across 19 campaigns with enough volume to measure.

The genuine share ranges from 8.8% to 54.3%, a spread wide enough that a campaign manager quoting the pooled 34.6% figure for their own campaign could be off by a large margin in either direction. The median genuine share, 28.1%, sits below the pooled mean, which means the pooled figure is being pulled upward by a smaller number of high-genuine campaigns, likely ones with more interested or not-interested recipients already in the mix, per the segment pattern above.

No single campaign dominates the pool: the largest contributes 21.8% of all replies, and the average campaign contributes 32.68 replies out of 2,418 total, across 74 campaigns. So the headline is not an artifact of one oversized campaign swamping the rest, it genuinely reflects variation that exists across a broad set of 19 measured campaigns.

The honest reading is two-sided. The population-level claim, that genuine replies are a minority of all replies, holds up across campaigns, out_of_office or auto_reply_other is the median outcome, not an edge case. But the exact share for any one campaign depends heavily on list composition and prior segmentation, enough that a team should measure its own campaigns' class mix rather than import the pooled figure wholesale.

Calibration: how much to trust the labels

The classification was done by a language model, claude-haiku-4-5, reading each reply against a written rubric. That is faster and more content-aware than a keyword filter, but it is not infallible, so a sample was checked against independent agreement.

On 2,418 labelled units, the calibration check reports agreement_pct of 87.1%, stored precision of 84.43%, and stored recall of 86.04%. In plain terms: when two independent passes were compared, they agreed on the class 87.1% of the time. Precision of 84.43% means that of the units labelled a given class, that share were confirmed correct; recall of 86.04% means that of the units that truly belonged to a class, that share were actually caught by the label.

No mitigation was layered on top of the raw model output: 0 mitigations were applied. The published percentages are the model's direct labels, not a post-hoc adjusted estimate.

What this means for the headline figure: an agreement rate of 87.1% is high but not perfect, which means a modest share of units are plausibly mislabeled, in either direction, between genuine and the two automated classes. This is most important at the boundary between genuine and auto_reply_other, where a short, terse human reply ('not interested, remove me') can resemble an automated unsubscribe confirmation, and least important for the out_of_office class, which tends to carry distinctive structural markers (date ranges, alternate contacts) that are easier to classify consistently.

The honest caveat: precision and recall figures reported here are stored, aggregate figures rather than broken out per class, so it is not possible from this study alone to say whether genuine is labelled more or less reliably than out_of_office specifically. A reader who needs class-specific error rates for a high-stakes decision should treat the pooled 87.1% as an upper bound on confidence for any single class, not an exact figure for each.

What to do differently on Monday

Three concrete changes follow from this study, in order of how cheap they are to make.

Stop reporting raw reply count as engagement. If a weekly report currently lists 'replies' as a single number, split it into at least genuine versus automated before it reaches anyone making a budget or headcount decision. At the pooled rate, roughly two out of three replies in an unsorted inbox are not from a person; a team that doesn't split the count is reporting noise as signal roughly twice as often as signal.

Check segment composition before comparing campaigns. Because the neutral segment runs close to 94.8% out_of_office while the not-interested segment runs 97.1% genuine, two campaigns with identical messaging but different list composition, more cold outreach to unsegmented contacts versus more follow-up to already-engaged ones, will show very different raw reply rates for reasons that have nothing to do with message quality. Compare genuine-reply rate within matched segments, not raw reply rate across campaigns with different list histories.

Don't assume your campaign sits at the pooled average. The spread across campaigns, 8.8% to 54.3% genuine, is wide enough that the pooled 34.6% is a population description, not a forecast for any one list. A team should pull its own campaign's class mix and compare it to the median of 28.1% rather than the pooled mean.

What this study cannot tell a team to do: it does not say which message angle produces more genuine replies, because it classifies what a reply is, not why it was sent. A follow-up study would need to hold the class constant and vary subject lines or sender identity to make that causal claim, something this dataset was not designed to support.

Reading the published dataset yourself

The underlying data is the 2,418 labelled replies (from 2,419 rows, with 1 excluded), each carrying a class label, a segment label where one existed prior to the reply, and a campaign (group) identifier. A reader working with the raw export should keep a few things in mind.

Check denominators before quoting a percentage. The unsubscribe or hostile segment has only 43 units, small enough that its 93% auto_reply_other figure should be quoted alongside that denominator every time, not as a bare percentage. The interested segment, at 158 units, is similarly modest. Contrast both against the unlabelled segment's 638 units, which carries far more statistical weight.

Distinguish segment from campaign. These are two different groupings in the data (five segments versus 74 campaigns) and they answer different questions. Segment tells you about recipient history; campaign tells you about list and messaging. Don't average across campaigns within a segment without checking whether one or two campaigns dominate that segment's count, the same concentration check performed for the whole dataset in an earlier section.

Treat the class label as model output, not ground truth. Every class value in the export came from claude-haiku-4-5 applied against the written rubric, with 0 corrections layered on afterward. The calibration figures, 87.1% agreement over 2,418 checked units, are the right error bars to carry into any downstream analysis. If you re-slice this data by a dimension not covered here, such as sender domain or time of day, treat any resulting percentage with the same caution given to the segment and campaign cuts above: check the count behind it before trusting the rate.

The measurement window is fixed. All figures here cover 2025-12-01 to 2026-10-19 only. Reply composition is plausibly seasonal, end-of-year auto-replies spike for different reasons than summer ones, so figures from a different window are not guaranteed to match.