Methodology

Engines and exact versions

A study that does not name the model version it ran against is not reproducible, because the answer changes when the model does.

Engine	Model version

Run window, UTC: [object Object].

Method

What counts as one unit. One unit is one inbound reply to a cold email, stored as a row in BusySeed's warehouse. The study pulled 2,419 such rows from the measurement window 2025-12-01 to 2026-10-19 and excluded 1 row that could not be classified (for example, empty body text). That leaves 2,418 labelled units, drawn from 74 distinct campaigns run across ten sending workspaces.

Why a reply, not a thread. The unit of analysis is the individual reply message, not the email thread or the recipient. A single recipient could in principle generate more than one reply (an out-of-office followed later by a genuine one), and each would be labelled separately. This is the right unit for the question asked, because the question is about what a reply event looks like when it lands, not about any one person's eventual disposition.

The three classes. Every unit was sorted into exactly one of three classes, defined by a written rubric:

Class	What it is	Share
out_of_office	Automated absence notice	43.3%
genuine	A person wrote this	34.6%
auto_reply_other	Bounces, ticket receipts, departure notices	22.1%

Who did the labelling. Classification was performed by claude-haiku-4-5, a language model, applied independently to each unit against the written rubric, rather than by a rules-based filter keyed on subject lines or sender headers. This choice trades the perfect consistency of a keyword filter for the ability to read a message's actual content and intent, at the cost of needing a calibration check, covered later in this study.

Why label at all, rather than count replies as replies. Because the entire point of the study is that 'reply' is not a single event type. Collapsing the three classes back into one count is exactly the measurement failure being diagnosed.

Limitations we volunteer

Written by us, before anyone else found them.

Terms used in this study

reply_class

A column already present in the data warehouse that attempts to classify each reply, but one whose labelling process is not documented or audited on this system.

out_of_office

An automated message sent by an email system to say the recipient is currently absent, with no judgment from the recipient about the offer itself.

auto_reply_other

A machine-generated reply that is not an absence notice: bounce notifications, support ticket receipts, or notices that someone has permanently left a role.

genuine

A reply actually composed by a person, regardless of whether their tone is positive, negative, or neutral.

Batch API

A mode of calling a language model that processes a large set of inputs together rather than one at a time, used here to label every reply in the corpus in a single run.

calibration

Comparing this study's independent labels against the platform's existing reply_class column to see how often the two agree, used as a check rather than as proof that either one is correct.

precision and recall

Two ways of scoring a classifier against a reference: precision is how often its positive calls were actually right, recall is how many of the true cases it managed to catch.

segment (ai_interest_value)

The platform's own label for how interested a lead seems based on their reply, used here to check whether automated replies cluster in particular interest categories rather than spreading evenly.

group spread

The range of a class's share across individual campaigns, reported alongside the overall average so a reader can see whether the headline number reflects a consistent pattern or is driven by a few unusual campaigns.

References

Sources this study reads against. Every link was fetched and confirmed reachable at publication.

1. Cold Email Benchmark Report 2026 Instantly Headline industry benchmark reporting reply rates without separating human replies from automated ones; the figure our study is checking.
2. How to detect automatically generated emails arp242.net Practical explainer of the Auto-Submitted header standard and its limits in real mail systems.
3. I Analyzed 30.31 Million Cold Emails: 13 Stats That Matter in 2026 Saleshandy Describes a vendor pipeline that runs every reply through an AI classifier into categories including Auto-Reply and Out of Office, the same type of classification task our study performs.
4. The State of Cold Email 2026 Hunter.io Reports overall reply incidence and a quantified human-editing-versus-automation effect on reply rate.
5. Cold Email Benchmarks 2026: The Real 3.43% Reply Rate Data (and What Good Actually Looks Like) lemlist Secondary source compiling Instantly's and other vendors' reply-rate figures and commenting on methodology variance across benchmarks.
6. Cold Email Statistics (2026): 60+ Data Points from 2M+ Emails Sales.co Provides a finer reply-type breakdown (positive, negative, auto-reply, referral, neutral, other) comparable to our class and segment structure.
7. RFC 3834: Recommendations for Automatic Responses to Electronic Mail IETF Canonical standard defining the Auto-Submitted header used to mark automated responses, the technical basis for distinguishing bot replies from human ones.
8. Cold Email Response Rates: What 1,288,605 Emails Say Sales.co Independent study quantifying what share of cold email replies are automated versus human across a large sent-email dataset.
9. Cold Email Open and Reply Rates: 2026 Benchmark EmailBison Explains why published reply-rate benchmarks diverge by multiples, citing denominator and inclusion-rule differences (all replies vs. human-only replies).
10. Out of Office Detection, Auto Pause, and Resume Saleshandy (documentation) Vendor documentation showing a real-world implementation of auto-reply detection, including stated platform-specific blind spots such as Yahoo/AOL threading.

Data

The complete row-level dataset is published open and ungated under CC BY 4.0. Every number in this study can be recomputed from it.