Questions about this study

In this corpus, 34.6% of replies were classified as genuine, meaning a person wrote them. The rest were automated: 43.3% were out-of-office notices and 22.1% were other automated messages like bounces, ticket receipts, or departure notices. This covers 2,418 replies out of 2,419 rows in BusySeed's warehouse (one row excluded) over the window 2025-12-01 to 2026-10-19, each labelled independently by Claude Haiku 4.5 against a written rubric. So when someone reports a reply count for a campaign, the bulk of it may not represent human engagement at all.

Each reply was labelled by an AI model, Claude Haiku 4.5, working from a written rubric, not by a human reader checking each message. It assigned one of three classes: out_of_office (an automated absence notice), auto_reply_other (bounces, ticket receipts, departure notices, and similar non-human replies), and genuine (a message a person actually wrote). This labelling has been checked against a stored software column already in the platform, agreeing on 87.1% of a sample of 2,418 replies, but it has not yet been checked against a human reader. That comparison is planned, with a sample of disagreements set aside for it.

It's a fair concern, and the honest answer is that this hasn't been validated against a human reader yet, only against the platform's own existing reply_class column. Against that stored column, the two methods agree on 87.1% of a sample of 2,418 replies, and the stored column shows 84.43% precision and 86.04% recall when measured against this new labelling. That's a reasonable but imperfect agreement, not proof of ground truth, since both could share the same blind spots. Until a human-reader check happens, treat this as a careful machine classification rather than a verified fact.

Not quite, this measures the composition of replies that do arrive, not how often campaigns get replies in the first place. Send volume per campaign isn't in this dataset, so there's no reply rate here, only a breakdown of what a reply contains once it lands. A campaign could have a very low reply rate overall and still see most of those replies be genuine, or a high reply rate dominated by out-of-office notices. These are two separate questions, and this study only answers the second one.

Out_of_office is specifically an automated absence notice, the kind of message someone's email client sends while they're on vacation or away from their desk. Auto_reply_other is a broader catch-all for other non-human replies: bounce notifications, automated ticket receipts from a support system, departure notices saying someone has left the company, and similar messages. Both are automated and neither represents a person engaging with the email, but they're generated by different systems for different reasons, which is why the study keeps them as separate classes rather than lumping all automation together.

It varies enormously. Across 19 campaigns, the genuine share ranges from 8.8% on the low end to 54.3% on the high end, with a median of 28.1%. So the overall 34.6% figure is an average across very different campaigns, not a number you should expect any single campaign to hit. A campaign sitting near the low end of that range isn't necessarily broken, and one near the high end isn't necessarily a template to copy, without knowing why they differ.

Yes, strongly so. Replies marked interested are 91.8% genuine, essentially all human, out of 158 replies in that segment, with zero out-of-office notices in that group. Compare that to replies marked neutral, which are 94.8% out-of-office out of 1,067 replies. This suggests the platform's interest label and the genuine/automated split are telling a very similar story from two different angles, with automated noise concentrating almost entirely in the segment a human would also describe as uninteresting.

Stop counting raw reply totals as a proxy for engagement, since on average roughly two-thirds of replies in this corpus were automated. If a dashboard reports reply counts without splitting out genuine from automated, a spike could just mean more absence notices, not more interest. A reasonable next step is to filter or weight replies by the interest label before reporting them, since neutral-labelled replies are overwhelmingly out-of-office (94.8%) while interested-labelled ones are overwhelmingly genuine (91.8%). That alone would strip out most of the automated noise without needing the full classifier.

That's a reasonable worry given how much the genuine share varies by campaign. The data spans 74 campaigns across ten sending workspaces, averaging 32.68 replies each, and the single largest campaign contributes 21.8% of all replies, so no one campaign dominates the total. But the spread itself, from 8.8% to 54.3% genuine share across campaigns, means the aggregate 34.6% figure is shaped by whatever mix of campaigns happened to be in this window. A different mix of campaigns could shift that overall number meaningfully, which is why the per-campaign range matters as much as the average.

Genuine means the rubric judged that a person wrote the message, as opposed to it being generated automatically by an email client or ticketing system. It doesn't say anything about the length, tone, or effort behind that message, a one-line