Does labeling a reply "interested" or "not interested" tell you whether a human wrote it?

Cold email platforms typically let a sales rep tag an incoming reply with an outcome: interested, not interested, neutral, or unsubscribe/hostile. This study's labelling of 2,418 replies lets us check something useful: do those human-applied sentiment tags line up with whether a reply was actually written by a person, as opposed to an autoresponder?

The answer is yes, strongly. Among the 158 replies tagged "interested," 91.8% were independently classified as genuine human replies (145 of them), with zero classified as out_of_office. Among the 512 replies tagged "not interested," 97.1% were genuine (497 replies), again with zero out_of_office. The "unsubscribe or hostile" segment, smaller at 43 replies, runs the other way: 93% of those were classified as auto_reply_other rather than genuine, which likely reflects automated unsubscribe-confirmation messages being swept into that segment rather than actual hostile replies from people.

Compare that to the "neutral" segment, 1,067 replies, where only 1.8% were genuine and 94.8% were out_of_office. And the largest group by far, 638 replies with no sentiment tag applied at all, splits 27.3% genuine, 67.6% auto_reply_other, and 5.2% out_of_office.

What this tells you. When a sales rep or an automated system assigns a definite outcome label like "interested" or "not interested," that label is almost always attached to a reply a human actually wrote. That makes sense mechanically: an autoresponder saying someone is on vacation gives a rep nothing to categorize as interested or not, so those replies tend to stay unlabelled or get swept into a generic neutral bucket instead. The sentiment-tagging workflow, in other words, is implicitly doing some of the same filtering this study's classifier does explicitly, just as a side effect of reps only bothering to tag replies that look like real conversations.

What this does not tell you. The "unlabelled" segment is the largest single segment in the dataset at 638 replies, more than four times the size of "interested" and more than the "not interested" and "neutral" segments combined are compared to it. That segment is mostly automated (67.6% auto_reply_other plus 5.2% out_of_office), but it still contains 174 genuine human replies that never received an outcome tag. If a team only looks at labelled segments to judge campaign performance, it will miss those replies entirely, whether because the rep hadn't gotten to them, the workflow didn't force a label, or the reply fell into some ambiguous middle ground a rep didn't know how to categorize.

The practical implication: sentiment labels applied during a sales workflow are a reasonably good proxy for "a human wrote this," but they are not a complete one. A team auditing reply quality should still run genuine replies through independent classification rather than assuming the unlabelled pile is just more of the same.