Cold Email Reply Classification Explained

Every agency treats an inbound reply as an opening to engage. We run outbound for 50+ B2B companies, have handled over 95,000 positive replies this year, and the campaigns that book the most conversations are the ones answering the fewest messages. Below, the 11 category taxonomy, the routing table that decides what happens to each reply, the boundary cases that break naive classifiers, and how to measure whether yours is working.

What Is Cold Email Reply Classification?

A reply arrives. Somebody, or something, has to decide what it is before anything else can happen. A buyer who writes "yes send it over" and a buyer who writes "do you really think we need this" both replied, both used plain English, and the right next move for each is the opposite of the right next move for the other.

At 300 emails a month, a founder reads every reply and the decision is free. At 15,000 emails a month per client across 50+ campaigns, nobody is reading every reply, and the decision has to be a function with a defined output. That function is the classifier. It sits between the inbox webhook and whatever composes the response, and it is the single most underbuilt layer in most outbound programs.

Reply Classification

The step in an outbound pipeline that reads an inbound reply, identifies the sender's intent, and labels the message with one category from a fixed taxonomy. It produces a label and a confidence score, nothing else. It does not write the response and it does not decide the action.

Routing Table

The fixed mapping from each category to exactly one action: send this asset, move to this stage, suppress this contact, queue for this date, or do nothing. One label, one action, no branching logic inside the classifier. The separation is what makes the system auditable when something goes wrong.

The reason this matters more in 2026 than it did in 2022 is volume math. Instantly's 2026 cold email benchmark report puts the average reply rate at 3.43 percent, with the top quartile at 5.5 percent and elite campaigns above 10 percent. Our book sits at 4.6 percent. At that rate a single client sending 15,000 invites a month produces close to 700 replies, and only a minority of those are the ones worth answering. The positive reply rate is the number that predicts booked conversations, and it moves on classification quality long before it moves on copy.

Speed is the other half of the argument. Harvard Business Review's study of online sales leads found that firms contacting a lead within an hour were nearly 7 times more likely to qualify it than firms waiting one hour longer, and more than 60 times more likely than firms waiting a day. You cannot hit a sub-hour response window with a human reading a shared inbox twice a day. You hit it with a classifier that labels in seconds and a router that fires without asking permission.

What Are the 11 Reply Categories?

One reply, one primary label. These 11 cover roughly 99 percent of real inbound messages on a B2B campaign, and each one exists because it routes differently from every other one. If two categories share an action, collapse them.

• Positive. Clear interest with no specific question attached. "Sounds good." "Tell me more." "Send it over." "Happy to take a look." The buyer has agreed to receive the next thing and has not told you what they want to know.
• Question. A sincere request for information the buyer would use to decide. Scope, mechanism, timeline, fit, rates, who else you have worked with. The test is whether a real answer changes what they do next.
• Call-request. An outright ask for a meeting. "Can we jump on something Thursday." This skips the tell-me-more step and deserves its own label because the action is a booking link, not an asset.
• Referral. The recipient routes you to a different person. "That would be Dana, copying her." Highest-value category per message in most campaigns, and the one most taxonomies bury inside "other" where it quietly dies.
• Objection. Pushback with no sincere question inside it. "We already have an agency." "We tried this and it did not work." "No budget this year." It reads like a question sometimes. It is not asking for information.
• Banter. Sarcasm, snark, jokes, mockery, rhetorical challenges. Frequently carries a question mark, which is exactly why naive classifiers mislabel it as a question and then answer it earnestly.
• Not-now. A deferral with a date or a trigger attached. "Circle back after our fundraise." "Ask me in Q3." The date is the whole value of the category.
• Hard-no. Explicit disinterest in plain language. "Not interested." "Please stop." No date, no door left open.
• Out-of-office. An autoresponder. Vacation, leave, a role change, a departure notice. Some of these carry a named alternate contact, which makes them referrals in disguise.
• Unsubscribe. An explicit removal request. "Remove me." "Take me off this list." This is a compliance event, not a sales event.
• Other. Gibberish, bounced auto-notices that are not OOO, internal forwards with no instruction, a one-word reply with no readable intent. Keep this category small. A rising "other" share is the earliest signal that the taxonomy has drifted away from what the inbox is actually receiving.

Two design notes that save rework later. First, hard-no and unsubscribe stay separate even though both end in suppression, because only one of them is a legal event and the audit trail has to show which. Second, not-now and hard-no look nearly identical on the surface and route in opposite directions, which is why handling not interested replies is a classification problem before it is a copy problem.

Which Reply Categories Should Get an Automated Response?

Four categories earn an automated send. The other 7 get logged, labeled, and routed in silence. That ratio is the part most teams get backwards, and fixing it has produced a larger lift for us than any copy test we have run this year.

Here is the full routing table. One row per category, one action per row.

The suppression rows are not optional and not a style choice. The FTC's CAN-SPAM compliance guide requires an opt-out request to be honored within 10 business days, and requires the opt-out mechanism to keep working for at least 30 days after the message goes out. A classifier that labels an unsubscribe as an objection is a compliance failure wearing the costume of a routing bug.

The commercial case for suppressing hard-nos is just as hard-edged. Google's sender guidelines tell bulk senders to keep the user-reported spam rate in Postmaster Tools below 0.10 percent and to never reach 0.30 percent. Mailing people who told you to stop is the fastest route to that number, and once you are over it you are looking at domain recovery rather than campaign tuning. Classification is a deliverability control, which is the part nobody puts in the org chart.

On the 7 silent categories, the intuition that drives the wrong default is reasonable on its face: a reply is engagement, and engagement deserves a response. What the data says is that reply quality is bimodal. The buyer is either sincere, which is positive, question, call-request, or referral, or the buyer is dismissive or distracted, which is everything else. Across 95,000 positive replies this year, engaging the second group converted at a rate that did not justify a single send, and it carried a tail risk the first group cannot offset. One screenshot of your system earnestly answering a joke undoes a quarter of goodwill.

Objections are the interesting exception, and the one people argue with. We still do not auto-reply to them. We log the exact wording, because an objection is free market research about the ICP definition and the offer, and because handling an objection well is a human conversation, not a template. Gong's analysis of sales email data found that pitching cuts reply rates by as much as 57 percent. An auto-rebuttal to an objection is the purest form of pitching there is.

How Do You Classify the Boundary Cases?

Clear cases are not where classifiers fail. A reply that says "not interested, remove me" is labeled correctly by a regular expression. The 5 boundary cases below are where the accuracy actually lives, and where a generic prompt will quietly cost you bookings for months.

Boundary 1: sarcasm that looks like a question. This is the hardest class in the taxonomy and the research says so. A 2026 study on sarcasm detection in Springer's AI journal measured GPT-4o at a macro-F1 of 67.47 percent on the MUStARD++ benchmark, with Llama 3-70B at 63.59 percent, while purpose fine-tuned transformer models reached the low 90s. Work on LLM sarcasm annotation presented at Interspeech points the same direction. In plain terms, a general model is a coin flip and a half on sarcasm unless you write the rule into the prompt yourself.

So write it in. A reply is banter if any of these 5 signals fire, regardless of what the surface text looks like:

• Mocking tone or a rhetorical question structure. "What do you say to that." "Really." "How is that working out."
• Sarcastic agreement or feigned enthusiasm. "Oh sure, send me everything you have got."
• Jokes, taunts, or ribbing. Laughter markers, emoji-heavy one-liners, a punchline with no ask.
• A challenge that re-poses your own premise back at you with no follow-up question attached.
• A dismissive single-clause comeback aimed at the sender rather than at the offer.

A question is sincere, and earns a reply, only when the buyer is asking for information they would use to decide. Rates, scope, timeline, mechanism, fit, references, next steps. The question mark is not the signal. The decision-usefulness of the answer is the signal.

Boundary 2: multi-intent replies. "What does this run, and can we talk Thursday." That is a question and a call-request in one message, and a single-label classifier drops one of them. The fix is a secondary labels array on the output schema plus a composer that answers the question first and confirms the meeting second. In our testing this cut misrouted replies by roughly two-thirds, and multi-intent messages turned out to be some of the highest-intent mail in the whole inbox.

Boundary 3: the gatekeeper forward. An assistant replies on behalf of the buyer, or the recipient copies in a colleague with no comment. The label is referral, the action is a new contact and a fresh thread, and the original recipient gets suppressed so the same company does not get two parallel sequences. Getting this wrong is the single most expensive miss in the taxonomy, because the forward is a warm hand-off and reaching a decision maker is the hard part of the job.

Boundary 4: soft no versus dated no. "This is not a priority right now" has no date. "Not until after we close the raise" has a trigger. The first is a hard-no and gets suppressed. The second is a not-now, gets a date extracted, and goes into a dated queue for exactly one touch. Both read as rejection, and re-engaging cold leads only works when the queue is populated by the second kind and never the first.

Boundary 5: the out-of-office that is really a referral. "I am out until the 14th, for anything urgent contact Priya at priya@" is labeled out-of-office by almost every classifier in production, and that is a miss. Parse the named alternate, create the contact, open the thread. The same parse handles the departure notice, where the right move is a new contact at the same account rather than a bounce-driven bounce cleanup.

How Do You Measure Classifier Accuracy?

One blended accuracy number is worse than no number, because it averages away the errors that cost you something. A classifier at 94 percent overall that misses 1 unsubscribe in 10 is a liability. Set the target per category, and set it against the cost of each direction of error.

The measurement loop itself is 4 steps and takes under an hour a week.

1. Sample. Pull 100 classified replies at random from the last 7 days, stratified so every category appears at least 5 times. Random-only sampling under-represents the rare categories, which are the expensive ones.
2. Label blind. A human assigns the category without seeing the machine label. Seeing it first anchors the reviewer and the number comes back flattering and useless.
3. Build the confusion matrix. Rows are the human label, columns are the machine label. Read the off-diagonal cells, not the headline percentage. The cell that matters is always a specific confusion, usually banter landing in question or hard-no landing in objection.
4. Feed the misses back as examples. Every miss becomes a few-shot example in the system prompt, verbatim. This is the whole improvement mechanism, and it compounds week over week.

Track one leading indicator alongside the matrix: the share of replies landing in "other". When it climbs, the inbox has started receiving something the taxonomy does not describe, and the matrix will not tell you that for another 2 weeks. The same instinct that drives deliverability monitoring applies here, because both are early-warning systems on a layer nobody looks at until it fails.

How Do You Actually Build One?

A reply classifier is one model call with a cached system prompt and a constrained output. It is not a fine-tune, it is not a vector store, and it is not a multi-agent anything. Five configuration decisions carry almost all of the performance.

1. Pick a small fast model. Classification is a labeling task, not a writing task. A Haiku-class or Mini-class model runs it at a fraction of the latency and cost of a frontier model with no measurable accuracy loss on a well-specified taxonomy. Save the larger model for composing the reply, where the writing quality actually shows.
2. Set temperature at or near 0. This is not a stylistic preference. Research on LLMs for text annotation found stance classification accuracy rising from 53.7 percent to 70.7 percent as temperature dropped from 0.9 to 0.2. A creative classifier invents categories. You want the boring one.
3. Write the prompt as examples, not as definitions. The same body of annotation research found that adding a single example per class lifted average accuracy from roughly 62 percent to 68 percent, and dynamic few-shot selection pushed it close to 90 percent. Give every category 3 to 5 real replies pulled from your own inbox, include the sarcasm rubric verbatim, and state the cost asymmetry as a rule: when in doubt, take the silent path.
4. Constrain the output to a schema. Use the structured output mode your provider offers, as documented in OpenAI's structured outputs guide, so the model has to return one of your 11 labels instead of a sentence about labels. The schema is 4 fields: category, secondary_labels, confidence, and reasoning. The reasoning string is for humans and it is the column you read when accuracy slips.
5. Cache the system prompt. It does not change between leads, which makes it the textbook caching candidate. Anthropic's prompt caching documentation is explicit about the pattern: put the stable instructions at the front of the request, the variable content at the back, and set the breakpoint between them. Done right this is most of your cost per reply.

For context, pass the original email body, the full reply body, and at most the prior 2 messages in the thread. More history than that adds tokens and subtracts accuracy, because the model starts classifying the conversation instead of the message in front of it.

End to end, from webhook receipt to send, a well-built path runs in 10 to 15 seconds. That is what makes a 15 minute response window on a positive reply a floor rather than a stretch, and in our data positive replies that get a personalized asset inside 15 minutes close at 31.2 percent against 8.4 percent for a bare booking link. The classifier is what buys that speed.

What Breaks a Reply Classifier?

Nearly every broken classifier we audit on an inherited campaign fails in one of 4 ways. The fixes are cheap. Finding which one you have is the work.

False positives on banter. The classifier sees a question mark, labels the reply a question, and the composer answers a taunt in good faith. Fix the temperature first, then paste the 5 signal rubric into the prompt, then add your 10 worst real examples. This is the failure with the longest tail of damage and the cheapest fix.

False negatives on vague positives. A buyer writes "yeah go ahead" or "sure" and the classifier calls it too thin to be positive. The model is not wrong on principle, it is just under-exampled. Ten real vague-yes replies in the prompt closes this gap in an afternoon, and it is usually worth a visible move in the reply rate benchmarks you report.

Taxonomy drift. The categories were written for the campaign you ran last year. The offer changed, the personalization changed, the list changed, and now 12 percent of replies land in "other" and nobody noticed because nobody has a dashboard for "other". Review the taxonomy quarterly against the actual inbox.

No human in the loop at all. The classifier ships once, works acceptably, and never gets touched again. Accuracy decays slowly and invisibly, which is the worst possible failure shape. Salesforce's State of Sales research has reps spending roughly 40 percent of their time actually selling, with the rest going to admin, data entry, and follow-up overhead. A classifier exists to buy back part of that 60 percent, and it only keeps buying it if somebody spends an hour a week on the confusion matrix. On the volume side, HubSpot's sales statistics note that 60 percent of buyers say no 4 times before saying yes, which is also why the not-now queue has to be real and dated rather than a folder.

What Changes When the Reply Is to a Podcast Invite?

Invite replies classify differently from pitch replies, and anybody moving from reverse outbound on a pitch to an invite will see the distribution shift inside a week. The ask is a compliment rather than a sale, so the dismissive categories shrink and the question category grows.

Three things change in practice. First, banter and hard-no drop as a share of replies, because almost nobody is rude to an invitation. Second, the question category splits into sub-questions that repeat constantly: what is the show, who listens, how long is it, what are we talking about, who else has been on. Those are canned-answer territory, not composed-reply territory. Third, the referral category jumps, because an executive who cannot do it will often hand you their co-founder or their head of marketing instead of declining outright.

So the invite classifier carries a small extra set of answer slots rather than a bigger taxonomy. We hold a fixed answer for each repeat question, the classifier picks the slot, and the composer stitches the answer above the booking link. That design is why podcast invite reply rates hold up where pitch reply rates sag, and it is the same reason the invite follow-up sequence reads nothing like a pitch follow-up. The copy that earns the reply in the first place matters just as much, which is covered in writing an invite that sounds human and in invite subject lines.

Downstream, the label is what the whole funnel runs on. A positive reply becomes a booked alignment call, the alignment call becomes a recorded conversation, and the recorded conversation is the unit our guarantee is written in: 30 recorded conversations with your ideal buyers in 90 days or your money back. Editing and publishing are included, invites go out by email only, and the client owns every recording on their own show. None of that holds if the classifier cannot tell a yes from a joke, which is why the CRM schema stores the label on the contact and attribution reads it back later.

The Practitioner Frame on Reply Classification

Reply classification produces no artifact anybody wants to look at. There is no deck slide, no creative review, no screenshot for the client Slack. It sits behind the inbox and the only people who ever notice it are the ones whose campaigns it has already quietly ruined. That is exactly why it is where the spare performance is.

The teams that get it right do 3 unglamorous things. They keep the taxonomy small enough that every category maps to one action. They run the 100-reply blind review every week and feed the misses back as examples. They bias toward silence on anything ambiguous, because the cost of a wrong send is asymmetric and always has been.

Eleven categories. Four that earn a send. Five signals on sarcasm. One small model, temperature near 0, cached prompt, constrained output, 15 second path. The rest of the stack gets the attention, from domain setup and warmup through verification and sending strategy, and all of it exists to produce replies that something then has to read correctly. Build the layer that reads them, review it weekly, and the back end of your outbound program stops being the ceiling on what the front end can do.