Blog / What we measured
Answer first: what reviewing every LinkedIn reply taught us
When someone replies to one of our profiles, our AI writes the answer and a second model reviews it before it is sent. In September the reviewer was turning down most of what the writer produced, and the reason was always the same.
The short version
- Replies that mentioned our product to someone who had not asked were approved about 1 time in 6.
- Replies that simply answered what the person said were approved about 2 times in 3.
- We stopped letting the writer decide when to mention us. It now does so only when the person asks, or says yes.
- Approval went from 39% to 80% and 83% on the next two runs.
How a reply gets written
Every one of our profiles has a real inbox with real people in it. When one of them writes back, our AI drafts an answer in the voice of the profile. Before anything is sent, the draft goes through code checks and then through a second model whose only job is to refuse anything a careful person would hesitate to send.
The reviewer's main rule is simple: answer what they said. It turns down a reply that pitches to someone who did not ask, invents a detail about them, or would make sense pasted into any conversation. It also refuses any link except our own website, and any email address or phone number.
What the reviewer turned down
Over three days we had 276 reviews. We split them two ways: had the person asked about us (what we do, what it costs, how it works), or at least described the problem we solve? And did the draft mention our product or our free preview?
| What the person had said | Draft mentioned us | Draft just answered |
|---|---|---|
| Nothing about the problem | 16% approved | 64% approved |
| Described the problem | 29% approved | 71% approved |
| Asked about us | 1 of 13 approved | 4 of 6 approved |
The first two rows were the ones we expected. Someone who writes "Hello" does not want a product pitch back, and someone who mentions that outbound is hard has not asked to be sold to either.
The last row surprised us. Even when a person had apparently asked, most mentions were turned down. When we read them, most of those people had not really asked. One had said our company's name in passing, and another was ending the conversation politely. Our old rules had treated both as an invitation.
The writer had been given a permission to decide for itself: "you may mention the preview if what they said is about finding prospects". It read that permission generously, every time. It offered the preview to people saying hello, to people looking for a job, and to someone who had already said yes to it.
What we changed
The decision moved out of the writer and into code that reads only the other person's own words:
- They asked what we do, what it costs or how it works. The writer answers, names LLM Leads once and says it works on it.
- They said yes to the preview. The writer gives them the next step and nothing else.
- They said hello. The writer says hello back and asks one light question about their work.
- They are looking for a job. The writer wishes them well and makes no promise.
- They are signing off. One warm line, and no new question.
- Anything else. The writer answers their point and does not mention us at all.
We also stopped showing the writer our product description unless it was allowed to use it. It had been told "background only, do not recite this", and it recited it anyway.
What happened next
| Review run, 26 September | Reviewed | Approved |
|---|---|---|
| Before the change | 18 | 39% |
| First run after | 50 | 80% |
| Second run after | 29 | 83% |
There was a cost, and it is worth saying. Far fewer replies now carry a link to our site, because far fewer people have asked for one. We would rather a stranger get an answer than a brochure, and we will know within a few weeks whether the conversations that follow do better.
If you write replies yourself
- Reply to the last thing they said. If your answer would fit any conversation, it is a template.
- Wait to be asked. "What do you do?" will come if the conversation is going well, and the answer lands better then.
- Read "yes" as a request for the next step, not as permission to pitch again.