"Colgate gets 90% accurate surveys from AI personas" drew 7,200 likes. The paper's 90% means something else
First seen on X 31 hours ago@politicalmath ♥ 7,208
"Colgate says they are generating 90% accurate user survey data by simply asking the LLM to role-play as their target demographics," PoliMath posted on Nov. 21, drawing 7,200 likes. A reply warning "this is a great way to get all the same product eventually" got 593.
The number traces to an October 2025 arXiv paper that tested 57 personal-care product surveys (9,300 human responses) from a leading company in that market. Digg summarized it as Colgate's AI "customers" reaching 90% of human test-retest reliability.
So 90% is not accuracy: it means the AI matched 90% of how consistently the same humans answer the same survey twice. And it is not simple role-play. Asking models for 1-to-5 ratings directly gives unrealistic distributions, so the authors had them write free-text answers and mapped those to ratings by embedding similarity (SSR). The resulting distributions scored above 0.85 on KS similarity, and the synthetic respondents explained their ratings.
Practitioners in the replies split: one marketer said persona-based research has worked "fantastic" since March; others warned that if people stop posting opinions online, the models lose their feedback.
Comments 0
Select a sentence in the story (long-press on phones) to comment on it.
“Colgate says they are generating 90% accurate user survey data by simply asking the LLM to role-play as their target demographics That has some pretty profound short-ter…”
