What is a conversational survey, and when does it beat a questionnaire?
A conversational survey, or AI interview, replaces a fixed list of questions with a short written conversation. An AI interviewer asks an open question, listens to the answer and follows up on what the participant actually said, for instance by asking for a concrete example. It beats a form with open questions when you want to understand why people feel the way they do, and you don't know in advance what answers to expect.
How an AI interview works
An AI interview usually starts just like an ordinary survey: with one score question, such as eNPS or NPS. Then it changes character. Instead of moving on to the next question on the list, the interviewer asks about the answer that was just given. An example:
Score: 6
InterviewerThank you. What's the main reason it's a 6 rather than a higher score?
ParticipantThe workload. It's just too much.
InterviewerToo much work, then. Can you tell me what's mainly behind that?
ParticipantMostly the rush jobs that keep landing in between. Since the new planning system came in, I don't know in the morning what I'll be doing that day.
InterviewerCould you give me an example from the last few weeks?
A form would have given you "workload". The conversation gives you a cause you can act on: the new planning system, and the rush jobs that cut across it. That's the whole idea: keep following up until the answer is concrete enough to do something with.
Afterwards, every conversation is analysed: each answer is linked to one or more themes and given a sentiment. Across hundreds of conversations you can see which topics come up most and how people talk about them, with the quotes alongside.
How it differs from an open question on a form
An open question in a questionnaire looks as if it does the same job, but there are real differences:
| Open question on a form | AI interview | |
|---|---|---|
| Follow-up | Not possible. What's written is what you get. | Asks for a reason, an example or a clarification. |
| Length of answers | Often a few words. | Longer and more specific, because someone responds to them. |
| Order and topics | Fixed, decided by you. | Follows what matters to the participant, within your research goal. |
| Analysis | Done by hand, or a word cloud. | Grouped into themes automatically, with sentiment. |
What the research says
There's now academic research on this, and the picture is more nuanced than many vendors' marketing suggests.
- Xiao and colleagues (2020) had around 600 people complete a survey: half through a standard online questionnaire, half through a chatbot that asked open questions, interpreted the answers and followed up where needed. The chatbot produced markedly higher engagement, and the more than 5,200 free-text answers were better: more informative, relevant, specific and clear.
- Kim, Lee and Gweon (2019) compared a chatbot with a web survey, and a formal tone with a casual one. In the chatbot, people gave more differentiated answers and showed less satisficing: quickly settling on an answer that seems good enough. A casual tone helped, but only in the chatbot.
- Zarouali and colleagues (2024), on the other hand, found no evidence in a two-week study that a chatbot is a better tool than an online questionnaire. The questionnaire did better on several measures, and participants rated the chatbot less favourably.
Our takeaway: the chat window itself isn't where the gain is. The gain is in following up on open answers. A chatbot that asks the same closed questions in a different format adds little. An interviewer that listens and probes does.
When a conversation works better
An AI interview is the better choice when:
- you want to understand why, not just how many. A score tells you where you stand; the conversation tells you what to change.
- you don't know in advance what answers to expect. A questionnaire measures what you thought of; a conversation also finds what you didn't.
- you want to hear from a lot of people. Twenty in-depth interviews are doable. Four hundred aren't, unless the interviewer never tires.
- you want a shorter survey. One score question and a few minutes of conversation often replace a long list of statements.
When a questionnaire works better
A conventional questionnaire is still the better choice when:
- you need precise numbers on many fixed topics, for example to compare twenty statements per department with last year.
- the answer options are fixed, as in a vote or a choice between two options.
- it needs to be a single click, such as a thumbs up or down after a help article.
A combination is often strongest: a fixed score question for the trend, and a conversation for the why.
What to watch out for
Be open that it's an AI. The EU AI Act (Article 50) requires providers to make sure people know they are interacting with an AI system, unless that's already obvious. Law aside, participants who find out later feel misled.
Watch for leading follow-ups. An interviewer, human or AI, can nudge someone in a particular direction. A good AI interviewer follows up neutrally ("what do you mean by that?") and doesn't suggest answers.
Check the conclusions. An automatically written summary is a starting point, not a final verdict. When a conclusion surprises you, always read the quotes behind it.
Know where the conversations go. Conversations are processed by a language model. Ask your provider which model, where it runs, who sees the data and whether it's used to train models.
Keep it short. A few minutes on one or two topics yields more than a quarter of an hour on everything.
How it works in ScoreStory. After the score question (NPS, eNPS, CSAT or CES), Vox, the AI interviewer, holds the conversation. It briefly sums up what it heard and asks for the reason and for concrete examples, in a tone you choose: warm, direct, empathetic or analytical. Participants type or speak their answers, in any of eight languages; spoken answers are transcribed and the recording itself is never stored. The invitation page and the score question both say that the interviewer is an AI. Answers are linked to themes automatically and labelled positive, neutral, negative or mixed, and a short written conclusion tells you what stands out. Which parties process the conversations is set out in our privacy statement. More on the product page and in why ScoreStory.
Frequently asked questions
Do participants know they're talking to an AI?
They should. The EU AI Act requires providers to make sure people know they are interacting with an AI system, unless that's already obvious. In ScoreStory it's stated on the invitation page and alongside the score question.
How long does an AI interview take?
Usually a few minutes. Longer is rarely better: the aim is to really understand one or two topics, not to cover everything.
Can an AI interview be anonymous?
Yes, if the system is built for it: no names stored with the conversations, no link to the invitation and no results for groups that are too small. What anonymity really takes is covered in our guide to anonymous employee surveys and the GDPR.
Does an AI interview replace a real researcher?
Not entirely. It replaces the part a researcher can't scale: holding and sorting hundreds of short conversations. Interpreting the results, deciding what to do and checking whether a conclusion holds remain human work.
Sources
- Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions, Ziang Xiao et al., ACM Transactions on Computer-Human Interaction 27(3), 2020
- Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality, Soomin Kim, Joonhwan Lee and Gahgene Gweon, CHI 2019
- Comparing Chatbots and Online Surveys for (Longitudinal) Data Collection, Brahim Zarouali et al., Communication Methods and Measures 18(1), 2024
- Regulation (EU) 2024/1689 (AI Act), Article 50, EUR-Lex
Measure the score, then ask why
ScoreStory asks for one score and then holds a short conversation about the reasons behind it. Your first 10 interviews are free, with no credit card and no end date.