AI Changed How I Interview Customers, but Not Why

I promised to share which of my original Running Lean interviewing rules AI would break first. I’ve pushed it back twice since, so this whole post is that answer.

This is the fourth post in Running Lean: AI Edition. Same principles, new tactics, re-tested one at a time on a live product.

The short version is that my interviewing rules come in two kinds. The strategies are evergreen and hold no matter what tools you use, but the tactics keep changing. AI is the biggest change to the tactics yet. None of the strategies broke, but the first tactic to go was my evenings of tagging patterns by hand. AI also gave me one new rule I didn’t have before.

My rules didn’t jump straight from 2011 to AI, though, so here’s the short history first.

Fifteen years of interviewing rules

In 2011, I wrote up what I’d learned about talking to customers in a post called The Achilles Heel of Customer Development. It had five rules:

1. Build a frame around learning, not pitching.
2. Create a script.
3. Start with people you know.
4. Take someone along with you.
5. Record your learning right after every interview, and debrief later.

The 2012 edition of Running Lean added one more that I still stand by: “Don’t ask customers what they want. Measure what they do.”

Back then, the interviews were framed like a science experiment. You wrote your best guess at the customer and their problem on a Lean Canvas. Then you stated your top problems to the customer, asked them to rank them, and looked for evidence that validated or invalidated your guess.

In February 2015, I co-hosted a joint Running Lean and Switch workshop in Austin with Bob Moesta, Chris Spiek and Ervin Fowlkes. That workshop started to change how I interview. Bob’s jobs-to-be-done work starts from the idea that people hire products to get a job done. One of my biggest takeaways was that when customers hire your product, they fire something else.

So over the next couple of years, the goal of the interview shifted from testing my guess at their problem to uncovering their switch story. That’s what triggered them to look for something better, what they were trying to achieve, and what they fired along the way. In 2017, I published an updated problem interview script built around triggers and desired outcomes. I also combined the two models I’d learned from Bob and his team into the Customer Forces Canvas.

By the time I rewrote Running Lean for its third edition in 2022, I recommended not even bringing up the word “problem” in these interviews. The objective isn’t to validate problems, but to discover them.

Then AI happened. Since then, founders have analyzed over 2,000 interviews in LEANSpark across 12 languages, and I’ve been running my own interviews through it, too.

Fifteen years of interviewing rules, one stop at a time.

Looking back across all of it, the rules sort into the strategies I settled on and kept, and the tactics that keep changing with the tools.

The why holds. The how keeps changing.

The strategies held

These are the rules about why you interview and what you’re listening for. Once I settled on each one, it held, because the conversation is still yours to have.

1. Learn, don’t pitch. The moment you start pitching, customers stop telling you about their world and start reacting to your idea.
2. Uncover the switch story. This is the strategy that came last, from my work with Bob in 2015, and it has held ever since. You’re listening for what triggered them, what they were trying to achieve, and what they fired.
3. Facts and past behavior over opinions. What someone did last month is evidence, but what they say they’d do next month is a guess, however sincere.
4. Talk to real people. Start with people you know and work outward. AI doesn’t find people for you to talk to, which is still where most founders get stuck.

The Customer Forces Canvas from 2017. The switch story it captures is still what I listen for.

What AI does change is that your bad habits become visible. LEANSpark reads every interview for how you ran it, not just what the customer said. It caught me more than once. On my own calls in September, it flagged the moment I stopped asking questions and started showing what we were building. It even quoted me saying, “so this is an example of what the admin view will look like,” which is a pitch by another name.

From one of my own calls. The flag quotes me, and suggests what to ask instead.

I wrote this rule and teach it, but I still slipped. You can’t hear your own pitch while you’re giving it.

The tactics keep changing

The tactics are the how, which covers how you script, record, debrief, find patterns and build stories, and who you take along. These have changed many times, and most of the changes came from me, long before AI.

1. Scripts. In 2017, I removed the problem-ranking step, since customers don’t often understand their own problems well enough to rank them. By 2022, my script had become a meta-script, because the best script for discovery is no script.
2. Recording. In 2012, I told founders to avoid recording, because it made some people self-aware and I never went back to listen anyway. By 2022, I recommended recording whenever the prospect is okay with it.
3. Debriefing and stories. In 2011, it was a short form after every interview. By 2022, it was 5 to 10 minutes after each one to turn your notes into a Customer Forces Canvas. Then you looked for the stories that keep repeating.
4. Who you take along. In 2011, I took my wife along to keep the learning objective, but most founders interview alone. Now LEANSpark’s read of how you ran each interview does part of that job.

My Problem Interview Script before and after the 2017 rewrite. The tactics were changing long before AI.

AI is the biggest tactics change yet. When I was testing CloudFire, it took me six weeks to interview 30 moms about how they shared photos and videos of their kids. After every interview, I spent a few minutes filling out a short form while it was still fresh. Then I debriefed with my wife, who had come along. And then came the evenings, which I spent tagging patterns across all 30 interviews by hand.

That was the part of the process that took the most discipline, and it’s also the part most founders skip, which is understandable because it’s tedious work. Even with the Customer Forces Canvas, turning every interview into a story took real effort. So the interviews happened, but the notes sat in a doc nobody re-read.

Now, you upload the recording or the transcript, and the write-up is ready within minutes. It includes the customer’s story of what pushed them, what pulled them, and what held them back. The quotes are pulled word-for-word from the conversation, so you can always click through to what the customer actually said. And as the interviews pile up, the AI does the first pass at grouping people who tell the same story. That used to be my evenings of tagging by hand.

On a test account. Each finding carries the customer’s own words, and the time stamp opens the transcript at that moment.

I still review the patterns myself, because that’s where the judgment is, but I no longer spend my evenings finding them.

AI changed how I interview customers, but not why.

One new rule

Running my own interviews through LEANSpark also taught me a rule I didn’t have before.

If you can’t tell who said it, it isn’t evidence.

Many interviews arrive as pasted notes or transcripts without speaker names. In a back-and-forth conversation, it’s surprisingly easy to lose track of who said what. Your own words can end up credited to the customer. And your own words are the one thing that can never count as evidence for your idea.

In other words, it’s the facts-over-opinions strategy again, but now the tools make it easy to break without noticing.

So now, when LEANSpark can’t tell who’s speaking, it asks you instead of guessing. I recommend recording with speaker labels where you can, or simply telling the AI who’s who.

On a test account: when the speakers aren’t clear, LEANSpark asks which one is you before grading anything.

What this means for you

1. Keep the strategies. Learning instead of pitching, the switch story, and facts over opinions matter as much as ever. If I still slip on them, you probably do too.
2. Update your tactics. Record (with permission), make sure you know who’s speaking, and let the AI do the write-up, the story, and the first pass at patterns.
3. Spend the time you saved on the next conversation. Start with the first one this week.

What’s next

Every interview I’ve described so far is a deliberate one. You pick the belief, the people and the questions, then run the study over a couple of weeks. That works well, but it can only test what you already suspect.

The other thing I’ve learned over these years is that some of my best insights came from conversations I never planned. Around 2015, a fisherman in Chile signed up for Lean Canvas at a time when almost everyone signing up was a tech founder, and one call with him changed how I saw our customers.

In the next post, I’ll tell that story and show why continuous discovery, from every signup call, support thread and chance conversation, belongs right next to your deliberate interviews.