Thinking before building: framing an AI problem

A support team at a mid-sized software company spent three months building an AI chatbot. They launched it, and within two weeks customers were furious. The bot gave vague answers, looped in circles, and made people angrier than before. When the team finally looked at the data, they found something embarrassing: 80% of incoming questions were the same 15 questions, all answered in their existing help docs. Customers just couldn't *find* those docs.

They didn't need a chatbot. They needed better search.

This is the most common way AI projects fail. The technology isn't the culprit; the framing is. Someone says "let's add AI," and the team starts building before anyone asks "what problem are we actually solving?"

The trap: solution-first thinking

"Let's add a chatbot" is a solution. It is not a problem.

When you start with the solution, you skip the most important step: understanding what is broken and for whom. You end up building something impressive that nobody needed.

Solution-first sounds like this:

• "Let's use AI to summarize our reports."
• "We should build an AI agentAI agentSoftware that pursues a goal on its own: it plans steps, uses tools and takes actions with limited human input.View full definition → for sales."
• "Can we add a GPT to our website?"

Problem-first sounds like this:

• "Our managers spend 4 hours a week reading reports they skim anyway."
• "Sales reps waste 30 minutes per call hunting for product specs."
• "New visitors can't tell what our product does in under 10 seconds."

Notice the difference. The second list describes pain, who feels it, and how much. That gives you something to measure and solve. The first list is a tool looking for a job.

A simple framing method

Before you build anything, write down four things. This takes 20 minutes and saves months.

1. The Job

What is the specific task someone is trying to get done? Be concrete.

Bad: "Improve customer support."

Good: "Help a customer find the answer to a common question without waiting for a human."

2. who has the problem

Name the actual person. A frustrated customer at 11pm? An overworked support agent? A manager drowning in tickets? Each one points to a different solution.

3. how you'll know it worked

We cover how to define a measurable success criterion and audit your data in detail in the lesson "Scoping: Data and Success Criteria."

4. the simplest thing that could work

This is the big one. Force yourself to ask: **what is the *least* amount of technology that solves the problem**? Often the answer involves no AI at all, or a tiny slice of it.

For the support team, the simplest fix was a smart search bar over their existing docs. Cheaper, faster, and it actually worked.

Walking through the support example

Let's redo the failed chatbot project the right way.

The Job: A customer wants the answer to a routine question (password reset, billing date, export limits) right now.

Who: Customers, often outside business hours, who would rather self-serve than wait.

Success metric: 50% of common questions resolved without a human, measured by ticket volume dropping.

Simplest thing that could work: A search box that understands plain-language questions and returns the right help article. Not a conversation. A search.

This is where a technique called semantic searchsemantic searchSearch that matches on meaning and intent rather than exact keywords, so a query and a result can connect even when they share no words.View full definition → fits. Semantic search means matching by *meaning*, not just keywords. A customer types "I can't log in" and it finds the article titled "Resetting your password," even though none of those words match. Traditional keyword search would miss it.

You can build a basic version of this with an embedding model (a tool that turns text into a list of numbers representing its meaning, so similar meanings sit close together). Here is a minimal example using OpenAI's APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → in Python:

from openai import OpenAI
client = OpenAI()

# Your existing help articles
docs = [
    "Resetting your password: click Forgot Password on the login screen.",
    "Your billing date is the day you first subscribed.",
    "You can export up to 10,000 rows on the free plan.",
]

def embed(text):
    return client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    ).data[0].embedding

# A customer's question
question = "I can't log in to my account"
q_vec = embed(question)
doc_vecs = [embed(d) for d in docs]

# Find the closest article by meaning
import numpy as np
scores = [np.dot(q_vec, d) for d in doc_vecs]
best = docs[int(np.argmax(scores))]
print("Best match:", best)

This returns the password article, even though the customer never said "password." That is the whole problem solved, with no chatbot, no conversation flow, and far less risk of the AI inventing wrong answers.

For a clear, free walkthrough of how embeddings and semantic search work, see OpenAI's embeddings guide.

When you actually do need the chatbot

Problem-first thinking does not mean "never build AI." It means build the right thing.

If your analysis showed that most questions were *unique*, involved back-and-forth ("my export failed, here's my error, what do I do?"), and required reasoning across multiple documents, then a conversational AI assistant earns its place.

The test is simple. Ask: does the problem require a conversation, or just an answer?

• Just an answer: search, a summary, a single AI call.
• A conversation: a chatbot or assistant.

Most teams assume "conversation" by default. Most problems only need "answer."

A quick decision checklist

Run any AI idea through these questions before building:

1. Can I state the problem without naming a technology? If you can only describe it as "an AI thing," you don't understand it yet.
2. Who feels the pain, and how often? Rare pain rarely justifies a project.
3. What's the dumbest solution that might work? Try that first. A spreadsheet, a better search bar, a saved prompt template.
4. What does success look like as a number? If you can't measure it, you can't improve it.
5. What happens when the AI gets it wrong? A wrong FAQ answer is annoying. A wrong medical or legal answer is dangerous. Higher stakes mean simpler, more controlled designs.

Sizing the solution to the problem

Once you know the problem, match the tool to it. Going overboard wastes money and adds failure points. Going too small leaves the pain unsolved. Here is a rough ladder, from simplest to most complex:

Level 0: No AI

A better-organized FAQ page. A search filter. A template. Sometimes the real problem is that information is hidden, not that it's missing. Always check this first.

Level 1: A single AI call

One prompt, one answer. "Summarize this document." "Classify this email as urgent or not." No memory, no conversation. Cheap, predictable, easy to test.

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Classify the support email as 'billing', 'technical', or 'other'. Reply with one word."},
        {"role": "user", "content": "My card was charged twice this month."}
    ]
)
print(response.choices[0].message.content)  # billing

This one snippet could automatically route every incoming email to the right team. That alone might solve the support team's real bottleneck.

Level 2: AI plus your own dataown dataData collected directly from your own customers and prospects through your own channels: your most reliable and privacy-compliant source.View full definition →

This is the semantic search example from earlier, often called RAGRAGA method that lets an AI model answer using your own documents, retrieving relevant passages before generating a response instead of relying only on training data.View full definition → (Retrieval-Augmented Generation: the AI looks up relevant info from your documents, then answers using it). Good when answers must come from *your* content, not the model's general knowledge.

Level 3: A conversational assistant

Multi-turn chat, memory of the conversation, maybe the ability to take actions. The most powerful and the most expensive to build and maintain. Reserve it for problems that need dialogue.

The lesson: start at the lowest level that solves the problem. You can always climb up. Climbing down (ripping out an over-built chatbot) is painful and public.

Why this saves you

Framing first feels slow. It is actually the fastest path. The support team lost three months building the wrong thing. The right framing would have pointed them at a search box they could have shipped in two weeks.

Every hour spent on the four framing questions (the job, who, the metric, the simplest fix) saves you weeks of building something nobody asked for.

Key Takeaways

• Start with the problem, never the tool. If you can only describe your idea as "let's add AI," you haven't framed it yet. State the pain, who feels it, and how often.
• Define one measurable success metric before building. "Cut 'how do I' tickets by 40%" beats "improve support" because you can actually check it.
• Always try the dumbest solution first. Better search, a saved prompt, a routing rule. Many "AI projects" are solved at Level 0 or Level 1 without a chatbot in sight.
• Match complexity to the problem. Need an answer? Use search or a single AI call. Need a conversation? Then, and only then, build an assistant.
• Ask what happens when the AI is wrong. Higher stakes demand simpler, more controlled designs, not flashier ones.