CASE STUDY . 02 . DESIGNING WITH AI: THEN AND NOW . CONCEPT STUDY
Same bot. New engine. The design problem flipped.
In 2022, the hard part was teaching the bot what to say. In 2026, the hard part is teaching it what not to.
ROLE
Concept study, building on my AI design work at Reece
COMPANY
Reece USA
YEAR
2022 reimagined for the LLM era
FOCUS
Conversational design. AI guardrails. Trust.
WHAT FLIPPED WITH LLMs
Then, the bot returned results. Now, it can hold a conversation about the job.
My 2022 customer chat could find "centerset bathroom faucet." It could not answer the question underneath: "I'm replacing a faucet on a small vanity with three holes, four inches apart, what fits?" An LLM can. It can ask the clarifying question, narrow the options, and explain the tradeoffs like a good counter person would.
So the design job flipped. From matching words to catalog items, to deciding what the assistant should handle, what it must never guess, and when a counter person takes over.
Then
2022, before the LLM era
Three AI features. Predictable, but limited.
At Reece we designed three AI powered experiences. Each solved a real job, and each ran into the same limitation.
01. CUSTOMER
Reece Chat
Searched products in natural words inside the mobile app, with guided choices upfront: Branch hours, find a part, search a product.
Customers saw what the bot could do before typing something it couldn't handle.
02. CUSTOMER
Reece Vision
Photograph a part, find the same part, similar options, or a replacement.
It solved the question a contractor can't type: what is this, and what replaces it?
03. EMPLOYEE
Employee support
A chatbot on ServiceNow serving 1,500 employees across HR and IT.
Internal facing, high volume, held to the same trust bar as the customer tools.
2022. GUIDED ENTRY: THE BOT'S LIMITS, UPFRONT
2022. CONVERSATIONAL SEARCH, WITH AN ESCAPE TO THE CATALOG
All three taught the same lesson.
The hardest design problem was never the happy path. It was the moment the system reached its limit: the search that didn't match, the photo the model wasn't sure about, the question outside the bot's intents.
Designing those moments well was the difference between AI people trust and AI people avoid.
Now
2026, the LLM era . a concept redesign
The redesign: three zones of trust.
Where AI advises freely. Where it must use live data. Where a human takes over.
ZONE 1
Open Conversation
the AI advises freely
The LLM works like a knowledgeable counter person.
ZONE 2
Exact data only
never guess, always pull live data
The assistant must never guess these from memory. It should pull live answers from the catalog and inventory systems and show them exactly.
ZONE 3
Handoff
a human takes over
A warm handoff to the branch with the full conversation attached. In 2022, this was the "View all results" link.
NOW. THE GUARDRAILS . CONCEPT . the logic I would write, not a shipped prompt
The zones are a design idea until something enforces them.
In a real build, the system prompt is what holds the line. It is where "teaching it what not to say" actually lives. Here is the logic I would write for this assistant.
# the assistant's standing rules
ADVISE freely on the job - understand, clarify, compare, explain what's installable
NEVER guess price, stock, specs, or compatibility - pull live and show exactly, or say you are checking
HAND OFF on big orders, returns, or a frustrated customer - pass the full conversation to the branch
# if unsure whether a number is current, do not state it. fetch it, or route.
The hardest rule is the second one. A fluent wrong price is worse than no price, because it reads as certain. Grounding is not a technical detail here. It is trust.
NOW. MEASURING QUALITY
A bot that answers everything is easy. A bot you can trust is the work.
Four signals I would design in from day one.
SIGNAL 01
Containment
How often the assistant resolves a question without a human. The measure of real usefulness, not just activity.
SIGNAL 03
Handoff quality
Whether the human picksup with the context intact, so the customer never has to start over.
SIGNAL 02
Correction rate
How often customers correct or rephrase. A rising rate is a sign the answer keeps missing the real question.
SIGNAL 04
Purchase confidence
Whether conversations end in a confident purchase or an abandoned one. The outcome the whole thing is for.
HOW DESIGN SHOULD BE LED NOW
What the flip asks of the people leading the work.
Reflecting on the flip, here is how I think design needs to be led now, and what I would tell a designer standing where I stood in 2022.
01
Design the loop, not the headcount.
The employee bot worked because of a loop, not a bigger team. The bot took the routine, designers took the moments it could not, and what people kept re-asking shaped the next build. The job now is to design that loop.
02
Set the rules upfront.
Reece Chat's guided choices shaped every conversation that followed. The leverage is in the few rules the whole system follows, not in reviewing every output.
03
Read the rewrites.
Reece Vision's value was watching where the model was unsure. Now the signal is how much a team rewrites the AI. A high rewrite means the brief is wrong, not the people.
04
Design for the second step.
The hard part was always the handoff at the limit, never the answer. Lead toward what happens downstream, not just the answer in hand.
05
Name what stays human.
The bot cleared the routine so people could spend judgement where it counts. Designers should be told plainly what is theirs to protect. The research, the trust calls, the relationships.
THE REFLECTION
When I look back at 2022 and forward to now, the strange part is how little the principles moved. Know the user's real question. Design for the failure, not the success. Earn trust one conversation at a time. The LLM did not rewrite those rules. It only changed which one is hardest. Conversational design from before the LLM era is not outdated. It is the foundation. The flip did not replace the craft. It raised the stakes on getting it right.