Designing graceful failure in an LLM-powered voice ordering system
At Square, I worked on an LLM-powered voice agent that allowed customers to place orders over the phone. A key part of the experience involved sending a payment link via SMS to complete the transaction.
However, a significant number of customers were calling from landlines—creating a hard system constraint: landlines cannot receive text messages, making order completion impossible within the designed flow.
This is how I designed a graceful failure state that preserved user trust, avoided reputational risk for sellers, and maintained conversational continuity.
The problem
The voice agent depended on SMS to complete transactions, but:
Landline callers could not receive payment links
The system had no reliable fallback for completing orders
Customers were unaware of this limitation at the time they called
This created a high-risk scenario:
Failed transactions
Frustrated customers
Potential erosion of trust in sellers
The challenge
How might we handle a hard system limitation without breaking the conversation or damaging customer trust?
We needed to:
Clearly communicate the limitation
Avoid blaming the seller or suggesting unsupported alternatives
Prevent the interaction from becoming a dead end
Keep the experience coherent and conversational
Key constraints
No SMS fallback for landline users
Cannot assume sellers support alternate ordering methods
Must avoid assigning blame to the seller
Must maintain conversational flow
Must avoid dead-end interactions that lead to user drop-off
My role
As the lead content designer on this workstream, I partnered with product design and product management to define:
When and how the agent should communicate system limitations
How to structure messaging within a dynamic, LLM-driven interaction
How the agent should behave after delivering a failure state
My approach
1. Timing the intervention
We determined that the agent should only communicate the limitation after clear order intent was expressed.
Users often call businesses for reasons unrelated to ordering (e.g., hours, location). Surfacing the limitation too early would introduce unnecessary friction and confusion.
This decision ensured the system remained context-aware, rather than prematurely restrictive.
2. Framing the limitation
I drafted a concise, neutral message to communicate the constraint:
“We’re unable to take your order because it looks like you’re calling from a phone number that can’t receive text messages.”
Key considerations:
Be direct without over-explaining
Avoid implying user error
Avoid assigning responsibility to the seller
Maintain a calm, matter-of-fact tone
3. Designing for recovery, not dead ends
Instead of ending the interaction or instructing the customer to call back from a mobile device, I designed the agent to immediately continue the conversation.
After communicating the limitation, the agent would:
Ask a contextually relevant follow-up question
Tailor the question based on the user’s expressed intent
For example:
Customer: “Can I order a large pepperoni pizza?”
Agent: “We’re unable to complete orders from this number because it appears to be a landline.”
Agent: “Would you like to hear more about our menu or location?”
This approach:
Preserved engagement
Reduced user frustration
Allowed the system to remain helpful despite constraints
4. Establishing guardrails
I explicitly avoided:
Suggesting alternative ordering methods we couldn’t guarantee
Introducing steps that would shift burden onto the customer
Creating ambiguity about what the system could or couldn’t do
This ensured:
Clear expectations
Protection of seller reputation
Consistency across interactions
The outcome
While this work focused on a specific edge case, it established a broader pattern for handling constraints in AI-driven interactions:
System limitations should be context-aware and selectively surfaced
“Failure states” should still provide forward momentum
AI agents should prioritize engagement and clarity over rigid task completion
What I learned
Designing for AI systems requires thinking beyond static messaging and into behavior over time
The timing of a message can be as important as its content
Graceful failure is a critical part of building trust in AI-powered experiences
Even when a task cannot be completed, the system should remain useful and responsive