AI Automation: From Manual to Automated
AI Automation: From Manual to Automated
AI Automation: From Manual to Automated
Designing AI-powered workflows for enterprise conversational AI
Designing AI-powered workflows for enterprise conversational AI
Designing AI-powered workflows for enterprise conversational AI
Company
Company
Microsoft
(Nuance Communications)
Microsoft
(Nuance Communications)
My Role
My Role
Principal UX Designer,
Product Owner
Principal UX Designer,
Product Owner
Collaborators
Collaborators
Product Management,
Engineering, Research
Product Management,
Engineering, Research
Timeframe
Timeframe
6 months,
Presented at iCubed Conference
Ahead of the Curve
Ahead of the Curve
Ahead of the Curve
Designed and shipped AI automation for enterprise conversational AI prior to the official release of Microsoft Copilot.
Designed for Humans
Designed for Humans
Designed for Humans
Designed AI automation with a deliberate human-in-the-loop equation to balance speed with accuracy.
+
+
+
About
•
•
Nuance Mix is a platform for building conversational AI experiences used by Fortune 500 and Global 2000 companies.
Nuance Mix is a platform for building conversational AI experiences used by Fortune 500 and Global 2000 companies.
Nuance Mix is a platform for building conversational AI experiences used by Fortune 500 and Global 2000 companies.
•
•
AI-powered automation replaced laborious, manual, and unsustainable processes for building and managing conversational AI models at scale.
AI-powered automation replaced laborious, manual, and unsustainable processes for building and managing conversational AI models at scale.
My Role
My Role
My Role
•
•
Sole designer and product owner on this work.
Sole designer and product owner on this work.
Sole designer and product owner on this work.
•
•
Partnered with PM and Engineering to understand AI feasibility and translate capability into design.
Partnered with PM and Engineering to understand AI feasibility and translate capability into design.
Partnered with PM and Engineering to understand AI feasibility and translate capability into design.
•
•
Incorporated human-in-the-loop review requirements informed by UX research.
Incorporated human-in-the-loop review requirements informed by UX research.
Incorporated human-in-the-loop review requirements informed by UX research.
•
•
Designed and shipped AI-powered automation features for automatic data classification, automatic data labeling, a combined single-step workflow for both, and AI-powered generated data.
Designed and shipped AI-powered automation features for automatic data classification, automatic data labeling, a combined single-step workflow for both, and AI-powered generated data.
Designed and shipped AI-powered automation features for automatic data classification, automatic data labeling, a combined single-step workflow for both, and AI-powered generated data.
•
•
Presented the shipped work at the Nuance iCubed Conference.
Presented the shipped work at the Nuance iCubed Conference.
Presented the shipped work at the Nuance iCubed Conference.
THE CHALLENGE
The Manual Cost of Building AI Models
The Manual Cost of Building AI Models
There is an irony at the heart of this work: the people building AI-powered products were doing it entirely by hand.
There is an irony at the heart of this work: the people building AI-powered products were doing it entirely by hand.
•
•
Conversational AI models require large volumes of accurately classified and annotated data to perform. Previously, all of it was built manually.
Conversational AI models require large volumes of accurately classified and annotated data to perform. Previously, all of it was built manually.
•
•
Every piece of data and each data point had to be classified one at a time through multiple clicks. At enterprise scale, this was slow, repetitive, and unsustainable for teams that needed to move fast and ship accurate models.
Every piece of data and each data point had to be classified one at a time through multiple clicks. At enterprise scale, this was slow, repetitive, and unsustainable for teams that needed to move fast and ship accurate models.
Principles That Guided the Design
New. Exciting. Powerful.
AI Automation.
Principles That Guided the Design
Get users to the feature first.
Get users to the feature first.
Get users to the feature first.
•
•
AI automation was the most significant capability added to the platform since its launch.
AI automation was the most significant capability added to the platform since its launch.
•
•
For non-technical users, the design prioritized reaching that capability immediately: approval was still required, but woven in, not a gate.
For non-technical users, the design prioritized reaching that capability immediately: approval was still required, but woven in, not a gate.
Expert users won't trust what they can't verify.
Expert users won't trust what they can't verify.
Expert users won't trust what they can't verify.
•
•
Research told us experienced users would not act on AI output without reviewing it first.
Research told us experienced users would not act on AI output without reviewing it first.
•
•
Their experience was designed as an explicit review board, making the approval step visible and central.
Their experience was designed as an explicit review board, making the approval step visible and central.
New. Exciting. Powerful. AI Automation.
New. Exciting. Powerful.
AI Automation.
New. Exciting. Powerful. AI Automation.
Product and engineering brought AI automation capability to the table. The question wasn't whether to build it. It was where it belonged.
Product and engineering brought AI automation capability to the table. The question wasn't whether to build it. It was where it belonged.
Product and engineering brought AI automation capability to the table. The question wasn't whether to build it. It was where it belonged.
The Challenge
The Challenge
The Challenge
•
•
How do you eliminate thousands of manual clicks without sacrificing the accuracy enterprise models demand?
How do you eliminate thousands of manual clicks without sacrificing the accuracy enterprise models demand?
•
•
And how do you bring that capability to both expert and non-technical users without compromising either experience?
And how do you bring that capability to both expert and non-technical users without compromising either experience?
The Solution
The Solution
The Solution
•
•
AI-powered automation to classify and annotate data in bulk, with human review before anything was committed to the model.
AI-powered automation to classify and annotate data in bulk, with human review before anything was committed to the model.
•
•
AI suggested data classification was also introduced for non-technical users, keeping the beginner experience intact without exposing the full expert automation suite.
AI suggested data classification was also introduced for non-technical users, keeping the beginner experience intact without exposing the full expert automation suite.
Non-technical Users
Develop Tab Design
Develop Tab Design
Auto-Annotation
Auto-Annotation
Expert Users
Optimize Tab Design
Optimize Tab Design
Auto-Intent and Auto-Annotation
Auto-Intent and Auto-Annotation
Design System Scalability
Design System Scalability
Design System Scalability
New color tokens added to design system to distinguish AI-suggested from user-assigned data classification. Built to scale if both states ever needed to coexist.
New color tokens added to design system to distinguish AI-suggested from user-assigned data classification. Built to scale if both states ever needed to coexist.
•
•
Blue = User-assigned
Blue = User-assigned
Blue = User-assigned
•
•
Orange = AI-suggested.
Orange = AI-suggested.
Orange = AI-suggested.
Fast. Scalable. AI-Generated Data.
New. Exciting. Powerful.
AI Automation.
Fast. Scalable. AI-Generated Data.
This work was built in partnership with Microsoft Copilot.
This work was built in partnership with Microsoft Copilot.
This work was built in partnership with Microsoft Copilot.
The Challenge
The Challenge
The Challenge
•
•
How do you generate the large volumes of training data AI models require without relying on manual workarounds outside the platform?
How do you generate the large volumes of training data AI models require without relying on manual workarounds outside the platform?
•
•
And how do you bring that capability to both expert and non-technical users without compromising either experience?
And how do you bring that capability to both expert and non-technical users without compromising either experience?
The Solution
The Solution
The Solution
•
•
A dedicated beginner experience, already designed and in place, provided the foundation for AI-generated sample sentences to land.
A dedicated beginner experience, already designed and in place, provided the foundation for AI-generated sample sentences to land.
•
•
A dedicated expert experience, in a newer place, which provided the scalable foundation for AI automation to land.
A dedicated expert experience, in a newer place, which provided the scalable foundation for AI automation to land.
Non-technical Users
Develop Tab Design
Develop Tab Design
Expert Users
Optimize Tab Design
Optimize Tab Design
Data informed: Human-in-the-Loop
Data informed: Human-in-the-Loop
Data informed: Human-in-the-Loop
•
•
Human-in-the-loop review was a data-informed design decision from proactive user research.
Human-in-the-loop review was a data-informed design decision from proactive user research.
•
•
Users must approve AI-suggested data and classifications before anything is committed to the model.
Users must approve AI-suggested data and classifications before anything is committed to the model.
Nuance iCubed Conference
Nuance iCubed Conference
Nuance iCubed Conference
The work was presented at Nuance's internal R&D conference in under the title Mix NLU #beastmode. A nickname given by corporate vice president, David Ardman that followed the work from internal meetings to the conference stage. It was the most highly attended session at the event.
The work was presented at Nuance's internal R&D conference in under the title Mix NLU #beastmode. A nickname given by corporate vice president, David Ardman that followed the work from internal meetings to the conference stage. It was the most highly attended session at the event.
Impact
Impact
Impact
Ahead of the Curve
Ahead of the Curve
Ahead of the Curve
•
•
Designed and shipped AI automation for enterprise conversational AI prior to the official release of Microsoft Copilot.
Designed and shipped AI automation for enterprise conversational AI prior to the official release of Microsoft Copilot.
•
•
Built in partnership with Microsoft Copilot before it was available to the world.
Built in partnership with Microsoft Copilot before it was available to the world.
Designed for Humans
Designed for Humans
Designed for Humans
•
•
Designed AI automation with a deliberate human-in-the-loop equation to balance speed with accuracy.
Designed AI automation with a deliberate human-in-the-loop equation to balance speed with accuracy.
•
•
Human-in-the-loop review was a data-informed design decision driven by proactive user research.
Human-in-the-loop review was a data-informed design decision driven by proactive user research.
Built for Enterprise
Built for Enterprise
Millions of Clicks
to Bulk Approval
•
•
Classifying and annotating training data required thousands of manual clicks per model. AI automation replaced that entirely.
Classifying and annotating training data required thousands of manual clicks per model. AI automation replaced that entirely.
•
•
AI-generated sample sentences allowed Fortune 500 and Global 2000 companies to build and scale conversational AI models faster.
AI-generated sample sentences allowed Fortune 500 and Global 2000 companies to build and scale conversational AI models faster.