Speech-to-Order System
"Alternative communication methods for the deaf and hard-of-hearing in customer service interactions"
Problem Space
The current coffee ordering experience can be stressful because customers find it difficult to express themselves clearly to the barista and the barista feels frustrated that he or she cannot effectively communicate back to the customers. For the purposes of this project, we will be designing Odina for Starbucks. However, the application for this system can be implemented in various service industries.
Design Solution
A solution that can bring baristas closer to DHOH customers without creating addition technological and communicative barrier or distance.
Challenges
The design challenges for this project are to see through the lens of our target audience, to interview DHOH participants, and to design a system that is consistent with Starbuck’s principles.
Role
Lead UI / UX Designer
Usability Testing
Duration
3 months
Tools
Sketch
Principle
Invision
Team
Dawn Ferguson
Jeff Matarrese
Susanne Kirchner
Kent Tsai
1 | Design Question
“How might we help DHOH (deaf and hard of hearing) customers communicate more effectively and efficiently in customer service interactions? ”
2 | User Research
In oder to understand how customers felt about the current ordering process, we interviewed multiple participants with the DHOH participants being our primary audience, coffee shop baristas, and industry experts.
Research Methods
Contextual Inquiry
We recruited and conducted three observations with deaf/hard of hearing, watching them ordering and picking up their drinks to better understand the flow, pain points, and opportunities in context.
Surveys
We selected two groups to interview: Starbucks retail employees and subject matter experts. Complementing our observations, these two groups gave us a diverse sampling of attitudinal data from the cashier/employee viewpoint, as well as professional experience in the field of accessibility product management.
Interviews
We created a Qualtrics survey asking DHOH participants about their experiences at coffee shops, the methods they most often use to order, and the methods they would ideally use to order. We received 17 responses and our key findings are listed below. I proposed and led an expert interview with an industry accessibilities manager to conduct heuristic evaluation on current ordering system.
Key Findings
DHOH Customers
Feel embarrassed when barista cannot understand them.
Ordered the wrong drink and cannot confirm order detail.
Do not wish to stand out with their impairment or hold up the line.
Not sure if their drinks were called. Afraid to pick up someone else’s order.
Coffee Shop Barista
Worried that they cannot communicate with DHOH customers.
Wish they knew some basic ASL to communicate
Expert Interview
It is important to remove unnecessary physical barrier between barista and customers.
Starbucks is striving to be the 3rd Place for all customers.
Persona
3 | Ideation & Sketching
Affinity Diagram
In this exercise, the team brainstormed possible solutions and group similar ideas together to form affinity groups. Some of the popular ideas include: Speech to text software, simplified ASL training, speech to ASL, touchscreen countertops, etc.
Top 3 Sketch Solutions
A simple badge worn by the barista to indicate to the customers that they know ASL and a sticker label with customer’s name and drink detail
Dual screen POS - The customer can see a real time drink builder and a chat box that translates speech to text and a writing pad area for customers to write back
Real time animated drink builder that visually puts together the drink with every ingredient ordered so customer can know exactly what is in their drink or what they’ve ordered
*Final sketch illustrations drawn by Dawn.
ASL badge worn by barista
2 way POS display with speech to text feature
Real time order display with ingredients
4 | Prototyping & Usability Testing
Paper prototype
After coming up with our top 3 sketch prototypes, we took a step back and reevaluated the problem space and design question to see which idea solves what we are going after. We’ve decided that the most important problem that we are trying to solve is to remove barrier and to create a two-way communication channel, and reduce friction from the ordering experience. As a result, we decided to go with revamped POS as our design direction.
Our final paper prototype included:
Real time drink builder
Writing pad
Speech to text (from barista to customer)
Usability Testing
Usability Testing Findings:
DHOH users do not prefer to use the writing pad because it calls out their need for assistance
They are not sure how to go back to change drinks
DHOH users still prefer to have face-to-face interaction
Users do not wish to create a separate assistive machine just for them
5 | Design Solution
From our usability results, we made iterations to our prototype design and implemented new features to address user feedback from our research and usability testing. Then I was responsible for designing the overall user flow as well as led the UI design for our final solution using Sketch, Principle and Invision.
Our final solution is not aimed to replace human. Our goal is to create a complementary tool that takes away the current physical, communicative, and emotional barriers so the the barista can better attend to the DHOH customer person-to-person. We aim to simplify the current ordering process and remove the POS that current stands in between barista and the customer, which causes anxiety on both sides. With the Starbuck’s mission to make Starbuck’s the “3rd Place” we have reinvented the POS interaction and UI so the staff can focus on the people, not machines.
6 | Key Features
Side-by-side Ordering Experience
Changing the physical space of conventional ordering experience by removing POS in between customers and barista
A.I. Keyword Detection
Automatically listens and picks up menu-related keywords
Automatic Order Builder
Using A.I., the system generates customers order filling in detected information
Real-time Order Visualization
Customers can view order detail including images and price breakdown
7 | Future Steps
Designing order pick-up experience
Research and development into voice recognition software
In-store testing
Adding and testing language recognition software to help customers with language barriers
8 | Updates
Keeping up with the technology: On February 4th, 2019, Techcrunch released an article on Google’s new accessibility feature, Live Transcribe. This app transcribes speech to text in real-time and is available in 70 languages and dialects. Making this available on Android devices offers multiple use cases for the deaf and hard of hearing community (one of the use cases highlighted in the video is the coffee ordering experience!).
Words can’t describe how excited I am too see this feature being incorporated into more and more everyday technology. Speech to text was the core inspiration for our project. Our design solution aims to make speech-to-text as seamless and natural as possible during human interaction. I am looking forward to see how Live Transcribe will be used and how it will evolve in the future to help more people beyond the DHOH community.