The rise of the conversation design team
Conversation design teams have largely been a function of the professionalization of the conversational AI industry. Back in 2014, it was normal to see assistant teams composed entirely of developers. The ROI of conversational AI was still being proven, so vertical tools and small teams of developers were the extent of investment in the space. As the ROI of conversational AI increased, so did the investment in the category, and subsequent rise in professionalization of both the tools and talent to implement it. With greater adoption of conversational assistants across brands, the investment in not only the technology, but the user experience and its design, became increasingly important.
At its core, conversation design is the user experience (UX) design of automation.
It is no secret that automation cuts costs. Whether it’s customer support or sales, every company on Earth would automate as much as possible if they could. Why don’t they? Because the drop in quality of customer experience, and subsequently business, would nullify the benefits of automation cost savings. If this wasn’t the case, everything would already be automated! Conversation design is what allows automation to take place at greater scale by maintaining or even raising the customer experience whilst automating increasingly large chunks of the business.
The move to federated conversation design teams
As investment in conversational AI grew, brands began to shift their conversational AI approach from business-line-centric conversational AI solutions, to dedicated in-house conversational AI teams that own end-to-end assistant design across business lines.
Before this shift, departments like customer support, sales, and marketing independently introduced conversational experiences for their lines of business. Each team managed their own vendors, technology, brand persona, standard of experience, and most importantly, talent.
This approach made sense when conversational AI was still nascent and time to value, measured by time to market, was the most important factor. But, without in-house product teams capable of rapid iteration and continuous improvement, the ceiling of value that could be unlocked by conversational AI was capped.
Vertical vendors bought by business lines were expensive, with teams often needing professional services for enablement (which increased the total ownership cost). Without in-house talent, business lines were at the mercy of buying an expensive tool, only to require expensive implementation or consulting services from that same vendor to maintain their bot. This approach is still the standard approach for most large conversational AI companies. Sell a tool that’s hard to use to upper management, then charge the team actually responsible for implementation on their professional services because the tool is difficult to use.
So, companies began to move to a federated team approach where an in-house product team directly served the business units. This model lets brands:
- Iterate quickly and continuously with in-house talent.
- Generate material cost savings through streamlining their technology vendor stack across all business units and avoiding added professional service costs.
- Reduce redundancy of work through shared technology frameworks and components
- Create and maintain a consistent brand experience for customers.
This move to in-house product teams from business lines is still not only ongoing, but accelerating. Most mature CxD teams fall under the Design organization, working cross functionally with Engineering and Product. When CxD is smaller, it’s common to see these teams fall under Product.
The problem with the conversation designer title
The problem with the conversation designer title today is that it encompasses too much. As Jason Bejot puts it, “it’s an overloaded term”. Teams outside of CXD rarely understand exactly what conversation design is in the first place, and even if they do, it’s hard to understand its components outside the visually obvious “script writing and flowcharting”. By breaking down the conversation design role into its functional components, we can help define the role.
At its core, an assistant has three component parts:
- Dialog model (manager) is the conversation logic of the assistant, managing flow state
- NLU model provides the ability to understand what the user says
- Response model provides the content responses for the assistant
Conversation designers today could be expected to work on any one, or all three, of these components of the assistant, depending on the structure, process and expectations of their team. With smaller teams, conversation designers are often required to go as far as creating the requirements for the Assistant itself, bleeding into the responsibilities of a product owner. This leads to wildly varying definitions of “conversation designer” and what is actually expected of them.
Full stack conversation designers
This is where the term “full stack conversation designer” has begun to take off.
Amber Prause is a Full Stack Conversation Designer at Gordon Food Service and defines her role on Linkedin as being able to “execute along the entire design process, from strategy to final UI.” In other words, she can design, manage, and iterate on all three components of the assistant—from NLU to dialog design and copy. The necessity for a term like full stack conversation designer highlights the same necessity for roles working within conversation design’s component parts.
In software development, Full stack developers are those who can work frontend, backend, and infrastructure—all components necessary to ship an application end to end. Full stack in software gets its meaning from the combination of, and direct contrast to, its component parts—frontend, backend, and infrastructure engineers. But for conversation design, we just have conversation designer and full stack conversation designer. Clearly, we as an industry, know there are component roles within full stack CXD, as otherwise, “full stack” would be unnecessary. Yet, we haven’t actually defined these component roles across our industry in a standard way.
Breaking down conversation design
This realization led us to develop a framework to make sense of the varying component roles and responsibilities in the CxD space, thanks largely to active customer feedback.
The general framework splits conversation design into three sub-disciplines, where CxD is the overarching full stack skillset. Each component skill set maps to the main three components of the assistant:
- Dialog design - design and management of the conversational logic and flow
- NLU design - design and management of the NLU model
- Response design - design and management of response content including visuals
Breaking down conversation design into its component parts allows us to think about future specialized roles that work on each component within the broader conversation design team.
NLU design focuses on discovering, designing and maintaining the Assistant’s NLU model(s). A rough workflow for an NLU designer includes intent discovery through clustering, utterance generation, conflict mapping, model testing, and handoff to dialog design or development. On some teams, the NLU Designer might also act as the business analyst or product owner responsible for determining the functionality of the assistant, informed by their utterance clustering of live customer data.
Dialog design focuses on the creation and management of the conversation logic that drives the Assistant. The rough workflow of a dialog designer includes determining the primacy of intents if not already specified, writing sample scripts for these primary intents, creating happy path flows with error pathing, user testing the Assistant to determine gaps, and handing off to development or response design.
Response design focuses on the creation and management of the response content delivered by the Assistant, as well as the management of the Assistant persona. Unlike NLU design and dialog design where separate roles focusing on each component make sense given their distinct skill sets, response design is more situational and will often be a function of the dialog designer's role. This is because creating great response content for the Assistant requires understanding of the dialog flow, and vice versa. Response design merits its own role within the conversation design team when the complexity and variety of prompts begins to span many languages, modalities, personas, or internal stakeholders (like marketing or legal). With a great level of response complexity and variation, such as complex language internationalization, splitting response design from dialog design can remove the burden of content management from the dialog designer.
The workflow for the dialog designer becomes setting up the flow with a barebones version of the design, then having the response design team adding prompt variation, languages, match prompt copy to their Assistant’s persona guide, etc. The most obvious need for splitting response design from dialog design comes with multimodal experiences - ie Assistants that incorporate visual elements. It is unreasonable to expect the dialog designer to also be a fantastic visual designer. So, having someone working with the conversation design team responsible for visual responses alone makes sense.
The evolution of conversation design teams
This framework makes it possible to map the evolution of a conversation design team in terms of team structure and specialization vs simply the aggregate count of people.
When teams are small and looking to get started in conversation design, hiring a single full stack conversation designer is usually the best approach. These people can be hard to find, but they are worth their weight in gold as they can design all three components.
As the requirements of the team grow, the team begins to specialize from everyone being a conversation designer to component roles depending on the team's needs. Eventually, the conversation design team becomes an organization with subteams for each component role. All of which focus on independent components of the assistant. This structure can be applied to a single channel assistant like IVR, or spread across multiple channels depending on the objectives of the assistant.
From slow waterfall to fast independent iteration
With this new framing of conversation design, workflows can be more easily defined. Even if your conversation design team has the right structure and people, your toolset and workflow can make or break your ability to iterate on your assistant quickly and continuously.
What you want to avoid is waterfall workflows where work is passed with dependencies from team to team. Usually in conversational AI, that waterfall workflow looks like this:
The NLU design team passes work to the dialog design team, who then passes work to the response design team, who then passes it to development. Worst of all, each team has to wait for production analytics to come back to inform their next iteration.
By giving clear division to each subteam within conversation design, teams can work independently through their own measure and refinement loop, shortening the time to iteration.
In this model, each subteam has a mandated set of metrics they are tasked with improving, and can both get production feedback on their work and iterate independently of other teams. The goal is independent workflows, not silos. Each team should have visibility into each other’s work but retain the ability to work independently of each other with clear function-specific goals in mind.
Megan Hopkins put this nicely in noting: “Ideally, team metrics align with the goals of the greater organization so that teams have the freedom to experiment and iterate continuously whilst maintaining alignment with the overarching strategic roadmap.”
For example, a company automating customer support could measure conversation design team output by:
- Dialog design measures their work against transcripts for resolution rate and drop-off
- NLU design measures against intent coverage and their K-fold score
- Response design measures Assistant NPS, transcripts, or even fallbacks and no-replies
This independent measurement, paired with the right tooling for each team, allows conversation design teams to iterate faster across the entire assistant than today's traditional waterfall iteration loop.
The shift from development to conversation design
Above, you may have noticed I used “design” as the designation for all three components within conversation design (NLU design, dialog design, response design). This is because if a CxD team had the perfect tools and workflow, there would be no gap between design and development. Ultimately development and conversation design should be seen as equal partners, with development managing the “how” and design managing the “what” and “why”.
A conversation design team should be tasked with uncovering customer needs, designing solutions, and iterating on them, while the development team implements these designs and maintains and scales its architecture.
Developers are equal counterparts to designers in the workflow, but they shouldn’t overlap in the responsibility of assistant design and creation. Developers also don’t want to constantly implement iterative changes to the assistant (but they often do want to, and should give, design feedback). Their focus should be on the fun bits —the architecture, technology stack, and integrations.
Frankly, it’s a waste of these developers’ time to be writing hundreds of conditional statements, building and maintaining a rigid dialog manager, updating copy - or spending hours trying to understand, or “translate” a design split between multiple docs and teams. What developers should be spending their time on is higher ROI, more engaging functions like building new APIs and feature sets to enable new Assistant competencies.
Most companies have this backward and focus on developer implementation first, then have conversation design “clean up,” after which is far more difficult to do. Instead, design across all three components of the assistant should lead the workflow, with development focusing on making the design a reality.
Conversation design is the next frontier of design
There are no limits to how big conversation design could become. It is clear that an increasing percentage of our daily interactions with brands, big and soon small, will be through assistants—across every facet of consumer life, from customer support to conversational commerce. As this transformation happens, conversation design teams will continue to be set up to design, own, and iterate these assistant experiences. I don’t think it’s science fiction to imagine the majority of brand interactions we have in five years will be through assistants, across a number of channels.
With this growth, we as an industry need to better specify roles and responsibilities within the conversation design practice so that our industry can scale to achieve its potential impact on billions of people. I hope this framework serves as a starting point for a discussion on how we as an industry can scale, standardize, and legitimize.
Special thanks to Peter Isaacs for taking the time to bounce these ideas around with me early in their development (plus for extensive edits).
And thanks to the amazing industry leaders, and friends, who helped shape this framework:
Cathy Pearl, Gina Riley, Lisa Falkson, Molly Dickinson, Tara Nair Shah, Peter Isaacs, Ali Pinch, Simon Wolf, Amber Prause, Michael Barnes, Greg Bennett, Elaine Anzaldo, Eunji Jeong, Joseph Suskin, Hillary Black, Brian Smith, Megan Hopkins, Kane Simms, Jesús Martín, Olivera Bay
.svg)