Crafting a Visual QA Process for Marketers to Optimize User Flows in Braze
Crafting a Visual QA Process for Marketers to Optimize User Flows in Braze
Building 0 to 1 Quality Assurance Functionality
Building 0 to 1 Quality Assurance Functionality
Within 2 quarters of releasing this feature, we saw 35% product adoption.
Within 2 quarters of releasing this feature, we saw 35% product adoption.
Role
User Experience (UI/UX)
Strategic Thinking
Scoping MVP
Design Vision
Research Partnership
Timeline
November 2022 - March 2024 (GA Release)
Domain
Quality Assurance and Testing
Team
Product Manager
User Researcher
Tech Lead
Engineering Team
Design Lead (Me)
Overview
Marketers often face challenges in efficiently testing aspects of their campaigns, such as segments, messages and settings. They spend significant time manually setting up and triggering canvases for test runs, or they may forego this altogether and opt to visually inspect the canvas setup. This process can take multiple teammates and several days.
My team, Messaging Experience, took on what turned into a year-plus initiative to help bring more of the Quality Assurance (QA) process into Braze, leading to streamlined processes and reduced friction in the marketer’s workflow, as well as less reliance on more technical resources such as engineering teams.
Marketers often face challenges in efficiently testing aspects of their campaigns, such as segments, messages and settings. They spend significant time manually setting up and triggering canvases for test runs, or they may forego this altogether and opt to visually inspect the canvas setup. This process can take multiple teammates and several days.
My team, Messaging Experience, took on what turned into a year-plus initiative to help bring more of the Quality Assurance (QA) process into Braze, leading to streamlined processes and reduced friction in the marketer’s workflow, as well as less reliance on more technical resources such as engineering teams.
Background
Marketers use Canvas to create cross-channel experiences for their users. Think of it as a marketing campaign no-code flow builder, where you can create different paths based on actions your users take. After a marketer has completed setting up their Canvas, they will test it to ensure it’s setup correctly. They’ll check to make sure messages look as they should, and all of their links go to the right places. Many marketers will also create test users or send to their email to ensure they meet the criteria for receiving communications. This is one of the last steps in the marketer’s workflow as they’re working in Braze.Testing and quality assurance has consistently been ranked one of the hardest parts of a marketer’s workflow in Braze. On average, it can take up to four hours for a marketer to QA one canvas. Now, think about how many different campaigns a marketer might run at one time - this equates to many frustrating hours checking, testing and ensuring everything within the canvas is configured correctly.After results from a customer satisfaction survey, our team prioritized automating the QA workflow as a large part of our 2023 roadmap.
Marketers often face challenges in efficiently testing aspects of their campaigns, such as segments, messages and settings. They spend significant time manually setting up and triggering canvases for test runs, or they may forego this altogether and opt to visually inspect the canvas setup. This process can take multiple teammates and several days.
My team, Messaging Experience, took on what turned into a year-plus initiative to help bring more of the Quality Assurance (QA) process into Braze, leading to streamlined processes and reduced friction in the marketer’s workflow, as well as less reliance on more technical resources such as engineering teams.
Dig into the Details
Discovery
Partnering with User Research, we conducted multiple research sessions with 8 customers in Fall 2022 to better understand marketers’ pain points in QAing their canvases in Braze.
Marketers often face challenges in efficiently testing aspects of their campaigns, such as segments, messages and settings. They spend significant time manually setting up and triggering canvases for test runs, or they may forego this altogether and opt to visually inspect the canvas setup. This process can take multiple teammates and several days.
My team, Messaging Experience, took on what turned into a year-plus initiative to help bring more of the Quality Assurance (QA) process into Braze, leading to streamlined processes and reduced friction in the marketer’s workflow, as well as less reliance on more technical resources such as engineering teams.
Session 1
Focused on understanding the user journey
Session 1
Focused on understanding the user journey
Session 2
Focused on ranking pain-points
Session 2
Focused on ranking pain-points
insight.
insight.
Ultimately, it was loud and clear that we needed to solve the problem of testing canvas elements for our users. 6 out of 8 customers ranked the ability to forecast a canvas journey of a given user/segment as their most wanted functionality in Braze. Aside from this major signal, QA has been one of the most commonly brought up pain points from customers in ProductBoard, with nearly 40 insights.
I then created a low fidelity proof of concept in early 2023 to assess the appetite for a testing feature in Canvas. I fully scoped the functionality we’d include in the prototype. From research, we learned that users wanted a visual way to understand how users flowed through a canvas. We also knew users wanted to speed up the time between steps, and test with a testing group.
Ultimately, it was loud and clear that we needed to solve the problem of testing canvas elements for our users. 6 out of 8 customers ranked the ability to forecast a canvas journey of a given user/segment as their most wanted functionality in Braze. Aside from this major signal, QA has been one of the most commonly brought up pain points from customers in ProductBoard, with nearly 40 insights.
I then created a low fidelity proof of concept in early 2023 to assess the appetite for a testing feature in Canvas. I fully scoped the functionality we’d include in the prototype. From research, we learned that users wanted a visual way to understand how users flowed through a canvas. We also knew users wanted to speed up the time between steps, and test with a testing group.
Driving the Proof of Concept
I defined and aligned POC requirements and scope, validating core assumptions and de-risking product decisions to accelerate delivery of a viable solution.
I defined and aligned POC requirements and scope, validating core assumptions and de-risking product decisions to accelerate delivery of a viable solution.
Testing confirmed that the functionality would deliver significant value, reducing QA time by over 40%.
Of users mentioned they would use the functionality weekly, if not more
Mentioned the positive impact it would have by ridding their workflow
Specifically called out that it would improve their current QA process
Testing confirmed that the functionality would deliver significant value, reducing QA time by over 40%.
Of users mentioned they would use the functionality weekly, if not more
Mentioned the positive impact it would have by ridding their workflow
Specifically called out that it would improve their current QA process
Milestone Mapping
At this point, the scope of this first milestone was very ambiguous. I initiated a milestone mapping session with research, engineering and product partners to align on what would be included in milestone one of this feature.
Cutting Scope
Due to a high level of engineering effort that would push out the release timeline, we scoped the work to not include showing ineligibility reasons, which meant that marketers tested in this flow with test users, every user would successfully go down the path, regardless of whether they actually were eligible for each step.
results.
Unsurprisingly, users found the explanation of ineligibility crucial for understanding the utility of the testing functionality. Without it, we were not confident the feature would be adopted.
A large majority of users explicitly called out that they like to be able to have control over paths where there is more than one option, such as a variant or action path.
Grounding Trade-Off Decisions in User Feedback
To keep our first milestone on track, we explored ways to deliver value without significant tradeoffs. We decided to launch with partial testing capabilities, followed by quick iterations to enable full Canvas testing. To ensure this release would resonate with users, we conducted another round of testing. Based on the feedback, I also refined the information architecture for how test results are displayed, as well as moving the experience to a full page.
Grounding Trade-Off Decisions in User Feedback
To keep our first milestone on track, we explored ways to deliver value without significant tradeoffs. We decided to launch with partial testing capabilities, followed by quick iterations to enable full Canvas testing. To ensure this release would resonate with users, we conducted another round of testing. Based on the feedback, I also refined the information architecture for how test results are displayed, as well as moving the experience to a full page.
Releasing to GA
After several months of early access across a broad set of ~35 customers, we announced General Availability in March 2024. Within one month of GA, we saw these results:
This incredible product adoption speaks to both the need of a testing feature in Braze, while also ensuring we solved the right problem, designed the right way.
Nearly 20%
Nearly 20%
of our customer base has used the feature at least once
of our customer base has used the feature at least once
10% of customers
10% of customers
have used the feature more than once in 90 days
have used the feature more than once in 90 days
“This is like one of the coolest things y’all have put together in the last few years… Short of being able to edit the connections in an existing canvas, this is the best feature I’ve seen in 3 years.”
- BiltRewards
“Killer Feature”
- SSENSE
“This is like one of the coolest things y’all have put together in the last few years… Short of being able to edit the connections in an existing canvas, this is the best feature I’ve seen in 3 years.”
- BiltRewards
“Killer Feature”
- SSENSE
Design Vision
Since releasing this feature as Generally Available, I have worked on multiple different concepts for enhancing this functionality, including being able to preview message content on hover and testing across multiple audiences (shown below). We are currently collecting feedback and have plans to enhance the Preview User Paths in the coming quarters so that users can visualize how multiple users flow through a Canvas. Our next steps will include building out this functionality as well as identifying opportunities through ongoing customer feedback.
Since releasing this feature as Generally Available, I have worked on multiple different concepts for enhancing this functionality, including being able to preview message content on hover and testing across multiple audiences (shown below). We are currently collecting feedback and have plans to enhance the Preview User Paths in the coming quarters so that users can visualize how multiple users flow through a Canvas. Our next steps will include building out this functionality as well as identifying opportunities through ongoing customer feedback.
