Crafting a Visual QA Process for Marketers to Optimize User Flows in Braze

Overview

Marketers often face challenges in efficiently testing aspects of their campaigns, such as segments, messages and settings.  They spend significant time manually setting up and triggering canvases for test runs, or they may forego this altogether and opt to visually inspect the canvas setup. This process can take multiple teammates and several days.

My team, Messaging Experience, took on what turned into a year-plus initiative to help bring more of the Quality Assurance (QA) process into Braze, leading to streamlined processes and reduced friction in the marketer’s workflow, as well as less reliance on more technical resources such as engineering teams.

Role

I collaborated with Product and UX research to conduct research and map out project milestones. I took charge of all design aspects, ranging from low-fidelity wireframes to ensuring alignment with engineering for accurate implementation of designs.

Timeline

Nov 2022 - March 2024

How might we help marketers visualize how their users flow through a Canvas?

Background

Marketers use Canvas to create cross-channel experiences for their users. Think of it as a marketing campaign no-code flow builder, where you can create different paths based on actions your users take. After a marketer has completed setting up their Canvas, they will test it to ensure it’s setup correctly. They’ll check to make sure messages look as they should, and all of their links go to the right places. Many marketers will also create test users or send to their email to ensure they meet the criteria for receiving communications. This is one of the last steps in the marketer’s workflow as they’re working in Braze.

Testing and quality assurance has consistently been ranked one of the hardest parts of a marketer’s workflow in Braze. On average, it can take up to four hours for a marketer to QA one canvas. Now, think about how many different campaigns a marketer might run at one time - this equates to many frustrating hours checking, testing and ensuring everything within the canvas is configured correctly.

After results from a customer satisfaction survey, our team prioritized automating the QA workflow as a large part of our 2023 roadmap.

Research

Partnering with User Research, we conducted multiple research sessions with 8 customers in Fall 2022 to better understand marketers’ pain points in QAing their canvases in Braze.

Ultimately, it was loud and clear that we needed to solve the problem of testing canvas elements for our users. 6 out of 8 customers ranked the ability to forecast a canvas journey of a given user/segment as their most wanted functionality in Braze. Aside from this major signal, QA has been one of the most commonly brought up painpoints from customers in ProductBoard, with nearly 40 insights.

I then created a low fidelity proof of concept in early 2023 to assess the appetite for a testing feature in Canvas. I fully scoped the functionality we’d include in the prototype. From research, we learned that users wanted a visual way to understand how users flowed through a canvas. We also knew users wanted to speed up the time between steps, and test with a testing group.

Lo-Fidelity Testing

Results.

Testing helped validate that testing functionality would be extremely valuable, cutting QA time by 40% or more.

87% of users said they would use this feature weekly, if not more.

63% of users mentioned the positive impact this functionality would have by ridding their workflow of re-testing, proofing, and making errors.

38% of users specifically called out that it would improve their current QA process. This functionality would remove the need for a staging environment, help marketers build canvases in Braze faster and eliminate the need for any workarounds.

Milestone Mapping

At this point, the scope of this first milestone was very ambiguous. I initiated a milestone mapping session with research, engineering and product partners to align on what would be included in milestone one of this feature.

Coming out of the session, we moved forward with minimizing the delay between steps and the action path trigger event. We discarded showing ineligibility reasons because of its level of effort. Keep an eye out for when this loops back into the project!

Pre-MVP Evaluative Testing

Now ready to test with our scoped first milestone, I created flows for various scenarios to see how users would respond. A big question on my mind was how valuable this feature would be without showing ineligibility reasons.

Unsurprisingly, users found the explanation of ineligibility crucial for understanding the utility of the testing functionality. Without it, we were not confident the feature would be adopted.

Five out of seven customers mentioned wanting to know the criteria that caused a user to exit.

“It’s a little barren to be honest.”

One customer would rank the feature 5/5 useful if it included more step detail.

Four out of seven customers explicitly called out that they like to be able to have control over paths where there is more than one option, such as a variant or action path.

Iterating on the First Milestone

In order to still deliver our first milestone on time, we strategized how we could make it valuable without making too many tradeoffs. We landed on shipping the ability to partially test, with quick fast follows to fully test all of Canvas. To ensure users would find this iteration more valuable, we went through another round of testing. I also updated the information architecture of how we would display test results, based on users’ feedback in the last round of testing.

Results

Our testing results were extremely positive. Two customers (Bilt, Lumos) said they would use the feature every time they built a Canvas. Three customers (Ro, WealthSimple, ASOS) said they would use it multiple times a week, and of those, two (ASOS, WealthSimple) said they would use it up to several times a day. More sophisticated customers like Etsy and SSENSE would use it for extra assurance on complex canvases. However, our customers really said it all...

“This is like one of the coolest things y’all have put together in the last few years… Short of being able to edit the connections in an existing canvas, this is the best feature I’ve seen in 3 years.” - BiltRewards

“Killer Feature” - SSENSE

Releasing to GA

After several months of early access across a broad set of ~35 customers, we announced General Availability in March 2024. Within one month of GA, we saw these results:

2665 companies with the feature turned on

This incredible product adoption speaks to both the need of a testing feature in Braze, while also ensuring we solved the right problem, designed the right way.

Nearly 20% of our customer base has used the feature at least once (528 companies)

10% of companies have used the feature more than once in 90 days (287 comp)

Design Vision

Since releasing this feature as Generally Available, I have worked on multiple different concepts for enhancing this functionality, including being able to preview message content on hover and testing across multiple audiences (shown below). We are currently collecting feedback and have plans to enhance the Preview User Paths in the coming quarters so that users can visualize how multiple users flow through a Canvas. Our next steps will include building out this functionality as well as identifying opportunities through ongoing customer feedback.