A planned product release mid-study created an unexpected opportunity
A leading Silicon Valley crowdfunding platform for creators and their subscribers wanted to benchmark their B2B dashboards — ensuring creators could find key information about their accounts and make decisions about their businesses. The client had a product release planned in the middle of the study, with updates to this functionality included. As a result, we determined that benchmarking in two parts was necessary: a baseline study first, then a second study using the updated functionality before release.
- Benchmark the client’s platform (desktop experience) for their B2B audience, creators.
- Work with the client’s internal teams to develop a test plan and recruit interviewees.
- Conduct remote interviews and usability tests, then share findings, insights, and recommendations with the product team.
Process
Building a grading system that made subjective experience measurable
Collaborating with the client’s Head of Product, Head of UX Research, and Head of Customer Care, we developed and tested the interview guide, including core questions and screenshots of expected answers. We also established the grading criteria for responses, tracked in a spreadsheet that became a heat map:
- Answered / found quickly = green
- Answered / found with some difficulty = yellow
- Not found = red
Why expanding recruitment criteria led to richer findings
The client’s customer care and community managers handled initial recruitment outreach. I followed up to schedule interviews and coordinate incentives. The product team initially wanted only new users (accounts under two months old), but recruitment proved difficult. We therefore expanded the criteria to include users with approximately one year of account history — a change that ultimately produced richer findings.
Independent facilitation: why an outside researcher sees what internal teams miss
I conducted each interview remotely and independently, using Lookback to capture participant expressions and on-screen actions. During sessions, I kept notes minimal to stay present and focused on observation.
From raw video to heat map: a synthesis process built for speed and pattern recognition
Within two days of each interview, I replayed the videos and transcribed responses into corresponding cells on a master spreadsheet. I then graded each cell by color and built the heat map, color-coding each response per the criteria above and bolding key text in red to flag issues at a glance.
Part 2 of the study (left side of heat map) showed a 17% improvement over Part 1 (right). Cells in red for the third question revealed a navigation label issue which was corrected and re-tested prior to launch.
After completing all interviews for each study part, I analyzed the full dataset for patterns and trends, noting these in a separate sheet within the document. For each part of the two-part study, I delivered a report and presentation covering all discovery, participant personas, and recommendations.
The findings were presented to the product team twice. The second round of findings enabled the team to adjust a navigation label before release — catching a defect before it shipped. I also delivered an overarching case study at the end showing the complete outcome of both studies.
17% improvement — and a navigation defect caught before it shipped
The second part of the study showed a 17% improvement over the first — creators found key information about their businesses faster and more easily.
Although the client had an internal UX Research team, hiring an independent consultant offered specific advantages:
- First, I was able to look at the product with fresh eyes and facilitate sessions with less bias.
- Additionally, it was easier to deliver insights and recommendations that might have been difficult for internal stakeholders to hear.
- Finally, participants may have felt more comfortable being candid with an outside researcher.
What I’d do differently, and how the tools have caught up since
I divided each video into clips, uploaded them to the client’s storage drive, and linked them cell-by-cell into the heat map spreadsheet. This was one of the most time-consuming parts of the process — and potentially unnecessary. For a future project, I would use markers within the larger video instead, unless the client confirmed they planned to watch individual clips.
Transcription was the other time-intensive step. Automated transcriptions via Descript required significant cleanup — roughly 4-5 hours for every hour recorded. That said, the cleanup process forced deep familiarity with the content, which made analysis easier. For future projects, Rev.com was worth testing to find the most efficient method.
2021 Update:
After evaluating Rev.com and several other options, I returned to Descript for a follow-up study. The product had improved significantly: automated transcriptions were far more accurate, reducing cleanup to approximately three hours per one-hour interview. Exporting video clips directly from Descript also eliminated the need for separate editing software.
2023 Update:
Currently, Dovetail and MURAL handle review, tagging, and synthesis — reducing analysis time to 1-2 hours per interview. In-depth transcription cleanup is no longer necessary.
A long-form case study of this project is available for presentation in person.
“That was a really fun and detailed presentation to watch. I also just reviewed an in-depth research doc for another company and thought yours was much more engaging. I can’t wait to read your doc over again.”
“Kim and I worked together in two capacities. Here, she helped us benchmark usability quality through attitudinal assessments. Kim's core strength is her ability to take very large and somewhat vague challenges and distill them into just the right execution proposals that executives can effectively act on. She champions research methodology as much as she is meticulous about her outcomes.”
