Prototype Usability Study

I proposed, designed, recruited, moderated, and synthesized findings for a usability study to identify usability issues and opportunities in two potential prototypes for a B2B SaaS analytics software with the goal of providing actionable recommendations to move forward with the product.

Overview

End‑to‑end solo researcher responsible for planning, execution, analysis, and stakeholder alignment.

  • Study design & research plan

  • Recruitment & scheduling

  • Moderation & live note‑taking

  • Session recording & tagging

  • Quantitative + qualitative analysis

  • Stakeholder readout & decision support

My Role

Background

A software team had developed two distinct prototypes for the same B2B SaaS product experience and needed a decisive, evidence‑based recommendation on which approach to carry into beta. I designed and ran a moderated, in‑person usability study to evaluate each prototype across core jobs‑to‑be‑done. The study captured both behavioral data (time‑on‑task, completion, error types) and perception data (SEQ per task, SUS post‑test), and surfaced actionable usability issues to inform the go‑forward build.

Study Design

Each session used 5 scenario‑based tasks that reflected expected end‑user flows. After each task, participants answered the Single Ease Question (SEQ). At the end, participants completed the System Usability Scale (SUS). Sessions were recorded and time‑stamped to correlate comments with events.

I recruited 8 testers for each prototype (16 total), balancing role familiarity and technical literacy to reflect a realistic range of experience. All sessions were moderated in person to capture rich qualitative signals and reduce environmental confounds.

Key Findings

What Data was Captured

  • Single Ease Question (SEQ) per task

  • System Usability Scale (SUS) post-test

  • Time-on-task & completion rates

  • Critical incident logging

  • Qualitative theming & affinity mapping

  • Think-aloud protocol recordings

What the Data Showed

  • Both prototypes exhibited major usability deficiencies; a lack of affordances and in-app guidance left users confused about what information was interactable and what was static.

  • Prototype A encountered issues with data table presentation, with users desiring more control over the data visualizations.

  • Prototype B showed lower usability in most observed areas, but saw much higher usability in the interactive AI system.

  • Testers of both prototypes expressed a desire for a deeper level of drill-down data than currently available, and frustration with navigation and non-clickable data.

Top Usability Themes

Data Visualization

Interactive AI

Navigation & Discoverability

Recommendations & Impact

After synthesizing the data from both studies, I delivered a set of actionable, evidence-based recommendations that directly informed the product team’s next steps.

For Prototype A, I identified usability friction tied to unclear interaction design and limited data manipulation options. I recommended adding clear visual affordances to clickable elements, fixing non-responsive components that users expected to engage with, and expanding grid functionality to include features such as double-sorting and visible sum totals, core behaviors users relied on for validation and analysis.

For Prototype B, my focus shifted to functional completeness and system feedback. I advised re-introducing expected features such as data export, and improving the AI assistant’s perceived responsiveness through clearer loading states and progress feedback.

The product team adopted my recommendations, moving Prototype A forward to beta after implementing the proposed interaction and grid updates. Additionally, they ported the AI features from Prototype B, which my research showed to be more intuitive and trustworthy from a user standpoint.

This outcome demonstrated how structured usability testing and evidence-driven recommendations can shape product direction, resulting in a hybrid solution that combined the structural clarity of Prototype A with the superior AI experience of Prototype B.

“This is amazing. I have so much to go off of now, I’ve already got the team working on all of these findings.”

— Prototype Product Owner

Outcomes

  • Product team advanced a single prototype into beta with a clear fix list and prioritized design debt.

  • Engineering received actionable acceptance criteria derived from observed breakdowns (labels, feedback, error handling).

  • Executive stakeholders aligned on a data‑backed decision, reducing debate and accelerating the roadmap.

Next

Birdr App