Usability Frameworks

Comprehensive frameworks and methodologies for planning, conducting, and analyzing usability tests to improve user experience.

When to Use This Skill

Auto-loaded by agents:

• research-ops
• For usability testing and heuristic evaluation

Use when you need:

• Planning usability tests

• Conducting user testing sessions

• Evaluating interface designs

• Identifying usability problems

• Testing prototypes or live products

• Applying Nielsen's heuristics

• Measuring usability metrics

Core Concepts

What is Usability Testing?

Usability testing is a method for evaluating a product by testing it with representative users. Users attempt to complete typical tasks while observers watch, listen, and take notes.

Purpose: Identify usability problems, discover opportunities for improvement, and learn about user behavior and preferences.

When to use:

• Before development (testing prototypes)

• During development (iterative testing)

• After launch (validation and optimization)

• Before major redesigns

The Five Usability Quality Components (Jakob Nielsen)

• Learnability: How easy is it for users to accomplish basic tasks the first time?

• Efficiency: How quickly can users perform tasks once they've learned the design?

• Memorability: Can users remember how to use it after time away?

• Errors: How many errors do users make, how severe, and how easily can they recover?

• Satisfaction: How pleasant is it to use the design?

Usability Testing Methodologies

1. Moderated Testing

Setup: Researcher guides participants through tasks in real-time Location: In-person or remote (video call)

Best for:

• Early-stage prototypes needing clarification

• Complex products requiring guidance

• Exploring "why" behind user behavior

• Uncovering emotional reactions

Process:

• Welcome and set expectations

• Pre-task questions (background, experience)

• Task scenarios with think-aloud protocol

• Post-task questions and discussion

• Wrap-up and thank you

Advantages:

• Rich qualitative insights

• Can probe deeper into issues

• Observe non-verbal cues

• Clarify misunderstandings immediately

Limitations:

• More time-intensive (30-60 min per session)

• Researcher bias possible

• Smaller sample sizes

• Scheduling logistics

2. Unmoderated Testing

Setup: Participants complete tasks independently, recorded for later review Location: Remote, on participant's own schedule

Best for:

• Mature products with clear tasks

• Large sample sizes needed

• Quick turnaround required

• Benchmarking and metrics

Process:

• Automated instructions and consent

• Participants record screen/audio while completing tasks

• Automated post-task surveys

• Researcher reviews recordings later

Advantages:

• Faster data collection

• Larger sample sizes

• More natural environment

• Lower cost per participant

Limitations:

• Can't probe or clarify

• May miss nuanced insights

• Technical issues harder to resolve

• Participants may skip think-aloud

3. Hybrid Approaches

Combination methods:

• Moderated first impressions + unmoderated task completion

• Unmoderated testing + follow-up interviews with interesting cases

• Moderated pilot + unmoderated scale testing

Nielsen's 10 Usability Heuristics

Quick reference for evaluating interfaces. See references/nielsens-10-heuristics.md for detailed explanations and examples.

• Visibility of system status - Keep users informed

• Match between system and real world - Speak users' language

• User control and freedom - Provide escape hatches

• Consistency and standards - Follow platform conventions

• Error prevention - Prevent problems before they occur

• Recognition rather than recall - Minimize memory load

• Flexibility and efficiency of use - Accelerators for experts

• Aesthetic and minimalist design - Remove irrelevant information

• Help users recognize, diagnose, and recover from errors - Plain language error messages

• Help and documentation - Provide when needed

Think-Aloud Protocol

What It Is

Participants verbalize their thoughts while completing tasks, providing real-time insight into their mental model.

Types

Concurrent think-aloud: Speak while performing tasks

• More natural thought flow

• May affect task performance slightly

Retrospective think-aloud: Review recording and explain thinking after

• Doesn't disrupt natural behavior

• May forget or rationalize thoughts

Facilitating Think-Aloud

Prompts to use:

• "What are you thinking right now?"

• "What are you looking for?"

• "What would you expect to happen?"

• "Is this what you expected?"

Don't:

• Ask leading questions

• Provide hints or solutions

• Interrupt natural flow too often

• Make participants feel tested

See references/think-aloud-protocol-guide.md for detailed facilitation techniques.

Task Scenario Design

Good task scenarios are critical to meaningful usability test results.

Characteristics of Good Task Scenarios

Realistic: Based on actual user goals Specific: Clear endpoint/success criteria Self-contained: Provide all necessary context Actionable: Clear starting point Not prescriptive: Don't tell them how to do it

Example Transformation

Poor: "Click on the 'My Account' link and change your password"

• Too prescriptive, tells them exactly where to click

Good: "You've heard about recent security breaches and want to make your account more secure. Update your account to use a stronger password."

• Realistic motivation, clear goal, doesn't prescribe path

Task Complexity Levels

Simple tasks (1-2 steps): Establish baseline usability Medium tasks (3-5 steps): Test core workflows Complex tasks (6+ steps): Evaluate overall experience and error recovery

See assets/task-scenario-template.md for ready-to-use templates.

Severity Rating Framework

Not all usability issues are equal. Prioritize fixes based on severity.

Three-Factor Severity Rating

Frequency: How often does this issue occur?

• High: > 50% of users encounter

• Medium: 10-50% encounter

• Low: < 10% encounter

Impact: When it occurs, how badly does it affect users?

• High: Prevents task completion / causes data loss

• Medium: Causes frustration or delays

• Low: Minor annoyance

Persistence: Do users overcome it with experience?

• High: Problem doesn't go away

• Medium: Users learn to avoid/work around

• Low: One-time problem only

Combined Severity Ratings

Critical (P0): High frequency + High impact Serious (P1): High frequency + Medium impact, OR Medium frequency + High impact Moderate (P2): High frequency + Low impact, OR Medium frequency + Medium impact, OR Low frequency + High impact Minor (P3): Everything else

See assets/severity-rating-guide.md for detailed rating criteria and examples.

Usability Metrics

Quantitative Metrics

Task Success Rate: % of participants who complete task successfully

• Binary: Did they complete it? (yes/no)

• Partial credit: Did they complete most of it?

Time on Task: How long to complete (for successful completions)

• Compare to baseline or competitor benchmarks

Error Rate: Number of errors per task

• Define what counts as an error for each task

Clicks/Taps to Task Completion: Efficiency measure

• More relevant for well-defined tasks

Standardized Questionnaires

SUS (System Usability Scale):

• 10 questions, 5-point Likert scale

• Score 0-100 (industry avg ~68)

• Quick, reliable, easy to administer

• Good for comparing versions or benchmarking

UMUX (Usability Metric for User Experience):

• 4 questions, lighter than SUS

• Similar reliability

• Faster for participants

SEQ (Single Ease Question):

• "Overall, how difficult or easy was the task to complete?" (1-7)

• One question per task

• Immediate subjective difficulty rating

Other scales:

• SUPR-Q (for websites)

• PSSUQ (post-study)

• NASA-TLX (cognitive load)

Qualitative Insights

Observed behaviors:

• Hesitations and confusion

• Error patterns

• Unexpected paths

• Verbal frustrations

Verbalized thoughts (think-aloud):

• Mental model mismatches

• Expectation violations

• Pleasantly surprising discoveries

Sample Size Guidelines

For Qualitative Insights

Nielsen's recommendation: 5 users finds ~85% of usability problems

• Diminishing returns after 5

• Run 3+ small rounds instead of 1 large round

• Iterate between rounds

Reality check:

• 5 is a minimum, not ideal

• Complex products may need 8-10

• Multiple user types need 5 each

For Quantitative Metrics

Benchmarking: 20+ users per user group A/B testing: Depends on effect size and desired confidence Statistical significance: Use power analysis calculators

Planning Your Usability Test

1. Define Objectives

What decisions will this research inform?

• Redesign priorities?

• Feature cut decisions?

• Success of recent changes?

2. Identify User Segments

Who needs to be tested?

• New vs. experienced users?

• Different roles or use cases?

• Different devices or contexts?

3. Select Tasks

What tasks represent success?

• Most critical user goals

• Most frequent tasks

• Recently changed features

• Known problem areas

4. Choose Methodology

Moderated, unmoderated, or hybrid?

• Consider timeline, budget, research questions

5. Create Test Script

See assets/usability-test-script-template.md for a ready-to-use structure including:

• Welcome and consent

• Background questions

• Task instructions

• Probing questions

• Wrap-up

6. Recruit Participants

• Define screening criteria

• Aim for 5-10 per user segment

• Plan for no-shows (recruit 20% extra)

• Offer appropriate incentives

7. Conduct Pilot Test

• Test with colleague or friend

• Validate timing

• Check recording setup

• Refine unclear tasks

8. Run Sessions

• Stay neutral and encouraging

• Observe without interfering

• Take detailed notes

• Record if permitted

9. Analyze and Synthesize

• Code issues by severity

• Identify patterns across participants

• Link issues to heuristics violated

• Quantify task success and time

10. Report and Recommend

• Prioritized issue list

• Video clips of critical issues

• Recommendations with rationale

• Quick wins vs. strategic fixes

Integration with Product Development

When to Test

Discovery phase: Test competitors or analogous products Concept phase: Test paper prototypes or wireframes Design phase: Test high-fidelity mockups Development phase: Test working builds iteratively Pre-launch: Validate before release Post-launch: Identify optimization opportunities

Continuous Usability Testing

Build it into your process:

• Weekly or bi-weekly test sessions

• Rotating focus (new features, established flows, mobile vs. desktop)

• Standing recruiting panel

• Lightweight reporting to team

Ready-to-Use Templates

We provide templates to accelerate your usability testing:

In assets/ :

• usability-test-script-template.md: Complete moderator script structure

• task-scenario-template.md: Framework for creating effective task scenarios

• severity-rating-guide.md: Detailed criteria for rating usability issues

In references/ :

• nielsens-10-heuristics.md: Deep dive into each heuristic with examples

• think-aloud-protocol-guide.md: Advanced facilitation techniques and troubleshooting

Common Pitfalls to Avoid

• Leading participants: "Was that easy?" → "How would you describe that experience?"

• Testing the wrong tasks: Tasks that aren't real user goals

• Over-explaining: Let users struggle and discover issues naturally

• Ignoring severity: Fixing cosmetic issues while critical issues remain

• Testing too late: After it's expensive to change

• Not iterating: One-and-done testing instead of continuous improvement

• Confusing usability with preference: "I like green" ≠ usability issue

• Sample bias: Testing only power users or only complete novices

Further Learning

Books:

• "Rocket Surgery Made Easy" by Steve Krug

• "Handbook of Usability Testing" by Jeffrey Rubin

• "Moderating Usability Tests" by Joseph Dumas

Online resources:

• Nielsen Norman Group articles

• Usability.gov

• Baymard Institute research

This skill provides the foundation for conducting effective usability testing. Use the templates in assets/ for quick starts and references/ for deeper dives into specific techniques.