Every technique in this track — nudges, hooks, engagement mechanics — is a hypothesis about what will improve a real metric, not a guarantee. Testing is how a design decision moves from "we believe this helps" to "we measured that it does."
Split real users into two groups at random: one sees the current design (the control), the other sees the new design (the variant). Measure the metric that matters (signups, completion rate, time on task) for both groups, and compare. Because the split is random, any real difference between the groups is attributable to the design change, not to who happened to see it.
| Type | What changes | When to use it |
|---|---|---|
| A/B (or A/B/n) testing | One element, isolated — a button color, a headline, a flow step | Testing one specific change with a clear before/after |
| Multivariate testing | Multiple elements at once, testing combinations | Understanding how several changes interact — needs far more traffic to reach significance |
Rather than one big redesign tested all at once, incremental testing ships smaller, individually-testable changes in sequence — each one's impact is measurable on its own, and a change that hurts the metric can be reverted without losing everything else that was working.
| Mistake | Why it's a problem |
|---|---|
| Calling a result significant too early | Small sample sizes produce noisy results that often reverse with more data |
| Testing during an unusual period | A holiday, a marketing campaign, or an outage skews behavior for both groups unevenly |
| Optimizing for the wrong metric | A change that increases clicks but decreases actual task completion is not really a win |
| Running too many variants at once without enough traffic | Splits the sample too thin to reach a confident conclusion for any of them |
Ties back to the design process
This closes the loop with the Double Diamond process from earlier in this course — Define and Develop propose a solution, Deliver ships it, and testing is how the loop confirms it actually worked before calling it done.