Every product team I know peeks. Monday standup, someone opens the analytics dashboard, squints at a variant chart, and asks: “Is confetti winning yet?”

That habit is human. It is also how classical A/B tests quietly lie to you — unless you design for peeking on purpose.

This post is a practitioner guide with a real worked example from my personal SaaS: a 50/50 experiment on what trial users see right after they create their first core object in the target product. Confetti celebration vs an entry-plan pricing-card offer. Numbers below are from product analytics as of 23 August 2026 (~one week of traffic). The point is not “we found a winner.” The point is how to read early data without crowning noise.

The Classical Contract (And Why We Break It)

A fixed-horizon test is a deal with the math:

1. Choose one primary metric and a minimum detectable effect.
2. Compute how many users you need for ~95% confidence and ~80% power.
3. Randomize, run, do not stop early because the chart looked exciting on day four.
4. Analyse once at the end.

That deal keeps your false-positive rate honest. Break step 3 — check every Friday and ship when p < 0.05 first appears — and you inflate winners. Lucky streaks get crowned. Losers get killed for noise.

This is the peeking problem. It is not pedantry. It is the difference between “we learned something” and “we dramatised a random walk.”

Three Ways to Run Experiments

Think in operating modes, not academic camps.

1. Fixed horizon (classical)

Best when traffic is high, the decision can wait, and the team will actually leave the dashboard alone. You pre-commit to N users or N days, then one analysis. Clean. Rare in early-stage products because patience is scarce and volume is low.

2. Sequential testing

Built for looking early and often. Each peek spends some of your “error budget.” Thresholds adjust so that checking weekly does not silently raise your chance of a false win. Names you will hear: sequential probability ratio tests (SPRT), always-valid p-values, peeking-safe confidence sequences.

You do not need to derive them from scratch. You need the posture: monitoring is allowed; naive p-value shopping is not.

3. Bayesian testing

Instead of asking “is the null dead?”, you track a running belief:

Probability that variant A is better than B on the primary metric — written P(A > B).

Example: after enough data, P(A > B) = 0.93. Ship if your bar is 0.95 and hold. Or stop for futility if it wobbles around 0.55 for a month. Peeking is natural here because you are updating a posterior, not repeatedly performing a one-shot ritual that assumed you would not look.

Bayesian does not magically need less data. A weak effect with tiny volume still looks like a coin flip. It just answers a question product people already ask in English.

Worked Example: First-Create Activation Offer (activation_offer_v1)

On my personal SaaS, trial users who create their first core object in the target product see one of two experiences (stable 50/50 assignment from user id):

• control_confetti — celebration + confetti, nudge to open product analytics / continue the trial.
• pricing_offer — entry-plan pricing cards with an upgrade CTA (or continue trial).

Experiment id: activation_offer_v1. Instrumenting events: activation_offer_exposed and activation_offer_action (with variant + action), then the usual funnel events pricing_viewed, checkout_started, subscription_payment_succeeded.

Primary metric (decision metric): exposure → paid within 14 days.
Secondary: in-modal actions, pricing viewed, checkout started.
Window in this snapshot: exposures from 16–23 Aug 2026 (~7 days). Roughly ~20 exposures/day total (~10 per arm).

1) Exposures

Variant	Unique users exposed
control_confetti	87
pricing_offer	73
Total	160

Slightly uneven arms are normal with hash assignment at this volume — not a bug by itself.

2) What people did inside the modal

Variant	Action	Users	% of exposed
confetti	open_analytics	20	23.0%
confetti	continue_trial	12	13.8%
pricing_offer	continue_trial	69	94.5%
pricing_offer	upgrade_clicked	3	4.1%

This is the loudest early signal. The pricing-card modal is not “interesting but undecided.” It is mostly dismissed. A 4% upgrade click rate on a hard sell at the dopamine moment of first successful create is a product smell, not a rounding error.

3) Downstream funnel (from exposure)

Pricing / checkout counted within 7 days of exposure; paid within 14 days. Unique users.

Variant	Exposed	→ Pricing	→ Checkout	→ Paid
control_confetti	87	11 (12.6%)	2 (2.3%)	3 (3.4%)
pricing_offer	73	4 (5.5%)	3 (4.1%)	2 (2.7%)

Two readings matter:

• Confetti → more later pricing views (12.6% vs 5.5%), even without an on-modal upgrade wall.
• Paid is 3 vs 2. That is a coin flip. Anyone who “ships confetti because paid is higher” is peeking without a design.

Caveat you will hit in any SaaS funnel: client checkout_started and server subscription_payment_succeeded can disagree because of identity merge / webhook timing. On confetti, paid (3) > checkout (2) in this window — treat paid as ground truth, and do not over-interpret checkout→paid ratios until identity is clean.

4) What Bayesian / sequential thinking says with these numbers

Pretend primary = expose→paid, uniform Beta(1,1) prior (uninformative):

• Confetti: 3 paid / 87 → posterior roughly Beta(4, 85), mean ≈ 4.5%
• Pricing offer: 2 paid / 73 → posterior roughly Beta(3, 72), mean ≈ 4.0%

Simulate draws from those two betas and ask how often confetti’s rate exceeds the offer’s. You get something in the neighbourhood of a coin — call it P(confetti better) ≈ 0.55–0.60, nowhere near a 0.95 ship bar. Sequential testing would say the same: keep running; do not stop.

So why dig into the experiment at all? Because secondary behaviour is already decisive for redesign, even when paid is not:

• ~95% dismiss the pricing cards.
• ~4% click upgrade on the offer.
• Confetti users explore analytics and find pricing later more often.

That is how you use early peeks without lying: ship/kill on the primary only when the rule fires; use mid-funnel behaviour to change the losing creative or to form a directional product gut — labelled as directional, not “significant.”

5) How long until a “real” call?

At ~10 exposures per arm per day:

Goal	Target per arm	Rough wait from this snapshot
Directional mid-funnel (CTR / pricing)	~300	~3 more weeks
Stronger mid-funnel	~500	~6 weeks
Industry-ish paid call at ~3% baseline	~2,000	many months (not practical without more traffic)

Practical plan for this test: keep measuring ~3 weeks for mid-funnel; treat paid as a guardrail; default UX toward confetti (or confetti + a soft link) while the pricing-card treatment gets rewritten or retired as a peer variant.

Ship / Kill Rules You Can Copy

Steal this template. Put it in the experiment brief before launch — I should have pasted it next to activation_offer_v1 on day zero.

• Primary metric: “expose → subscription_payment_succeeded within 14 days.”
• Guardrails: trial completion, refunds, support load.
• Minimum runtime: at least one full weekly cycle (often two).
• Sample target: fixed N, or “run until decision rule fires.”
• Bayesian ship rule: ship A if P(A > B) ≥ 0.95 for three consecutive daily snapshots; else keep running until the calendar stop.
• Futility rule: after minimum runtime, if P(A > B) sits ~0.40–0.60 and the absolute gap is below your minimum interesting lift, stop and keep control.
• Peeking policy: weekly review allowed; early ship without the rule firing is not — unless you explicitly reframe as a non-statistical product decision.

For this SaaS’s current volume, a smarter primary for the next iteration of the offer might be upgrade_clicked or expose→checkout, with paid as guardrail. Demanding paid significance on ~80 users per arm is how founders torture themselves.

When to Use What

• Fixed horizon — high traffic, patient stakeholders, one clear look at the end.
• Sequential — you will monitor weekly and want frequentist guarantees that survive peeking.
• Bayesian — you want decisions in probabilities (“91% chance A is better”) and a natural language for ship/futility.
• Neither + honesty — low-traffic learning tests (this first-create week). Call them directional. Redesign on behaviour. Do not fake a p-value on 3 vs 2 paid.

What This Is Not

This is not a licence to stop the moment the chart flatters you. Bayesian and sequential methods still punish small samples. They punish peeking less dishonestly than naive classical tests. If expose→paid is ~3% and you want a 20–30% relative lift, you are still in “thousands of users or months of runway” territory on a small personal SaaS — exactly where this experiment sits today.

The upgrade is intellectual hygiene: match the statistical tool to how your organisation actually behaves, and separate product taste from early behaviour from statistical crowning of a paid winner.

A Closing Rule

If you will peek weekly, say so in the experiment design and pick Bayesian or sequential rules up front. If you will not peek, lock a horizon and keep your hands off the dashboard. The worst option is classical math plus product-team habits — a ritual that looks rigorous while rewarding whoever refreshed analytics at the luckiest hour.

On this first-create activation test: do not call a paid winner yet; do treat the pricing-card dismissal rate as a redesign signal; write the ship bar before the next variant ships. That loop — instrument, peek honestly, decide with a pre-written rule — is the whole game.

Further reading if you want the deeper rails: Statsig on sequential testing, Evan Miller on Bayesian A/B testing, and the literature on always-valid p-values / confidence sequences. Use them as implementation references — the operating principle above is what most teams are missing.