Surfaces and Assignment

The onboarding test ran for two weeks and showed a 9% lift in activation. Then someone noticed that a third of the users in the treatment arm had also appeared in the control arm, because the flag was keyed to a device cookie before signup and to a user id after it, and the two never met.

Half the treated users had seen both versions. The 9% was a comparison between two groups that had traded members, and there is no estimator that recovers an effect from that.

The decision it encodes

Which visitor is in which arm, on which surface, and did they see it?

Three questions, and a feature flag tool answers the first one by default and the other two not at all. The lifecycle system had the same problem with a stable identifier and solved it with a hash and a registry. Here the identifier changes under the visitor mid-journey, which is the whole difficulty.

The surface is the scope

Growth surfaces are the parts of the product a visitor meets on the way from arriving to paying, where the growth program is accountable for what happens: the signup flow, onboarding, the pricing page, the paywall, the trial-to-paid step. Which team that is varies by company; the surfaces do not. Naming them is the first act of the system, because a test can only be interpreted against the surface it ran on. A pricing page test whose variant also changed the checkout has two treatments and one result.

The list is short on purpose. A product team runs the same machinery on the rest of the product with a different owner, and the boundary between the two is worth drawing explicitly so a test on the paywall does not collide with a test on the feature behind it.

Assignment before an account exists

A visitor on the signup flow has no user id. Assignment has to key on something that exists, which is a first-party cookie or a device identifier, and the same hash-and-salt discipline applies: a stable id hashed with the test’s salt gives a deterministic arm, so the visitor sees the same variant on every visit.

The failure is at signup. The moment the visitor becomes a user, the product starts identifying them by user id, and a flag evaluated on user id will re-assign them. The fix is not clever: the assignment made on the anonymous id has to be written to the user record at signup and read from there ever after, so the arm follows the person across the join. Every flag tool can do this and almost none does it unless told.

Exposure is separate from assignment

Assigning a visitor to the pricing page variant does nothing until they reach the pricing page. Most assigned visitors never do. The readout that compares all eligible assigned visitors is intention to treat, it is unbiased for the effect of shipping the change to that population, and it is the default estimand of this system.

Two narrower reads are available, each under a condition. A triggered read restricts the analysis to visitors who reached the surface, and it is valid when the trigger is defined before the treatment can affect it and identically in both arms: randomize everyone, analyze the ones who later reached the pricing page, when reaching the pricing page is not something the variant could change. A complier read scales the intention-to-treat effect by the difference in exposure rates between arms, and it estimates the effect on the visitors whose exposure the assignment changed, under the assumption that the change did nothing to anyone it never reached. Neither is “the effect on everyone we observed exposed”, which is not a quantity randomization identifies. Both need exposure logged at the moment the variant rendered, and a team that does not log it cannot tell a weak change from a rarely seen one. The exposure component in lifecycle makes the same argument for messages that were sent and never opened.

What the assignment record carries

Field	What it decides
Anonymous id and user id	Both, linked at signup, so the arm can be followed across the join and a visitor who appears in two arms is found rather than averaged over.
Test, variant, and salt	Which test this assignment belongs to and how it was drawn, so it can be reproduced.
Assignment time	When the visitor entered the test, since a visitor assigned in week one and one assigned in week six have had different amounts of time to convert.
Exposure time and surface	Whether and when the variant rendered. Absent for most assigned visitors, and the absence is the point.
Flag version	What the variant looked like at the time, because variants get edited mid-test and a readout across two versions is a readout of neither.

Maturity, honestly staged

Stage 1: named surfaces and one flag tool. The scope exists and every test on it is randomized by a deterministic hash. Most teams are here and believe they are further.

Stage 2: assignment persisted through signup. The arm is written to the user at the join and read from there. Contamination stops being the default.

Stage 3: exposure logged at render. Intention to treat and the effect on the exposed can both be reported, and a rarely seen variant stops looking like a weak one.

Stage 4: flag versions and cross-device rates in the record. A readout can be restricted to one version of the variant, and the assumption above is measured every time rather than believed.

Deterministic assignment across an identity change Technical

Assignment is a hash of salt + ”:” + anonymous_id reduced to a bucket, the same construction the lifecycle registry uses. The salt is per test, so two tests on the same surface do not share buckets, and the hash is stable across runs, so an assignment can be recomputed later without a lookup.

At signup the assignment is materialized: the bucket computed from the anonymous id is stored on the user, and every subsequent flag evaluation reads the stored value rather than recomputing from the user id. That one rule is what keeps the arms fixed. Where a user later appears with a second anonymous id, on another device, the stored value wins and the new id is linked to it, which is the right behavior for the readout and has to be handled explicitly because the tool will otherwise assign afresh.

Intention to treat compares all eligible assigned visitors by arm and is reported for every test. The complier effect divides that by the difference in exposure rates between arms, which is the Wald estimator the lifecycle system uses; it identifies the effect among visitors whose exposure the assignment changed, under exclusion (the change does nothing to a visitor who never reached the surface), relevance, and monotonicity, and the assumptions are recorded with the estimate. A triggered read is reported only when the trigger is pre-treatment and symmetric across arms, and the dilution between a triggered and an intention-to-treat estimate is not a simple ratio for ratio metrics, which is one more reason the intention-to-treat number is the headline.

Context

Best suited for

Products where the surfaces before purchase are owned by marketing or growth, traffic is high enough to randomize visitors rather than sessions, and a flag tool already exists but nobody has checked what it does at signup.

Data requirements

• A first-party anonymous identifier stable across sessions on one device
• A user record the assignment can be written to at signup and read from thereafter
• Render-time exposure events per surface, distinct from page views
• Flag configuration history, so a variant edited mid-test is a new version rather than a silent change

What changes by context

Product-led

The natural home. Signup, onboarding, and the upgrade prompt are the growth surfaces and the product itself, so the boundary with product experimentation has to be drawn by owner rather than by screen.

B2C subscription

Paywall and pricing tests reach identified users, so assignment can key on user id from the start and the signup join matters less. Cross-device is the larger problem, since a subscriber browses on one device and pays on another.

B2B SaaS

The visitor is usually the wrong unit, since two people from one company must see one price. Assignment moves to the account or the domain, which the next component covers, and the sample shrinks by an order of magnitude.

B2C ecommerce

Guest checkout means many visitors never become identified at all, so the anonymous id carries the whole test and cookie loss is the dominant threat to the sample.

When it breaks down

• Surfaces where consent rules prevent a persistent anonymous identifier, so a returning visitor is a new visitor and the arm cannot be held.
• Traffic too low to randomize visitors on a surface, where the session becomes the unit and the same person sees both variants on successive visits.
• Products where the surface cannot be varied per visitor, such as a public pricing page indexed by search engines, where a visible split is a cloaking risk and the test has to be geo or time based instead.