Defining Events That Survive Contact With Operations

Qualified leads are up 34% year on year and the team is having a good quarter.

In March, sales asked for the qualification bar to be lowered so reps could work more accounts. In July, a form field became optional, which stopped disqualifying anybody who left it blank. In September the CRM added a second route to the qualified stage for accounts coming from events.

The 34% is arithmetic performed on a definition that changed three times inside the window. Nobody did anything wrong, nobody hid anything, and the number is uninterpretable.

An event is an agreement first and data second

This is the part that makes measurement in operational systems different from measurement in a warehouse of immutable logs. A purchase is a fact. A qualified lead is a decision somebody made about a record, encoded in a rule that another person can change on a Tuesday for a reason that has nothing to do with measurement and everything to do with running the business.

That is not a defect to be engineered away. It is what those systems are for. The failure is treating the resulting series as though it were a physical measurement.

Where a definition lives, and how each copy drifts
Copy	How it changes without anyone deciding to
The written definition	Ages. Written once at a planning offsite, never revised, and increasingly unrelated to the other two.
The system rule	A validation rule, a required field, a workflow. Edited by an administrator solving an operational complaint.
What people actually do	The real definition. Moves when a target changes, when a manager sets a different bar, or when a new hire is trained by somebody else.
The reporting query	A fourth copy nobody counted, written by an analyst who interpreted the other three.

Any two of these disagreeing is a discrepancy somebody can chase. All four agreeing is the unusual state, and it does not persist without maintenance.

The test: could two people classify the same record the same way?

An event definition is measurable if two competent people, given the same record and the definition, put it in the same bucket. That is a low bar and a surprising number of the metrics a business runs on fail it.

Where the answer is no, the metric is recording a judgment, and the series moves when the judges change. Sales headcount turnover, a new manager, or a compensation change will all move it without anything happening in the market.

What to do, in order of how much it costs

Date every definition change. A dated log of what changed and when, kept anywhere at all. Cheap, unglamorous, and it converts an unexplainable trend into an explained one months later when somebody asks.

Require a comparison to state whether the definition held. Any year-on-year or before-and-after number carries a note saying whether anything changed inside the window. If it did, the comparison is between two different things and should be labeled that way rather than adjusted into comparability.

Version the definition and keep the raw ingredients. If the underlying fields are retained, a changed definition can be recomputed backwards, and the series can be rebuilt on a consistent basis. This is the only move that recovers history, and it only works if the ingredients were kept, which is a decision made long before anybody needs it.

Separate the operational metric from the measurement metric. Sales can have a qualification bar that moves with what they need to work. Measurement can hold a fixed definition alongside it, computed from the same fields, that nobody is allowed to tune. Both are legitimate and they should not be the same column.

Recomputing history, and when it is not possible Technical

A definition change is recoverable if the inputs it depends on were stored and are still stored. Then the old period can be recomputed under the new rule, or the new period under the old one, and the difference between those two numbers is the size of the definition effect, separated from everything else that happened.

It becomes unrecoverable in two common cases. When the definition depends on a judgment that was never written to a field, such as a rep deciding an account is worth pursuing, there is nothing to recompute from. And when the fields themselves were overwritten rather than versioned, which is the normal behavior of a CRM, the state at the time of classification is gone. Both are arguments for capturing change history at the point the account graph is built, rather than when somebody first needs it.

Where history cannot be rebuilt, what remains is a series with a break in it, annotated on the date of the change, and no year-on-year figure spanning the break. That is less satisfying than a number and it is the only form that does not mislead.

Context

Best suited for

Any metric produced by an operational system rather than by an immutable log, which covers most of what marketing and sales report on.

Data requirements

A dated log of definition, rule, and workflow changes, kept outside the system that changed
The raw fields a definition depends on, retained rather than only the derived flag
Change history on those fields, so the state at classification is recoverable

What changes by context

B2B SaaS: Qualification bars move with pipeline coverage targets, so the definition tightens when pipeline is plentiful and loosens when it is not. That makes the metric move against the business cycle, which is the most confusing direction it could move in.
Sales-led: The real definition lives in what reps do rather than in the rule, so it changes with headcount, training, and compensation. A comp change is a definition change and is almost never logged as one.
Lifecycle-led: Activation and engagement definitions are usually owned by whoever built the dashboard, and there are often several in use at once. The problem here is less drift over time than three concurrent definitions nobody has reconciled.
Product-led: Definitions are computed from instrumentation, so a release can change the metric silently. Tying definition versions to deploy versions is the equivalent of the change log elsewhere.

When it breaks down

Metrics computed purely from immutable transactions, such as payments, where the definition is not a judgment and this problem largely disappears.
Organizations unwilling to keep any change log, where trends in these metrics cannot be interpreted and should not be quoted.

What this does not establish

That definitions should be frozen. Operational definitions need to change; the argument is for recording when they do and for keeping a separate measurement definition that does not.
That a stable definition makes a metric causal. A perfectly consistent series of qualified leads still contains no counterfactual and says nothing about what marketing caused.
That recomputation always resolves it. It works only where the inputs survived and the judgment was written down, and neither is typical.
Any estimate of how much of a given trend is definition drift. That is measurable on a specific series with a specific change log, and cannot be assumed from anywhere else.