KPI Definition Drift Triage

KPI Definition Drift Report

STATE: DEFINITION_DRIFT_CONFIRMED
Confidence: High that the documented contracts differ; no conclusion yet about pipeline health.

The two values are not calculations of one shared contract. Material differences in numerator, denominator, time boundary, refund treatment, entity grain, and refresh cutoff can explain part or all of the gap. The supplied evidence does not establish that either implementation is defective.

Value snapshot

Contract diff

Ranked causes

1. Confirmed: Different denominators mean the two metrics are not semantically equivalent.
2. Confirmed: Paid versus settled-net-of-refund numerator logic changes inclusion.
3. Confirmed: UTC versus America/New_York changes period boundaries.
4. Possible: Refresh cutoff and late-arriving records add a secondary data-timing difference.
5. Possible: Test-traffic treatment differs because the Finance rule is unknown.

Minimum verification checks

1. Recalculate both outputs using one frozen event set and one shared period boundary.
2. Produce counts for paid orders, settled orders, refunded orders, sessions, and eligible users before calculating either ratio.
3. Compare distinct users, sessions, and orders to expose grain and repeat-activity effects.
4. Align test-traffic filters and record the exact refresh cutoff for both sources.
5. Re-run each existing formula unchanged after the aligned snapshot to separate semantic drift from pipeline defects.

Draft canonical contract — decision proposal, not approved truth

• Proposed name: Monthly purchaser conversion rate
• Purpose: Measure the share of eligible users who complete at least one settled paid order in the reporting month
• Draft numerator: Distinct eligible users with at least one settled paid order
• Draft denominator: Distinct eligible users with a qualifying visit in the same period
• Time basis: One business-approved timezone and explicit inclusive/exclusive boundaries
• Refund rule: Requires owner decision on same-month versus lifetime attribution
• Grain: Distinct user
• Owner: Unresolved; executive reporting and Finance owners must designate one accountable approver

Unresolved decisions

• Is the intended business concept session conversion, purchaser conversion, or order frequency?
• Which status establishes completion: paid, settled, or another accounting state?
• Which timezone and refund-attribution policy governs board reporting?
• Which team owns the canonical contract and version history?

Reconciliation readback

After the owners approve one contract, calculate both sources from the same frozen population, time boundary, status rule, refund rule, and refresh cutoff. The figures should then agree within an explicitly approved rounding tolerance. Until those decisions are made, do not label either existing value as the canonical KPI.