goagentdata

Where does a number go wrong?

These five steps take your data from the right problem to the right decision at the right time, and each one can go wrong without a warning.

Each step below names what goes wrong, a guardrail and a check you can run on your own data.

First data person on the team? Start with the five draft checks in steps 2 to 4, and run them this week.

  1. 1Choose the problem
  2. 2Collect what you need
  3. 3Choose the number
  4. 4Count it right
  5. 5Read it right
Where it goes wrongHow to guard it
  1. 1

    Choose the problem

    problem
    Goes wrong
    You answer a question no decision depends on.
    Example
    A startup charts site visits while nobody asks why early customers cancel.
    Guardrail
    Before building anything, find the gate: the problem that, once solved, lets you move on, such as a cost, the funnel or how long customers stay. Write it as a question with its decision.
    Check
    Finish this sentence: if the answer is X we do A, and if it is Y we do B.
    Into the company brain
    Write down the question and the decision it serves.

    Write a short question brief: the gate, the decision, the candidate metrics, the cheapest data that could answer it, and an owner. Give agents the brief, not a schema to explore.

  2. 2

    Collect what you need

    collect
    Goes wrong
    Data you will need is never recorded, gets overwritten, or stops arriving.
    Example
    A startup edits plans in place, and a year later nobody can say how long customers stayed on the starter plan.
    Guardrail
    Give each customer one ID that links every system, keep each plan and status change with its date, and check daily that data arrives.
    Check
    Next to each thing you collect, write the question or number it serves now or in the coming months. Anything with no answer is collecting for its own sake. Then ask to see one customer's old plan and the date it changed.
    Into the company brain
    Write down what you collect and why.

    One internal customer ID, with every external ID mapped to it. created_at and updated_at, soft deletes, and a history or snapshot table for status and plan. UTC timestamps, raw data kept, money stored with currency. Freshness and row count alerts go to a named person.

    Draft check for your coding agent: keep-historyDraft

    A change overwrites the past, and last quarter's answer is gone.

    # keep-history
stage: Before launch, First paying customers, Growing
step: Collect what you need
guards: A change overwrites the past, and last quarter's answer is gone.
business_model: all
version: 0.1 (draft)
last_reviewed: 2026-10-13
rule:  every table has created_at and updated_at (UTC).
rule:  deletes are soft: set deleted_at, keep the row. An erasure request is the exception: remove or mask that person's personal fields.
rule:  a plan or price change adds a history row.
check: change a plan on a test account; read back the old and new plan with their dates.
    Draft check for your coding agent: one-customer-idDraft

    Each system has its own customer ID, so joins drop or double people.

    # one-customer-id
stage: Before launch, First paying customers, Growing
step: Collect what you need
guards: Each system has its own customer ID, so joins drop or double people.
business_model: all
version: 0.1 (draft)
last_reviewed: 2026-10-13
rule:  each customer gets one internal ID at the first sign-up or deal.
rule:  every external ID, in billing, the CRM and support, maps to that internal ID.
check: pick three customers; find each one in billing and in the CRM through the internal ID alone.
  3. 3

    Choose the number

    measure
    Goes wrong
    You track a number that looks like the goal and is not, or one name carries two meanings.
    Example
    A campaign is judged on sign-ups, a vanity number, while the customers it brings leave within a month.
    Guardrail
    Run the company on one outcome and its few drivers. Give each number a written meaning, a decision and an owner.
    Check
    Ask which decision changes if this number moves, then ask two people to define it. If the answers differ, you have two numbers.
    Into the company brain
    Write down the definitions.

    Metric file fields: definition, population with exclusions (test and internal accounts by default, pilots in or out on purpose), grain, time zone, decision, owner, version. Vanity test: remove the metric, and if no decision, owner or action changes, retire it.

    Read: Why your tools disagree on active customers

    See an example metric tree

    Paying customers at month end

    New paying customers this month

    Share of last month's customers still paying

    Days from signup to first value

    Hours to first support reply

    This is only an example. Yours depends on your business.

    Outcome: The single outcome the company is run on, chosen for its stage. Every box is a number with a unit and a period.

    Drivers: Together they add up to the outcome: last month's customers times the share kept, plus the new ones.

    Levers: Numbers one team can move directly. Each rolls up to one driver, and that link is a hypothesis until tested.

    What every number carries
    • What it meansOne written meaning that everyone shares.
    • Who is countedWho is included and excluded, such as test accounts.
    • Which periodThe dates and the time zone it covers.
    • The decision it informsThe decision this number is there to inform.
    • Who owns itOne named person answers questions about it.
    Draft check for your coding agent: metric-filesDraft

    A number's meaning lives in one head and changes quietly.

    # metric-files
stage: Before launch, First paying customers, Growing
step: Choose the number
guards: A number's meaning lives in one head and changes quietly.
business_model: all
version: 0.1 (draft)
last_reviewed: 2026-10-13
rule:  each metric has one file with its definition, who is counted and excluded, period and time zone, the decision it informs, and an owner.
rule:  a change to a definition goes through review and raises the version.
check: pick a number on a dashboard; find its file, its owner and the decision it informs.
  4. 4

    Count it right

    compute
    Goes wrong
    The right number is calculated wrong and still looks fine.
    Example
    An overnight import fails and reruns, so a day of orders counts twice.
    Guardrail
    Calculate from the written definition on a copy of the data, make imports safe to rerun, and leave test accounts out. Every dashboard and agent reads the number from the same place, and that place says when the number is known to be broken.
    Check
    Reproduce one number a second way, such as the finance total, then count the test accounts inside it. There should be none.
    Into the company brain
    Write down the checks and their status.

    Idempotent loads with unique keys. Check row counts before and after every join to catch fan-out. is_test set on the account and inherited by its users and events. Reconcile revenue to billing. Analytics run on a replica or nightly export, never the primary. Agents get read-only roles and a scratch schema, and propose changes by pull request. Serve every metric from one layer (metric files or a semantic layer) with a status per metric, so a late source or a broken count warns every tool and agent before it answers.

    Draft check for your coding agent: flag-test-accountsDraft

    Test accounts inside real numbers show growth that is not there.

    # flag-test-accounts
stage: Before launch, First paying customers, Growing
step: Count it right
guards: Test accounts inside real numbers show growth that is not there.
business_model: all
version: 0.1 (draft)
last_reviewed: 2026-10-13
rule:  every account has is_test, set when it is created; its users and events inherit it.
rule:  every metric leaves is_test accounts out by default.
check: count active accounts with and without test accounts; the headline active account number should equal the smaller count.
    Draft check for your coding agent: no-analytics-on-primaryDraft

    Queries on the live database can slow the product, and their answers shift and cannot be reproduced.

    # no-analytics-on-primary
stage: Before launch, First paying customers, Growing
step: Count it right
guards: Queries on the live database can slow the product, and their answers shift and cannot be reproduced.
business_model: all
version: 0.1 (draft)
last_reviewed: 2026-10-13
rule:  analysis and agent queries run on a replica or nightly export, never the live primary database.
rule:  the agent gets read-only credentials.
check: list the connections the agent and the dashboards use; none points at the primary.
  5. 5

    Read it right

    conclude
    Goes wrong
    A correct number is read the wrong way.
    Example
    Feature users stay longer, so the team pushes everyone into the feature. They were staying anyway.
    Guardrail
    Before you act, name one other explanation and compare like with like. A person reads the agent's conclusion before anyone acts on it.
    Check
    Write one other explanation and one fact that would tell them apart.
    Into the company brain
    Write down how to read the number.

    Use cohorts instead of running totals. Exclude partial periods, and look at seasonality, segment mixes that reverse a trend, the denominator and selection bias. Have the agent cite the metric file behind every claim in a draft, and have a person approve.

Safe by default

  • Read-only agent on a copy
  • Own login per person
  • Only the data you need
  • Personal details removed on request

Read-only roles, one named login per person per tool, secrets in a store, personal fields tagged, retention written per personal field and an erasure path. The rules guide the agent, and the access settings enforce them.

Each check will tell your agent to start read-only, propose changes for a person to approve, and never write to a live system. You can read each one now. None is released yet. Draft checks for your coding agent sit in steps 2, 3 and 4. The other two steps are judgment calls that a person owns.

Defaults: read-only credentials, changes only through pull requests a person approves, no writes to production.

What is a company brain, and what has to come first?

Everything you wrote down in steps 1 to 5 is your company brain.

A company brain is the shared record of definitions, decisions and context that your people and their agents read from. It only gives right answers when the data underneath was collected and counted right.

Keep it in one place, where people and agents read the same thing.

Goes wrong
An agent answers from a definition that changed last month, or from a guess, and the answer looks right.
Guardrail
Keep one shared place for definitions, decisions and context, with an owner and a date on each entry, and have agents read from it before they answer.
Check
Ask a colleague and your agent what counts as an active customer. If the answers differ, the record is missing or nobody reads it.

Keep the record as files in a repository: metric files, a short decision log and a note for each number that turned out wrong. Point agents at it in their instructions, and review every change by pull request.

Check your setup in 2 minutesBefore launchFirst paying customersGrowingGet new posts by email