goagentdata

Where did that number come from? A lookup order for your data agent

TL;DR. Make your data agent say where each number came from. Also make it hand a question to a named owner when it cannot answer from trusted material, instead of guessing. The source is one of three: an approved definition or saved calculation, a certified table, or raw data the agent combined itself. A fixed lookup order (definitions first, then approved queries, then certified tables, then raw tables) makes that label meaningful, and the handoff can turn an unanswered question into a new approved definition.

The same question, with and without a lookup order Left: without a lookup order, the agent joins two raw tables on its own and returns 412 active customers with no source. Right: with a lookup order, the same question walks down four rungs, definitions, approved queries, certified tables and raw tables, and stops at rung 2, where an approved query returns 371 with the label source: rung 2, active_customers, owner named. A second question, churn by channel, finds nothing on any rung, so the agent returns no number and sends the question and its search trail to the named owner. Illustration of the method, not measured data. No lookup order How many active customers? The agent joins two raw tables and picks its own definition. ordersbilling_export 412 Source: none given. Owner: none given. A confident number. Nobody can say where it came from. Lookup order Q1 How many active customers? The agent stops at the first rung that fits Q1Q2 1 Definitions 2 Approved queries 3 Certified tables 4 Raw tables 371 Source: rung 2 active_customers owner named Q2 Churn by channel? No rung fits. No number. The question and the search trail go to the named owner. The owner's answer becomes a new approved query. The label shows the path, not that the number is right.
Illustration of the method, not measured data.

What goes wrong when an agent answers without a lookup order?

Picture a startup with its first paying customers. The founder asks a chat assistant connected to the billing export how many active customers there were last month. It counts customers with an order and answers in seconds, test orders and failed payments included. The number looks right, so it goes into the investor update.

Costly data mistakes are often silent, and agents assume, fill gaps and sound confident. Before AI assistants, one analyst often held the definitions in their head and knew which dashboard to trust and whom to ask. That analyst was the lookup order. Now answers reach the business before anyone with context sees them. I think the fix is to write the analyst's habits down and make the agent follow them, while a person still decides which definition is right.

What is a lookup order?

Scout24 described, in an article dated 22 October 2026, a fixed search order for its internal data agent: glossary, then approved queries run exactly as written, then certified tables, then less curated tables, with unanswerable questions routed to the responsible analyst. It is one company's account, not a study, and reports no accuracy numbers.

For a small team, I think four plain rungs are enough. The agent starts at the top and stops at the first rung that fits.

  1. Definitions. What each word means (why tools disagree), what is excluded, who owns it.
  2. Approved queries. A query is a saved question to the database. Written once by the owner of the number, or an engineer they ask, and run as written.
  3. Certified tables. A short list the data owner has checked and signed. "None yet" is valid.
  4. Raw tables. Only when rungs two and three do not cover the question and every word in it is already defined. The answer says this is the weakest source.

Rung one holds no numbers, and every rung below has to respect it. Data teams use other names: a glossary or semantic layer, verified queries, certified datasets, raw tables.

Add one sentence to the rule: say which rung you used. Anthropic's data team does both at its scale: its agents must try the governed metric layer first, and every answer ends with a footer naming the source tier, the data's freshness and the data model's owner (3 June 2026). The label shows the path, not that the number is right.

What should a data agent do when it cannot answer?

The rule: when a word in the question has no written definition, or the data needed is not there, stop and write a message for the named owner with the question and the search trail. A person sends it. Give no number. If you are the owner, the handoff is a to-do for you: writing the missing definition is the task, then ask again.

I think small teams skip this because "I don't know" feels like failure. For a number that reaches investors or billing, it is usually the cheaper outcome, and the owner's answer becomes one entry in the definitions file, so the next person gets an answer from rung two.

A practitioner's write-up on Substack (29 October 2026) requires a data-team ticket for missing-data cases in its test set. It reports accuracy rising from 60 to 98.6 percent (98.9 in the title) on 200 cases the author wrote initially, at one company, graded by a second model. It describes no human check of that judge and no per-change figure, so it is not proof that these rules work.

What changes by stage?

I think stakes and team setup drive these rules more than stage does. Stage decides who does the work. Before launch the builder writes the definitions and owns the numbers. With first paying customers each number gets a named owner, often not the person asking. With a first data person, that person owns the file.

What if you have no repository and no coding agent?

This version is for a founder or manager using a chat assistant with a spreadsheet export, such as a payment provider download. Paste the paragraph into the project or custom instructions, with your own definitions (the company and the owner below are made up).

Your export may have no test flag. Define test accounts the way you would spot them: your own email domain, or a list of customer ids you keep up to date. The assistant cannot send anything, so forward the message it writes for the owner.

Before you answer any question about our company numbers, follow this order and stop at the first step that fits.
1. Our written definitions, pasted below. They hold no numbers. They say what each word in the question means and who owns it. They apply at every step: whichever step you are on, every term in the question must have a written definition.
2. Our approved calculations, pasted below. If one fits, follow it exactly. Do not change it or improve it.
3. Any sheet I have marked as certified. If none is listed below, skip this step.
4. Any other file I give you, only if steps 2 and 3 do not cover the question and every term in it has a written definition. Tell me this is the weakest source.
End every answer with one line: "Source: step N, name of the definition or calculation, owner." If you did any calculation of your own, say so on that line.
If a term in the question has no written definition below, or the data needed is not in the files, do not give a number and do not estimate. Instead write a short message for the owner, which I will send. It contains: my question word for word, what you looked at at each step and what you found, and what is missing. Then stop. When the owner answers, suggest the new definition for them to approve; do not add it yourself.

Definitions
- Paid order: an order with status "paid". Refunded, failed and pending orders are not paid.
- Test account: a customer whose email ends in @shop-internal.example, plus any customer ids I list here: none yet. Never count their orders.
- Active customer: a customer with a paid order in the last 90 days, excluding test accounts. For a month, the 90 days end on the last day of that month and include it. Owner: Dana Rosen (head of operations).
Approved calculations
- Active customers: count the distinct customer_id values among orders that are paid, not from a test account, and dated within the 90 days up to and including the report date.
Certified sheets: none yet.
Who to ask: Dana Rosen.

What goes wrong, and how do you catch it?

The agent answers from a query of its own. If the question is worded differently from the approved query's name, it can write a new query that counts test accounts and failed payments.

GuardrailRung two says run the approved query as written, and the answer names it.

CheckAsk the agent for its source line and compare the query it names with the one in the file. Once a quarter, the owner of the number recomputes it by hand or in a spreadsheet.

The agent fills a gap nobody wrote. In our test (below) neither session without the rule invented a churn rate, but both proposed churn wordings of their own, and one offered to compute churn once a definition was agreed.

GuardrailNo written definition, no number.

CheckAsk a question the files cannot answer. Expect a message for the named owner and no number. If nobody is named for that area, naming one is the first fix.

The label is trusted more than it deserves. A rung-two answer looks safe, but the query may be stale. Anthropic reports that accuracy on its own offline evals drifted from about 95 percent at launch to about 65 percent over a month, before the team treated the guidance files as engineering work (3 June 2026).

GuardrailThe owner approves every change, and the file carries a review date.

CheckAsk the owner when they last reviewed the query, and rerun a sample of past questions.

For your data team

For your data team

This part gets technical. The rule guides the agent and the access settings enforce it: give the agent a read-only database login and keep a log of each answer's source line. A person approves any change to the definitions file in a pull request.

Repository version. For a team with a repository and a coding agent such as Claude Code or Cursor. Save the rule where the agent reads it, with the definitions file beside the code.

# Lookup order for data questions

Stage: First paying customers. Business model: all. Version 1.1. License MIT.
You answer questions about company data. Use read-only access only. Never write to the database or to any file under the data folder.

## The four rungs, in this order

Start at rung 1. Move down only when the rung above cannot answer the question. Stop at the first rung that fits.

1. Definitions and glossary (`definitions.md`). They hold no numbers. They say what each word in the question means, which domain it belongs to and who owns it. Definitions apply at every rung: whichever rung you are on, every term in the question must have a written definition.
2. Approved queries (`definitions.md`, section Approved queries). If one fits, run it exactly as written. Fill in the parameters only. Do not improve it, and do not rewrite it.
3. Certified tables (`definitions.md`, section Certified tables). Use only tables listed there.
4. Raw tables. Use only when rungs 2 and 3 do not cover the question and every term in it has a written definition. Tell the reader this is the weakest rung.

## Every answer says which rung it used

End each answer with a line in this form:

`Source: rung <1 to 4>, <name of the definition, query or table used>, owner <name>.`

If you wrote any query yourself, say so on that line.

## Hand off, do not guess

Stop and hand the question off, and do not produce a number, when any of these is true:
- a term in the question has no written definition, whichever rung you are on;
- the data needed is not in the tables you can read;
- two written definitions disagree.

A handoff is a short message for the owner named in `definitions.md` (or the contact listed under Who to ask). You write it and a person sends it. It contains:
1. the question, word for word;
2. the search trail: what you looked at on each rung and what you found;
3. what is missing (a definition, a column or a decision).

Do not include a tentative number, and do not offer a rough estimate. When the owner answers, propose the new definition or query as an addition to `definitions.md` for the owner to approve. Do not edit the file yourself.
# Definitions and approved queries

Company: Shop Example (made-up). Last reviewed: 2 Nov 2026.
Database: shop.db (SQLite), tables `orders` and `customers`. Read-only.

## Glossary

### Paid order
An order whose `status` is `paid`. Statuses `refunded`, `failed` and `pending` are not paid orders.

### Test account
An order with `is_test = 1`. Test orders come from the internal QA accounts and are never counted as business activity.

### Active customer
A customer who placed a paid order in the last 90 days, excluding test accounts. The window ends on the report date (for a month, the last day of that month) and counts the 90 days up to and including it.
- Owner: Dana Rosen (head of operations). Dana approves any change to this definition.
- Approved query: `active_customers` below.

## Approved queries

### active_customers
Parameter: `:as_of`, a date in YYYY-MM-DD form. Run exactly as written. Fill in the parameter only.

select count(distinct customer_id) as active_customers
from orders
where status = 'paid'
and is_test = 0
and order_date > date(:as_of, '-90 days')
and order_date <= :as_of;


## Certified tables
None yet.

## Who to ask
- Anything about customers, orders and revenue: Dana Rosen (head of operations).

The test. We ran these once on made-up data (40 customers, 156 orders, no channel column, no churn definition), in fresh Sonnet sessions. Run A had the rule file and the database, B the same files without the rule, C CSV exports and the chat paragraph, and D the same exports and definitions without the paragraph. All four gave the same active customer count and none gave a churn number. A and C named their source and owner and wrote a handoff with the search trail. B and D gave no source label and wrote no message carrying the trail. This is one run each on made-up data, not evidence of a general effect.

Check it in 10 minutes

  1. Pick one number a decision depends on, such as active customers.
  2. Write its definition (meaning, exclusions, period, owner) and the approved query, or in the chat version the calculation in words.
  3. Add the rule file or the paragraph.
  4. Ask the question. Expect your approved query's number and a source line naming the rung and owner.
  5. Ask one question the data cannot answer. Expect no number and a message for the owner.

Related: testing answers, one definition per metric, forming your own view and context in general. Try the two minute self-check.

Text CC BY 4.0. Code and rules MIT.

Spotted a mistake or a newer source? Tell us and we check it against the source. How we work.

Check your setup in 2 minutesAll postsGet new posts by email