AI makes data answers cheap. Judging them is the new data work

What happens when an answer gets cheap?

Before an AI assistant, a data question had a price. You wrote a ticket, waited for the analyst, and often decided the question was not worth it. In my experience, many questions were never asked.

Here is how I think about it. An assistant drops that price close to zero. A head of sales can ask which region is losing deals at eleven at night and get a clear answer in seconds. So she asks more, and so does everyone else. It is not only people. Other agents and scheduled dashboards now send questions to the same assistant, and nobody reads each answer.

Economists call this pattern the Jevons paradox. In 1865, W. S. Jevons argued in The Coal Question that more efficient use of coal led Britain to burn more coal, not less. When something gets cheaper to use, people often use much more of it. Data answers may follow the same pattern. That is an argument, not a measurement.

The spreadsheet is the closest example. After it arrived, NPR's Planet Money reported that bookkeeping and accounting clerk jobs fell while accountant jobs grew. The report gives no numbers. I read it this way: the routine work shrank, and the work of deciding what the numbers mean grew.

What does AI actually remove?

I think it removes a lot, and pretending otherwise makes the rest of this argument weaker.

• Routine queries. "How many sign-ups last week" no longer needs a person who writes SQL.
• One-off reports. The spreadsheet someone builds for a single meeting and never opens again.
• Simple lookups. The ticket queue that kept a junior analyst busy.
• First drafts of analysis. A first cut at why a number moved, which a person then checks.

It also helps the data person: it can draft the definitions file and the checks described below.

If your company has one product, one billing system and a founder who asks a few questions a week, this may be most of your data work. For you, AI can mean you need less data help, and for longer.

What does it add?

Here is the part that does not get cheaper. Each item is a decision about your business, not a technical task.

• Definitions. Does a customer on a free trial count as a customer? Does a refunded sale count as revenue? The model can find the table, but it cannot know an answer your company never decided.
• Trusted sources. When billing and your CRM (the tool where sales tracks customers) disagree on revenue, someone has to decide which one is the trusted number.
• Checking answers. The head of sales cannot read the query behind her answer. A wrong answer looks exactly like a right one, so she forwards it to whoever owns the data and asks whether it is right.
• Access. Once anyone can ask, someone has to decide what the assistant is allowed to see.

Better models will pick the right table more often. They will not settle whether a trial user is a customer, because that is a choice, not a fact in the data.

LinkedIn's engineering team describes this work at scale. Teams there write down business definitions, exceptions and known data problems, and owners review them the way they review code.

So does the data work grow or shrink?

My view is that the shape changes more than the size. The typing shrinks and the judging grows. Whether the total grows depends on three things:

1. How many people ask. One founder asking creates little review work. A sales team of ten asking daily creates a lot.
2. How much money depends on the answers. A wrong number in a team chat costs little. A wrong number in a board deck or a pricing decision costs a lot.
3. How many definitions are contested. If sales, finance and product each count customers differently, every answer carries that dispute.

Ten people asking daily, or a number that goes to the board, is high. One founder asking weekly is low.

If all three are low, AI saves you data work, and you need to do nothing more than keep a short glossary. If one is high, name an owner for the judgment. If two or three are high, make that owner a named person with set hours each week. Without an owner, the review falls to whichever engineer knows the data best.

What does this look like at each stage?

I think the ownership should grow in this order.

Before launch. Few people ask and little money depends on the answers. Write a one page glossary: what a customer is, what an active user is, and which accounts are tests. Then ask the AI freely.

First paying customers. Money now moves, and sales, support and marketing start asking. This is usually when one of the three conditions turns high. Name an owner for each definition, choose the trusted source for revenue, and start a short file of real questions with answers you have checked. In my experience, a few hours a week of an experienced person, part-time or fractional, is often enough. That person gets the time by no longer taking routine requests the assistant now answers well.

Growing with a first data person. The data person owns the definitions, the question file and the review of answers. They also publish which questions the assistant handles reliably and which still need a person.

What goes wrong, and how do you catch it?

Nobody owns the definition. Two colleagues ask for active customers and get different numbers, because each phrasing led the assistant to a different meaning.
Guardrail: one named owner per definition, kept in a written file that the assistant must cite.
Check: have a few colleagues ask the same question in their own words. The numbers should match and name the same definition.

The review falls to an engineer by accident. Nobody trusts the replies, so each one goes to the person who knows the data, in addition to their main work.
Guardrail: a named reviewer, plus a short public list of question types marked trusted or needs review.
Check: ask that person how much of last week went to checking data answers. If they cannot say, nobody is watching the load.

A confident wrong number reaches a board meeting. Picture a deck whose revenue growth includes test accounts and refunds. The board plans hiring around growth that is not there.
Guardrail: a person traces every number that leaves the company to its definition and its source.
Check: before the next board meeting, trace a few numbers from the deck to a written definition and to billing. Remove any number you cannot trace.

# metrics/active_customer.yaml
name: active_customer
meaning: account with at least one paid invoice dated in the calendar month (invoices.paid_at)
counts: status = paid and amount > 0; excludes is_test and is_internal
period: calendar month, UTC
source: billing_export.invoices
owner: <named person>
version: 3
changed: v3, refunded invoices no longer count as paid
restated: no, past months stay as reported
decided_by: <who settled the dispute, and when>

Check it in 10 minutes

1. Ask your assistant how many active customers you had last month. Ask again in different words. Compare the numbers.
2. Ask where "active customer" is defined and who decided it. If nobody can say, you have found your first job.
3. Score your company on the three conditions: how many people ask, how much money depends on the answers, how many definitions are contested.
4. Pick one number from your last board or investor update and trace it to billing.

No assistant yet? Do steps 2 to 4.

See where your setup stands with the two minute self-check. Then read the draft checks your coding agent can run.