PJ-02 · Product Judgment
Define good
Teams cannot trade off intelligently until the desired change and its quality bar are explicit.
This lesson builds on PF-02 Problem, mechanism, alternatives, PF-05 User and context boundary.
a goal that cannot be missed or ranked
A team spends three weeks improving onboarding. At review, half the room thinks it shipped well and half thinks it is not ready. Both sides point at the same screens.
The argument is not about the screens. It is about a sentence nobody wrote. The goal was "make onboarding better." Better is a direction. A direction has no end, so every person supplies their own end, and the review becomes a contest between private standards.
Now look at what this does to prioritization. The team also has a request for enterprise summaries and a performance problem in large workspaces. Someone asks which one matters more. Nobody can answer, because none of the three has a stated change or a stated bar. You cannot compare three directions. You can only compare three thresholds.
The failure is hard to see because the team is not confused about what to build. They are confused about what would count as having done it. Those look identical from inside a sprint, and completely different eight weeks later.
A direction does two kinds of damage at once, and they arrive at different times — one in the review, one in the prioritization meeting nobody connects back to it:
flowchart LR
G["Goal stated as a direction<br/>'make onboarding better'"] --> P["Each person supplies<br/>a private end"]
P --> R["Review becomes a contest<br/>of standards"]
G --> C{"Rank it against summaries<br/>and large-workspace P95?"}
C -->|"three directions,<br/>nothing to compare"| N["Prioritization stalls"]
G -.->|"what closes both"| T["Threshold<br/>amount · population · horizon"]
T --> D["A bar that can<br/>lose an argument"]The dashed edge is the whole lesson. One sentence, written before the work, removes both failures.
Noted has a live example. Activation is defined as creating a document and returning within seven days. Nobody defends this definition. So when activation falls 11%, no one can say whether that is a serious change in behaviour or a change in a number that never described the behaviour anyone cared about.
Ask your team: if this ships and works, what specifically is different, for whom, and how much of it is enough? If the answers vary across the room, you do not have a definition of good.
turn the direction into a threshold
"Good" is three separable statements. Teams usually write the first one, argue about the second, and forget the third.
| Part | The question it answers | What breaks when it is missing |
|---|---|---|
| Desired change | What behaviour or outcome is different, for which population, over what horizon | Work gets judged by what shipped, not by what changed |
| Quality bar | How much of that change is enough to be worth the cost | Endless polish, or shipping something too small to matter |
| Floor | What must not get worse in order for the change to count | A local win that creates a larger loss somewhere else |
Direction versus threshold
A direction is a verb with no end: increase, improve, reduce, simplify. A threshold is a direction plus a number and a horizon.
| Statement | Type | Can it be traded off? |
|---|---|---|
| "Improve activation" | direction | No — it never completes, so it always deserves more |
| "Reduce time to first useful document" | direction | No |
| "Recover half of the 11% decline, in the individual knowledge worker population, within one quarter" | threshold | Yes |
Only the third can lose an argument to another priority. That is the point. A direction cannot be outranked, which is why teams that speak in directions never finish a prioritization conversation.
The bar is set by consequence, not by ambition
How good is good enough depends on two things you already have from Phase 01:
- Who bears the cost of being wrong, from your decision statement.
- How reversible the work is. A reversible change can ship at a lower bar, because the correction is cheap. A one-way change needs a higher bar, because the correction is not available.
A team that applies the same bar to everything is either over-building reversible work or under-building irreversible work. Usually both at once.
Both versions below are about Noted's activation decline. Both sound like goals:
Weak — "Improve activation back to where it was."
A direction with a destination attached. It still has no population and no horizon, so it cannot be missed, and it cannot lose an argument to the large-workspace performance work. It also inherits a definition nobody on the team defends.
Strong — "Recover half of the 11% decline, in the individual knowledge worker population, within one quarter. Floors: small-team paid conversion does not fall, and the share of created documents that are reopened later does not drop."
The difference is not ambition. The strong version can be missed, in public, by an amount someone else can compute — and one of its floors protects a population that is not the one it is optimising.
The fraction — half — is chosen, not measured. The case gives you the 11%. It does not tell you how much of it is worth buying back, or at what cost. Step 6 of the Build asks you to defend that choice out loud.
The desired change needs a population
PF-05 gave you audience as a behaviour-and-context boundary. Apply it here. "Improve activation" across all of Noted is three different definitions of good, because individual knowledge workers, small teams, and enterprise team leads do not do the same thing in a first session. A single aggregate bar hides which population is moving and lets a gain in one hide a loss in another.
Boundary
Some work legitimately has no numeric bar in advance.
When you are building a capability nobody has used before, you often cannot know what good looks like until you watch a person use it. Demanding a threshold there produces an invented number, and invented numbers are worse than absent ones, because they get defended.
The honest move is not to skip the definition. It is to name the bar-setting moment: what you will observe, when, and what will convert the observation into a threshold. "We will set the bar after ten sessions with team leads, using whether they returned to the document in the following meeting" is a definition of good. "We will know it when we see it" is not.
Both of the left-hand paths below are legitimate. Only the right-hand one is a failure, and it is the one that feels most like open-mindedness:
flowchart TD
W["Work about to start"] --> Q{"Has anyone used this<br/>capability before?"}
Q -->|"yes"| T["Write the threshold now<br/>amount · population · horizon"]
Q -->|"no"| B["Name the bar-setting moment<br/>what you watch, and when"]
B -->|"observation converts<br/>into a number"| S["Threshold set — and you<br/>say that you tightened it"]
Q -->|"no, and the moment<br/>is never named"| F["'We will know it<br/>when we see it'"]
F --> N["No definition of good"]The difference between the two "no" branches is a single sentence written in advance. Without it, the work arrives at review with nothing to be judged against, which is where this lesson started.
The second limit: a definition of good that is too tight too early kills learning. If you set a threshold on exploratory work and the work misses it, you will usually learn that the threshold was wrong, not that the work was. Tighten the bar as the mechanism becomes clearer, and say each time that you have done it.
The third limit is about what this artifact is not. A threshold with floors looks like an experiment success criterion, and it is not one. Turning it into something a test could settle needs a baseline, a sample large enough to detect the effect you care about, and a validity check — separate work, covered in EV-06. A definition of good tells you what you are trying to buy. It does not tell you whether you could measure having bought it.
Practice on the Noted decision case
Read the Noted case if you have not already.
Take Signal 1: activation fell 11% over six weeks.
Activation is currently defined as creating a document and returning within seven days. Nobody defends the definition. The decline has not been segmented. Two changes shipped inside the window.
Take the inherited definition apart before you write a bar on top of it. Each clause counts something and misses something else:
| Clause in the current definition | What it counts | What it leaves out |
|---|---|---|
| Creating a document | That a document was started | Whether anything was produced that was worth keeping |
| Returning within seven days | A second visit inside a week | Whether the document was reopened, or only the product |
| Applied to everyone | One aggregate number | Which of the three populations moved, and in which direction |
Noted's job is capture, direct, produce. Read the middle column against that job: it is a record of activity, and the right-hand column is where the job lives. Your definition of good has to name which column you are buying.
Your task. Write a definition of good for Noted's activation, and be ready to defend every part of it.
- State the behaviour you actually care about, in plain words, before naming any metric. Noted's job is capture, direct, produce. Say which part of that job a newly activated user has completed.
- Say what the current definition includes that should not count, and what it excludes that should. Creating a document and returning is a measure of activity. Say whether it is a measure of the job.
- Pick one population, using PF-05 boundaries. Do not write a bar for "users."
- Write the desired change as a threshold: the amount, the population, the horizon.
- Write two floors — things that must not get worse for the change to count. At least one floor should protect a different population from the one you chose.
- State the bar's justification: why this number and not one half its size. "It felt right" is a legitimate answer only if you say it out loud.
- Say what your definition deliberately does not measure, so a reader does not assume more coverage than you are claiming.
Expect to be pushed on: whether your threshold was reverse-engineered from the 11% figure rather than from the job the product does; whether your population is a behaviour-and-context boundary or a plan tier in disguise; and whether your floors are real constraints or things you were never at risk of breaking.
Committed to a position? Check it against what a strong answer holds
- The behaviour is named before any metric, and placed on capture → direct → produce. An activated user has produced something and come back to it — not merely created a document and come back to the product.
- The inherited definition is taken apart in both directions: what it counts that should not (an empty document, a return to any page) and what it misses that should count (the document reopened, something kept). The answer says plainly that the current metric measures activity, not the job.
- One population from PF-05, with a reason for choosing it over the other two. Not "users", and not a plan tier.
- The threshold has all three parts — amount, population, horizon — and the justification for the amount is not "it gets us back to where we were". The answer says why this number and not half of it, even if the honest reason is judgment.
- Two floors, one of which protects a population other than the one being optimised, and each of which is something the intended change could actually break. A floor nothing threatens is decoration.
- The most common weak move is reverse-engineering the bar from the 11% figure. It is weak because it inherits a definition nobody on the team defends, so hitting the bar would prove only that an undefended number moved back.
Transfer the move to your product
Take one piece of work your team is currently doing, or is about to start.
Answer five questions:
- What behaviour or outcome will be different, for which population, over what horizon?
- How much of that change is enough to justify the cost, and why that amount?
- What must not get worse for the change to count?
- Is the work reversible? If yes, say why your bar is not lower. If no, say why your bar is not higher.
- What does your definition deliberately not measure?
Where you have no answer, write <unknown>. A missing threshold is a real
finding — it usually means the work was approved on a direction, and the
argument about whether it succeeded is still ahead of you.
Keep one reusable artifact
Definition of good
Write it for the next person who must make or review the decision. Include the choice, evidence, uncertainty, alternatives, owner, and revision trigger. Link it into your cumulative Product Decision Case.
Produce artifacts/PJ-02-definition-of-good.md using the template below.
Write it for the person who will review this work when it ships, including the version of you that will be tempted to move the bar afterwards. A definition of good written before the work is a commitment. The same words written after are a description.
This extends your Product Decision Case. PJ-01 committed you to a position. This artifact states what that position is buying, in an amount someone else can check.
artifacts/PJ-02-definition-of-good.md# Definition of good · <short name of the work> - **Date:** <YYYY-MM-DD> - **Product:** <product> - **Decision this supports:** <the decision or position this work serves> - **Owner of the consequence:** <person or role> - **Reversibility of the work:** <reversible | costly to reverse | one-way> ## The job behind the metric <In plain words, before any metric: what does the user get to do that they could not do before, or do reliably that they previously did by accident? If you cannot write this without naming a metric, the metric is standing in for a job nobody has stated.> ## Desired change - **Behaviour or outcome that changes:** <what is different> - **Population:** <a behaviour-and-context boundary, not a plan tier or a demographic> - **Horizon:** <over what period this change is expected to appear> ## Quality bar - **Threshold:** <the amount of change that is enough — a number and a horizon> - **Why this amount:** <what makes this worth the cost, and not half of it> - **Why the bar is not higher:** <what a higher bar would cost you> - **Why the bar is not lower:** <what a lower bar would fail to prove> <If you genuinely cannot set a threshold in advance, delete the four lines above and use the block below instead. Do not do both.> **Bar-setting moment (exploratory work only):** - **What will be observed:** <the behaviour, not the sentiment> - **When and with whom:** <count, population, date> - **What converts the observation into a threshold:** <the rule you will apply> ## Floors Things that must not get worse for the change to count. | # | What must not get worse | Population it protects | How it will be checked | |---|---|---|---| | 1 | <floor> | <who> | <observable> | | 2 | <floor — protecting a different population from the one above> | <who> | <observable> | ## Deliberately not measured <What this definition does not cover, so a reader does not assume more coverage than you are claiming. Naming this prevents the number being quoted later as if it settled something it never touched.> ## What the current definition gets wrong <If you are replacing an inherited definition, say what it counted that should not count, and what it missed. If you are keeping it, say why it survives.> ## Revision trigger <The observation and threshold that would make you change this definition of good — not the work, the definition. Definitions drift silently, and a definition that is never revisited becomes an inherited one for the next PM.> ## Gaps <Everything marked `<unknown>` above, listed as open questions with who could close them.>
Before you call it done
Six things a reviewer must be able to find. If one is missing, the artifact is not finished — it is a draft with a hole in it.
- Threshold: <the amount of change that is enough — a number and a horizon>
- Why this amount: <what makes this worth the cost, and not half of it>
- Why the bar is not higher:
- Why the bar is not lower:
<If you genuinely cannot set a threshold in advance, delete the four lines above and use the block below instead. Do not do both.>
Bar-setting moment (exploratory work only):
- What will be observed: <the behaviour, not the sentiment>
- When and with whom: <count, population, date>
- What converts the observation into a threshold:
Floors
Things that must not get worse for the change to count.
| # | What must not get worse | Population it protects | How it will be checked |
|---|---|---|---|
| 1 | |||
| 2 | <floor — protecting a different population from the one above> |
Deliberately not measured
<What this definition does not cover, so a reader does not assume more coverage than you are claiming. Naming this prevents the number being quoted later as if it settled something it never touched.>
What the current definition gets wrong
<If you are replacing an inherited definition, say what it counted that should not count, and what it missed. If you are keeping it, say why it survives.>
Revision trigger
<The observation and threshold that would make you change this definition of good — not the work, the definition. Definitions drift silently, and a definition that is never revisited becomes an inherited one for the next PM.>
Gaps
<Everything marked <unknown> above, listed as open questions with who could
close them.>
## Quality bar
Before saving, check all six exposures are present. Name any that are missing
rather than filling them in yourself:
| Exposure | Present when |
|---|---|
| Choice | The threshold is a number and a horizon, not a direction |
| Evidence | The job behind the metric is stated, and the threshold's justification is visible |
| Uncertainty | Either the bar is defended against being higher and lower, or a bar-setting moment is named |
| Alternatives | What the definition deliberately does not measure is stated, and floors name what could be lost |
| Owner | The consequence owner is named, and the population is a behaviour-and-context boundary |
| Revision trigger | A condition that would change the definition itself, not just the work |
Can the reasoning survive these tests?
Answer before you open the explanation. The tutor asks these one at a time and does not show the options’ verdicts until you commit.
01Three initiatives are on the table: improve activation, reduce large-workspace response time, and add meeting summaries. Each is described as a direction with no threshold. What is the practical consequence?
- AThe team will pick the one with the clearest user demand
- BNone of the three can lose the argument, so the choice is settled by who advocates hardest
- CThe team will over-invest in whichever one is technically easiest to start
- DThe team will need a scoring model to rank them
Reveal the answer
B. A direction never completes, so it always deserves more effort and can never be outranked. Option D is the answer a PM gives after learning prioritization vocabulary — but scoring three directions just moves the same undefined comparison into a spreadsheet and gives it a number.
02Noted defines activation as creating a document and returning within seven days. What is the strongest objection to using this definition to judge the 11% decline?
- ASeven days is too short a window for a document tool
- BThe definition has never been validated against a control group
- CAggregate activation should always be segmented before it is used
- DIt measures activity, and nothing establishes that the activity corresponds to the job the product does
Reveal the answer
D. Noted's job is capture, direct, produce. Creating a document and returning can happen without a document ever becoming useful. Option C is tempting and true as far as it goes, but segmenting a definition that does not describe the job just gives you the same wrong measure split four ways.
03A PM writes: "Good means recovering 60% of the lost activation within one quarter." What is the most important thing still missing?
- AThe population the threshold applies to, and what must not get worse for the recovery to count
- BA confidence interval on the 60% figure
- CThe engineering estimate required to hit it
- DAgreement from the founder that 60% is the right target
Reveal the answer
A. A threshold without a population lets a gain in one group hide a loss in another, and a threshold without floors lets the number be hit in a way that costs more than it earns. Option C is tempting because cost matters, but cost belongs in the trade-off, not in the definition of what good is.
04Noted is about to build its first AI-first onboarding. Nobody has watched a user go through it. What does defining good require here?
- ASet a threshold from the current activation baseline, then adjust it after launch
- BSkip the definition, since exploratory work is judged on learning rather than outcomes
- CName the bar-setting moment: what you will observe, when, and what converts the observation into a threshold
- DUse the enterprise team lead population, because their behaviour is the most consequential
Reveal the answer
C. This is the boundary. You cannot know the bar before you watch the behaviour, but "we'll know it when we see it" is not a definition. Option A is the disciplined-sounding trap: a number invented from an unrelated baseline gets defended later as if it had meant something.
05Two changes are proposed. One is a copy change on the empty document state, trivially reversible. One is a change to how documents are stored, costly to undo. A PM applies the same evidence bar to both. What is wrong?
- ANothing — a consistent bar is what makes decisions comparable across the roadmap
- BThe bar should follow reversibility, so the copy change is being over-built and the storage change under-built
- CThe storage change should not be a product decision at all
- DThe copy change needs a higher bar, because it touches the surface every new user sees
Reveal the answer
B. How good is good enough depends on what it costs to be wrong and whether the correction is available. Option A is the trained answer — consistency sounds like rigor, but a uniform bar means you are simultaneously over-testing cheap reversible work and under-testing the work you cannot take back.