Sales leaders often describe discovery notes as good or bad without saying what they mean. The judgment is real, but it is difficult to audit, and two managers can look at the same note and reach opposite conclusions. A completeness study replaces the general impression with a field-level score and, more importantly, with a check on whether the score is reproducible.

This paper describes a method for scoring sales note completeness using a codebook, independent reviewers, and an agreement statistic. It does not report a result from a local dataset. It describes how such a study should be designed and what its limits are, so that a team can run one without overstating what it learned.

Contents

The question and the unit

The question is not "are our notes good?" It is: for a declared population of discovery calls, what share contain the information another reviewer needs to explain the opportunity, and can two trained reviewers agree on that judgment?

The unit of analysis should be the call record, not the note sentence. A call may have several notes, a transcript, and a set of fields. The score should describe the record as a whole, because that is what the next representative sees when they open the opportunity.

The population should be declared by date, team, meeting type, and record status. A study of new-business discovery calls is not comparable with a study of renewal calls. If the team wants both, run two samples and report them separately.

Build the codebook before scoring

A codebook defines each field, what counts as complete, what counts as partial, and what counts as absent. It should include examples from real notes, with names and sensitive details removed. The examples matter more than the definitions, because reviewers interpret prose through analogy.

For each field, specify the minimum content. A "next step" is complete when it names an action, an owner, and a date. It is partial when it names an action but no date. It is absent when there is no next step. That three-level scale is more informative than a binary present or missing.

The codebook should also define what not to score. Grammar, length, and writing style are not completeness. A short note can be complete, and a long note can be empty of decisions. Keeping style out of the score protects the study from becoming a writing assessment.

Sample calls, not sentences

Select calls at random from the declared population, stratified by representative and by meeting type if those groups are likely to differ. A sample of thirty to fifty calls is enough for a local operational study; a larger sample supports narrower estimates.

Do not let representatives volunteer their best notes. The sample should come from the system, with a fixed inclusion rule. If a call has no note at all, it belongs in the sample as an absent record, because excluding empty records inflates the score.

Record the sampling frame, the selection method, and the number of records excluded. An excluded record needs a reason, such as a cancelled meeting or a test entry. The reasons should be few enough that a reader can see the population was not curated.

Keep reviewers independent

Two reviewers should score the same sample without seeing each other's results. Independence is what makes the agreement statistic meaningful. If one reviewer scores first and the other adjusts, the agreement measures conformity rather than clarity.

Train both reviewers on the codebook using examples that are not in the sample. Then let them score the sample separately. Record the scores before any discussion, including the reviewer identifier and the timestamp.

The reviewers should be able to mark "cannot determine" when the note is ambiguous. That option is important. Forcing a binary choice on an ambiguous record produces false confidence and hides a codebook problem that should be fixed.

Calculate agreement, not just completion

Report the completeness percentage and the agreement between reviewers. A simple percentage agreement is easy to understand, but it can be inflated by fields that are almost always absent. A chance-corrected statistic such as Cohen's kappa is more informative for two reviewers and a categorical scale.

The calculation should be done per field and for the overall record. Per-field agreement shows where the codebook is weak. Overall agreement shows whether the record-level score can be trusted. If one field has low agreement, the overall score should be reported with that weakness visible.

Do not treat a single threshold as universal. The acceptable level of agreement depends on how the score will be used. A score used for coaching can tolerate more disagreement than a score used in a compensation decision or a customer commitment.

Resolve disagreement without erasing it

After the independent scores are recorded, the reviewers should discuss the cases where they disagreed. The goal is to understand why, not to force a consensus that makes the statistic look better. The discussion should produce either a codebook change or a note that the case is genuinely ambiguous.

Keep a disagreement log with the record identifier, the field, both scores, the reason for the difference, and the resolution. This log is the most useful output of the study. It shows which definitions are unstable and which examples are missing.

If a codebook change is made, do not rescore the original sample with the new codebook and report the result as if it were the original study. Either report both versions or draw a new sample. Retrospective rescoring hides the learning that the disagreement produced.

Report the score with its uncertainty

A completeness score should be reported with the sample size, the population, the codebook version, and an interval. A point estimate without an interval invites false precision, especially when the sample is small.

Report the field-level results as well as the overall score. A team may have strong next steps and weak decision criteria, and the overall number hides that pattern. The field-level table is usually more actionable than a single percentage.

Describe the practical meaning of the result. A score of seventy percent does not tell a sales manager what to change. A statement that most notes name a next action but fewer than half name a date gives the manager a specific behaviour to coach.

Watch for gaming and construct drift

Once a completeness score becomes a target, representatives will optimize for the score. That can be healthy if the codebook reflects useful information. It can also produce notes that fill every field with minimal content, which raises the score without improving the handoff.

Guard against this by sampling for usefulness as well as presence. Ask a reviewer who did not attend the call to explain the opportunity from the record. If they cannot, the fields may be present but the note is not complete in any meaningful sense.

Construct drift is the slower problem. The business changes, the fields stay the same, and the score stops measuring what matters. Review the codebook at least annually and whenever the sales process changes. The NIST Cybersecurity Framework 2.0 describes continuous improvement as part of governance, and a measurement instrument needs the same discipline.

Limits of the method

The method measures what the codebook defines, not the quality of the conversation. A complete note can describe a poorly qualified opportunity, and an incomplete note can describe a strong one. Completeness is a necessary condition for a useful handoff, not a sufficient one.

The method is also labour intensive. Two reviewers scoring fifty calls is a real cost, and the cost grows with the number of fields. A team should start with the fields that downstream users actually read, then expand only if the study changes a decision.

Regulated recordkeeping regimes make a similar demand in other domains. The DOL fact sheet on FLSA recordkeeping requires employers to keep accurate records of hours and pay, and the NARA records management policy sets retention expectations for federal records. Neither governs sales notes, but both illustrate the underlying principle: a record is useful only when it is accurate enough and retained long enough to be relied on.

The NIST SP 800-53 audit and accountability controls are relevant to the evidence trail around the score, not to the content of the note. The NIST Privacy Framework is relevant because discovery notes can contain personal information that should be minimized and protected. The BLS occupation page and the SBA management guide provide role and operating context, not a standard for note quality.

Sources

  1. NIST SP 800-53 Rev. 5. https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final
  2. NIST Security and Privacy Controls. https://www.nist.gov/publications/security-and-privacy-controls-information-systems-and-organizations
  3. NIST Cybersecurity Framework 2.0. https://www.nist.gov/cyberframework
  4. NIST Privacy Framework. https://www.nist.gov/privacy-framework
  5. NARA records management policy. https://www.archives.gov/records-mgmt/policy
  6. DOL FLSA recordkeeping fact sheet. https://www.dol.gov/agencies/whd/fact-sheets/21-flsa-recordkeeping
  7. SBA, Manage your business. https://www.sba.gov/business-guide/manage-your-business
  8. BLS, Wholesale and manufacturing sales representatives. https://www.bls.gov/ooh/sales/wholesale-and-manufacturing-sales-representatives.htm