CRM duplicate rate is one of the most quoted and least defined measures in sales operations. A team reports that four percent of accounts are duplicated, another reports eleven percent, and a third reports that duplicates are not a problem. The three numbers may all be correct, because each was produced by a different matching rule.

This paper describes an observation protocol that makes the rule visible. It separates candidate generation from adjudication, estimates both false positives and missed pairs, and reports the result with the population and the standard used. The aim is not a universal duplicate rate. It is a number that another analyst can reproduce and a reviewer can challenge.

Contents

The question behind a duplicate rate

A duplicate rate can answer several different questions. It can estimate how many records represent the same real-world entity, how much work a cleanup would create, or how often a matching rule fires. These are related but not identical, and a report should say which one it addresses.

The entity question is the most common: what share of account records refer to the same organisation? The work question is narrower: how many merge decisions would the team have to make? The rule question is technical: how often does the current duplicate detection job produce a candidate pair?

Each question implies a different denominator. For the entity question, the denominator is the account population. For the work question, it is the candidate queue plus the duplicates the rule misses. For the rule question, it is the pairs the rule evaluated. Mixing denominators produces a number that cannot be interpreted.

Define the unit and the population

The unit is usually the account record, but the definition of an account matters. A holding company and its subsidiary may be one commercial entity for some purposes and two for others. A franchise may share a brand and a domain while operating as a separate buyer. The protocol should define the entity boundary before counting.

The population should be declared by creation date, status, and source. Active accounts created in the last three years is a different population from all accounts including inactive and imported records. If the goal is cleanup workload, include the inactive records. If the goal is current data quality, restrict the population and say why.

Exclusions need reasons. Test records, deleted records, and records with incomplete mandatory fields can be excluded, but each exclusion changes the denominator. Report the count excluded and the effect on the rate. An exclusion applied silently is indistinguishable from a convenient result.

Generate candidates with declared blocking keys

Record linkage at scale rarely compares every pair. It uses blocking keys to reduce the comparison space, such as a normalised domain, a postal code prefix, a phone suffix, or a company name token. The choice of blocking key determines which duplicates can be found and which cannot.

A domain block will miss duplicates where one record uses a free email address and the other uses a corporate domain. A name block will miss abbreviations and trading names. A phone block will miss records with no phone. The protocol should list the blocking keys and the known blind spots before the count is produced.

The blocking rule should be versioned. When a key is added or a normalisation changes, the candidate set changes, and the historical rate is no longer directly comparable. The version identifier belongs in the report, alongside the date range and the population definition.

Adjudicate without a hidden standard

A candidate pair is not a duplicate until a person decides it is. The adjudication standard should be written down with examples: same legal entity and same buying unit, parent and subsidiary, shared domain but separate entities, same contact at two companies, and uncertain.

Two reviewers should be able to reach the same decision on the same pair. If they cannot, the standard is incomplete. The fix is usually a sharper entity definition or an additional example, not more training on an ambiguous rule.

Record the adjudication for each sampled pair, including the reason and the reviewer. The reason codes make the result explainable and allow a later analyst to see which rule produced the false positives. Without them, the only output is a percentage that nobody can audit.

Estimate precision and recall

A duplicate count based only on the candidates that were reviewed measures precision, not prevalence. It says how many of the flagged pairs were true duplicates. It says nothing about the duplicates the rule never flagged.

Estimating recall requires a second pass with a different method. One option is to sample records and search for possible matches using a broader key, then adjudicate the results. Another is to compare the current rule with a more permissive rule and review the additional pairs. Both approaches cost time and both produce an estimate with uncertainty.

The protocol should report precision and recall separately, or state clearly that only precision was measured. A single "duplicate rate" with no recall estimate is a lower bound, not a prevalence estimate. Presenting it as the true rate overstates confidence.

Design the sample

Sampling should be stratified by the characteristics that affect matching: source system, creation period, country, and whether the record has a domain, phone, and postal address. A simple random sample can miss a small but badly duplicated import.

For precision, sample from the candidate queue. For recall, sample from the population and search each sampled record for potential matches. The two samples answer different questions and should not be combined into one number without explanation.

State the sample size and the selection method. A sample of fifty pairs can support a rough operational estimate but not a precise percentage for a large database. The report should give an interval or, at minimum, describe the uncertainty in plain language.

Record the merge decision

The observation protocol should connect to the merge process. Each adjudicated duplicate that is merged needs a record: the survivor, the losing record, the matched fields, the reviewer, the date, and the field-level decisions. This record is the evidence that the rate is being acted on.

The merge record also supports a later recheck. If a rule is changed, the team can compare the old decisions with the new candidates and see whether the change improved precision or merely moved the queue. The NIST Security and Privacy Controls treat auditability as a control property, and a merge decision is exactly the kind of event that should be reconstructable.

Keep the losing record in an archive rather than deleting it. A duplicate merge that later proves wrong is much easier to reverse when the original values are available.

Report the result with its rule

A useful report contains the population, the blocking keys, the normalisation rules, the adjudication standard, the sample design, the precision estimate, the recall estimate or its absence, and the date range. It also contains the operational recommendation: which rule to change and what evidence supports the change.

The report should avoid a single headline percentage unless the caveats fit beside it. A statement such as "the duplicate rate is four percent" invites comparison with a report that used a different rule and a different entity definition. A statement such as "four percent of active accounts created since 2024 matched the v3 domain-plus-phone rule, with an estimated precision of..." is harder to misread.

The NIST Privacy Framework frames data handling as a risk management activity, which is relevant because duplicate records often carry different consent states and different contact histories. The merge decision is not only a data quality decision; it can change what the organisation is permitted to do with a contact.

Limits and governance

No duplicate rate is permanent. Sources change, imports introduce new patterns, and the sales team creates records under time pressure. A rate measured once is a baseline, not a control. The protocol should specify a recheck cadence and the events that trigger an earlier review, such as a new data source or a CRM migration.

The NARA records management policy and the SBA management guide are not duplicate-detection sources. They support the surrounding governance: records need retention rules, and small businesses need documented operating processes. The BLS sales occupation page describes the role context in which records are created, not the quality of those records.

The main limitation is that adjudication is human and therefore contestable. The protocol reduces the contest by making the standard explicit, but it cannot eliminate it. A reviewer who disagrees with the entity definition will produce a different rate, and the report should make that disagreement visible rather than averaging it away.

Sources

  1. NIST Privacy Framework. https://www.nist.gov/privacy-framework
  2. NIST Security and Privacy Controls. https://www.nist.gov/publications/security-and-privacy-controls-information-systems-and-organizations
  3. NIST Cybersecurity Framework 2.0. https://www.nist.gov/cyberframework
  4. NARA records management policy. https://www.archives.gov/records-mgmt/policy
  5. SBA, Manage your business. https://www.sba.gov/business-guide/manage-your-business
  6. SBA, Strengthen your cybersecurity. https://www.sba.gov/business-guide/manage-your-business/strengthen-your-cybersecurity
  7. BLS, Wholesale and manufacturing sales representatives. https://www.bls.gov/ooh/sales/wholesale-and-manufacturing-sales-representatives.htm
  8. CISA Secure by Design. https://www.cisa.gov/securebydesign