Skip to main content
Entity resolution is the process of deciding whether two or more records describe the same real-world person, company, or place. It’s the foundation for deduplicating your data and for linking your records to other datasets, including PDL’s. Real-world data is messy. Typos, inconsistent formats, nicknames, and missing fields are the norm. No single technique handles all of it, so effective matching pipelines layer several together. Resolving records isn’t the same as merging them:
  • Record linking establishes that records refer to the same entity, usually by assigning them a shared entity ID.
  • Record merging combines linked records into one consolidated representation.
You can link without merging. Many teams keep the original source records, their provenance, and any conflicting values, so they can audit or undo a link later.

Three categories of techniques

This guide covers 11 techniques in three groups.

Prepare your data

Clean, normalize, and structure your records so you can compare them reliably.

Match on identity signals

Find matching entities with shared identifiers and combinations of identifying attributes.

Match on similarity

Find potential matches when records are incomplete, inconsistent, or approximate.

Most matching pipelines follow the same general shape. Treat this as a starting point, not a fixed sequence. Not every pipeline needs every technique. A few principles apply at every step:
  • Start simple. Use reliable, inexpensive techniques such as exact identifier matches first. Save sophisticated techniques for the records that remain unresolved.
  • Treat every signal as evidence, not proof. A shared name, address, phone number, or domain can be a coincidence, and so can a high similarity score. Link records when several independent signals agree and none conflict.
  • Set decision rules on purpose. Choose thresholds based on how much a wrong match costs you. See Evaluating match quality.
  • Review the middle. Send borderline matches to review rather than linking them automatically.

Worked example

Here’s how the techniques build on each other for two records that describe the same person.
1

Standardize

Both phone numbers normalize to +14155550132. Company names normalize to acme once you lowercase them and set aside legal suffixes. See Standardization.
2

Map nicknames

“Bob” is a common nickname for “Robert,” so the first names are compatible. See Nicknames.
3

Evaluate the signals

No single signal here proves identity. “Smith” is common, and phone numbers get reassigned. The combination is what makes the link credible.

Technique quick reference

Find your data problem, then jump to the technique. The roles are typical uses, not the only ones.

Part 1: Prepare your data

1. Data cleaning and preprocessing

Remove duplicate rows and fix obvious errors before you try to match anything. Otherwise, the same mistake carries into every later step. Example dataset Issues
  • IDs 1 and 3 are duplicate rows for Alice Johnson.
  • ID 5 has a typo: “Bbo Smith” should be “Bob Smith.”
Solution
  • Remove rows that are identical on every field, such as IDs 1 and 3.
  • Prefer linking over deleting when rows share only some values, such as the email on IDs 2 and 5. Different people can share an inbox.
  • Flag likely typos for manual review instead of correcting them automatically. Many names aren’t dictionary words, so a spell-checker can “fix” a name that was already correct.
IDs 2 and 5 aren’t duplicate rows. They’re two rows that probably describe the same entity, and the shared email is what reveals it. Matching on email catches a typo that name matching alone would miss. Keep the original values alongside your cleaned ones. You’ll need them to audit matches and to recover from a correction that turns out to be wrong.

2. Standardization and normalization

The same value can be written many ways. Convert everything to one consistent format so you can compare records directly.
  • Phone numbers: convert to E.164 format. Numbers without a country code need a default region, so record which region you assumed.
  • Addresses: use consistent abbreviations and country codes. For example, “123 Main Street, Anytown, US” and “123 Main St., Anytown, U.S.A.” become the same value.
The phonenumbers Python library handles parsing and formatting:
Normalize phone numbers
Output
phonenumbers.parse raises NumberParseException on values it can’t parse, so catch it and keep the raw value for review.
Avoid transformations that throw away meaningful differences. Stripping “Apt 12B” from an address makes two different households look identical. Standardize into a new field and keep the original. For addresses, use an address standardization API or library. For PDL’s company, school, and location names, the Cleaner APIs return the standardized values PDL uses.

3. Parsing and decomposition

Splitting a combined value into its components makes each part comparable on its own. Email example Address example Implementation: use regular expressions or a parsing library. For emails, split at the @ symbol, then parse the local part for name patterns.
Name patterns in email addresses are a hint, not a fact. Addresses like jdoe@ or info@ won’t parse into a name, so treat parsed names as lower confidence than names from a dedicated field.

Part 2: Match on identity signals

4. Unique identifiers

When both datasets share a reliable identifier, match on it directly. This is the fastest and highest-confidence approach. Dataset A Dataset B Joining on Unique ID links A1002 and A1003. A1004 exists only in Dataset B, so it’s a new record. A1001 (Laura Martin) and A1004 (laura.martin@example.ca) are probably the same person, but the IDs don’t match. A unique ID only tells you what does match. Records that don’t join may still be candidates for the fuzzier techniques below. An identifier only works when both sources use the same namespace. Customer ID A1002 in your CRM and A1002 in a vendor file may refer to different people. Emails and profile URLs are strong signals, but they aren’t always unique or permanent. People share inboxes, change jobs, and update their profile URLs. Implementation: normalize the identifier on both sides, then join on it. Before you accept the results, check for identifiers that match more than one record on either side. Those aren’t clean one-to-one links, and they usually point to duplicates in one of the sources.

5. Cross-field matching with composite keys

When no single field is reliable, combine several. A composite key is unlikely to collide by chance, even if each field alone is ambiguous. Example Solution
  1. Standardize dates to YYYY-MM-DD.
  2. Build a composite key from first name + last name + date of birth.
  3. Account for first-name variations (“Annie” and “Anne”), as covered in the next section.
Composite keys are only as good as the fields inside them. Plan for three failure modes:
  • Collisions: common names and shared birthdays still produce false matches in large datasets.
  • Missing attributes: a record without a date of birth can’t form the key. Decide whether to skip it or fall back to a weaker key.
  • Conflicts: if names match but dates of birth differ, treat that as evidence against the match, not as missing data.

6. Nicknames and name variations

People rarely use their formal name everywhere. Map nicknames to a common form and allow for small misspellings. Solution
  • Nickname mapping: “Bill” and “Billy” both map to “William.”
  • Spelling correction: reconcile “Thompson” and “Thomson.”
  • Fuzzy matching on last name: catch remaining minor differences.
Treat a nickname mapping as a possible alias, not a fact. “Bill” usually means William, but someone named Bill may have no other name. Some nicknames, such as “Alex,” map to several formal names. Use the mapping to generate candidates, then confirm with other signals. Implementation: maintain a nickname dictionary that maps informal names to formal names, and use a spelling correction algorithm for last names.

7. Geocoding and location matching

Two addresses can look different but point to the same place. Converting them to coordinates makes them directly comparable. Example A geocoding service converts both to the same latitude and longitude, so you can treat them as one location. Implementation
  • Use a geocoding API such as Google Maps or OpenStreetMap.
  • Compare coordinates within an acceptable margin of error.
  • For US metros and cities, use PDL’s canonical location data to keep place names consistent.
Treat a shared location as a supporting signal, since family members, roommates, and companies in the same building share addresses.

8. Custom rules for business data

For company records, rule-based matching that combines several signals is often more reliable than any single similarity score. Rules
  • Normalize legal suffixes like “LLC” and “Limited” before comparing names.
  • Normalize websites by removing the protocol, www., and trailing paths.
  • Normalize phone numbers to E.164.
  • Prioritize unique keys such as LinkedIn URL and LinkedIn ID.
  • If the normalized phone number and website both match, treat the records as a strong match candidate. Check for a parent or subsidiary relationship before you link them.
Suffix normalization helps comparison, but it doesn’t establish legal equivalence. “Tech Solutions LLC” and “Tech Solutions Limited” can be separate legal entities in different countries, and subsidiaries often share a parent’s domain. Implementation: define a rule set specific to business naming conventions and contact details, and apply it inside your matching logic.

Part 3: Match on similarity

9. Phonetic algorithms

Phonetic algorithms convert names into codes based on how they sound. Names with the same code are potential matches, even when spelled differently. Example: Double Metaphone The codes match, so the records are flagged as potential matches. Implementation: use a phonetic encoding library to generate codes, then compare them. Phonetic codes also make good blocking keys. Most phonetic algorithms were designed for English pronunciation. They work less well for names from other languages and for names transliterated from other scripts, so test them on a sample of your own data. Use phonetic matches to find candidates, then confirm them with a second signal such as email or location.

10. Fuzzy matching

Fuzzy matching measures how close two strings are. It catches typos and transpositions that exact matching misses. Example: edit distance Edit distance counts the smallest number of single-character edits that turn one string into the other. The Damerau-Levenshtein distance is 2, because it counts a transposition as one edit. Standard Levenshtein distance counts a transposition as two edits, so it reports 3. Check which variant your library uses before you set a threshold.
The RapidFuzz Python library computes both distances and a normalized similarity score:
Compare two names
Output
A similarity score measures how alike two strings are, not how likely the records are to describe the same person. A fuzz.ratio of 89.7 doesn’t mean an 89.7% chance of a match. Implementation: compute a similarity score and set a threshold. A looser threshold catches more matches but adds false positives. See Evaluating match quality.

11. Machine learning approaches

When rules get too complex to maintain, a supervised model can learn from labeled examples and output a match score. Example
  • Features: name similarity score, email match (0/1), address similarity score, phone match (0/1)
  • Labels: record pairs marked “Match” or “No Match” from prior knowledge
  • Model: train a classifier, such as a random forest, then score new record pairs
Implementation
  1. Extract features from record pairs.
  2. Train the model on labeled pairs that represent the data you’ll match in production.
  3. Evaluate on a held-out set of labeled pairs the model didn’t train on.
  4. Tune the decision threshold to balance precision and recall.
Most candidate pairs are non-matches, so labeled data is usually heavily imbalanced. A model that predicts “No Match” for everything can look accurate. Measure precision and recall instead of accuracy, and make sure your labeled set includes enough true matches and hard non-matches.
The techniques above make good features. Similarity scores from fuzzy and phonetic matching, plus exact-match flags from standardized fields, are often what the model learns from.

Matching at scale

Comparing every record in one dataset against every record in another gets expensive fast. Two datasets of one million records each produce one trillion possible pairs. Blocking reduces that search space. You assign each record one or more blocking keys, and compare only records that share a key. Common blocking keys include:
  • Normalized company domain
  • Last name plus country
  • A phonetic encoding of the name
  • Company name plus location
In this example, blocking cuts 64 comparisons to 16. On real datasets, the reduction is usually far larger. Blocking has a trade-off. If two records that describe the same entity land in different blocks, you never compare them, and the match is lost. A typo in a last name is enough to cause it. To reduce that risk, combine several blocking keys and compare any pair that shares at least one.
This example blocks on last name plus country, and separately on phone number. Dana Lee and Dana Leigh have different last names, so the first key misses them. The phone key catches them.
Generate candidate pairs
Output
The code compares 2 candidate pairs instead of all 6. Robert Smith in v-2 is never compared with crm-1, because the country and phone number both differ. Score the candidate pairs with the techniques above.

Evaluating match quality

Every matching pipeline makes two kinds of mistakes:
  • False positives link records that describe different entities.
  • False negatives fail to link records that describe the same entity.
Two metrics summarize them:
  • Precision is the share of accepted matches that are correct. False positives lower it.
  • Recall is the share of all true matches that you found. False negatives lower it.
A match threshold is the decision boundary between accepting and rejecting a match. Raising it improves precision and lowers recall. Lowering it does the opposite. The right balance depends on the cost of each mistake. A workflow that automatically merges CRM records should favor precision. A workflow that suggests candidates for manual review can favor recall. Move the threshold below to see the trade-off on ten labeled record pairs. At a threshold of 0.90, every accepted match is correct, but you find only half of the true matches. At 0.70, you find five of six, but two of the seven accepted matches are wrong. Similarity scores aren’t necessarily calibrated probabilities. A score of 0.9 from one method doesn’t mean a 90% chance of a match, and it doesn’t mean the same thing as 0.9 from a different method. Measure what each score means on your own data.

Validate your matching logic

  1. Assemble a representative sample of record pairs with known labels, including hard cases.
  2. Run your matching logic on the sample, and measure precision and recall.
  3. Look at the errors. Adjust thresholds and rules based on what you find.
  4. Validate the adjusted logic on a separate labeled sample, so you don’t tune to one set of examples.
  5. Monitor results over time. Source data changes, and a threshold that worked last quarter may not work now.

Entity resolution with PDL data

The same techniques apply when you link your records to PDL data. Check exact field names and formats in the Person Schema and Company Schema.

Match people across datasets

The strongest signals are professional profile URLs and email addresses. Normalize LinkedIn URLs to PDL’s format, which drops the protocol and www. (see linkedin_url). Without a strong identifier, combine name with employer, school, or location.
  • Person Enrichment returns the single best match, with a likelihood score you can filter with min_likelihood.
  • Person Identify returns several scored candidates, so you can compare them yourself.

Match company records

The strongest signals are websites and company profile URLs. Normalize domains the way PDL does for website. Names are weaker signals. Check alternative_names and alternative_domains for aliases and name changes. Use ultimate_parent and all_subsidiaries when a subsidiary shares its parent’s domain.

Maintain resolved entities over time

Store the PDL id next to your internal identifier, and send it back as pdl_id to the Person or Company Enrichment API instead of matching again. IDs can change between builds, so check the changelogs for merged or deleted IDs before you re-enrich. See Persistent IDs, the Person Changelog, and the Company Changelog.

Choose an implementation approach


Person Schema

Exact field names and values for person records.

Company Schema

Exact field names and values for company records.

Persistent IDs

How PDL person IDs persist from one build to the next.

Cleaner APIs

Standardize company, school, and location names to PDL’s values.