Fuzzy Duplicate Finder for CSV

updated 31 August 2026

How to use it

1. Paste or drop the CSV. The column names appear under the panes — click the ones that identify a record (usually a name plus an email, or a company plus a postcode). Pick none and every column is used, which is stricter than it sounds.
2. Leave Do on show the groups for the first run. You get only the rows involved in a near-match, grouped, each with its similarity to the row that would survive.
3. Move the threshold until the groups look right. 88% is a good start; lower catches more and starts to catch coincidences.
4. Switch to delete the duplicates when you trust it, and download.

What "similar" means here

Two steps, and the first does most of the work.

1. Normalise

The chosen columns are joined into one key, and then — depending on the ignore toggles — lower-cased, stripped of punctuation, collapsed to single spaces, and decomposed to remove accents (José becomes jose, not jos). Everything but digits is the one that is off by default: it throws away all non-digits, which is exactly right for phone numbers, account IDs and postcodes written six different ways, and useless for names.

2. Measure

What survives normalisation is compared with an optimal string alignment distance — Levenshtein, plus a move for swapping two adjacent characters. That extra move matters more than it sounds: transposition is the commonest typo there is, and plain Levenshtein charges Lovelace → Lovelcae as two edits, scoring 83% on a 12-character key and falling under any sensible threshold. Counted as one edit it scores 92% and matches.

The score is 1 − distance / length of the longer string, so it is a percentage of the key that had to change.

How it stays fast

Comparing every row with every other row is quadratic: 20,000 rows is 200 million comparisons and a hung tab. Two things prevent that.

• Blocking. Rows are bucketed by the first four characters of their normalised key. Two strings that share no four-character prefix cannot clear an 80% threshold on any realistic key length, so they are never compared.
• A second pass on sorted tokens. Lovelace, Ada and Ada Lovelace share no prefix and are eight edits apart as written — but sort the words in each and they are identical. So the whole sweep runs twice, once on the key as written and once on its words in alphabetical order, and the two results are unioned. Reordered names are the duplicate class that a single-pass matcher always misses.

Grouping is by union-find, so a chain of matches ends up in one group: if A matches B and B matches C, all three are one record even when A and C are not similar to each other on their own. The comparison count is reported in the status bar. Input is capped at 20,000 rows.

The three modes

• Show the groups — only the rows in a near-match, with group, role and similarity columns in front. This is the review step, and it is the default on purpose: deleting rows on a similarity score you have not looked at is how a mailing list loses real customers.
• Flag every row — the whole file back, with the same three columns added and left blank for rows in no group. Sort by dup_group in a spreadsheet, or filter dup_role = duplicate. Nothing is removed.
• Delete the duplicates — one row survives each group and the rest are dropped. The original columns come back unchanged.

Which row survives

• First — the earliest in file order. Predictable, and right when the file is chronological.
• Last — the most recently appended, which in an append-only export is the most current.
• Fullest row — the one with the most characters across all its non-empty cells. Fuzzy duplicates are very often the same record entered twice with different fields filled in, and this keeps the more complete one.

Example

name,email,city
Ada Lovelace,[email protected],London
ada  lovelace,[email protected],london
Ada Lovelcae,[email protected],
Grace Hopper,[email protected],New York

At 88%, matching on name, in show the groups mode:

group,role,similarity,name,email,city
1,keep,100.0%,Ada Lovelace,[email protected],London
1,duplicate,100.0%,ada  lovelace,[email protected],london
1,duplicate,91.7%,Ada Lovelcae,[email protected],

The second row scores 100% because casing and double spacing are removed before anything is measured. The third is one transposition away. Grace Hopper is not in a group and does not appear.

Limits and gotchas

• A score is not a decision. Jon Smith and Jan Smith score 88% and are two people. Review before deleting; that is what the default mode is for.
• Short keys are noisy. On a three-character key one edit is 33%, so nothing matches; on a two-character key everything does. Match on enough columns that the key has length.
• The blocking prefix can miss a match. Two records that differ inside their first four characters and in word order will not be paired. Lowering the threshold does not help — normalising more, or matching on a column that starts the same, does.
• Nothing is written back to the matched cells. The surviving row keeps its own values; this does not merge fields from its duplicates. Merge does that, on an exact key.
• Capped at 20,000 rows, and the status bar says when the cap bit.

Privacy

100% client-side. The file is parsed and compared in your browser; nothing is uploaded. See the privacy policy.

Related tools

• Remove duplicates from CSV — exact matching, when the rows already agree.
• Find duplicates — report them without changing the file.
• Trim whitespace and clean numbers — normalise upstream so exact dedupe is enough.
• Merge CSV files — combine files on a shared key.
• Value counts — see how often each value occurs before you decide what a duplicate is.