Text cleanup: whitespace, duplicates, and invisible characters
Why two identical-looking strings fail to match, and the safe order of operations for cleaning a list.
Two strings can look identical on screen and still fail an equality check. The difference is almost always invisible: a zero-width space copied from a web page, trailing tabs from a spreadsheet export, or a nonbreaking space in a name. Cleaning text is mostly about making those differences visible and then removing the ones that do not belong.
Step 1: remove invisible characters
Zero-width spaces and joiners, direction marks, soft hyphens, byte-order marks, and nonbreaking spaces are legitimate in some contexts and poison in others. Each category is controlled separately in Remove invisible characters, so you can keep the ones you meant to have.
Step 2: normalize whitespace
Trim whitespace handles leading and trailing spaces and tabs per line, and Remove empty lines drops blank rows, optionally including lines that contain only whitespace. Doing this before any comparison prevents the most common false mismatch.
Step 3: decide on case explicitly
Case folding is a decision, not a cleanup. If identifiers should be compared case-insensitively, convert both sides with Case converter before the comparison rather than hoping one list already matches.
Step 4: deduplicate and sort
Remove duplicate lines keeps the first occurrence of each line and reports how many were removed. Sort lines offers natural ordering, where item2 comes before item10instead of after it. Sorting also makes the result easy to eyeball.
Step 5: compare and verify
Compare lists produces three results: items in both lists, items only in the first, and items only in the second. When two blocks of prose or configuration need comparing rather than two lists, Text diff reports added, removed, and unchanged lines.
Step 6: check the result
Before you ship cleaned data, verify the shape. Word counter confirms line and word counts, and Word frequency surfaces repeated tokens that might indicate remaining duplicates or stray headers.
The whole workflow runs in your browser, which matters when the text is a customer list, an internal export, or anything else you would rather not paste into a server-based tool.