Text cleanup: whitespace, duplicates, and invisible characters

Two strings can look identical on screen and still fail an equality check. The difference is almost always invisible: a zero-width space copied from a web page, a trailing tab from a spreadsheet export, or a non-breaking space in a name. Lookups fail, deduplication misses obvious duplicates, and imports reject “valid” values. Cleaning text is mostly about making those differences visible and removing the ones that do not belong, in the right order.

Where the invisible characters come from

• Zero-width space (U+200B): inserted by websites and CMSs to allow line breaks in long words or URLs. It survives copy-paste and is invisible everywhere.
• Byte-order mark (U+FEFF): added at the start of files saved as “UTF-8 with BOM”, for example Excel’s “CSV UTF-8”. It sticks to the first value, so the first column header no longer matches.
• Non-breaking space (U+00A0): produced by word processors, often between numbers and units or in names. It looks like a space but does not match one.
• Soft hyphen (U+00AD): an invisible hyphenation hint from word processors and PDFs.
• Direction marks: invisible characters that control text direction. They are legitimate in Arabic and Hebrew text, and have been abused to disguise source code.

Step 1: remove invisible characters

Remove invisible characters removes zero-width spaces, word joiners, soft hyphens, and byte-order marks by default, and reports how many it found. That count alone answers the question “is there something hidden here?”. Joiners and direction marks are separate options, because they are needed in emoji sequences and right-to-left scripts. Non-breaking spaces can be converted to ordinary spaces.

Step 2: normalize whitespace

Trim whitespace removes leading and trailing spaces and tabs from each line, which are the most common cause of “Berlin” not matching “Berlin ”. Remove empty lines drops blank rows, optionally including lines that contain only spaces. Doing this before any comparison prevents the most common false mismatches.

Step 3: decide on case explicitly

Case folding is a decision, not a cleanup. Email addresses and most identifiers should be compared case-insensitively. Product codes and passwords should not. If case should not matter, convert both sides with the case converter before comparing, or use the case-insensitive option in the comparison tool, rather than hoping one list is already consistent.

Step 4: deduplicate and sort

Remove duplicate lines keeps the first occurrence of each line and preserves the original order. Matching is exact, which is why the earlier steps matter. After trimming and lowercasing, “apple”, “Apple ”, and “APPLE” all collapse into one. Sort lines offers natural ordering, where item2 comes before item10, and sorts accented letters with their base letter. Sorting also makes the result easy to scan.

Step 5: compare

Compare lists produces three results: items in both lists, items only in the first, and items only in the second. Typical uses are finding subscribers missing from a CRM, or checking which SKUs a supplier dropped. When two blocks of prose or configuration need comparing rather than two sets of items, text diff reports added, removed, and unchanged lines.

A worked example

Two email lists that “should” overlap completely:

List A: [email protected], [email protected], [email protected]
List B: [email protected], [email protected]␣, [email protected]

Here ␣ marks a trailing space. A naive comparison finds no common entries except by luck. With trimming and case-insensitive matching, the result is: in both, bob and carol; only in A, alice; only in B, dave. That is the answer people expected in the first place.

Step 6: check the result

Before shipping cleaned data, verify its shape. The word counter confirms line counts, so 1,000 lines in should become the expected number out, and word frequency surfaces repeated tokens that might indicate leftover duplicates or stray header rows.

The whole workflow runs in your browser, which matters when the text is a customer list, an internal export, or anything else you would rather not paste into a server-based tool. For tabular data with several columns, the same principles apply column by column. See the CSV cleaning guide.