Remove duplicate lines
Clean up a list by removing repeated lines. Keep the first occurrence, in its original order.
How it works & useful details
A cleaner list in one step
Paste one item per line, select Remove duplicates, then copy or download your result. The first occurrence of every line stays in place.
What counts as a duplicate?
Matching is case-sensitive and preserves spaces. “Apple” and “apple” remain separate. Repeated blank lines become one blank line. Line endings are normalized to LF.
Up to 500,000 characters per run. Your text is never uploaded.
About this tool
Remove duplicate lines cleans a list by deleting repeated lines while keeping the first occurrence in its original order. It is the standard step before importing a list into a system that rejects duplicates.
Comparison is exact by default, so lines differing only by case or trailing spaces count as different. Use the case converter or trim tool first if you need looser matching.
How to use it
- Paste the list with duplicates.
- Run the cleanup to keep only the first occurrence of each line.
- Check the reported count of removed lines.
- Copy or download the cleaned list.
Worked example
The list
apple
banana
Apple
apple
cherry
"banana " (with a trailing space)becomes apple, banana, Apple, cherry, "banana ". The second lowercase "apple" is removed, because it is an exact repeat of the first line. "Apple" and "banana " stay, because matching is exact: case and spaces count.
Getting the matches you expect
Exact matching is deliberate: it never merges lines that differ, so no data is lost unexpectedly. If near-duplicates should count as the same, normalize first:
- Trim whitespace to remove leading and trailing spaces.
- Remove invisible characters, which hide in text copied from web pages and PDFs.
- Convert everything to lowercase with the case converter if capitalization does not matter, as with email addresses.
The first occurrence of each line keeps its position, so the original order is preserved. Sort afterwards if you want an alphabetical list.
Frequently asked questions
Is case ignored when comparing lines?
No. Comparison is exact, so "Apple" and "apple" are treated as different lines.
Are blank lines removed too?
Blank lines are subject to the same rule: the first one is kept and later ones are removed. Use the Remove empty lines tool to drop them entirely.
What is the limit?
Up to 500,000 characters of text.