Remove duplicate CSV rows
Remove duplicate CSV rows using selected key columns while keeping the first or last match.
How it works & useful details
Formats & limits
Import UTF-8 text up to 1.5 MB or paste up to 500,000 characters. CSV accepts comma, tab, semicolon or pipe separators, doubled quotes and line breaks inside quoted fields. All rows must have the same number of fields. A final line ending is ignored; interior blank rows are real rows. A leading UTF-8 BOM is removed. Up to 10,000 rows including the header, 100 columns, 100,000 cells and 50,000 characters per cell. Output is limited to 2,000,000 characters.
Live previews pause above 50,000 input characters; exports still process the full bounded input. The table pages through 15 rows and 12 columns at a time. Export includes every result row. Processing is synchronous and bounded; cancellation is checked before and after a transformation.
Data details
Text comparisons use Unicode code-unit order, with optional case folding. Numeric sorting/filtering requires finite numbers in every selected cell and uses JavaScript decimal precision; large integer identifiers need text mode. No trimming or spreadsheet formulas are evaluated.
CSV-to-JSON defaults to strings. JSON object export requires unique, nonempty headers. JSON input uses object keys in discovery order (integer-like keys follow JSON property order), missing properties become empty cells, and arrays must be rectangular. Primitive arrays get one “Value” column. Null becomes the literal text “null” by default; nested objects/arrays become compact JSON strings. Empty strings, missing values and optional empty nulls are indistinguishable in CSV. Mixed row shapes are rejected. JSON numbers use JavaScript precision; unsafe integers are rejected, so quote large IDs. Duplicate JSON names use the last value.
Transpose allows up to 100 source rows including the optional header, and exports all transposed cells as data. Spreadsheet formula protection is opt-in: it prefixes formula-like cells with an apostrophe, visibly changing their data. Spreadsheet programs differ; review results before opening them.
About this tool
Remove duplicate CSV rows finds rows that share the same values in the key columns you choose and keeps one of them. It is the quickest way to de-duplicate a mailing list or an export before importing it elsewhere.
You decide which columns define a duplicate and whether to keep the first or last match. The comparison runs locally; no row ever leaves your device.
How to use it
- Import a CSV file or paste the text.
- Select the key columns that should identify a duplicate row.
- Choose whether to keep the first or the last occurrence.
- Check the preview and export the de-duplicated CSV.
Worked example
The sample table contains "Ada, Design, 92, true" twice. Comparing all columns removes the second copy and keeps 4 of 5 rows.
Comparing only the Team column is a different question: "keep one row per team". It keeps the first Design row (Ada) and the first Engineering row (Lin) and removes 3 rows. Choose "keep last" instead to retain the most recent row per key, which is useful when a file is sorted by date and you want the latest record per customer.
Choosing key columns
A duplicate is only a duplicate with respect to some key. For a contact list, the email address is usually the right key, since names are written inconsistently. For order lines, the order ID plus the product ID might be the key. Comparing all columns only removes rows that are identical in every field.
Values must match exactly, so "[email protected]" and "[email protected] " are different. Clean those first: trim spaces and normalise case in the spreadsheet editor, or export the key column, clean it with the text tools, and join it back. Row order is preserved for the rows that stay.
Frequently asked questions
Is matching case-sensitive and whitespace-sensitive?
Comparison uses the cell values exactly as written, so casing and spacing matter. Normalize those first if you need looser matching.
How many rows were removed?
The workspace reports the removed count so you can confirm the result matches your expectation before exporting.
What are the limits?
Up to 10,000 rows, 100 columns, and 100,000 cells, with a 2,000,000 character export limit.
Is the header row compared?
No. With "first row is a header" enabled, the header is kept and only data rows are compared.