Remove Duplicate Lines
Delete repeated lines from a list and keep the first of each.
Results are provided as is, for general information. Check anything important before relying on it. See the Terms of Use.
About this tool
Paste a list with one item per line and remove the repeats. The first occurrence of every line is kept, in its original position and spelling, and later copies are dropped, so the order of your list is preserved. Three options change what counts as a duplicate: case sensitivity (Apple and apple), whitespace around each line (' a ' and 'a'), and whether blank lines are kept or removed. The counts show how many lines went in, came out, and were removed. Everything runs in your browser.
When to use it
Questions
Which copy of a duplicate is kept?
The first one. Every later line that matches it is removed, and the survivors stay in their original order. If you turn off case sensitivity, the first spelling wins, so 'Apple' followed by 'apple' leaves 'Apple'.
Does it sort my list?
No. Order is preserved. If you also want the result alphabetised, run it through the Sort Lines tool afterwards.
Are lines that differ only by trailing spaces duplicates?
Not by default, because they are different strings. Turn on 'Ignore spaces around lines when comparing' to treat them as the same. The kept line is left exactly as you pasted it.
What happens to blank lines?
By default the first blank line is kept and later ones are removed like any other repeat. Tick 'Remove blank lines' to drop them all, including lines that contain only spaces.
How do I dedupe and sort in one workflow?
Dedupe here, then run the result through the Sort Lines tool — dedupe preserves original order on purpose, so sorting is deliberately a separate step. The composition works in either order for a clean list, but dedupe-first is the habit that scales: paste, clean, then choose the sort mode the list needs (alphabetical, natural for filenames, numeric for tallies).
Why did 'a ' and 'a' both survive?
They are different strings, and by default comparison is exact — an invisible trailing space is a difference the tool respects rather than hides. Switch on the surround-whitespace option to fold them together; the line kept is still copied out exactly as you pasted it, so the option changes what counts as a duplicate, never the spelling of what remains.
Can it compare case-insensitively but keep the first spelling?
Yes — that is precisely what the case option does: 'Apple' and 'apple' count as one, and the first occurrence's casing is what survives. The comparison folds case; the output never rewrites a line. It also does not pick the most common variant — first-seen wins, because picking a winner would mean guessing your convention.
Are blank lines treated as duplicates?
Like any other value: the first is kept, later blank lines dropped as repeats. The 'remove blank lines' option drops them all instead — useful when the paste came from a table and formatting noise is not worth keeping. Lines holding only spaces count as blank for that option; content-bearing lines are never dropped by it.
How large a list can it take?
No fixed cap — the work is a single linear pass over the lines with a set for seen values, so hundreds of thousands of lines finish in the browser in well under a second. The practical limit is the paste itself: for very large logs, chunk by section and each chunk dedupes cleanly. The in/kept/removed counts update with each run, so the effect is visible as you go.
What is it not for?
Multi-column dedupe: it compares whole lines, so for CSV data it identifies identical rows only — rows sharing a value in one column do not collapse. To dedupe by a single field, extract that column alone (paste it standalone), dedupe, then rejoin; or use a spreadsheet's row-unique filter when entire-row semantics are genuinely what you want.