Free Duplicate Line Remover Online
Remove duplicate lines from text instantly. Keep first occurrence, last occurrence, or remove all duplicates. 100% client-side — your text stays private.
How do you remove duplicate lines from text? Deduplication compares every line against the lines already seen and keeps only one copy of each distinct value. The comparison rule decides the result: whether leading and trailing spaces are stripped before comparing, and whether Apple and apple count as the same line. This tool applies that rule to a whole block at once and reports how many lines it removed.
How to Use the Duplicate Line Remover
- Paste your lines — Drop the list, log or export into the input panel on the left. Every newline is treated as one record, and the text never leaves your browser.
-
Set the comparison rule — Trim whitespace ignores leading and trailing spaces before comparing; Case-sensitive decides whether
ACMEandacmeare the same line. Both change which rows count as duplicates. - Choose what to keep — First occurrence keeps the earliest copy, Last occurrence keeps the latest, and Remove all duplicates discards every line that appeared more than once, leaving only the values that were unique to begin with.
- Sort the output if you need to — Sort alphabetically reorders what survives. Leave it unchecked and the remaining lines stay in the order they appeared in the source.
- Check the counts — The cards under the panels show total lines in, unique lines out, and how many were removed — a quick sanity check that the rule you picked did what you expected.
- Copy or run a second pass — Copy puts the output on your clipboard. Swap moves it back into the input so you can rerun it with different options.
How Line Deduplication Works
The text is split on newline characters into an array of lines, and each line is reduced to a comparison key before anything is compared. The key is what actually determines whether two lines are duplicates, and the two checkboxes control how it is built.
key = trimWhitespace ? line.trim() : line → caseSensitive ? key : key.toLowerCase()Each key is then looked up in a set of keys already seen. A line whose key is new is written to the output and its key is recorded; a line whose key has been seen is counted as a duplicate. Because the lookup is a hash set rather than a comparison of every line against every other line, the work grows in proportion to the number of lines rather than to its square — a hundred thousand lines take roughly a hundred times as long as a thousand, not ten thousand times.
Note that the key only affects matching. The line written to the output is always the original, untrimmed, original-case text — trimming is used for the comparison and is not applied to what you copy. If you want the whitespace actually gone from the result, run the output through the Whitespace Remover afterwards.
The Three Keep Modes
The Keep dropdown is the setting people most often get wrong, because two of the three modes return one copy of each value and the third returns something quite different. Using the five-line list apple, banana, apple, cherry, banana as input:
| Keep mode | Output | What it is for |
|---|---|---|
| First occurrence | apple, banana, cherry |
The default. Preserves the order in which values were first seen — right for mailing lists, tag lists and imports where the earliest record wins. |
| Last occurrence | apple, cherry, banana |
Keeps the final copy in the position where it appeared. Useful for append-only logs and config files where the later entry overrides the earlier one. |
| Remove all duplicates | cherry |
Drops every value that appeared more than once, leaving only values that occurred exactly once. This is a difference report, not a deduplicated list. |
Remove all duplicates is not "keep one of each". If you are cleaning a list and every line happens to be repeated, this mode returns an empty result. Choose First occurrence for ordinary deduplication.
Reading the Counts
Three numbers describe what happened. Total Lines is the count of lines in the input, including blank ones — a trailing newline at the end of a paste produces one empty line, so this number is often one higher than you expect. Unique Lines is the number of lines in the output. Duplicates Removed is the difference between the two, which is the count of lines discarded, not the count of distinct values that had repeats. A value that appeared four times contributes three to that figure under First occurrence.
Blank lines are compared like any other line. With Trim whitespace on, a line containing only spaces produces an empty key and is therefore a duplicate of a genuinely blank line, so consecutive blank lines collapse to one. If you want every blank line gone rather than deduplicated, use the Empty Lines Remover before or after this step.
Frequently Asked Questions
It keeps the earliest copy of each distinct line and discards every later repeat, so the surviving lines stay in the order they were first seen. This is the mode you want for ordinary deduplication of a mailing list, a tag list or an import file. "Last occurrence" does the mirror image: it keeps the final copy, positioned where that final copy appeared, which suits append-only logs where a later entry supersedes an earlier one.
Because that mode is not "keep one of each" — it removes every line that appeared more than once and returns only the lines that occurred exactly once. Given apple, banana, apple, cherry, banana it returns just cherry. It is a report of the values unique to your list, which is useful for spotting one-offs but wrong for cleaning a list. Switch to "First occurrence" for that.
With the box unchecked (the default) both lines are lowercased before comparison, so Hello, HELLO and hello collapse into one and the first spelling encountered is the one kept. Check the box and each spelling is treated as a distinct line. Leave it off for emails, domain names and tags, which are conventionally case-insensitive; turn it on for anything case-significant such as identifiers, hashes or Base64 strings.
It strips leading and trailing whitespace from each line before comparing, so hello and hello match. It also removes tabs and the carriage return left behind by Windows CRLF line endings, which is why files pasted from Notepad usually need it on. It only affects the comparison — the lines written to the output keep their original spacing exactly as you pasted them.
Almost always an invisible difference. Check for a trailing tab or non-breaking space (turn on Trim whitespace), a double space in the middle of the line (trimming does not touch interior spacing — run the Extra Spaces Remover first), curly quotes versus straight quotes, or accented characters that differ from their plain equivalents. Case is another candidate if you enabled case-sensitive matching.
Yes, unless you check Sort alphabetically. Under "First occurrence" the output follows the order of first appearance; under "Last occurrence" each surviving line sits where its final copy was. Sorting is applied after deduplication and compares the trimmed text, using the same case setting as the deduplication itself, so a case-insensitive run sorts case-insensitively too.
It works on CSV as long as each record occupies exactly one line — the whole row is compared as a single string, so two rows are duplicates only when every field matches. It will not work on CSV files containing quoted fields that span multiple lines, because those are split at the newline. JSON is not line-oriented at all; format and compare it with the JSON Diff tool instead.
No. Splitting, comparing and rejoining all happen in JavaScript inside your own tab, with no upload and no server round trip. That makes it safe for customer lists, internal logs and anything else you would not paste into a hosted service. The only data that leaves the page is what you choose to put on your clipboard with the Copy button.
Use Cases
Merging Two Exports
You pasted last month's CRM export under this month's and need one combined list. First occurrence keeps the earlier record for every contact that appears in both files.
Finding the Distinct Errors in a Log
A crash log repeats the same stack trace hundreds of times. Deduplicating the message lines turns thousands of rows into the handful of distinct failures worth investigating.
Cleaning an Email List Before Import
Mailing platforms bill per contact and reject files with repeats. Leave case-sensitivity off so Sam@site.com and sam@site.com are recognized as the same address.
Spotting One-Off Entries
Run "Remove all duplicates" over a list of transaction references or SKUs to surface only the values that appear exactly once — often the anomalies worth checking by hand.
Collapsing a Config File
A settings file has accumulated repeated directives over time. Last occurrence keeps the version that actually takes effect and drops the dead earlier copies above it.