How duplicates are detected
Each line is trimmed of leading and trailing spaces, then compared exactly — the match is case-sensitive, so “Apple” and “apple” both survive. Empty lines are skipped entirely. The first occurrence is kept and the counter reports how many repeats were dropped: from a, b, a, c that is 3 kept, 1 removed.
Everything runs in your browser — the list you paste is never uploaded, transmitted or stored on a server, so you can safely de-duplicate customer emails or internal URLs.
If you want a different flavour of the same thing, try the word frequency counter.
Typical use cases
Cleaning email or keyword lists before an import, de-duplicating URLs from a crawl, tidying CSV columns pasted from a spreadsheet, and merging two lists into one. Because order is preserved, you can safely de-duplicate a ranked list without losing its ranking. Count what is left with the word counter.
A worked example: paste 1,200 exported subscriber emails, get 1,043 unique lines back and see that 157 repeats were dropped, before the import tool rejects the file. Watch for trailing commas or quotes copied from a spreadsheet — “apple,” and “apple” are different lines and both survive.
The random list picker covers the same ground with a different mechanism.
Order-preserving vs sorted output
Command-line sort | uniq requires sorting first, which destroys the original sequence. This tool uses a hash set instead, so it is a single pass and keeps sequence intact. If you then need the survivors on one line, feed them into remove line breaks.
Frequently asked questions
- How many lines remain from a, b, a, c?
- Three: a, b and c. The second “a” is removed as a duplicate.
- Is the comparison case-sensitive?
- Yes. “Apple” and “apple” are treated as two different lines.
- Does it sort the output?
- No. Lines are returned in their original order, with only later repeats removed.
- What happens to blank lines?
- They are ignored and never appear in the unique list.
- Does it detect lines that differ only by trailing spaces or punctuation?
- Leading and trailing spaces are trimmed before comparison, so “apple ” and “apple” are treated as the same line. Any other difference, including a trailing comma or a quotation mark from a spreadsheet, counts as a different line.