Remove duplicate lines
Paste a list, remove repeated lines and sort what is left.
Runs entirely in your browser
Paste lines to begin.
Text up to about 5 MB. The result updates as you change the options.
How to remove duplicate lines
- Paste your list into the input, one item per line, or open a .txt file.
- Choose how lines are compared, which copy to keep and, if you want, a sort order.
- Copy the result or download it as a .txt file.
Removing duplicates and sorting lines
Lists collect repeats: email addresses pasted from several sources, keywords, log lines, file names. This tool reads the text one line at a time, remembers each line it has seen and drops the later copies (or the earlier ones, if you choose to keep the last occurrence). The lines that survive stay in their original order unless you pick a sort.
The report above the buttons shows how many lines went in, how many came out and how many were removed as duplicates, so you can check the result against what you expected. Empty lines removed by the option are reported separately.
Tips
- To clean up the lines first, use Find and replace, then come back here.
- Check what changed between two versions of a list with Compare text.
- Only need the list in order, with more sorting choices? Use Sort lines alphabetically.
- Count what is left with the Word counter.
Questions
What counts as a duplicate?
Two lines are duplicates when they are exactly the same, character for character. Turn on "Ignore upper and lower case" to treat Apple and apple as the same line, and "Trim spaces before comparing" to ignore spaces and tabs at the start and end of each line.
Does trimming change the lines in the result?
Yes. When trimming is on, the spaces at the start and end of every line are removed from the lines you get back as well as from the comparison.
What happens to empty lines?
With "Remove empty lines" on, lines that are empty or hold only spaces are dropped. With it off, empty lines are kept exactly as they are and are never counted as duplicates of each other.
What is natural sort?
Natural sort understands numbers inside text, so item2 comes before item10. A to Z sorts character by character using your browser's language rules, which puts item10 before item2. Sort by length puts the shortest lines first and keeps the original order between lines of equal length.
Is the shuffle random?
It uses your browser's secure random number generator with a Fisher-Yates shuffle, so every order is equally likely. Press Shuffle again for a new order.
Is my list uploaded?
No. Everything is done in your browser and nothing is stored.