Cleaning a merged list
Combining two lists of contacts or tags produces overlaps. Deduplicating keeps one of each in the original order.
Remove Duplicate Lines
Remove duplicate lines from text in your browser, keeping the first occurrence and the original order.
Maintained by Roshan.
Use Remove Duplicate Lines
Converts lines to unique lines in this tab, with nothing sent anywhere.
Runs entirely on your deviceThe conversion happens in this browser tab. Nothing you type is uploaded, logged or stored, which is worth knowing for anything you would not paste into a search box.
Explore text tools
Find related browser-local tools for nearby tasks without starting another search.
Browse all text toolsAbout Remove Duplicate Lines
Removing duplicate lines is how you clean a list that has accumulated repeats - an export that ran twice, a merged set of addresses, a log with the same entry recorded many times, a collection of tags with obvious overlaps. The task is simple to describe and has one meaningful choice: what to do with the order. A tool that sorts as it deduplicates scrambles a list whose order mattered. This one keeps the first appearance of each line and removes only the later repeats, leaving everything else exactly where it was. So a list that was in a deliberate order stays in that order, minus the duplicates, rather than being rearranged as a side effect of cleaning it.
Work through the input
- Each line is compared against the ones already seen, and a line that has appeared before is dropped.
- The first occurrence of each line is kept; only the later repeats are removed.
- The original order of the surviving lines is preserved, so a deliberately ordered list is not rearranged.
- Comparison is exact, so lines must match fully to count as duplicates - a difference in spacing makes them distinct.
- Blank lines are treated like any other line for the purpose of matching.
When remove duplicate lines is useful
Tidying a log
A log with a repeated entry collapses to the distinct lines, making the real variety visible.
Check the output
- Matching is exact and case-sensitive, so Apple and apple are treated as different lines, as are lines differing only in trailing spaces.
- It removes whole-line duplicates. Near-duplicates that differ slightly are kept, since they are not identical.
- It preserves order rather than sorting; to sort as well, run the sort tool afterward.
- Trailing whitespace can make two visually identical lines count as different; trim first if that is a problem.
- This removes repeats; it counts nothing and reports no statistics.
Questions about Remove Duplicate Lines
Which duplicate is kept?
The first occurrence. Later repeats are removed, and the surviving lines keep their original order.
Is it case sensitive?
Yes. Matching is exact, so Apple and apple are treated as different lines.
Does it sort the lines?
No. It preserves order deliberately. Run the sort tool afterward if you also want them sorted.
Why are two identical-looking lines both kept?
They probably differ in trailing whitespace. Trim the text first so they match exactly.
Related tools
Sort Lines
Sort lines of text alphabetically in your browser, with natural numeric ordering and a reverse option.
Remove Line Breaks
Remove line breaks from text in your browser, collapsing it into a single line without merging words.
Remove Extra Spaces
Remove extra spaces from text in your browser, collapsing runs and trimming each line.