Remove duplicates from a list

Paste a list. Remove the repeated lines, see only the values that repeat, count how often each one appears, or keep only what appears once. Extra spaces and different case do not hide a repeat.

Your list

Nothing pasted yet

0Pasted

0Unique

0Duplicated values

0Extra copies

How to match

The list is separated by

Count values as the same even if they differ in

Hidden characters are non-breaking spaces, zero-width characters and tabs inside a value. Leading zeros make 00123 a repeat of 123.

Numbers Excel has shortened

Excel replaces a long ID with a short form such as 3.5E+14 and the rest of the digits are gone. Two different IDs can end up looking the same, so these are never treated as repeats of each other.

Result —

Show as

How to remove duplicates from a list

Paste your list in the box, or drop a text or CSV file on it. One value per line is the default. Switch to commas or tabs under How to match.
Read the counters. They show how many values you pasted, how many are different, how many values appear more than once and how many extra copies there are.
Pick what to see: the list without repeats, only the values that repeat, a count for each value, or only the values that appear once. Then copy it or save it.

The first copy of a value is the one that stays, so your list keeps its order. Matching ignores the things that most often hide a repeat: spaces at the ends, doubled spaces, hidden characters and upper or lower case. Leading zeros are an option. The note under the counters says how many values each option changed.

To check one list against another instead, use compare two lists. If you need the result in another shape, such as quoted values or a different separator, send it through the list converter. If the list came from Excel and some IDs lost their zeros or carry hidden characters, clean up the text and IDs first.

Count how often each value appears

Choose Count each to get a table of value, count and the line where the value first appeared. The values that appear most often come first. Values with the same count stay in the order they first appeared. Paste the table into a sheet and it lands in three columns, or save it as a CSV.

Excel's Remove Duplicates button

In Excel, select the column, then choose Data and Remove Duplicates. It works well on clean data and misses repeats in data that is not clean.

Spaces at the end. AB12 and AB12 look the same and are different values to Excel, so both stay.
Non-breaking spaces. Web pages and other systems add them. Microsoft's TRIM page says the function "does not remove this nonbreaking space character", so TRIM does not rescue you here.
Case. The button ignores case, so ab12 and AB12 count as repeats and only the first one survives. That is a surprise when the case mattered.
It changes your sheet. The rows are deleted from the sheet itself. Work on a copy if you need the original.

Flag duplicates with COUNTIF

To see the repeats before you delete anything, put a formula next to the first value and fill it down. This one is TRUE for every copy after the first.

=COUNTIF($A$2:A2,A2)>1

This one is TRUE for every copy, including the first.

=COUNTIF($A:$A,A2)>1

COUNTIF reads a long number stored as text as a number and compares only its first 15 digits, so two different 19-digit SIM numbers (ICCIDs) can show as duplicates. For IDs that long, use this instead:

=SUMPRODUCT(--($A$2:$A$5000=A2))>1

Google Sheets: UNIQUE

In Google Sheets, one formula returns the list without repeats:

=UNIQUE(A2:A)

It ignores case, like Excel's button, and it does not trim spaces, so clean the column first or paste it here.

Questions

How do I remove duplicate lines from a list?

Paste the list in the box and leave Remove duplicates selected. The first copy of each value stays, in the order you pasted it, and every later copy goes. Copy the result or save it as a text file.

Why did Excel's Remove Duplicates leave duplicates behind?

Excel treats AB12 and AB12 followed by a space as two different values, and it does the same for a non-breaking space, which TRIM cannot remove. Case does not matter to it. This tool trims spaces and removes hidden characters before it compares, so those repeats are found.

How do I count how many times each value appears?

Choose Count each. You get every value with its count and the line where it first appeared, with the most repeated values first. Copy it into Excel as three columns, or save it as a CSV.

Which copy is kept, the first or the last?

The first. Each value stays where it first appeared and the later copies are dropped, so the order of your list is unchanged. The value is shown as you pasted it, apart from spaces and hidden characters that were cleaned.

Is my list uploaded anywhere?

No. The work happens in your browser and nothing you paste is sent to a server, saved or put in the address. Only your option choices are remembered on your device. The privacy policy has the detail.

If duplicates keep coming back because several people keep one shared list, Mbule.Tech can build your team a tracker that stops them at the door.