How to Remove Duplicates from List in Python

List is a useful data structure in Python. It can be used to store various data types and perform many different tasks. Sometimes you may need to remove duplicates from list in Python. There are many ways to do this in Python. In this article, we will look at how to remove duplicates from list in Python.

How to Remove Duplicates from List in Python

Here are the steps to remove duplicates from list in Python.

1. Using dict.fromkeys

In this approach, we will simply create a dictionary using the list items as keys. This will drop all duplicates since there cannot be duplicate keys in dictionaries. And then get a list of this dictionaries keys. Of course, there is an overhead of creating a dictionary in this approach but it works for medium-sized lists. Here is an example.

>>> list1=["a", "b", "a", "c", "c"]
>>> dict1=dict.fromkeys(list1)
>>> dict1
{'a': None, 'c': None, 'b': None}
>>> dict1.keys()
['a', 'c', 'b']

Here is a single line command for the same

>>> dict.fromkeys(list1).keys()
['a', 'c', 'b']

2. Using set & list functions

You can also remove list duplicates using sets. In this case we simply convert list with duplicates to set first, and then convert this set to list. Please note, list function is available in Python 3+ only.

>>> res=list(set(list1))
['a', 'c', 'b']

3. Using for loop

This is the most basic method to remove duplicates from list, using for loop. In this case, we simply loop through the list and add elements to another list, if they are not already present in it.

>>> res = []
>>> for i in test_list:
    if i not in res:
        res.append(i)
>>> res
['a', 'c', 'b']

In this article, we have looked at 3 different ways to remove duplicates from list. There are many other ways to do this in Python. We have only looked at the most common and easiest ones.

Also read:

How to Get Key from Value in Python
How to Flatten List of Lists in Python
How to Find Element in List in Python
How to Parse CSV in Shell Script
How to Find All Symlinks in Folder