Using Python Sets for Membership Testing and Deduplication

Hello! Last lesson, you used a dictionary when each item needed an associated count or value. Sometimes the question is simpler: Have I seen this item before? A Python set keeps track of distinct items without storing a value for each one.

In this lesson, you’ll use sets for membership checks and duplicate removal. Allow about 35–40 minutes, including time to run the examples. If you’re studying in 30-minute sessions, you can leave the order-preserving approach for the next day.

Keep only what is present

A set contains each distinct element at most once. Watch the selected parts of What are Sets in Python? Python Tutorial for Absolute Beginners from Programming with Mosh for a quick demonstration of creating a set, changing it, and checking membership.

Try the operations together:

seen = {"red", "blue", "red"}

print(len(seen))       # 2
print("red" in seen)   # True
print("green" in seen) # False

seen.add("green")
seen.add("red")        # Already present; no second "red" is stored
print(len(seen))       # 3
print("green" in seen) # True

Use curly braces for a nonempty set, but use set() for an empty one. Empty braces, {}, create a dictionary:

seen = set()
print(type(seen))  # <class 'set'>

other = {}
print(type(other)) # <class 'dict'>

Unlike a list, a set does not provide positions. You cannot use seen[0], and you should not rely on the order in which its elements print or appear during iteration. This matters when removing duplicates: a set can preserve which values occurred, but not their original sequence.

Check membership as data arrives

Suppose a stream of task codes may contain repeats. A set can record the codes encountered so far:

codes = ["A1", "B2", "A1", "C3"]
seen = set()

for code in codes:
    if code in seen:
        print(code, "is a repeat")
    else:
        print(code, "is new")
        seen.add(code)

Before each iteration, seen contains the distinct codes from earlier iterations. Thus code in seen tells you whether the current code has appeared before. Adding a repeated code would not change the set, but separating the two cases lets the program respond differently.

The in operator may look familiar from dictionaries. For a dictionary, it tests keys; for a set, it tests elements. Use a dictionary such as {"A1": 2} when you need a count or other associated value. Use a set such as {"A1"} when presence alone is enough.

Read two short passages from Real Python’s Sets in Python to clarify what a set can contain and why it is useful for repeated membership checks.

Like dictionary keys, set elements must be hashable. Integers and strings work; a tuple works if its contents are hashable. A list does not work as a set element:

visited = {(0, 1), (1, 0)}
print((0, 1) in visited)  # True

# visited.add([2, 3])    # TypeError: a list is unhashable
visited.add((2, 3))      # A tuple of integers works

For repeated membership checks, a set is generally a better fit than scanning a list each time: a list may need to inspect many elements to decide whether an item is present. You’ll put precise names to these efficiency differences in the complexity module.

Remove duplicates without losing track of order

The shortest way to obtain the distinct values from a list is to pass it to set():

codes = ["A1", "B2", "A1", "C3", "B2"]
unique_codes = set(codes)

print(len(unique_codes))  # 3
print("C3" in unique_codes) # True

This is enough when you need membership tests or the number of distinct codes. If you need a list afterward, list(set(codes)) produces one—but do not assume it has the same order as the input.

When the required output is a list in first-seen order, use the set for membership and a list for output:

def unique_in_order(items):
    seen = set()
    result = []

    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)

    return result


codes = ["A1", "B2", "A1", "C3", "B2"]
print(unique_in_order(codes))  # ['A1', 'B2', 'C3']
print(unique_in_order([]))     # []

Here the two collections have different jobs. seen answers whether an item has appeared; result retains the first occurrence of each item in input order. After each iteration, result contains exactly the first occurrences found so far. A repeated item fails the not in test, so it cannot be appended a second time. As with set(items), the items must be hashable.

Key takeaways

A set stores unique, hashable elements. Create an empty one with set(), use in or not in to test membership, and use add() to record an element. set(items) removes duplicates when order does not matter; pair a set with a result list when first-seen order does matter.

Next, you’ll define a class-based node and connect node objects using references—the first step toward building linked data structures.