Removing duplicates from Python lists is a common task that keeps data clean and predictable. This guide walks through reliable patterns so you can handle repeated values without losing order or performance.
Use the structured overview below to match your preferred technique with the constraints of your project, such as ordering needs or memory limits.
| Method | Preserves Order | Mutable or New List | Best Use Case |
|---|---|---|---|
| Loop with seen set | Yes | New list | Stable order and readable code |
| Dict.fromkeys | Yes (3.7+) | New list | Concise one-liner with good speed |
| Set seen with list comprehension | Optional | New list | Fast, order not required |
| In-place two-pointer | Yes | Mutates original | Memory constrained scenarios |
Loop and Seen Set Pattern
Step by step logic
The loop and seen set pattern scans the list once, adding items to a new list only when they have not been encountered before. This guarantees order preservation while keeping membership checks efficient.
Tradeoffs and scaling behavior
Memory usage increases slightly to store the set, but speed remains linear on average. It is a safe default when readability and stable order matter.
Dict Fromkeys One-Liner
Concise stable deduplication
In Python 3.7 and later, dict keys preserve insertion order, so list(dict.fromkeys(items)) removes duplicates in a single readable line and keeps the first occurrence of each value.
Compatibility considerations
If you support older Python versions or rely on duck typing for mappings, verify ordering guarantees in your runtime environment before adopting this as policy.
Set Based Filtering
When order does not matter
Using a set directly, such as list(set(items)), removes duplicates with minimal code. This approach trades order for speed and is suitable when only uniqueness is required.
Performance and memory
Set operations are highly optimized, making this option faster on large datasets, but the resulting list may be in any order and is not deterministic across runs.
In-place Mutation Strategy
Modifying the original list
By overwriting slice indices, you can deduplicate without allocating a second list. This in-place two-pointer method keeps the first occurrence and then shrinks the list to remove redundant elements.
Side effects and usage guidance
Because this approach mutates the input, ensure that other parts of your code do not rely on the original list identity remaining unchanged.
Recommended Practices
- Prefer
list(dict.fromkeys(items))for concise, ordered deduplication in modern Python. - Use a loop with a seen set when you want explicit control or need to apply custom equality logic.
- Choose set based filtering only when order is irrelevant and performance is critical.
- Apply in-place mutation carefully, ensuring no other code holds references to the original list structure.
- Write tests that validate behavior for edge cases, such as empty lists, all identical items, and mixed types.
FAQ
Reader questions
How can I keep the original list unchanged while removing duplicates?
Build a new list by iterating and tracking seen values with a set, or use list(dict.fromkeys(items)) to create a stable copy without modifying the source.
Will converting to a set always remove duplicates correctly?
Yes, a set eliminates duplicates because it only stores unique keys, but the output order is not guaranteed and may differ between runs.
What is the best approach for large datasets in terms of speed?
Set based filtering is generally fastest, followed by dict.fromkeys when order matters, with the loop and seen set pattern as a readable alternative that still scales linearly.
Can I remove duplicates based on a specific key in dictionaries inside a list?
Yes, track seen values derived from the chosen key while iterating, and build or overwrite the list based on whether that key value has already been encountered.