Duplicate records are one of the most common data quality issues in ETL pipelines. Events can be sent twice, API retries create duplicates, and merging data sources introduces overlaps.
Given a list of records, a set of key columns that define uniqueness, and a deduplication strategy, return the deduplicated list of records.
"first": keep the first occurrence of each unique key
"last": keep the last occurrence of each unique key
"most_complete": keep the record with the fewest None values. Ties are broken by keeping the first occurrence.
The output list must preserve the order of first appearance of each unique key in the input. For example, if key X first appears at index 2 and key Y first appears at index 0, then key Y's selected record comes before key X's selected record in the output.
Return a list of the selected records (as dicts).
Input:
records = [ {"id": 1, "name": "Alice", "email": "alice@test.com"}, {"id": 2, "name": "Bob", "email": "bob@test.com"}, {"id": 1, "name": "Alice Smith", "email": "alice.s@test.com"}, ] key_columns = ["id"] strategy = "first"
Output:
[{"id": 1, "name": "Alice", "email": "alice@test.com"}, {"id": 2, "name": "Bob", "email": "bob@test.com"}]
Two records share id=1. Strategy "first" keeps the earliest occurrence.
Input:
records = [ {"id": 1, "name": None, "email": "a@test.com", "phone": None}, {"id": 1, "name": "Alice", "email": "a@test.com", "phone": "555-1234"}, ] key_columns = ["id"] strategy = "most_complete"
Output:
[{"id": 1, "name": "Alice", "email": "a@test.com", "phone": "555-1234"}]
The second record has 0 None values vs 2 in the first, so it wins.
Use an ordered dictionary or a list to track the first-appearance order of unique keys.
For "most_complete", count None values across all fields, not just key columns.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: string
Duplicate records are one of the most common data quality issues in ETL pipelines. Events can be sent twice, API retries create duplicates, and merging data sources introduces overlaps.
Given a list of records, a set of key columns that define uniqueness, and a deduplication strategy, return the deduplicated list of records.
"first": keep the first occurrence of each unique key
"last": keep the last occurrence of each unique key
"most_complete": keep the record with the fewest None values. Ties are broken by keeping the first occurrence.
The output list must preserve the order of first appearance of each unique key in the input. For example, if key X first appears at index 2 and key Y first appears at index 0, then key Y's selected record comes before key X's selected record in the output.
Return a list of the selected records (as dicts).
Input:
records = [ {"id": 1, "name": "Alice", "email": "alice@test.com"}, {"id": 2, "name": "Bob", "email": "bob@test.com"}, {"id": 1, "name": "Alice Smith", "email": "alice.s@test.com"}, ] key_columns = ["id"] strategy = "first"
Output:
[{"id": 1, "name": "Alice", "email": "alice@test.com"}, {"id": 2, "name": "Bob", "email": "bob@test.com"}]
Two records share id=1. Strategy "first" keeps the earliest occurrence.
Input:
records = [ {"id": 1, "name": None, "email": "a@test.com", "phone": None}, {"id": 1, "name": "Alice", "email": "a@test.com", "phone": "555-1234"}, ] key_columns = ["id"] strategy = "most_complete"
Output:
[{"id": 1, "name": "Alice", "email": "a@test.com", "phone": "555-1234"}]
The second record has 0 None values vs 2 in the first, so it wins.
Use an ordered dictionary or a list to track the first-appearance order of unique keys.
For "most_complete", count None values across all fields, not just key columns.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: string