ETL Deduplication
ETL Deduplication
Duplicate records are one of the most common data quality issues in ETL pipelines. Events can be sent twice, API retries create duplicates, and merging data sources introduces overlaps.
Given a list of records, a set of key columns that define uniqueness, and a deduplication strategy, return the deduplicated list of records.
Strategies
-
"first": keep the first occurrence of each unique key
-
"last": keep the last occurrence of each unique key
-
"most_complete": keep the record with the fewest None values. Ties are broken by keeping the first occurrence.
Output Order
The output list must preserve the order of first appearance of each unique key in the input. For example, if key X first appears at index 2 and key Y first appears at index 0, then key Y's selected record comes before key X's selected record in the output.
Return a list of the selected records (as dicts).
Examples
Input:
records = [ {"id": 1, "name": "Alice", "email": "alice@test.com"}, {"id": 2, "name": "Bob", "email": "bob@test.com"}, {"id": 1, "name": "Alice Smith", "email": "alice.s@test.com"}, ] key_columns = ["id"] strategy = "first"
Output:
[{"id": 1, "name": "Alice", "email": "alice@test.com"}, {"id": 2, "name": "Bob", "email": "bob@test.com"}]
Two records share id=1. Strategy "first" keeps the earliest occurrence.
Input:
records = [ {"id": 1, "name": None, "email": "a@test.com", "phone": None}, {"id": 1, "name": "Alice", "email": "a@test.com", "phone": "555-1234"}, ] key_columns = ["id"] strategy = "most_complete"
Output:
[{"id": 1, "name": "Alice", "email": "a@test.com", "phone": "555-1234"}]
The second record has 0 None values vs 2 in the first, so it wins.
Hint 1
Use an ordered dictionary or a list to track the first-appearance order of unique keys.
Hint 2
For "most_complete", count None values across all fields, not just key columns.
Requirements
- Group records by the values of the key columns
- Apply the correct strategy to select one record per group
- Preserve first-appearance order of unique keys in the output
- Handle composite keys (multiple key columns)
Constraints
- 1 <= len(records) <= 10000
- 1 <= len(key_columns) <= 5
- strategy is one of "first", "last", or "most_complete"
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: array
Accepts: string
ETL Deduplication
ETL Deduplication
Duplicate records are one of the most common data quality issues in ETL pipelines. Events can be sent twice, API retries create duplicates, and merging data sources introduces overlaps.
Given a list of records, a set of key columns that define uniqueness, and a deduplication strategy, return the deduplicated list of records.
Strategies
-
"first": keep the first occurrence of each unique key
-
"last": keep the last occurrence of each unique key
-
"most_complete": keep the record with the fewest None values. Ties are broken by keeping the first occurrence.
Output Order
The output list must preserve the order of first appearance of each unique key in the input. For example, if key X first appears at index 2 and key Y first appears at index 0, then key Y's selected record comes before key X's selected record in the output.
Return a list of the selected records (as dicts).
Examples
Input:
records = [ {"id": 1, "name": "Alice", "email": "alice@test.com"}, {"id": 2, "name": "Bob", "email": "bob@test.com"}, {"id": 1, "name": "Alice Smith", "email": "alice.s@test.com"}, ] key_columns = ["id"] strategy = "first"
Output:
[{"id": 1, "name": "Alice", "email": "alice@test.com"}, {"id": 2, "name": "Bob", "email": "bob@test.com"}]
Two records share id=1. Strategy "first" keeps the earliest occurrence.
Input:
records = [ {"id": 1, "name": None, "email": "a@test.com", "phone": None}, {"id": 1, "name": "Alice", "email": "a@test.com", "phone": "555-1234"}, ] key_columns = ["id"] strategy = "most_complete"
Output:
[{"id": 1, "name": "Alice", "email": "a@test.com", "phone": "555-1234"}]
The second record has 0 None values vs 2 in the first, so it wins.
Hint 1
Use an ordered dictionary or a list to track the first-appearance order of unique keys.
Hint 2
For "most_complete", count None values across all fields, not just key columns.
Requirements
- Group records by the values of the key columns
- Apply the correct strategy to select one record per group
- Preserve first-appearance order of unique keys in the output
- Handle composite keys (multiple key columns)
Constraints
- 1 <= len(records) <= 10000
- 1 <= len(key_columns) <= 5
- strategy is one of "first", "last", or "most_complete"
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: array
Accepts: string