Before loading raw data into a warehouse, ETL pipelines must validate records against a schema to catch corrupt or malformed entries early. Bad data that slips through can silently poison downstream models and analytics.
Given a list of records and a schema definition, validate each record and report which ones pass or fail, along with the specific errors found.
Missing column: the column defined in the schema does not exist in the record
Null check: if the value is None and the column is not nullable
Type check: the value's type must match the expected type. The "float" type accepts both int and float values. Use Python's type() for checking (so booleans are not treated as ints).
Range check: if min/max bounds are specified and the value is numeric, it must fall within [min, max] inclusive
If a column is missing, skip remaining checks for it. If a value is None and nullable, skip type and range checks. If type check fails, skip range check.
Use these exact formats: "{column}: missing", "{column}: null", "{column}: expected {type}, got {actual_type}", "{column}: out of range"
Return a list of tuples (record_index, is_valid, errors) where record_index is 0-based, is_valid is a boolean, and errors is a list of error message strings (empty if valid).
Input:
records = [{"name": "Alice", "age": 30, "score": 95.5}, {"name": "Bob", "age": 25, "score": 88.0}] schema = [ {"column": "name", "type": "str", "nullable": False}, {"column": "age", "type": "int", "nullable": False, "min": 0, "max": 150}, {"column": "score", "type": "float", "nullable": False, "min": 0, "max": 100}, ]
Output:
[(0, True, []), (1, True, [])]
Both records pass all validation checks.
Input:
records = [{"age": "old", "score": None}] schema = (same as above)
Output:
[(0, False, ["name: missing", "age: expected int, got str", "score: null"])]
Three errors found: name column is missing, age has wrong type, score is null but not nullable.
Use type(value) instead of isinstance() to distinguish booleans from integers.
Process checks in order and skip later checks when an earlier one fails for a column.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Before loading raw data into a warehouse, ETL pipelines must validate records against a schema to catch corrupt or malformed entries early. Bad data that slips through can silently poison downstream models and analytics.
Given a list of records and a schema definition, validate each record and report which ones pass or fail, along with the specific errors found.
Missing column: the column defined in the schema does not exist in the record
Null check: if the value is None and the column is not nullable
Type check: the value's type must match the expected type. The "float" type accepts both int and float values. Use Python's type() for checking (so booleans are not treated as ints).
Range check: if min/max bounds are specified and the value is numeric, it must fall within [min, max] inclusive
If a column is missing, skip remaining checks for it. If a value is None and nullable, skip type and range checks. If type check fails, skip range check.
Use these exact formats: "{column}: missing", "{column}: null", "{column}: expected {type}, got {actual_type}", "{column}: out of range"
Return a list of tuples (record_index, is_valid, errors) where record_index is 0-based, is_valid is a boolean, and errors is a list of error message strings (empty if valid).
Input:
records = [{"name": "Alice", "age": 30, "score": 95.5}, {"name": "Bob", "age": 25, "score": 88.0}] schema = [ {"column": "name", "type": "str", "nullable": False}, {"column": "age", "type": "int", "nullable": False, "min": 0, "max": 150}, {"column": "score", "type": "float", "nullable": False, "min": 0, "max": 100}, ]
Output:
[(0, True, []), (1, True, [])]
Both records pass all validation checks.
Input:
records = [{"age": "old", "score": None}] schema = (same as above)
Output:
[(0, False, ["name: missing", "age: expected int, got str", "score: null"])]
Three errors found: name column is missing, age has wrong type, score is null but not nullable.
Use type(value) instead of isinstance() to distinguish booleans from integers.
Process checks in order and skip later checks when an earlier one fails for a column.
Sign in to take notes on this problem
Accepts: array
Accepts: array