Pandas Interview Questions for Data Scientists, with Code

What is the difference between a Series and a DataFrame?

A Series is a single labelled column of values with one index, and a DataFrame is a table of Series that share the same row index, with one label per column. Selecting one column with single brackets returns a Series, while a list of column names in double brackets returns a DataFrame, even when the list holds only one name. That difference matters because many DataFrame methods, such as merge, do not exist on a Series. Practise both forms with Column Selection and Multi-Column Selection.

import pandas as pd

df = pd.DataFrame({"user": ["a", "b", "c"], "clicks": [3, 5, 2]})
clicks_series = df["clicks"]
clicks_frame = df[["clicks"]]

What is the difference between loc and iloc?

The loc indexer selects rows and columns by label, and iloc selects them by integer position. A slice with loc includes its end label, while a slice with iloc stops before its end position, so the two lines below return the same two rows from a DataFrame whose index is 10, 20 and 30. The confusion shows up most after filtering or sorting, when the labels no longer match the positions. Practise both with Loc vs iLoc.

scores = pd.DataFrame({"score": [90, 75, 60]}, index=[10, 20, 30])
by_label = scores.loc[10:20, "score"]
by_position = scores.iloc[0:2, 0]

How do you filter rows on several conditions?

You filter rows on several conditions by combining boolean Series with the operators &, | and ~ and wrapping each condition in parentheses. The Python keywords and and or raise a ValueError on a Series, because pandas cannot reduce a whole column to one True or False. The isin method replaces a chain of equality checks, and query offers a string syntax that some people find easier to read. The Boolean Indexing problem drills this.

active_us = df[(df["country"] == "US") & (df["orders"] >= 3)]
paid = df[df["plan"].isin(["pro", "plus"])]
active_us_query = df.query("country == 'US' and orders >= 3")

How do you find and handle missing values?

You find missing values with isna, which returns True for NaN, None and NaT, and then decide per column whether to fill them with fillna or drop the rows with dropna. Comparing a value with NaN using == is always False, so isna is the only reliable test. Interviewers also check that you know how aggregations treat gaps, since count and mean skip missing values while the sum of a group with only missing values is 0. Practise the choices with Handle Missing Values.

missing_per_column = df.isna().sum()
df["revenue"] = df["revenue"].fillna(0)
df = df.dropna(subset=["user_id"])

How do you fix columns that have the wrong data type?

You fix data types with to_datetime for dates, to_numeric for numbers stored as text, and astype for everything else, such as category for repeated labels or Int64 for integers with gaps. Passing errors="coerce" to to_numeric turns values such as n/a into NaN instead of raising an error, which you can then count. The nullable Int64 type keeps whole numbers as integers when some are missing, where a plain integer column with gaps is stored as float64. Start with Data Types Overview, then practise the conversions with Change Data Types.

df["signup_date"] = pd.to_datetime(df["signup_date"])
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")
df["plan"] = df["plan"].astype("category")
df["visits"] = df["visits"].astype("Int64")

How do you remove duplicate rows and keep the latest record?

You remove duplicates with drop_duplicates, using subset to name the columns that define a duplicate and keep to choose which copy survives. To keep the latest record per user, sort by the timestamp first and keep the last row for each user. Counting duplicates with duplicated before you drop them is a good habit, because a large count usually means the table has a different grain than you assumed. Practise it with Drop Duplicates.

repeat_count = df.duplicated(subset=["user_id", "event_date"]).sum()
latest = (
    df.sort_values("updated_at")
    .drop_duplicates(subset="user_id", keep="last")
)

How do groupby and agg work together?

The groupby method splits a DataFrame into groups by one or more keys, and agg computes one or more summaries per group, returning one row per group. Named aggregation, where each keyword names an output column and gives a column and a function, produces readable column names without renaming afterwards. Know the difference between count, which skips missing values, size, which counts every row, and nunique, which counts distinct values. Start with GroupBy Basics, then Aggregation Functions and Multi-Level GroupBy.

summary = (
    orders.groupby("customer_id")
    .agg(
        n_orders=("order_id", "count"),
        revenue=("amount", "sum"),
        avg_order=("amount", "mean"),
    )
    .reset_index()
)

What is the difference between transform and agg?

The agg method returns one row per group, while transform returns a result with the same index as the original DataFrame, repeating each group's value on every row of that group. That makes transform the right tool when you need a group statistic next to each row, such as each order's share of its customer's total spend. It plays the same role as a SQL window function with PARTITION BY and no ORDER BY.

orders["customer_total"] = orders.groupby("customer_id")["amount"].transform("sum")
orders["share"] = orders["amount"] / orders["customer_total"]

How do you get the top N rows per group?

You get the top N rows per group by sorting on the group key and the ranking column, then taking the first N rows of each group with groupby and head. When values tie at the cutoff, head still returns exactly N rows and drops the other tied rows, so if the question says ties should all be kept, use rank with method="dense" and filter on the rank instead. The StrataScratch alternatives guide shows the dense rank version next to the SQL query, and the SQL version of the tie question is RANK and DENSE_RANK.

top3 = (
    df.sort_values(["department", "salary"], ascending=[True, False])
    .groupby("department")
    .head(3)
)

What merge types does pandas support, and what does validate do?

The merge method supports inner, left, right, outer and cross joins through its how argument, and it matches rows on the columns given in on, or in left_on and right_on when the names differ. The validate argument checks the relationship you expect, such as many_to_one, and raises a MergeError if the right table has duplicate keys that would multiply rows. Adding indicator=True creates a _merge column that tells you which rows found a match, which is the quickest way to answer a question about customers with no orders. Practise it with Merge DataFrames.

merged = orders.merge(
    customers,
    on="customer_id",
    how="left",
    validate="many_to_one",
    indicator=True,
)
unmatched = merged[merged["_merge"] == "left_only"]

When do you use concat instead of merge?

You use concat to stack DataFrames that hold the same kind of rows, such as one file per month, and merge to join tables on a key. By default concat stacks rows and aligns columns by name, filling NaN where a column is missing from one input, and ignore_index=True gives the result a fresh index. With axis=1 it places DataFrames side by side, aligned on the index. Collect pieces in a list and call concat once, because concatenating inside a loop copies the growing DataFrame every time. The Concatenate DataFrames problem covers both directions.

all_months = pd.concat([jan, feb, mar], ignore_index=True)
side_by_side = pd.concat([features, labels], axis=1)

How do pivot_table and melt reshape data?

The pivot_table method turns long data into wide data by spreading the values of one column across new columns and aggregating any repeated pairs, and melt does the reverse by turning columns into rows. The plain pivot method raises a ValueError when an index and column pair appears more than once, which is why pivot_table with an explicit aggfunc is the safer choice. The fill_value argument replaces the NaN that appears where a combination has no data. Practise with Pivot Tables, Melt Wide to Long and Cross Tabulation, which counts combinations of two categorical columns. The unstack method is a third option that moves an index level into the columns, and Unstack Long to Wide covers it.

wide = sales.pivot_table(
    index="region",
    columns="month",
    values="revenue",
    aggfunc="sum",
    fill_value=0,
)
long = wide.reset_index().melt(
    id_vars="region",
    var_name="month",
    value_name="revenue",
)

How do you compare each row with the previous one and compute moving averages?

You compare a row with the previous one using shift, which moves values down by one position, and you compute moving averages with rolling followed by an aggregation such as mean. Sort by time first, and group by the entity when the data holds several of them, so values never leak from one store or user into the next. In SQL terms shift is LAG, cumsum is a running total, and rolling is a window frame. A rolling window can also be a time span such as 7D on a DatetimeIndex, which handles missing days correctly. The SQL versions are LAG and LEAD and Running Totals.

daily = daily.sort_values(["store", "date"])
by_store = daily.groupby("store")["revenue"]
daily["prev_revenue"] = by_store.shift(1)
daily["change"] = daily["revenue"] - daily["prev_revenue"]
daily["avg_7d"] = by_store.transform(lambda s: s.rolling(7, min_periods=1).mean())
daily["running_total"] = by_store.cumsum()

When should you use apply, and when a vectorised operation?

You should use a vectorised operation whenever one exists, and keep apply for logic that cannot be written with column arithmetic, numpy functions or the built-in string and date methods. A vectorised expression runs in compiled code over the whole column, while apply with axis=1 calls a Python function once per row, which is far slower on large DataFrames. numpy.where handles a two-way condition and numpy.select handles several, which covers most of the cases people reach for apply to solve. The Apply Custom Transforms problem covers the cases where apply is the right tool.

import numpy as np

df["total"] = df["price"] * df["qty"]
df["tier"] = np.where(df["total"] >= 100, "high", "low")
df["band"] = np.select(
    [df["total"] >= 100, df["total"] >= 30],
    ["high", "mid"],
    default="low",
)

How do you make pandas code faster on a large dataset?

You make pandas code faster by reading only the columns you need, choosing compact data types, replacing row loops and apply with vectorised operations, and building results in one step instead of growing them in a loop. A category dtype stores repeated labels as small integer codes, which saves memory and speeds up groupby on those columns. The iterrows method is the slowest way to touch every row, so treat it as a last resort. When the data does not fit in memory, say so and suggest reading it in chunks or aggregating in the database first.

events = pd.read_csv(
    "events.csv",
    usecols=["user_id", "event", "ts"],
    dtype={"event": "category"},
    parse_dates=["ts"],
)

Do data scientist interviews ask pandas questions?

Many data scientist interviews include a pandas or Python exercise, either live or as a take-home, and it often repeats a question from the SQL round with a DataFrame. Analytics roles tend to ask for filtering, grouping, merging and reshaping, while modeling roles add feature preparation such as handling missing values and changing data types. Ask your recruiter whether your loop has a pandas round and whether you can look up documentation during it.

Should you answer a data question in SQL or in pandas?

Answer in the language the interviewer asks for, and if you get to choose, use the one the team works in day to day. SQL fits questions about data that already sits in a database, and pandas fits questions that need several steps of cleaning, reshaping or custom logic in Python. Being able to translate one into the other is a common follow-up, so practise both versions of the same question.

Which pandas functions should you know for an interview?

You should know loc and iloc, boolean filtering with isin and between, isna and fillna, astype and to_datetime, groupby with agg and transform, merge and concat, pivot_table and melt, sort_values, drop_duplicates, rank, shift, rolling and cumsum. A typical interview task combines several of these, so practise chaining them on one small DataFrame until you can write them without looking anything up.

Can you use loops or apply in a pandas interview?

You can use apply or a loop when the logic truly needs it, but interviewers usually expect a vectorised answer first and often ask about the difference. Column arithmetic, numpy.where, numpy.select and the built-in string and date methods cover most row-wise logic without apply. If you do use apply, say why the vectorised version would not work and what it costs on a large DataFrame.

Is there a timed pandas practice test?

Yes, TensorTonic has a 60-minute Pandas + NumPy Sprint in its Data Scientist track that covers pandas and NumPy fundamentals. The free Data Science Mock Interview is a 75-minute sitting with five problems that mix SQL, pandas, NumPy, math and ML, and it ends with a scored report.