Kaggle vs Implementing ML From Scratch: Which Builds Which Skill?

Is Kaggle enough to learn ML, or should you implement from scratch?

Kaggle alone is not enough to learn ML. Competitions train applied modeling on real data, such as cleaning, features, validation and experiments. Implementing from scratch trains the algorithms, metrics and numerics underneath the library calls. Most learners need both, and since ML coding interviews test the second skill more often, build it first and use Kaggle to apply it.

Kaggle is a platform for data science competitions and datasets, where you build models with any libraries you like and are scored on held-out data. Implementing from scratch means writing the algorithm yourself in NumPy or PyTorch and testing it against a reference. The essay on learning machine learning from scratch makes the case for learning this way, so this guide sticks to the skills each approach builds.

Which skills does Kaggle build, and which does implementing from scratch build?

Kaggle builds the applied skills above the library call, and implementing from scratch builds the skills below it. Picture one churn-prediction project as a core sample, with the line where you write model.fit as the ground surface. Above it is work a competition rewards, and below it is work a library does for you.

The two coverage shapes barely overlap, which is why neither approach is enough alone. The top layer is left empty by both, because a competition and a from-scratch problem both hand you the target and the metric, and choosing them is a skill you build on your own projects or at work.

Is Kaggle worth it?

Kaggle is worth it for applied practice, where it teaches habits that real modeling work needs. The datasets come with missing values, inconsistent formats and imbalanced classes, so you learn to explore before you model. In many tabular competitions, feature work moves the score more than model choice, which teaches you to test each feature and keep a log.

Competitions also teach validation discipline. The public leaderboard scores part of the test data and the final ranking uses a separate private part, so tuning to the public score can drop your final place sharply. Reading the strongest solutions after a competition ends is one of the most useful things you can do.

The limit is that you can get a strong score while every model stays a black box. You can place well with a boosting library and still not know how a decision tree scores a candidate split, and from-scratch practice is where you learn that, starting with the lesson on information gain.

What does implementing ML from scratch teach that library code hides?

Implementing from scratch teaches the exact definitions, edge cases and numerical details that a library call hides. ROC AUC is a good example. It equals the probability that a randomly chosen positive is scored above a randomly chosen negative, with ties counted as one half, and that gives a short implementation using ranks.

import numpy as np

def roc_auc(y_true, scores):
    y = np.asarray(y_true).astype(bool)
    s = np.asarray(scores, dtype=float)
    order = np.argsort(s, kind="mergesort")
    s_sorted = s[order]
    ranks = np.empty(len(s))
    i = 0
    while i < len(s):
        j = i
        while j + 1 < len(s) and s_sorted[j + 1] == s_sorted[i]:
            j += 1
        ranks[order[i:j + 1]] = (i + j) / 2 + 1
        i = j + 1
    n_pos, n_neg = y.sum(), (~y).sum()
    return (ranks[y].sum() - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)

Writing this yourself shows why tied scores need average ranks, why AUC ignores the decision threshold, and why it is undefined when one class is missing, since the last line then divides by zero. Those are the follow-up questions interviewers ask. You can practice it in AUC-ROC from Scratch and read the integration and AUC lesson for the area-under-the-curve view.

The same thing happens with losses. A from-scratch log loss returns infinity or NaN the first time a probability hits exactly 0 or 1, and fixing it teaches you why libraries clip probabilities. With a sigmoid output, cross-entropy gives a gradient of p minus y with respect to the logit, which stays large when the model is confidently wrong, while mean squared error multiplies that gradient by the sigmoid's slope and lets it vanish. The cross-entropy lesson works through it. For the deepest layers of the core sample, the Research section rebuilds papers in tested stages and the CUDA practice path covers kernels.

From-scratch practice has its own limit. Problems use clean, well-specified inputs so the tests can check one idea at a time, and each one has a right answer, while applied work on real data usually does not.

How does one mistake show the difference between the two?

Data leakage shows the difference best, because Kaggle teaches you that it happened and from-scratch practice teaches you where it entered. In this well-known textbook example, the data is pure noise and the labels are random, so no model can beat a coin flip. Choosing features on the full dataset before cross-validation still produces an impressive score.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5000))
y = rng.integers(0, 2, size=100)

X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
print("leaky", cross_val_score(LogisticRegression(), X_leaky, y, cv=5).mean())

pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
print("honest", cross_val_score(pipe, X, y, cv=5).mean())

The leaky version prints about 0.85 and the honest pipeline prints about 0.5. The feature selector saw every label, including those in each validation fold, so it picked the 20 noise columns that happen to match them. Redraw the noise below and the gap stays.

On Kaggle, this lesson usually arrives when a strong validation score collapses on the private leaderboard. If you have built your own folds in K-Fold Split, you already know that anything fitted on the labels belongs inside the fold loop.

Which matters more for ML interviews?

Implementation skill matters in more interview rounds, although it depends on the round. Based on what candidates commonly report, ML interview loops include some mix of the rounds below.

A strong competition result can open a conversation, and the interviewer will then ask you to explain and implement what you used. The ML engineer interview preparation guide walks through each round, and ML System Design covers the round neither approach trains. To rehearse under a clock, try the ML Engineer Mock Interview or Data Science Mock Interview, or see the other mock sets on the assessments page.

Which should you do first for your goal?

Most learners should start with from-scratch practice, and the right balance after that depends on the role you want.

• Complete beginners should start with short problems such as Implement Sigmoid in NumPy and Implement Gradient Descent for a 1D Quadratic, then enter a beginner competition.
• Career switchers aiming at ML engineer roles should lean on implementation through Cracking ML and Cracking Deep Learning, then finish one competition end to end for a project story.
• Aspiring data scientists should balance the two, since the role leans on applied analysis and its interviews probe statistics.
• Research engineers should work mostly from scratch, reproducing papers in the Research section and the Build LLM from Scratch plan.
• GPU and performance engineers should work almost entirely from scratch, following the CUDA Kernels and Triton Kernels plans.

The apply-first camp has a fair point for data science roles, although a few weeks of from-scratch basics make other people's pipelines far easier to learn from.

How do you combine Kaggle and from-scratch practice?

Combine them by letting each competition stage send you back to one thing you implement yourself. This sequence works for any competition.

1. Before you enter, implement the core models you expect to use, such as logistic regression and tree splits.
2. At the start, implement the competition's metric yourself and confirm it matches the official score on a few submissions.
3. While you compete, build grouped or time-based folds where the test set demands it, and keep every fitted step inside the fold loop.
4. When a model surprises you, go back to how it works. A feature selection step that looks too good is usually leakage, and a boosted model that overfits sends you back to how trees split.
5. After it ends, implement one technique from the strongest solutions and write a short postmortem you can use in interviews.

Forking a top public notebook without rebuilding its key steps teaches very little. For how this kind of coding practice fits next to DSA practice, see LeetCode for machine learning.

What else do learners ask about Kaggle and from-scratch practice?

Which problems should you practice first?

Each of these problems builds one from-scratch skill that makes a competition easier. Then browse the problem library, or work through Cracking ML in order.

• Logistic Regression Training Loop has you write the sigmoid, the log loss gradient and the update yourself.
• K-Fold Split (Indices Only) has you build the folds that every validation scheme starts from.
• Compute AUC (Area Under ROC) has you compute one of the most common competition metrics from scores and labels.
• Binary Log Loss per Sample in Python has you compute a loss that many binary classification competitions use as their metric.

Related guides

• Learning machine learning from scratch
• How to prepare for an ML engineer interview
• ML coding interview questions: 20 to implement
• How to explain an ML project in an interview
• How to approach an ML take-home assignment