TensorTonicTensorTonic
Problems
Study PlansLabsProjectsInterviewPricingFeedback

Implement ML Papers From Scratch

Each track breaks one landmark paper into small steps. You implement every step in Python in the browser, and tests check your code before you move on. If you are new, start with the Transformer, or read which ML papers to implement from scratch to choose a track.

Transformers and LLMs

Convolutional networks

Sequence models and embeddings

Generative models

BERT: Pre-training of Deep Bidirectional Transformers

BERT: Pre-training of Deep Bidirectional Transformers badge
1

WordPiece Tokenization

Subword tokenization with ## prefix

Hard
2

Segment Embeddings

Token type IDs for sentence pairs

Easy
3

Masked Language Modeling

15% masking with 80-10-10 strategy

Medium
4

Next Sentence Prediction

Binary classification pre-training task

Hard
5

BERT Pooler

[CLS] token extraction for classification

Easy
6

Fine-tuning Architecture

Adding task-specific heads

Hard
Loading architecture visualization...