Text chunking splits a sequence of tokens into fixed-size chunks with optional overlap between consecutive chunks. This is a fundamental preprocessing step in NLP pipelines, especially in retrieval-augmented generation (RAG) systems where documents must be split into manageable segments for embedding and retrieval.
Given a list of tokens, a chunk size, and an overlap count, split the tokens into chunks.
Return the chunks as a list of token lists.
Input: tokens = ["a", "b", "c", "d", "e", "f"], chunk_size = 3, overlap = 0
Output: [["a", "b", "c"], ["d", "e", "f"]]
Explanation: A step of three produces two non-overlapping chunks.
Input: tokens = ["a", "b", "c", "d", "e", "f", "g"], chunk_size = 3, overlap = 1
Output: [["a", "b", "c"], ["c", "d", "e"], ["e", "f", "g"]]
Compute the distance between chunk starts as chunk_size minus overlap.
Slice from each start position and stop after the first chunk that reaches the end.
Accepts: array
Accepts: number
Accepts: number
Text chunking splits a sequence of tokens into fixed-size chunks with optional overlap between consecutive chunks. This is a fundamental preprocessing step in NLP pipelines, especially in retrieval-augmented generation (RAG) systems where documents must be split into manageable segments for embedding and retrieval.
Given a list of tokens, a chunk size, and an overlap count, split the tokens into chunks.
Return the chunks as a list of token lists.
Input: tokens = ["a", "b", "c", "d", "e", "f"], chunk_size = 3, overlap = 0
Output: [["a", "b", "c"], ["d", "e", "f"]]
Explanation: A step of three produces two non-overlapping chunks.
Input: tokens = ["a", "b", "c", "d", "e", "f", "g"], chunk_size = 3, overlap = 1
Output: [["a", "b", "c"], ["c", "d", "e"], ["e", "f", "g"]]
Compute the distance between chunk starts as chunk_size minus overlap.
Slice from each start position and stop after the first chunk that reaches the end.
Accepts: array
Accepts: number
Accepts: number