Text chunking splits a sequence of tokens into fixed-size chunks with optional overlap between consecutive chunks. This is a fundamental preprocessing step in NLP pipelines, especially in retrieval-augmented generation (RAG) systems where documents must be split into manageable segments for embedding and retrieval.
Given a list of tokens, a chunk size, and an overlap count, split the tokens into chunks.
Input:
tokens = ["a", "b", "c", "d", "e", "f"], chunk_size = 3, overlap = 0
Output:
[["a", "b", "c"], ["d", "e", "f"]]
With no overlap and step = 3, the tokens are split into two non-overlapping chunks of size 3.
Input:
tokens = ["a", "b", "c", "d", "e", "f", "g"], chunk_size = 3, overlap = 1
Output:
[["a", "b", "c"], ["c", "d", "e"], ["e", "f", "g"]]
With overlap = 1 and step = 2, each consecutive chunk shares its last token with the next chunk's first token. This overlap provides context continuity.
Compute step = chunk_size - overlap. Then iterate from i = 0, stepping by 'step' each time. At each position, take a slice tokens[i:i+chunk_size]. Stop once the current chunk reaches the end of the list.
Use a for loop: for i in range(0, len(tokens), step). Append tokens[i:i+chunk_size]. Break if i + chunk_size >= len(tokens) to avoid producing redundant trailing chunks that are entirely covered by the previous chunk.
Sign in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number
Text chunking splits a sequence of tokens into fixed-size chunks with optional overlap between consecutive chunks. This is a fundamental preprocessing step in NLP pipelines, especially in retrieval-augmented generation (RAG) systems where documents must be split into manageable segments for embedding and retrieval.
Given a list of tokens, a chunk size, and an overlap count, split the tokens into chunks.
Input:
tokens = ["a", "b", "c", "d", "e", "f"], chunk_size = 3, overlap = 0
Output:
[["a", "b", "c"], ["d", "e", "f"]]
With no overlap and step = 3, the tokens are split into two non-overlapping chunks of size 3.
Input:
tokens = ["a", "b", "c", "d", "e", "f", "g"], chunk_size = 3, overlap = 1
Output:
[["a", "b", "c"], ["c", "d", "e"], ["e", "f", "g"]]
With overlap = 1 and step = 2, each consecutive chunk shares its last token with the next chunk's first token. This overlap provides context continuity.
Compute step = chunk_size - overlap. Then iterate from i = 0, stepping by 'step' each time. At each position, take a slice tokens[i:i+chunk_size]. Stop once the current chunk reaches the end of the list.
Use a for loop: for i in range(0, len(tokens), step). Append tokens[i:i+chunk_size]. Break if i + chunk_size >= len(tokens) to avoid producing redundant trailing chunks that are entirely covered by the previous chunk.
Sign in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number