Assign time, height, and width coordinates to an ordered list of multimodal segment dictionaries. The type field selects one schema:
Start a shared offset o at zero. A text token at zero-based position j receives
(t,h,w)=(o+j,o+j,o+j).Text advances the offset by its length. A separator receives (o, o, o) and advances it by one. An image token at zero-based row r and column c receives
(t,h,w)=(o,o+r,o+c).The image advances the offset by max(H,W), where H and W are its grid height and width. A video token at frame f, row r, and column c receives
(t,h,w)=(o+fs,o+r,o+c).Here, t, h, and w are the three coordinate axes; s is frame_stride and F is the frame count. Video advances the offset by max((F−1)s+1,H,W). Emit visual tokens in frame, row, column order. A segment with any zero-sized dimension emits no tokens and does not advance the offset.
Return a Python dictionary with coordinates, a CPU int64 PyTorch tensor of shape (3, packed_length). Its rows contain time, height, and width, respectively; no emitted tokens gives shape (3, 0).
Input: segments = [{"type":"text","length":2},{"type":"image","height":2,"width":2},{"type":"separator"}]
Output: {"coordinates":[[0,1,2,2,2,2,4],[0,1,2,2,3,3,4],[0,1,2,3,2,3,4]]}
Explanation: Text advances to offset two, the image spans two spatial positions, and the separator starts at four.
Input: segments = [{"type":"image","height":1,"width":3}]
Output: {"coordinates":[[0,0,0],[0,0,0],[0,1,2]]}
Input: segments = [{"type":"video","frames":2,"height":1,"width":2,"frame_stride":2}]
Output: {"coordinates":[[0,0,2,2],[0,0,0,0],[0,1,0,1]]}
dict.get reads an optional field with a default value.
range generates zero-based token or grid indices.
Tensor.transpose exchanges the token and coordinate axes.
Accepts: array
Assign time, height, and width coordinates to an ordered list of multimodal segment dictionaries. The type field selects one schema:
Start a shared offset o at zero. A text token at zero-based position j receives
(t,h,w)=(o+j,o+j,o+j).Text advances the offset by its length. A separator receives (o, o, o) and advances it by one. An image token at zero-based row r and column c receives
(t,h,w)=(o,o+r,o+c).The image advances the offset by max(H,W), where H and W are its grid height and width. A video token at frame f, row r, and column c receives
(t,h,w)=(o+fs,o+r,o+c).Here, t, h, and w are the three coordinate axes; s is frame_stride and F is the frame count. Video advances the offset by max((F−1)s+1,H,W). Emit visual tokens in frame, row, column order. A segment with any zero-sized dimension emits no tokens and does not advance the offset.
Return a Python dictionary with coordinates, a CPU int64 PyTorch tensor of shape (3, packed_length). Its rows contain time, height, and width, respectively; no emitted tokens gives shape (3, 0).
Input: segments = [{"type":"text","length":2},{"type":"image","height":2,"width":2},{"type":"separator"}]
Output: {"coordinates":[[0,1,2,2,2,2,4],[0,1,2,2,3,3,4],[0,1,2,3,2,3,4]]}
Explanation: Text advances to offset two, the image spans two spatial positions, and the separator starts at four.
Input: segments = [{"type":"image","height":1,"width":3}]
Output: {"coordinates":[[0,0,0],[0,0,0],[0,1,2]]}
Input: segments = [{"type":"video","frames":2,"height":1,"width":2,"frame_stride":2}]
Output: {"coordinates":[[0,0,2,2],[0,0,0,0],[0,1,0,1]]}
dict.get reads an optional field with a default value.
range generates zero-based token or grid indices.
Tensor.transpose exchanges the token and coordinate axes.
Accepts: array
