Implement Autoregressive Decoding with a KV Cache

Name: TensorTonic
Author: TensorTonic

Decode autoregressively with an append-only key-value cache so previously processed tokens are never recomputed.