Compute the 50th, 95th, and 99th percentile of a set of observed inference latencies, using linear interpolation between the two closest ranks.
