TechiSeek

VendorsFor Vendors

On the Fundamental Limits of Prefill-Based Decoding Length Prediction

arXiv cs.AI

On the Fundamental Limits of Prefill-Based Decoding Length Prediction

Operators using prefill-based decode-length predictors should stop treating reported accuracy as a property of the predictor alone. The paper shows that when the target model, dataset, or sampling configuration changes, accuracy collapses even though the predictor is unchanged, because benchmarks conflate predictor quality with task hardness.

Implementation details

  • Derive bounds that every prefill-side method must obey, using only prefill-phase hidden states as input.
  • Residual departures from normality (mild right skew, slight excess kurtosis) match those predicted by the sampling-noise correction.

Examples

  • A broad design space of predictors (MLP, EGTP, FiLM, soft-label, and Mixture-of-Experts variants) converges to similar performance close to the bound under fixed model, data, and sampling conditions.
  • Code and analysis tools are available at the URL referenced in the paper.

Known limitations

  • Current benchmarks conflate predictor quality with task hardness.
Read More