Top-p sampling

Last updated Mar 15, 2023 Edit Source

• Decoding

• 등장 배경: The probability distributions we sample from are dynamic. (참고: Top-k sampling)
  • When the distribution $P_t$ is flatter, a limited $k$ removes many viable options.
  • When the distribution $P_t$ is peakier, a high $k$ allows for too many options to have a chance of being selecte.
• 이 문제를 해결할 방법: Top-p sampling
  • Sample from all tokens in the top $p$ cumulative probability mass (i.e., where mass is concentrated).
  • Varies k depending on the uniformity of $P_t$.