GenAI for Beginners:  What is the Top-p sampling in a model

When working with Large Language Models (LLMs), it is essential to understand specific key parameters that influence the model’s behaviour. Two of the most critical parameters are:

1. Temperature
2. Top-P (nucleus)

In the last part, we discussed the Temperature parameter. Read here –

GenAI for Beginners:  What is the Temperature parameter in a model

Top-p sampling, also called nucleus sampling, is a method used to control how random or focused a model’s output is. Instead of picking from all possible next words, it looks at the most likely ones that together make up a set percentage of probability (like 90%). Then, it randomly picks the next word from that smaller group.

The value of Top-p sampling varies between 0 and 1.

The Top-p sampling of an LLM,

1.  It is a method for selecting the next word in a sentence generated by a language model.
2. It looks at only the most likely words — the ones that together make up a chosen percentage of probability (called p).
3. This group of likely words is called the “nucleus”.
4. Instead of checking every possible word, it picks from this smaller, more focused group.
5. The p value goes from 0 to 1 (or 0% to 100%) — a higher p means more possible words to choose from, and more randomness.

How does temperature work?

Its value ranges from  0 to 1, which controls the focus and creativity of the model’s responses.

Low values (e.g., 0.1):

• The model sticks closely to the most likely words.
• Output is more focused, predictable, and consistent.
• It only chooses from the top 10% most likely word options.

High values (e.g., 0.9)

• The model explores more word options.
• The output is more creative, diverse, and sometimes unexpected.
• It picks from the top 90% most likely word options.

It is a good idea to keep the top-p value as mentioned below:

While working with a model, you can pass the Top-p value as shown below :

response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Complete this sentence: The weather today is"}],
    top_p=0.1  # Very focused
)

OpenAI recommends either using temperature or the top_p value

I hope you now have a basic understanding of the top-p parameter of models and in next post, we will delve into other topics. Thanks for reading.