🛡️

Running AI agents in production? Harness governs spend, access, and audit trails—so your team maintains control while agents safely handle production workflows. Visit →

LLM Settings & Parameters

When querying a Large Language Model, the model does not output tokens directly. Instead, it generates a raw vector of unnormalized scores (logits) across its entire vocabulary. Generation parameters control how these logits are scaled, filtered, and sampled to produce the next token.

📐 1. Logit Sampling Mathematics

Temperature Scaling

Temperature ( $T$ ) alters the shape of the probability distribution before tokens are selected. Logits ( $z_i$ ) are divided by the temperature value prior to applying the Softmax function:

\[p_i = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}\]

Low Temperature ( $T \to 0$ ): Amplifies the difference between logits. The token with the highest logit dominates the probability mass, making generation highly deterministic (greedy decoding).
High Temperature ( $T > 1.0$ ): Flattens the distribution, reducing the gap between high-probability and low-probability tokens. This increases output variety and creativity, but raises the likelihood of grammar errors or hallucinations.

Top-k Sampling

Top-k restricts the sampling pool to the $k$ tokens with the highest logits. Any token outside this top group has its probability set to $0$ .

Best For: Preventing the model from selecting highly inappropriate or nonsensical words (the “long tail” of low-probability tokens).

Top-p (Nucleus) Sampling

Top-p restricts the sampling pool to the smallest set of tokens whose cumulative probability exceeds the threshold $p$ (e.g., $p = 0.9$ ).

Advantage over Top-k: The number of candidate tokens dynamically expands or contracts depending on the model’s confidence. If the model is certain of the next word, the candidate pool shrinks to just 1 or 2 tokens. If the model is uncertain, the pool expands.

🚫 2. Repetition & Presence Penalties

To prevent the model from getting stuck in repetitive loops, penalties modify the raw logits ( $z_i$ ) based on whether and how often tokens have already appeared in the generated context sequence:

\[z'_i = z_i - (\text{frequency\_penalty} \times c_i) - (\text{presence\_penalty} \times \text{sign}(c_i))\]

$c_i$ represents the frequency count of token $i$ in the generated sequence.
$\text{sign}(c_i)$ is $1$ if the token has appeared at least once, and $0$ otherwise.
Frequency Penalty: Penalizes tokens proportionally to how many times they have already occurred, encouraging vocabulary diversity.
Presence Penalty: Applies a flat, one-time penalty for any token that has appeared, encouraging the model to introduce new topics.

💻 3. Python Logit Sampler Implementation

The Python script below illustrates how to implement Temperature, Top-k, and Top-p (Nucleus) sampling from scratch on raw model logits, simulating the logit processing layer of modern inference engines:

import numpy as np
from typing import List
 
def softmax(logits: np.ndarray) -> np.ndarray:
    # Stable Softmax subtraction to prevent floating-point overflow
    exp_logits = np.exp(logits - np.max(logits))
    return exp_logits / np.sum(exp_logits)
 
def sample_next_token(
    logits: List[float], 
    temperature: float = 0.7, 
    top_k: int = 50, 
    top_p: float = 0.9
) -> int:
    """
    Applies temperature scaling, Top-k filtration, and Top-p (nucleus)
    filtration to sample a token from raw logits.
    """
    # Convert input to numpy array
    logits = np.array(logits, dtype=np.float32)
    
    # 1. Apply Temperature Scaling
    # Avoid division by zero: if temperature is extremely low, perform greedy selection
    if temperature <= 1e-5:
        return int(np.argmax(logits))
        
    logits = logits / temperature
 
    # 2. Apply Top-K Filtration
    if top_k > 0:
        # Find the threshold value of the top K elements
        k = min(top_k, len(logits))
        top_k_indices = np.argpartition(logits, -k)[-k:]
        # Mask out all elements not in top_k indices by setting to -infinity
        mask = np.ones_like(logits, dtype=bool)
        mask[top_k_indices] = False
        logits[mask] = -np.inf
 
    # 3. Compute Softmax Probabilities
    probs = softmax(logits)
 
    # 4. Apply Top-P (Nucleus) Filtration
    if 0.0 < top_p < 1.0:
        # Sort probabilities in descending order
        sorted_indices = np.argsort(probs)[::-1]
        sorted_probs = probs[sorted_indices]
        
        # Calculate cumulative sum of probabilities
        cumulative_probs = np.cumsum(sorted_probs)
        
        # Identify indices to remove (cumulative probability is above threshold p)
        # We preserve the first element that crosses the threshold to ensure the pool is non-empty
        indices_to_remove = cumulative_probs > top_p
        indices_to_remove[1:] = indices_to_remove[:-1].copy()
        indices_to_remove[0] = False
        
        # Map back to original indices and mask out eliminated elements
        masked_indices = sorted_indices[indices_to_remove]
        probs[masked_indices] = 0.0
        
        # Renormalize remaining probabilities
        probs_sum = np.sum(probs)
        if probs_sum > 0:
            probs = probs / probs_sum
        else:
            # Fallback to greedy if numerical underflow occurs
            return int(np.argmax(logits))
 
    # 5. Sample token index from the filtered probability distribution
    sampled_index = np.random.choice(len(probs), p=probs)
    return int(sampled_index)
 
# Example Usage:
# raw_logits = [2.1, 1.5, 0.2, 5.4, 0.1, -1.2, 4.8]  # Raw output from transformer last layer
# next_token = sample_next_token(raw_logits, temperature=0.7, top_k=3, top_p=0.95)
# print(f"Sampled Token Index: {next_token}")

What is an LLM? — Context window limitations and KV caching details.
How LLMs Are Built — Deep dive into pre-training weights and fine-tuning datasets.
LLMOps (Operations) — High-throughput production serving engines (vLLM) and model quantization format comparisons.

LLMOps & Cost Optimization Multi-Modal AI