What is a Large Language Model?
A Large Language Model (LLM) is a deep learning program pre-trained on massive text corpora to understand and generate human-like text. Modern LLMs are built on the Transformer architecture, utilizing self-attention mechanisms to process tokens in parallel and capture long-range contextual relationships.
📐 1. Transformer Attention Flow
The diagram below illustrates how raw token streams are mapped to vector spaces, routed through multi-head self-attention and feed-forward blocks, and projected back to vocabulary logits:
🏗️ 2. Core Architectural Paradigms
Transformer models are divided into three primary topologies depending on their attention masking and intended downstream tasks:
Decoder-Only (Causal Language Modeling)
- Mechanism: Employs causal masking, meaning each token can only attend to previous tokens in the sequence. It prevents the model from “looking ahead” during training.
- Best For: Autoregressive text generation, conversational AI, and reasoning tasks.
- Examples: GPT-4, Claude 3.5, Llama 3, Mistral.
Encoder-Only (Masked Language Modeling)
- Mechanism: Utilizes bi-directional attention, allowing each token to attend to all other tokens in the sequence (both left and right). During training, random tokens are masked, and the model predicts them.
- Best For: Representation learning, text classification, named entity recognition (NER), and embedding generation.
- Examples: BERT, RoBERTa.
Encoder-Decoder (Sequence-to-Sequence)
- Mechanism: The encoder processes the input sequence bi-directionally to create a rich context representation, which is then passed to the decoder. The decoder generates the output sequence autoregressively using causal attention.
- Best For: Machine translation, document summarization, and text-to-text transformation tasks.
- Examples: T5, BART.
💾 3. Context Length & KV Caching
Context Window Constraints
The context window is the maximum token capacity (input prompt + generated output) the model can process in a single forward pass.
- Attention Complexity: In standard self-attention, computation scales quadratically,
$O(N^2)$, where$N$is the sequence length. This makes processing long contexts computationally expensive. - Recent Advances: Modern techniques like RoPE (Rotary Position Embeddings), FlashAttention, and Multi-Query Attention (MQA) allow models to support context windows of 128k to 1M+ tokens.
Key-Value (KV) Caching
During autoregressive text generation, the model predicts one token at a time. At each step $t$, computing attention requires generating Key ($K$) and Value ($V$) vectors for all prior tokens $0 \dots t-1$.
Without optimization, this leads to redundant, quadratic calculations. KV Caching solves this by storing the $K$ and $V$ vectors of past tokens in GPU memory (VRAM). At each generation step, the model only computes the Query ($Q$), Key ($K$), and Value ($V$) for the newly generated token, reducing the runtime complexity of generation to $O(N)$.
| Dimension | Without KV Cache | With KV Cache |
|---|---|---|
| Compute Complexity | $O(N^2)$ (Recomputes all tokens) | $O(N)$ (Only computes new token) |
| Memory Bandwidth Cost | Low | High (Constant reading/writing to VRAM) |
| GPU VRAM Overhead | None | High (Scales linearly with batch size and sequence length) |
🔗 Related Sections
- How LLMs Are Built — Pre-training, Supervised Fine-Tuning, and Alignment loops.
- LLM Settings & Parameters — Sampling controls, Temperature scaling, and logit manipulation.
- LLMOps (Operations) — High-throughput serving engines (vLLM) and model quantization.