How Large Language Models Are Built

The construction of production-grade Large Language Models is divided into three consecutive phases: Self-supervised pre-training, Supervised Fine-Tuning (SFT), and Preference Alignment.

📈 1. Training & Alignment Pipeline

The following flowchart maps the transition from raw text datasets to an aligned model suitable for conversational deployment:

🚨 2. The Three Stages of Training

Stage 1: Self-Supervised Pre-Training (Base Model)

• Objective: Learn grammar, world knowledge, and semantic representations by predicting the next token in a sequence.
• Compute scale: Extremely high (runs on thousands of GPUs for weeks/months, processing trillions of tokens).
• Outcome: A Base Model that completes sentences but does not follow instructions. If asked “Write a Python script to sort a list”, it might output another homework query rather than the code.

Stage 2: Supervised Fine-Tuning (SFT / Instruct Model)

• Objective: Teach the base model to follow commands, format outputs, and act as a conversational assistant.
• Dataset: Curated high-quality instruction-response pairs (e.g., <system_prompt> Write python code... \n <assistant> def sort...).
• Outcome: An Instruct Model that understands instructions and responds in the correct syntax format.

Stage 3: Preference Alignment (Production Model)

• Objective: Align model responses to match human values (helpfulness, truthfulness, safety) and prevent generation of harmful content.
• Methods:
  • RLHF (Reinforcement Learning from Human Feedback): Trains a separate Reward Model on human-ranked output pairs, then refines the SFT model using PPO (Proximal Policy Optimization). Highly complex and computationally heavy.
  • DPO (Direct Preference Optimization): Directly optimizes the policy model on preferred vs. rejected output pairs, bypassing the need for a separate reward model. Highly stable and computationally efficient.

⚡ 3. Training Efficiency & Parameter-Efficient Fine-Tuning (PEFT)

Training full models (billions of parameters) requires massive GPU memory. Developers use efficiency configurations to train models on single or multi-GPU environments:

LoRA (Low-Rank Adaptation)

LoRA freezes the base model weights $W_0$ and injects trainable rank decomposition matrices into the self-attention projection layers. This restricts training update modifications to a small low-rank space, reducing trainable parameter counts by up to 99% without degrading model capabilities:

\[W = W_0 + \Delta W = W_0 + B \cdot A\]

Where $W_0 \in \mathbb{R}^{d \times k}$ represents the frozen base weights, $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ are the low-rank updates, and the rank $r \ll d, k$ (typically $r = 8$ or $16$).

QLoRA (Quantized LoRA)

Extends LoRA by quantizing the base model weights to a special 4-bit NormalFloat (NF4) representation. It implements double quantization (quantizing the quantization constants) and utilizes paged optimizers to manage memory spikes during gradient updates. QLoRA enables fine-tuning a 70B parameter model on a single 48GB GPU.

FlashAttention

An IO-aware exact attention algorithm that accelerates training speed by 2x to 4x. Standard attention computes and stores the $N \times N$ attention matrix in GPU HBM memory. FlashAttention shards the computation using tiling, loading vectors into fast GPU SRAM memory and computing attention locally without writing the massive, intermediate attention matrix back to HBM.

Distributed Scale (FSDP & ZeRO)

When models do not fit on a single GPU, developers shard parameters across GPU clusters:

• ZeRO (Zero Redundancy Optimizer): Shards optimizer states (ZeRO-1), gradients (ZeRO-2), and model parameters (ZeRO-3) across data-parallel processes.
• FSDP (Fully Sharded Data Parallel): PyTorch’s native implementation of ZeRO-3, dynamically sharding model states during the forward and backward passes.

🔗 Related Sections

• What is an LLM? — Transformer layers, self-attention, and context length.
• LLM Settings & Parameters — Logit sampling math and custom Python implementation.
• LLMOps (Operations) — Deploying models to production using vLLM, speculative decoding, and quantization.