LLMs

LLM Tra in ing: Pretraining, Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF)

Training LLMs involves three stages: pretraining (learn language), SFT (follow instructions), and RLHF (align with humans). Understand each stage's purpose and methods.

SS

Soham Sharma

AI Engineer, Botmartz · July 17, 2026 · 1 min read

◈ Open in Google Colab ⌥ View Code on GitHub

Read Time

1 min

Failure Modes

5

Code Snippets

3

Runnable Notebook

1

LLM Training: Pretraining, Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF)

Stage 1: Pretraining Stage 2: Supervised Fine-Tuning (SFT)Stage 3: RLHF (Reinforcement Learning from Human Feedback)Conclusion

Training modern LLMs is a three-stage pipeline: pretraining (learn from internet-scale text), supervised fine-tuning (learn to follow instructions), and RLHF (align with human preferences). Each stage addresses a specific objective and constraint.

Stage 1: Pretraining

Predict the next token from previous tokens on massive text corpus.

Loss = -log P(token_i | token_1, ..., token_{i-1})

This stage learns language structure, factual knowledge, and reasoning patterns. It's computationally expensive (weeks on thousands of GPUs) but done once per base model.

Stage 2: Supervised Fine-Tuning (SFT)

Fine-tune pretrained model on instruction-following examples.

examples = [
{ "prompt": "What is 2+2?", "completion": "2+2 equals 4." }, { "prompt": "Translate to French: Hello", "completion": "Bonjour" } ]
# Fine-tune on instruction-response pairs # Loss = -log P(completion | prompt)

SFT teaches the model to follow instructions and be helpful. It's much cheaper than pretraining (hours on single GPU).

Stage 3: RLHF (Reinforcement Learning from Human Feedback)

Align model behavior with human preferences. Train a reward model, then use it to fine-tune via reinforcement learning.

1. Humans rate model outputs (e.g., "Response A is better than B")

Train reward model: reward_model(output) -> score
Use reward model to optimize model: max E[reward_model(output)]

RLHF reduces harmful outputs, makes models more helpful, and aligns with user intent.

Conclusion

The three-stage pipeline (pretraining → SFT → RLHF) is the recipe for state-of-the-art LLMs. Understanding each stage's purpose helps in fine-tuning for specific domains. Next: inference optimization—how to make LLMs faster and cheaper to run.

Closing Takeaways

✓

Measure retrieval precision and recall in isolation before touching the model.

✓

Chunk along document structure, not arbitrary character counts.

✓

Combine vector and keyword search — hybrid retrieval beats either alone.

✓

Treat evaluation as continuous infrastructure, not a launch-week report.

Try It Yourself

A runnable Google Colab notebook with the eval harness and hybrid search code from this post.

◈ Open in Colab ⌥ GitHub Repo

#LLMs#Training#RLHF#Fine-Tuning

0 views

SS

Soham Sharma

AI Engineer at Botmartz, building enterprise RAG and agent systems in production. Contributing to open-source libraries.

Discussion (0)

No approved comments yet. Be the first to share your thoughts!