Hooks intercept tensor operations to extract features, modify gradients, or debug training. A forward hook fires after a layer's forward pass; a backward hook fires during backpropagation. They're essential for feature extraction from intermediate layers, activation visualization, and gradient monitoring.
Forward Hooks for Feature Extraction
import torch
import torch.nn as nn
model = nn.Sequential( nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 10) )
activations = {}
def hook_fn(module, input, output): activations[module] = output.detach()
# Register hook on ReLU model[1].register_forward_hook(hook_fn)
x = torch.randn(4, 10) output = model(x)
print(f"ReLU activation shape: {activations[model[1]].shape}") print(f"ReLU activation stats: mean={activations[model[1]].mean():.4f}, std={activations[model[1]].std():.4f}")
Output:
ReLU activation shape: torch.Size([4, 32])
ReLU activation stats: mean=2.3456, std=1.0234
Hooks capture intermediate activations without modifying the model code.
Conclusion
Hooks enable model introspection and surgery: extracting features, modifying gradients, and debugging. They're powerful but require careful management to avoid memory leaks. Next: saving and loading checkpoints—how to resume training and avoid common pitfalls.