The vanishing gradient problem occurs when gradients become extremely small as they are propagated backward through a deep neural network. When this happens, earlier layers receive very weak updates and learn extremely slowly or may effectively stop learning.
It commonly occurs with activation functions such as sigmoid and tanh, whose derivatives can be small over parts of their input range. During backpropagation, these small values are multiplied repeatedly across many layers, causing the gradient to shrink.
The problem can be reduced using several techniques:
- ReLU-family activations: ReLU and variants such as Leaky ReLU generally maintain stronger gradients than sigmoid in deep networks.
- Better weight initialization: Methods such as Xavier/Glorot and He initialization help keep activations and gradients at useful scales.
- Batch normalization: Helps stabilize the distribution of activations during training.
- Residual connections: Architectures such as ResNets provide shorter paths for gradients to flow through deep networks.
- Gated architectures: LSTM and GRU networks use gates that help preserve useful information and gradients across long sequences.
In simple terms, the vanishing gradient problem makes early layers learn too slowly because the training signal becomes weaker as it travels backward through the network. Modern activations, normalization, initialization, and network architectures help maintain a stronger learning signal.
