The Attention Mechanism Before Transformers

• The RNN Encoder-Decoder VS the Attention Mechanism

• The Attention Layer

• The Bahdanau Attention

• The Luong Attention

• Implementing in Pytorch

• Implementing the Bahdanau Attention

• Implementing the Luong Attention

• Implementing the Decoder

• Putting everything together

---