Seq2seq, Bahdanau Attention, and Why Recurrence Hit a Wall

How do you translate a sentence from English to French using a neural network?

The input and output may have different lengths and word orders. "I am going to the store" becomes the shorter "Je vais au magasin." In "the black cat" → "le chat noir," the adjective moves after the noun. A translation model has to learn these correspondences without assuming one output token per input token.

In 2014, Cho et al. introduced an RNN encoder-decoder that scored phrase pairs for a statistical translation system. Sutskever et al. demonstrated end-to-end translation with a deep LSTM encoder-decoder on WMT'14 English–French. Both pass information from an encoder to a decoder; the design question here is how much of the source the decoder can access.

The encoder-decoder architecture

You take two RNNs (or LSTMs, in practice). The first one, the encoder, reads the source sentence one token at a time and builds up a hidden state. When it reaches the end of the input, its final hidden state is supposed to represent the "meaning" of the entire source sentence. That single vector is the context.

The second RNN, the decoder, takes that context vector as its initial hidden state and generates the target sentence one token at a time, feeding each predicted token back as input for the next step.

So the information flow looks like: source tokens $\rightarrow$ encoder RNN $\rightarrow$ single fixed-length vector $c$ $\rightarrow$ decoder RNN $\rightarrow$ target tokens.

The context vector $c$ has a fixed dimension regardless of source length. This creates a bottleneck: the final encoder state must preserve everything the decoder will need, even for a long input.

The effect of that bottleneck differed across systems. Cho et al. observed sharply degrading performance on longer sources in their basic RNN encoder-decoder; Bahdanau et al. saw the same pattern in their RNNencdec baseline. Sutskever et al. reported that their deep LSTM with reversed source sentences "did not have difficulty on long sentences." Attention offered a way to access source positions directly instead of relying on one final state.

The diagram compares how many source representations are available to the decoder. Attention compares $n$ source positions at each of $m$ decoder steps, costing $O(nm)$ with vector dimensions held fixed.

Bahdanau attention: learning to align

In 2015, Bahdanau, Cho, and Bengio proposed a mechanism they called "attention." The core insight was to let the decoder, at each generation step, compute a weighted combination over all of the encoder's hidden states, rather than relying only on the final one.

Bahdanau's encoder is bidirectional. Each annotation $h_j$ concatenates forward and backward recurrent states, giving source position $j$ context from both directions. The decoder attends over these annotations.

The encoder computes the annotations once for the source sequence. At decoder step $t$, using hidden state $s_{t-1}$, we compute:

Alignment scores (how relevant is each source position to this decoder step):

where $W_a$, $U_a$, and $v_a$ are learned parameters. This is called "additive" attention because we're adding transformed versions of the decoder state and encoder state, then passing through a tanh nonlinearity.

Attention weights (normalize scores into a distribution):

That's just a softmax over the source positions. Each $\alpha_{tj}$ tells us how much the decoder should "attend to" source position $j$ when generating target token $t$.

Context vector (the weighted sum):

The context vector $c_t$ can differ at each decoder step. When generating "chat," attention may concentrate on "cat"; for "noir," it may concentrate on "black." The weights let the decoder combine source information differently for each prediction.

The decoder then uses both $c_t$ and $s_{t-1}$ to compute the next hidden state and predict the next target token.

Attention as soft alignment

Collecting $\alpha_{tj}$ across target steps $t$ and source positions $j$ gives an alignment matrix. Each row shows the source weights used for one target prediction. In the paper, the network learned these alignments from parallel translation data without separate alignment labels.

Here's what that looks like for a simple English-to-French translation:

The crossed "black"/"cat" → "chat"/"noir" pattern illustrates adjective-noun reordering. The paper showed learned alignments with similar patterns. Alammar's illustrated post provides an animated walk-through of the routing.

A quick word on Luong variants

Luong et al. compared several scoring functions in 2015, including a dot product between the decoder and encoder states.

Dot-product scoring:

Just the raw dot product between decoder and encoder states. Fast, simple, no extra parameters. But requires the decoder and encoder hidden states to live in the same dimensionality.

General scoring:

A learned matrix $W$ projects the encoder states before the dot product.

Concat scoring (essentially the same as Bahdanau):

Concatenate the two vectors, transform them, and project the result to a scalar. Like Bahdanau's score, this includes a learned nonlinear transformation.

Luong's work helped normalize simpler multiplicative attention forms. Their strongest reported local-attention system actually used the "general" score function with the learned $W$, not raw dot product, but the multiplicative family as a whole proved competitive with additive attention while being faster. Two years later, the Transformer used a scaled dot product, mainly because it maps very efficiently to batched matrix multiplication and the scaling fixes large-dimension softmax saturation. We'll get there in Arc 4.

Worked example

For one decoder step, we can calculate the scores, weights, and context vector by hand.

Say our encoder has processed a 3-word source sentence and produced hidden states (using small 4-dimensional vectors for readability):

• $h_1 = [0.2, -0.1, 0.5, 0.3]$ (encoding of "the")
• $h_2 = [0.8, 0.4, -0.2, 0.1]$ (encoding of "cat")
• $h_3 = [0.1, 0.6, 0.3, -0.5]$ (encoding of "sat")

The decoder is at step 2, with hidden state $s_1 = [0.5, 0.3, -0.1, 0.7]$, and we want to compute which source words to attend to.

Using dot-product scoring for simplicity:

Softmax over $[0.23, 0.61, -0.15]$:

So the attention concentrates most on $h_2$ ("cat"), which makes sense if the decoder is trying to generate the corresponding French word. The context vector is:

That context vector is a blend of all three encoder states, weighted towards "cat." The decoder uses this, combined with its own hidden state, to predict the next output token.

Implementation: Bahdanau attention in PyTorch

This module implements additive attention. It accepts encoder states and one decoder state, then returns the context vector and attention weights.

import torch
import torch.nn as nn
import torch.nn.functional as F
 
class BahdanauAttention(nn.Module):
    def __init__(self, encoder_dim: int, decoder_dim: int, attn_dim: int = 128):
        super().__init__()
        self.W_enc = nn.Linear(encoder_dim, attn_dim, bias=False)
        self.W_dec = nn.Linear(decoder_dim, attn_dim, bias=False)
        self.v = nn.Linear(attn_dim, 1, bias=False)
 
    def forward(
        self,
        encoder_outputs: torch.Tensor,         # (batch, src_len, encoder_dim)
        decoder_hidden: torch.Tensor,          # (batch, decoder_dim)
        src_mask: torch.Tensor | None = None,  # (batch, src_len), True = real token
    ) -> tuple[torch.Tensor, torch.Tensor]:
        if encoder_outputs.size(1) == 0:
            raise ValueError("Each example needs at least one source token")
        if src_mask is not None and not src_mask.any(dim=-1).all():
            raise ValueError("Each example needs at least one unmasked source token")
 
        # Project encoder states: (batch, src_len, attn_dim)
        enc_proj = self.W_enc(encoder_outputs)
 
        # Project decoder state and broadcast over src positions: (batch, 1, attn_dim)
        dec_proj = self.W_dec(decoder_hidden).unsqueeze(1)
 
        # Additive scoring: tanh(W_enc * h_j + W_dec * s_t)
        energy = torch.tanh(enc_proj + dec_proj)
 
        # Scalar alignment scores: (batch, src_len)
        scores = self.v(energy).squeeze(-1)
 
        # Mask out padding positions before softmax so they get zero weight.
        # Without this, the decoder can attend to <pad> tokens.
        if src_mask is not None:
            scores = scores.masked_fill(~src_mask, float("-inf"))
 
        # Attention weights via softmax over source positions
        attn_weights = F.softmax(scores, dim=-1)
 
        # Context vector: weighted sum of encoder outputs
        context = torch.bmm(attn_weights.unsqueeze(1), encoder_outputs)
        context = context.squeeze(1)  # (batch, encoder_dim)
 
        return context, attn_weights

Why sequential processing hit a wall

Attention gives the decoder access to individual source states, but a recurrent encoder still computes those states in order.

An RNN processes tokens one at a time. To compute $h_5$, you need $h_4$. To compute $h_4$, you need $h_3$. This is an inherently sequential dependency chain. For a sentence with $n$ tokens, you need $O(n)$ sequential steps through the encoder, and you can't parallelize them.

RNNs can batch matrix multiplications across examples and hidden units. Within a single example, however, $h_t$ depends on $h_{t-1}$. Vaswani et al. described this as computation factored "along the symbol positions," which "precludes parallelization within training examples."

Adding hardware can process more examples together, but it does not remove this dependency within an example. The sequence still requires $n$ successive state updates.

Misconceptions

"Attention was invented for transformers." Bahdanau attention appeared as a preprint in 2014, three years before the transformer. The 2015 conference paper was about seq2seq translation with RNNs. The 2017 move was to keep this attention idea and drop the recurrence, and we'll cover that in Arc 4.

"Seq2seq models are obsolete and irrelevant." Transformer encoder-decoders retain the sequence-to-sequence abstraction. T5 casts tasks into text-to-text form; BART trains a denoising encoder-decoder. Their cross-attention layers let a decoder access the encoded source.

"Attention fixes the vanishing gradient problem." It adds shorter paths from decoder losses to source states. Gradients within a recurrent encoder still pass through successive state updates and can shrink or grow along that chain.

What's next

So far we've taken the input tokens as given. Arc 3 examines how text becomes those tokens, starting with bytes and Unicode and continuing through subword vocabularies, chat templates, and training batches.

The next post is Unicode, Bytes, and What Text Actually Is.

Additional reading (and watching)

• Sutskever, I., Vinyals, O., & Le, Q.V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS 2014.
• Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015.
• Luong, M.T., Pham, H., & Manning, C.D. (2015). Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015.
• Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. EMNLP 2014.
• Alammar, J. Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention).
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. The explicit framing that recurrence "precludes parallelization within training examples" comes from this paper, and it's the case for dropping recurrence entirely.
• Raffel, C., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5). JMLR 2020. The encoder-decoder abstraction, applied across every NLP task by casting it as text-to-text.
• Lewis, M., et al. (2020). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. ACL 2020. A standard Transformer-based encoder-decoder with a denoising pretraining objective.