Seq2seq, Bahdanau Attention, and Why Recurrence Hit a Wall

How do you translate a sentence from English to French using a neural network?

It sounds straightforward at first. You have an input sequence, you have an output sequence. But the input and output can be completely different lengths. "The black cat sat" is four words. "Le chat noir assis" is also four, but that's a coincidence. "I am going to the store" maps to "Je vais au magasin," which is shorter. And the word order can rearrange. This is variable-length to variable-length, with no guaranteed alignment between positions.

In 2014, Cho et al. and Sutskever et al. proposed the architecture that would dominate NMT for the next three years: the encoder-decoder, or seq2seq model. (Cho et al. introduced the RNN Encoder-Decoder and used it to score phrase pairs as an extra feature inside a statistical MT system; Sutskever et al. demonstrated the first large end-to-end LSTM-based seq2seq translation system on WMT'14 English-French.) The idea is clean enough to explain in one paragraph, and the bottleneck it introduces is what makes the rest of this post necessary.

The encoder-decoder architecture

You take two RNNs (or LSTMs, in practice). The first one, the encoder, reads the source sentence one token at a time and builds up a hidden state. When it reaches the end of the input, its final hidden state is supposed to represent the "meaning" of the entire source sentence. That single vector is the context.

The second RNN, the decoder, takes that context vector as its initial hidden state and generates the target sentence one token at a time, feeding each predicted token back as input for the next step.

So the information flow looks like: source tokens $\rightarrow$ encoder RNN $\rightarrow$ single fixed-length vector $c$ $\rightarrow$ decoder RNN $\rightarrow$ target tokens.

And there's the problem. That single vector $c$ has a fixed dimensionality, usually 256 or 512 or 1024 floats. It doesn't matter whether the source sentence is 5 words or 50 words. You're cramming the entire meaning of the input into the same-sized vector. I'll call this the bottleneck from here on. I think of it like the telephone game: you have to whisper the entire contents of a paragraph through a single breath. Short sentences come through fine; long ones come out garbled.

Cho et al. showed this empirically: in the basic RNN encoder-decoder setting they studied, performance degraded sharply as source sentences got longer. Bahdanau et al. later confirmed the same pattern for their RNNencdec baseline and used it as the explicit motivation for attention. Worth flagging the counterexample: Sutskever et al.'s deep LSTM with reversed source sentences reported it "did not have difficulty on long sentences." So the fixed-vector seq2seq family didn't uniformly fall apart at length; the stronger setups were more robust. But the fragile compression problem was real, and Bahdanau attention attacked it directly.

The shape of those two curves is the whole story for the first half of this post. The basic seq2seq baseline holds together while the source sentence fits comfortably inside the context vector, then degrades as soon as the encoder runs out of room. Adding attention greatly reduced the length penalty in the reported translation regime. It didn't make sequence length free, since attention still costs $O(nm)$ per step and long-context translation can degrade for other reasons, but it removed the need to squeeze the entire source through one final hidden state.

The question becomes: what if the decoder didn't have to rely on a single summary vector? What if it could look back at every position in the source sentence?

Bahdanau attention: learning to align

In 2015, Bahdanau, Cho, and Bengio proposed a mechanism they called "attention." The core insight was to let the decoder, at each generation step, compute a weighted combination over all of the encoder's hidden states, rather than relying only on the final one.

Bahdanau didn't just bolt attention onto a vanilla left-to-right encoder. The encoder is bidirectional. The annotation $h_j$ at each source position is the concatenation of a forward and a backward recurrent state, so it carries local focus on token $j$ along with surrounding context from both sides. The decoder then attends over those bidirectional annotations, not raw token embeddings or strictly left-to-right hidden states.

Here's the full mechanism. The encoder produces a hidden state (annotation) $h_j$ at each source position $j$. At decoder time step $t$, with decoder hidden state $s_{t-1}$, we compute:

Alignment scores (how relevant is each source position to this decoder step):

where $W_a$, $U_a$, and $v_a$ are learned parameters. This is called "additive" attention because we're adding transformed versions of the decoder state and encoder state, then passing through a tanh nonlinearity.

Attention weights (normalize scores into a distribution):

That's just a softmax over the source positions. Each $\alpha_{tj}$ tells us how much the decoder should "attend to" source position $j$ when generating target token $t$.

Context vector (the weighted sum):

This context vector $c_t$ is different at every decoder step. When the decoder is generating the French word "chat," the attention weights will concentrate on the English word "cat." When generating "noir," they'll shift to "black." The decoder dynamically focuses on whichever part of the source sentence is relevant right now. I'll call this moving focus the alignment window, another named handle for the rest of the post.

The decoder then uses both $c_t$ and $s_{t-1}$ to compute the next hidden state and predict the next target token.

The three sub-phases in that visualization are the entire mechanism. Scores, softmax, weighted sum. Every subsequent use of attention in the series (cross-attention in encoder-decoder stacks, attention inside speech models, attention inside vision-language models) is some variation on those three steps.

Attention as soft alignment

One nice property of this mechanism is that the attention weights form an alignment matrix. If you collect the weights $\alpha_{tj}$ for every decoder step $t$ and source position $j$, you get a matrix where each row shows which source words the decoder was looking at when generating that target word. And nobody had to label these alignments. The network learned them from parallel translation data alone.

Here's what that looks like for a simple English-to-French translation:

Notice the crossed pattern between "black"/"cat" and "chat"/"noir." In English the adjective comes before the noun; in French it's reversed. The attention mechanism figured this out without any explicit syntactic supervision. That was a significant result in 2015. (For an animated walk-through of the same routing, Alammar's illustrated post is a great reference.)

A quick word on Luong variants

Luong et al. proposed several alternative scoring functions in 2015, and they're worth knowing because the "dot-product" variant is the one that carried forward into later architectures.

Dot-product scoring:

Just the raw dot product between decoder and encoder states. Fast, simple, no extra parameters. But requires the decoder and encoder hidden states to live in the same dimensionality.

General scoring:

Introduces a learnable weight matrix $W$ that projects the encoder states before the dot product. More flexible than pure dot-product, still cheaper than Bahdanau's additive formulation.

Concat scoring (essentially the same as Bahdanau):

Concatenate the two vectors, transform, project to a scalar. The most expressive of the three variants, and also the most expensive per attention step.

Luong's work helped normalize simpler multiplicative attention forms. Their strongest reported local-attention system actually used the "general" score function with the learned $W$, not raw dot product, but the multiplicative family as a whole proved competitive with additive attention while being faster. Two years later, the Transformer used a scaled dot product, mainly because it maps very efficiently to batched matrix multiplication and the scaling fixes large-dimension softmax saturation. We'll get there in Arc 4.

Worked example

Let me walk through one decoder step concretely, because the equations above can feel abstract until you see numbers.

Say our encoder has processed a 3-word source sentence and produced hidden states (using small 4-dimensional vectors for readability):

• $h_1 = [0.2, -0.1, 0.5, 0.3]$ (encoding of "the")
• $h_2 = [0.8, 0.4, -0.2, 0.1]$ (encoding of "cat")
• $h_3 = [0.1, 0.6, 0.3, -0.5]$ (encoding of "sat")

The decoder is at step 2, with hidden state $s_1 = [0.5, 0.3, -0.1, 0.7]$, and we want to compute which source words to attend to.

Using dot-product scoring for simplicity:

Softmax over $[0.23, 0.61, -0.15]$:

So the attention concentrates most on $h_2$ ("cat"), which makes sense if the decoder is trying to generate the corresponding French word. The context vector is:

That context vector is a blend of all three encoder states, weighted towards "cat." The decoder uses this, combined with its own hidden state, to predict the next output token.

Implementation: Bahdanau attention in PyTorch

Here's a minimal implementation of the additive attention mechanism. This is just the attention layer itself, not the full encoder-decoder training loop.

import torch
import torch.nn as nn
import torch.nn.functional as F
 
class BahdanauAttention(nn.Module):
    def __init__(self, encoder_dim: int, decoder_dim: int, attn_dim: int = 128):
        super().__init__()
        self.W_enc = nn.Linear(encoder_dim, attn_dim, bias=False)
        self.W_dec = nn.Linear(decoder_dim, attn_dim, bias=False)
        self.v = nn.Linear(attn_dim, 1, bias=False)
 
    def forward(
        self,
        encoder_outputs: torch.Tensor,         # (batch, src_len, encoder_dim)
        decoder_hidden: torch.Tensor,          # (batch, decoder_dim)
        src_mask: torch.Tensor | None = None,  # (batch, src_len), True = real token
    ) -> tuple[torch.Tensor, torch.Tensor]:
        # Project encoder states: (batch, src_len, attn_dim)
        enc_proj = self.W_enc(encoder_outputs)
 
        # Project decoder state and broadcast over src positions: (batch, 1, attn_dim)
        dec_proj = self.W_dec(decoder_hidden).unsqueeze(1)
 
        # Additive scoring: tanh(W_enc * h_j + W_dec * s_t)
        energy = torch.tanh(enc_proj + dec_proj)
 
        # Scalar alignment scores: (batch, src_len)
        scores = self.v(energy).squeeze(-1)
 
        # Mask out padding positions before softmax so they get zero weight.
        # Without this, the decoder can attend to <pad> tokens.
        if src_mask is not None:
            scores = scores.masked_fill(~src_mask, float("-inf"))
 
        # Attention weights via softmax over source positions
        attn_weights = F.softmax(scores, dim=-1)
 
        # Context vector: weighted sum of encoder outputs
        context = torch.bmm(attn_weights.unsqueeze(1), encoder_outputs)
        context = context.squeeze(1)  # (batch, encoder_dim)
 
        return context, attn_weights

About 30 lines. Three learned projections, a tanh, a softmax, and a weighted sum. That's the attention layer only. A production seq2seq model also needs teacher forcing, beam search at inference, loss/label-smoothing handling, and proper integration with the decoder RNN. The piece above is the part that actually makes attention happen.

Why sequential processing hit a wall

Attention solved the bottleneck. Models could now translate long sentences accurately. But there was a deeper problem that attention couldn't fix: the recurrence itself.

An RNN processes tokens one at a time. To compute $h_5$, you need $h_4$. To compute $h_4$, you need $h_3$. This is an inherently sequential dependency chain. For a sentence with $n$ tokens, you need $O(n)$ sequential steps through the encoder, and you can't parallelize them.

The point isn't that GPUs are literally idle. RNNs do real GPU work each step: large matrix multiplies, batched across many examples and across many hidden units. What they can't do is parallelize across sequence positions within a single example, because $h_t$ depends on $h_{t-1}$. Vaswani et al. put this directly: recurrent models "factor computation along the symbol positions," which "precludes parallelization within training examples," and the constraint becomes critical at longer sequence lengths.

In 2014 this felt fine, because the sentences people were translating were short and the datasets were small. By 2016, people wanted to train on millions of sentence pairs, with sequences of 100+ tokens, on massive GPU clusters. The within-example serial dependence put a ceiling on how much hardware you could throw at one training example: more GPUs gave you more batch parallelism, but the per-example latency floor came from the recurrence itself.

Attention made the model more accurate. It did not make training faster. Every encoder still had to grind through the input sequence step by step. Every decoder still had to generate one token at a time.

There were other issues too. Even with LSTMs and attention, gradient flow over very long sequences was still lossy. Attention itself computed over all encoder states at every decoder step, $O(n \cdot m)$ work per sequence. But the real killer was the serial dependence within an example. As GPUs got wider and more parallel each year, that constraint forced training time to scale with sequence length in a way you couldn't engineer around.

Misconceptions

"Attention was invented for transformers." Bahdanau attention predates the transformer by three years. The 2015 paper was about seq2seq translation with RNNs. The 2017 move was to keep this attention idea and drop the recurrence, and we'll cover that in Arc 4. The attention concept came from the RNN era.

"Seq2seq models are obsolete and irrelevant." The RNN implementation is mostly obsolete. The encoder-decoder abstraction is alive and well in Transformer-based models: T5 casts every task into a text-to-text format with a Transformer encoder-decoder; BART is a denoising sequence-to-sequence model with a standard Transformer-based NMT architecture. The conceptual framework of "encode a source, attend to it, decode a target" is the foundation for understanding the cross-attention layers in every modern translation, summarization, and speech model deployed today.

"Attention fixes the vanishing gradient problem." Attention gives the decoder a direct connection to every encoder hidden state, which helps gradient flow during backpropagation. But the encoder itself is still a recurrent network with all the gradient issues we discussed in earlier posts. The encoder's hidden states are still computed sequentially, and gradients flowing backward through the encoder still face the vanishing/exploding gradient problem. Attention is a patch on the decoder side, not a fix for recurrence.

What's next

This wraps up Arc 2. Across seven posts we went from n-grams counting surface statistics, through word2vec's learned geometry, into the RNN/LSTM era of sequential hidden states, and out the other side with the idea that made scale possible: a decoder that can look back at every source position with learned, soft weights.

The next arc steps one floor below the model. Arc 2 quietly assumed an answer to a question it never really earned: where do tokens come from in the first place? Arc 3 picks that question up and doesn't let go. It starts at bytes and Unicode, builds up through BPE and unigram, and ends at chat templates and the packing tricks that modern training runs depend on.

The next post is Unicode, Bytes, and What Text Actually Is.

Additional reading (and watching)

• Sutskever, I., Vinyals, O., & Le, Q.V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS 2014.
• Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015.
• Luong, M.T., Pham, H., & Manning, C.D. (2015). Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015.
• Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. EMNLP 2014.
• Alammar, J. Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention).
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. The explicit framing that recurrence "precludes parallelization within training examples" comes from this paper, and it's the case for dropping recurrence entirely.
• Raffel, C., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5). JMLR 2020. The encoder-decoder abstraction, applied across every NLP task by casting it as text-to-text.
• Lewis, M., et al. (2020). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. ACL 2020. A standard Transformer-based encoder-decoder with a denoising pretraining objective.