Recurrent Neural Networks and Sequence Modeling

What does a word embedding actually know about the sentence it's sitting in?

Nothing. An embedding for "bank" is the same vector whether the sentence is about a river or a savings account. A lookup table maps tokens to fixed vectors. Those vectors capture meaning in isolation, but they have no idea what came before them. "The cat sat on the" is, from the embedding table's perspective, five independent vectors floating in space.

So how do you build a model that actually reads a sequence, taking in tokens one at a time and building up some internal representation of what it's seen so far? Recurrent network ideas had been kicking around since the 1980s (Rumelhart, Jordan, Werbos), but Elman's 1990 paper "Finding Structure in Time" gave the simple, sticky version most people now picture: give the network a memory. Feed it one token, let it update an internal state, and pass that state forward to the next timestep. That's a recurrent neural network.

Each box below is the same network, unrolled across time. Hit play and watch the hidden state evolve as the RNN reads "the cat sat on the mat" one token at a time.

Notice how the hidden state bars change at every step. When "cat" comes in, the hidden state shifts dramatically from where it was after "the." By the time we reach "mat," the state has been shaped by the entire sequence. That vector at the final timestep is the RNN's compressed summary of everything it has read.

And I do mean compressed. This is where the key tension lives. The hidden state is a fixed-size vector, usually a few hundred dimensions in practice. It has to cram the entire history of the sequence into that one vector, overwriting itself at every step. It's lossy by design.

The RNN Equations

The math for a single timestep is short:

$x_t$ is the input embedding at time $t$. $h_{t-1}$ is the hidden state from the previous timestep. $W_h$ and $W_x$ are weight matrices (learned during training), $b$ is a bias vector, and $\tanh$ squashes the result into $[-1, 1]$.

Let me walk through one timestep with actual numbers so this is concrete. Say our hidden state is 2-dimensional (tiny, just for illustration) and our input embedding is 3-dimensional.

Previous hidden state $h_{t-1} = [0.7, -0.2]$. Current input $x_t = [1.0, 0.5, -0.3]$.

So the new hidden state is $[0.647, -0.050]$. The previous state and the current input got blended together through two matrix multiplications, summed, and squashed. The network does this at every single timestep, reusing the same weights $W_h$ and $W_x$ each time. That weight sharing across time is what makes it "recurrent."

Here's the same update, one piece at a time. Watch the bars line up with the numbers above.

To actually produce an output (say, predicting the next token), you tack on one more linear layer: $y_t = W_y \, h_t + b_y$, and run the result through softmax to get a probability distribution over the vocabulary. Standard stuff.

Training: Backpropagation Through Time

Now. To train an RNN, you unroll it across all timesteps and backpropagate the loss gradient through the entire chain. This is called Backpropagation Through Time, or BPTT.

Define the local one-step Jacobian $J_k = \partial h_k / \partial h_{k-1}$. For our $\tanh$ RNN:

where $\mathrm{diag}(1 - h_k^2)$ is the derivative of $\tanh$ (always between 0 and 1). The dependence of a late hidden state on an early one is a left-to-right product of those Jacobians (matrix order matters):

And during the backward pass, the loss gradient is transported one step at a time through the transposed Jacobians:

So the gradient is a long chain of matrix multiplications. And this is where the whole thing breaks.

Vanishing Gradients

Go back to the visualization above and toggle the gradient mode on, and watch what happens to those red arrows as they flow backward through time. They fade quickly.

Why? Think about what happens when you repeatedly multiply by the same matrix. The worst-case one-step amplification of gradient norm is controlled by singular values (the spectral norm of $W_h$); the long-run behavior of the repeated powers $W_h^n$ is controlled by eigenvalues (the spectral radius). For the simple intuition here, think of an effective per-step multiplier $\rho$: if $\rho < 1$ the signal tends to shrink exponentially across steps, and if $\rho > 1$ it tends to grow.

Let me make this really concrete with a 2x2 example. Take a simple weight matrix:

The largest eigenvalue of this matrix has magnitude about 0.6. Now multiply it by itself a few times:

After 10 multiplications, every entry is down by three orders of magnitude. The gradient signal from timestep $T$ back to timestep $t$ has been multiplied to nothing. And the $\tanh$ derivative (always $\leq 1$) makes it even worse, acting as an additional damping factor at every step.

This is the vanishing gradient problem. In practical terms, it means an RNN can't learn long-range dependencies. If the answer to "what word should come next?" depends on something that happened 50 tokens ago, the gradient carrying that information has been multiplied by something like $0.6^{50} \approx 8 \times 10^{-12}$ by the time it reaches those early weights. The model literally cannot learn to connect distant parts of the sequence.

Watch a gradient pulse propagate backward through an unrolled chain, shrinking at every hop. The dashed line is the rough floor below which the optimizer gets no useful signal.

The formal version of the same picture: three regimes on a log scale, with a slider for ρ so you can feel where the tipping point lives.

Bengio, Simard, and Frasconi laid this out formally in 1994, and it remained the central problem in sequence modeling for nearly two decades.

Exploding Gradients

The flip side: if the spectral radius of $W_h$ is greater than 1, the gradients don't vanish. They explode. The same repeated multiplication that drives small values to zero drives large values to infinity.

This one is actually easier to fix. The standard solution is gradient clipping: if the norm of the gradient vector exceeds some threshold, you rescale it.

In PyTorch, that's one line: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0). This keeps training stable, but it doesn't solve vanishing gradients. Clipping prevents explosions; it can't resurrect a signal that's already decayed to zero.

What RNNs Can and Cannot Learn

So here's where that leaves us in practice. Vanilla $\tanh$ RNNs trained with ordinary BPTT are solid at short-range pattern matching. Character-level language modeling, simple sentiment detection, POS tagging. If the relevant context is within the last 10-20 tokens, this kind of network can usually learn it.

But long-distance dependencies are difficult. Subject-verb agreement across a long relative clause ("The keys that the man who lived near the river lost were found"), coreference resolution across paragraphs, keeping track of a conversation topic. The error signal decays through repeated Jacobian products, so the model's effective memory is severely limited even though the hidden state technically carries information from every past timestep. Gated RNNs (LSTMs, GRUs), orthogonal/identity initializations, and attention-augmented recurrent designs were invented largely to relax this constraint, and we'll get to them in the next post.

Karpathy's 2015 char-rnn experiments are the most-cited demo here. Those experiments used multi-layer LSTMs, not vanilla RNNs. They could produce convincing fake Shakespeare and fake C code, learning syntax-like local structure and even some interpretable long-range cells (line lengths, quotes, bracket depth, as he and Johnson and Fei-Fei catalogued the next year), but still made coherence and consistency errors over longer spans. That's a useful upper bound on what a gated RNN was getting; a vanilla $\tanh$ RNN of the same size struggles even with the local structure those LSTMs handled cleanly.

Worked Example: A Character-Level RNN

The classic demonstration is a character-level language model in the style popularized by char-rnn. You feed the network one character at a time, and it predicts the next character. After training on enough text, it learns spelling, common words, and rudimentary syntax. (Historically, Mikolov et al. 2010 showed that word-level RNN language models could beat strong n-gram baselines in perplexity and speech-recognition WER. The character-level Shakespeare demo is a slightly different lineage that became famous through char-rnn.)

Here's a minimal implementation. This is about 40 lines of real, runnable PyTorch, nothing hidden.

import torch
import torch.nn as nn
 
class CharRNN(nn.Module):
    def __init__(self, vocab_size, hidden_size=128):
        super().__init__()
        self.hidden_size = hidden_size
        self.embed = nn.Embedding(vocab_size, hidden_size)
        self.rnn = nn.RNN(hidden_size, hidden_size, batch_first=True)
        self.head = nn.Linear(hidden_size, vocab_size)
 
    def forward(self, x, h=None):
        # x: (batch, seq_len) of token indices
        emb = self.embed(x)              # (batch, seq_len, hidden)
        out, h_new = self.rnn(emb, h)    # out: (batch, seq_len, hidden)
        logits = self.head(out)           # (batch, seq_len, vocab_size)
        return logits, h_new
 
# --- Training loop ---
text = open("input.txt", encoding="utf-8").read()
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for c, i in stoi.items()}
 
data = torch.tensor([stoi[c] for c in text])
 
model = CharRNN(vocab_size=len(chars))
optimizer = torch.optim.Adam(model.parameters(), lr=3e-3)
criterion = nn.CrossEntropyLoss()
 
seq_len = 64
for step in range(5000):
    # Random chunk from the corpus. Note: hidden state starts fresh each
    # chunk (truncated BPTT with no carry), so dependencies that cross
    # the 64-character boundary are not learnable in this demo.
    start = torch.randint(0, len(data) - seq_len - 1, (1,)).item()
    x = data[start : start + seq_len].unsqueeze(0)
    y = data[start + 1 : start + seq_len + 1].unsqueeze(0)
 
    logits, _ = model(x)
    loss = criterion(logits.reshape(-1, len(chars)), y.reshape(-1))
 
    optimizer.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
    optimizer.step()
 
    if step % 500 == 0:
        print(f"step {step:>5d} | loss {loss.item():.3f}")
 
# --- Generate from the trained model ---
@torch.no_grad()
def sample(prompt, n_chars=200, temperature=1.0):
    ids = torch.tensor([[stoi[c] for c in prompt]])
    h = None
    out = list(prompt)
    logits, h = model(ids, h)            # warm up hidden state on prompt
    last = ids[:, -1:]
    for _ in range(n_chars):
        logits, h = model(last, h)
        probs = torch.softmax(logits[:, -1] / temperature, dim=-1)
        nxt = torch.multinomial(probs, num_samples=1)
        out.append(itos[nxt.item()])
        last = nxt
    return "".join(out)
 
print(sample("ROMEO:", n_chars=200))

A couple of things to point out. The nn.RNN module handles the unrolling internally; you pass in a whole sequence and it loops through the timesteps for you. We clip gradients at norm 5 to prevent explosions. The training loop is grabbing random 64-character chunks, which is a form of truncated BPTT: the hidden state resets every chunk, so the gradient never flows across chunk boundaries and dependencies longer than 64 characters are unreachable in this setup. Real char-rnn implementations carry the hidden state forward across chunks of contiguous text to extend the effective horizon.

After a few thousand steps on something like Shakespeare, this model will produce output that looks vaguely like English. Spelled words, punctuation in roughly the right places, occasional real words strung together. But it won't maintain coherence for more than a handful of characters, because the vanilla RNN hidden state can't carry information far enough.

Misconceptions

"The hidden state contains the whole history." Technically yes, every past token has influenced it. Practically no. The hidden state is a fixed-size vector that gets overwritten every step, and the influence of a token from 30 steps ago has been diluted through 30 applications of $W_h$ and $\tanh$. You can prove with probing experiments that the network has already forgotten specific words a dozen or so timesteps after they appeared. The hidden state is a running summary, and the summary's resolution gets worse the further back you look.

"Vanishing gradients mean the loss goes to zero." No. The loss stays high because the model can't fit long-range patterns. What goes to zero is the gradient of the loss with respect to weights several timesteps back. The optimizer can see the recent part of the sequence just fine; it just cannot get a useful signal for how to adjust the weights to make better use of the distant past. So the model settles into a solution that uses only local context.

"RNNs process sequences in parallel because they share weights." Weight sharing and parallelism are different things. An RNN uses the same weights at every timestep, which means fewer parameters, but step $t$ has to finish before step $t+1$ can start, because $h_t$ feeds into the next step. That strictly serial dependency is the main reason training an RNN on a long sequence is so slow compared to architectures that can process all positions at once.

It's worth holding three separate things in mind here, because beginners often collapse them: information flow in the forward pass (old tokens influence later hidden states), gradient flow in training (the learning signal must travel backward through many Jacobians), and the capacity bottleneck (the hidden state is fixed-size and overwritten every step). Vanishing gradients are specifically about the second one. The first and third are real constraints too, and they don't go away just because you fix the gradient problem. This is the wedge LSTMs attacked directly: they added gated, additive memory paths so error signals could survive longer than in a plain $\tanh$ recurrence, but the fixed-size hidden state and the strictly serial compute remained.

What's next

The next post covers LSTMs, GRUs, and gated memory, how a few extra sigmoid gates turned the vanishing gradient problem from a showstopper into a solvable engineering challenge.

Additional reading (and watching)

• Elman, J.L. (1990). Finding Structure in Time. Cognitive Science, 14(2), 179-211.
• Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5(2), 157-166.
• Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the Difficulty of Training Recurrent Neural Networks. ICML 2013.
• Karpathy, A. (2015). The Unreasonable Effectiveness of Recurrent Neural Networks.
• Mikolov, T. et al. (2010). Recurrent Neural Network Based Language Model. INTERSPEECH 2010.
• Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735-1780. The original LSTM paper, introduced specifically to address insufficient, decaying error flow over long time lags.
• Karpathy, A., Johnson, J., & Fei-Fei, L. (2016). Visualizing and Understanding Recurrent Networks. ICLR 2016. The follow-up to the char-rnn blog: catalogues interpretable LSTM cells (line lengths, quotes, brackets, code structure) in character-level language models.
• Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017. The original case for abandoning recurrence in favor of a fully parallelizable sequence model.