Recurrent Neural Networks and Sequence Modeling

What does a word embedding actually know about the sentence it's sitting in?

A static embedding for "bank" is the same vector whether the sentence is about a river or a savings account. A lookup table maps tokens to fixed vectors. Those vectors reflect corpus-level usage, but the lookup does not depend on the preceding tokens. "The cat sat on the" is, from the embedding table's perspective, five independent vectors floating in space.

A recurrent neural network updates an internal state as it reads each token, then passes that state to the next timestep. Recurrent network ideas appeared in work by Rumelhart, Jordan, and Werbos in the 1980s; Elman's 1990 paper "Finding Structure in Time" developed the simple recurrent architecture used here.

Each box below is the same network, unrolled across time. Hit play and watch the hidden state evolve as the RNN reads "the cat sat on the mat" one token at a time.

The state must preserve whatever information later predictions need in the same number of coordinates at every step. Successive updates can retain some features and lose others.

The RNN Equations

$x_t$ is the input embedding at time $t$. $h_{t-1}$ is the hidden state from the previous timestep. $W_h$ and $W_x$ are weight matrices (learned during training), $b$ is a bias vector, and $\tanh$ squashes the result into $[-1, 1]$.

For this worked step, use a 2-dimensional hidden state and a 3-dimensional input embedding.

Previous hidden state $h_{t-1} = [0.7, -0.2]$. Current input $x_t = [1.0, 0.5, -0.3]$.

So the new hidden state is $[0.647, -0.050]$. The previous state and the current input got blended together through two matrix multiplications, summed, and squashed. The network does this at every single timestep, reusing the same weights $W_h$ and $W_x$ each time. That weight sharing across time is what makes it "recurrent."

To predict the next token, add an output projection: $y_t = W_y \, h_t + b_y$, and run the result through softmax to get a probability distribution over the vocabulary.

Training: Backpropagation Through Time

To train an RNN, you unroll it across all timesteps and backpropagate the loss gradient through the entire chain. This is called Backpropagation Through Time, or BPTT.

Define the local one-step Jacobian $J_k = \partial h_k / \partial h_{k-1}$. For our $\tanh$ RNN:

where $\mathrm{diag}(1 - h_k^2)$ is the derivative of $\tanh$ (always between 0 and 1). The dependence of a late hidden state on an early one is a left-to-right product of those Jacobians (matrix order matters):

For a loss attached at the final step, the gradient travels backward through the transposed Jacobians. If losses are attached at every step, their local contributions must also be added:

The gradient therefore passes through a long chain of matrix multiplications, which can make training unstable.

Vanishing Gradients

Repeated matrix multiplication can shrink or amplify a gradient. Singular values bound the one-step change in norm; for powers of a fixed matrix, eigenvalues control the long-run behavior. A constant scalar multiplier $\rho$ illustrates the two cases: below 1, repeated multiplication shrinks the signal; above 1, it grows.

Consider this $2 \times 2$ weight matrix:

The largest eigenvalue of this matrix has magnitude about 0.6. Now multiply it by itself a few times:

After 10 multiplications, the entries are roughly two orders of magnitude smaller. In this toy recurrence, gradients from distant steps become very small. And the $\tanh$ derivative (always $\leq 1$) makes it even worse, acting as an additional damping factor at every step.

This is the vanishing gradient problem. At an effective multiplier of 0.6 per step, a contribution traveling backward 50 steps shrinks by $0.6^{50} \approx 8 \times 10^{-12}$. It can become negligible beside gradients from recent steps, making distant dependencies difficult to learn.

Move the step slider to follow a scalar gradient contribution through the chain. Its multiplier is held at 0.62 to make the repeated multiplication visible.

Bengio, Simard, and Frasconi analyzed the difficulty of learning long-term dependencies with gradient descent in 1994.

Exploding Gradients

Gradients can also grow through repeated Jacobian products. A large spectral radius of $W_h$ can permit this, but is not sufficient on its own: the activation derivatives and the directions of the gradients matter too.

Gradient clipping limits the norm used by the optimizer: if the gradient exceeds a threshold, rescale it.

In PyTorch, that's one line: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0). This bounds the gradient norm used for the update. Clipping limits large gradients; it cannot recover ones that have vanished.

What RNNs Can and Cannot Learn

Vanilla $\tanh$ RNNs have been used for character-level language modeling, sentiment classification, and part-of-speech tagging. Their effective context length depends on the task, weights, initialization, and training procedure.

Long-distance dependencies, such as subject-verb agreement across a relative clause, require useful information to survive many updates. Learning those dependencies also requires a gradient through the intervening Jacobians. Gated RNNs, orthogonal or identity initializations, and attention provide ways to address parts of this problem.

Karpathy's 2015 char-rnn experiments are the most-cited demo here. Those experiments used multi-layer LSTMs, not vanilla RNNs. They could produce convincing fake Shakespeare and fake C code, learning syntax-like local structure and even some interpretable long-range cells (line lengths, quotes, bracket depth, as he and Johnson and Fei-Fei catalogued the next year), but still made coherence and consistency errors over longer spans.

Worked Example: A Character-Level RNN

The classic demonstration is a character-level language model in the style popularized by char-rnn. You feed the network one character at a time, and it predicts the next character. After training on enough text, it learns spelling, common words, and rudimentary syntax. (Historically, Mikolov et al. 2010 showed that word-level RNN language models could beat strong n-gram baselines in perplexity and speech-recognition WER. The character-level Shakespeare demo is a slightly different lineage that became famous through char-rnn.)

This PyTorch example trains a character-level RNN on short text chunks.

import torch
import torch.nn as nn
 
class CharRNN(nn.Module):
    def __init__(self, vocab_size, hidden_size=128):
        super().__init__()
        self.hidden_size = hidden_size
        self.embed = nn.Embedding(vocab_size, hidden_size)
        self.rnn = nn.RNN(hidden_size, hidden_size, batch_first=True)
        self.head = nn.Linear(hidden_size, vocab_size)
 
    def forward(self, x, h=None):
        # x: (batch, seq_len) of token indices
        emb = self.embed(x)              # (batch, seq_len, hidden)
        out, h_new = self.rnn(emb, h)    # out: (batch, seq_len, hidden)
        logits = self.head(out)           # (batch, seq_len, vocab_size)
        return logits, h_new
 
# --- Training loop ---
text = open("input.txt", encoding="utf-8").read()
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for c, i in stoi.items()}
 
data = torch.tensor([stoi[c] for c in text])
 
model = CharRNN(vocab_size=len(chars))
optimizer = torch.optim.Adam(model.parameters(), lr=3e-3)
criterion = nn.CrossEntropyLoss()
 
seq_len = 64
if len(data) <= seq_len:
    raise ValueError("input.txt must contain at least 65 characters")
for step in range(5000):
    # Random chunk from the corpus. Note: hidden state starts fresh each
    # chunk (truncated BPTT with no carry), so dependencies that cross
    # the chunk boundary have no direct gradient path in this demo.
    start = torch.randint(0, len(data) - seq_len, (1,)).item()
    x = data[start : start + seq_len].unsqueeze(0)
    y = data[start + 1 : start + seq_len + 1].unsqueeze(0)
 
    logits, _ = model(x)
    loss = criterion(logits.reshape(-1, len(chars)), y.reshape(-1))
 
    optimizer.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
    optimizer.step()
 
    if step % 500 == 0:
        print(f"step {step:>5d} | loss {loss.item():.3f}")
 
# --- Generate from the trained model ---
@torch.no_grad()
def sample(prompt, n_chars=200, temperature=1.0):
    if not prompt or temperature <= 0:
        raise ValueError("Use a nonempty prompt and a positive temperature")
    ids = torch.tensor([[stoi[c] for c in prompt]])
    h = None
    out = list(prompt)
    logits, h = model(ids, h)            # warm up hidden state on prompt
    for _ in range(n_chars):
        probs = torch.softmax(logits[:, -1] / temperature, dim=-1)
        nxt = torch.multinomial(probs, num_samples=1)
        out.append(itos[nxt.item()])
        logits, h = model(nxt, h)
    return "".join(out)
 
print(sample("ROMEO:", n_chars=200))

The nn.RNN module handles the unrolling internally; you pass in a whole sequence and it loops through the timesteps for you. We clip gradients at norm 5 to bound the update’s gradient norm. The training loop is grabbing random 64-character chunks, which is a form of truncated BPTT: the hidden state resets every chunk, so there is no direct training gradient path spanning more than 64 characters. Real char-rnn implementations carry the hidden state forward across chunks of contiguous text to extend the effective horizon.

Misconceptions

"The hidden state contains the whole history." A recurrence connects past tokens to the current state, but does not ensure their recovery. Repeated updates can preserve some features and lose others within the fixed-size vector.

"Vanishing gradients mean the loss goes to zero." A small gradient contribution from distant timesteps says nothing by itself about the loss value. It means those timesteps provide little signal for the current update. The model may still learn from recent context while struggling to improve predictions that depend on earlier inputs.

"RNNs process sequences in parallel because they share weights." Weight sharing and parallelism are different things. An RNN uses the same weights at every timestep, which means fewer parameters, but step $t$ has to finish before step $t+1$ can start, because $h_t$ feeds into the next step. That strictly serial dependency is the main reason training an RNN on a long sequence is so slow compared to architectures that can process all positions at once.

There are three separate constraints here: information flow in the forward pass (old tokens influence later hidden states), gradient flow in training (the learning signal must travel backward through many Jacobians), and the capacity bottleneck (the hidden state is fixed-size and overwritten every step). Vanishing gradients are specifically about the second one. The first and third are real constraints too, and they don't go away just because you fix the gradient problem. LSTMs addressed gradient flow: they added gated, additive memory paths so error signals could survive longer than in a plain $\tanh$ recurrence, but the fixed-size hidden state and the strictly serial compute remained.

What's next

The next post covers LSTMs, GRUs, and gated memory, how learned gates and an additive memory path help gradients travel through longer sequences.

Additional reading (and watching)

• Elman, J.L. (1990). Finding Structure in Time. Cognitive Science, 14(2), 179-211.
• Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5(2), 157-166.
• Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the Difficulty of Training Recurrent Neural Networks. ICML 2013.
• Karpathy, A. (2015). The Unreasonable Effectiveness of Recurrent Neural Networks.
• Mikolov, T. et al. (2010). Recurrent Neural Network Based Language Model. INTERSPEECH 2010.
• Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735-1780. The original LSTM paper, introduced specifically to address insufficient, decaying error flow over long time lags.
• Karpathy, A., Johnson, J., & Fei-Fei, L. (2016). Visualizing and Understanding Recurrent Networks. ICLR 2016. The follow-up to the char-rnn blog: catalogues interpretable LSTM cells (line lengths, quotes, brackets, code structure) in character-level language models.
• Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017. The original case for abandoning recurrence in favor of a fully parallelizable sequence model.