LoRA and Parameter-Efficient Fine-Tuning

19 min readbuilds on Supervised Fine-Tuning, Vectors, Matrices, and the Spaces They Live In and GPUs, Floating Point, and Why Precision Matters

Here's a number that does a lot of work. A 70-billion-parameter model, trained with Adam in mixed precision, needs roughly 1.1 TB of GPU memory just to hold the weights, gradients, and optimizer states during a single training step. That's fourteen 80 GB H100s, carefully sharded, before you've even started thinking about activations or batch size.

Now. Most of the time, a team that wants to "fine-tune" a model doesn't actually need to retrain all 70 billion parameters. They have a narrower task: internal docs Q&A, a specific code style, a product-specific persona. The question this post is about: can we get 99% of the quality by touching way less than 1% of the weights?

The answer, since roughly 2021, has been yes. And the technique that made it into every serving stack is called LoRA. This post walks through what LoRA does, why the "low-rank" part works, how QLoRA squeezed a 65B fine-tune onto a single GPU, and when you should (and shouldn't) expect PEFT to match full fine-tuning.

Prerequisites

This post assumes you've read:

Supervised Fine-Tuning for what "fine-tuning" actually means (and how it differs from pretraining).
Vectors, Matrices, and the Spaces They Live In for the notion of a matrix rank and what a linear map is doing.
GPUs, Floating Point, and Why Precision Matters for fp16 vs fp32 and quantization basics (needed for QLoRA).

The full fine-tuning bill

Let me put a real price tag on full fine-tuning before I try to sell you a cheaper option.

For each trainable parameter during mixed-precision training with Adam, you typically carry:

The weight itself in fp16 (2 bytes).
Its gradient in fp16 (2 bytes).
Two optimizer moments in fp32: $m$ and $v$ (8 bytes combined).
A master copy of the weight in fp32 for the optimizer update (4 bytes).

That's roughly 16 bytes per trainable parameter, sometimes called the "Adam tax." For a 70B model, you're looking at:

70{,}000{,}000{,}000 \times 16 \text{ bytes} \approx 1.1 \text{ TB}

In practice you shave that down with ZeRO-style optimizer sharding, paged offload, and gradient checkpointing, but the fundamental shape doesn't change. Full fine-tuning means every weight gets a gradient, every weight gets Adam states, and every weight participates in the optimizer step. Your memory bill scales with the whole model.

And here's the part that makes the bill feel worse. If you have five different fine-tuned variants (English-only chat, code assistant, legal summarizer, medical triage, SQL helper), you now store five separate 70B checkpoints. 1.4 TB of weights on disk. 700 GB of HBM if you want to serve all five at once. Every task duplicates a full copy of the base model.

That duplication is the problem LoRA addresses.

The intuition that cracked this open

Back in 2020, Aghajanyan et al. published a paper whose title is basically the punchline: "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning." Their result, stripped down: fine-tuning BERT doesn't need 110 million independent directions in parameter space. It needs something on the order of 200 to 2,000 directions. The rest of the update is zero-ish, or along directions the other updates already covered.

So when we "fine-tune" a big model, the change we're making, even though it's in principle free to touch every weight, in practice lives in a tiny subspace of the full parameter space.

That's the opening LoRA walked through. If the update $\Delta W$ is low-rank anyway, we don't need to parameterize it as a dense $d \times d$ matrix. We can parameterize it as a product of two skinny matrices and train those instead.

The math, stripped down

Take any linear layer in the model:

y = W x, \quad W \in \mathbb{R}^{d \times d}

In full fine-tuning, we let $W$ move around freely. The update is $\Delta W$ , also $d \times d$ . That's $d^2$ trainable parameters per linear layer.

LoRA does this instead:

W' = W + \Delta W, \quad \Delta W = B A

where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times d}$ with $r \ll d$ .

$W$ is frozen. We don't compute gradients on it. We don't store Adam states for it. It sits there like a rock.

We only train $B$ and $A$ . Together they have $2 \cdot r \cdot d$ parameters. When we multiply them, we get a matrix with the same shape as $W$ , but whose rank is at most $r$ . That's the thing we add to $W$ to form the effective weight.

A few numbers, because this is where it starts to feel like free money:

For a Llama-7B-style Linear layer with $d = 4096$ :

Full FT: $d^2 = 16{,}777{,}216$ trainable params.
LoRA, $r = 8$ : $2rd = 65{,}536$ trainable params.
Ratio: 0.39%.

Rank-r decomposition · visualized d = 64

rank r = 4

typical: 4 – 32

W (frozen) · d × d

frozen

64² = 4096 cells rendered

B · d × r

256 cells

A · r × d

256 cells

ΔW · d × d, rank ≤ 4

low-rank sum of r outer products

full fine-tune

16.78M

d² trainable

LoRA, rank 4

32.77K

2·r·d trainable

ratio

0.20%

2r / d

Slide the rank up. The grey W on the left is huge and stays frozen. The blue pair B · A is tiny and does all the training. At r=8, you're training 0.4% of what full fine-tuning would touch, and at a realistic model scale (d=4,096) you're moving from 16M parameters to 65K per layer.

A lot of people read LoRA as a compressed approximation of full fine-tuning, but it's really a different hypothesis about where the update should live. LoRA says "the update is in this low-rank subspace, find it." Full FT says "the update can go anywhere, find it." If the first assumption holds (and for task-specific fine-tuning it usually does), LoRA finds the update just as well with 1% of the cost.

What changes inside the forward pass

The LoRA forward pass has two branches running side-by-side. Let me walk through what runs on the GPU when a token flows through a LoRA-wrapped Linear layer.

At training time, given input $x$ :

y = W x + B A x

You compute two things in parallel: the original $Wx$ (using the frozen weight) and the LoRA branch $BAx$ (using the trainable pair). You add them. Two matmuls in place of one.

The gradient flow is the interesting bit. The gradient of the loss with respect to $W$ is never computed. Autograd sees $W$ has requires_grad=False and skips it. The gradient does flow through the $B A x$ branch to $A$ and $B$ , and those are the only weights the optimizer updates.

There's also a scaling factor that LoRA introduces: $\alpha / r$ , where $\alpha$ is a hyperparameter that controls the magnitude of the adapter's contribution. The practical reason: if you change $r$ mid-project, you don't want the effective update size to change. Dividing by $r$ keeps the magnitude stable across ranks. Typical choice: $\alpha = 2r$ , which gives an effective scale of 2.

So the full forward pass is:

y = W x + \frac{\alpha}{r} \cdot B A x

At inference time, you have a choice. You can keep the two branches separate (more flexible, zero base-model duplication, slight overhead), or you can fold them together once: $W_{\text{merged}} = W + (\alpha/r) B A$ , and then run the forward pass as a single matmul against $W_{\text{merged}}$ . That second mode is literally free at inference (same cost as the base model, zero overhead). The trade-off is that a merged model is specialized to that one adapter; you've baked the specialization in.

Worked example

Let me do one fully concrete calculation because the numbers are satisfying.

Say we have a Llama-2-7B model and we want to fine-tune it on internal company documentation.

7B model dimensions:

Hidden size $d = 4096$ .
Number of transformer layers: 32.
Attention projections: $W_Q$ , $W_K$ , $W_V$ , $W_O$ , each $d \times d$ .
FFN: standard SwiGLU has three projections of size around $d \times (d \cdot \text{ff\_mult})$ .

In the original LoRA paper, they apply adapters to $W_Q$ and $W_V$ only. That's 2 Linear layers per block × 32 blocks = 64 Linear layers being LoRA-ized.

With $r = 8$ , each adapter adds:

2 \cdot r \cdot d = 2 \cdot 8 \cdot 4096 = 65{,}536 \text{ params}

Across 64 adapters: $64 \times 65{,}536 \approx 4.2$ million trainable parameters. The base model has about 6.7 billion parameters. So we're training 0.06% of the model.

Memory during training:

Base weights (frozen, fp16): $6.7\text{B} \times 2 = 13.4$ GB.
Activations (seq 1024, batch 1): roughly 8–12 GB (depends on checkpointing).
Adapter weights (fp16): $4.2\text{M} \times 2 \approx 8.4$ MB. Negligible.
Adapter gradients (fp16): another ~8 MB. Still negligible.
Adapter Adam states (fp32 $m$ , $v$ ): $4.2\text{M} \times 8 \approx 34$ MB. Still negligible.

Total: roughly 25 GB. That fits on a single 40 GB A100 with room to spare for a larger batch or longer sequence. A full fine-tune of the same model would need more like 140 GB in aggregate (weights + grads + Adam states + activations across the whole thing), putting it well outside a single consumer-ish GPU.

The activations dominate, not the adapter weights. Which is exactly what you'd hope: the "cost" of the fine-tune is now just the cost of running forward and backward through a frozen model.

Picking the rank

Okay, so $r = 8$ . Why not $r = 2$ ? Why not $r = 64$ ?

In practice, rank selection follows a pattern the LoRA paper established and most later work confirmed. Quality rises quickly as $r$ goes from 1 to about 8 or 16, then flattens. For most instruction-tuning-style tasks, anywhere from $r = 4$ to $r = 32$ works fine. For narrower tasks (a single domain, a single format), you can often get away with $r = 4$ or even $r = 2$ . For tasks that require learning genuinely new capabilities (a language the base model doesn't know well, for instance), you'd reach for $r = 64$ or higher, or you'd honestly consider whether full fine-tuning is what you want.

Rank vs. task quality · stylized

current r = 1

Stylized curves from several LoRA follow-up papers. The single-task line saturates before r=8. Broad-capability adaptation (trying to make a small model generally smarter) plateaus well below full fine-tuning no matter how high you push r. The flat middle region is where most practitioners live.

The shape of these curves is the important thing, not the absolute numbers. The "single-task" curve is what gets people excited. You can match full fine-tuning quality on a narrow task with rank 8 or 16. The "broad capability" curve is what keeps people honest. LoRA was never designed to replace pretraining.

QLoRA: the 65B-on-one-GPU trick

LoRA cut the optimizer cost. It didn't cut the cost of holding the frozen base model in memory. For a 70B model, you're still paying 140 GB just to keep $W$ resident in HBM during training.

QLoRA, from Dettmers et al. in 2023, fixed that. The recipe:

Quantize the base model to 4-bit using a format called NF4 (normalized float 4, which spaces values to match a Gaussian distribution rather than uniformly). This takes the 140 GB frozen base down to about 35 GB.
Keep the LoRA adapters in fp16. They're tiny, so there's no reason to quantize them, and keeping them in higher precision matters for training stability.
Double-quantize the quantization constants. The NF4 codebook has per-block scales. Quantize those too. Small but meaningful win on a big model.
Page optimizer states to CPU when they don't fit in GPU memory, using NVIDIA's unified memory feature. The Adam states are all tiny anyway because LoRA adapters are tiny, but paged memory gives you headroom.

Training memory · 65B model · batch 1 · seq 1024

scale: 0 → 780 GB

For a 65B model, full fine-tuning needs around 560 GB just for training state. LoRA cuts that to ~180 GB by freezing the base. QLoRA cuts it further to 45 GB by quantizing the frozen base to NF4. The dashed lines show GPU ceilings: QLoRA fits on a single 48 GB A6000.

The forward pass in QLoRA dequantizes one linear layer's weights on-the-fly from NF4 back to bfloat16 for the matmul, then the bfloat16 activations flow through as usual. The dequantization is cheap because it's a per-block lookup, and modern kernels fuse it into the matmul itself.

What surprised me when I first read the QLoRA paper was the quality claim. On the MMLU benchmark, their 65B QLoRA variants matched or beat the full 16-bit baselines after Guanaco-style instruction tuning. The 4-bit base wasn't a quality floor. It was, in retrospect, just a smaller hardware bill. QLoRA is basically LoRA's "does this still work if we crush the frozen part into 4 bits" check, and the answer was yes.

Merging, and what it buys you

Once training is done, you have a choice. You can keep the adapter separate and compose it with the base at inference time (as $y = Wx + BAx$ , with $W$ frozen and $BA$ living off to the side). Or you can precompute $W' = W + BA$ and replace $W$ in the model outright.

The merged form is mathematically identical at inference. Zero overhead. Same latency as the base model. The downside: you've picked one adapter. The model is now specialized to that task.

The separate form has two wins:

One base model, many tasks. You keep one copy of $W$ resident, and you swap a small adapter (a few MB each) per request. A team like Hugging Face or Replicate can host hundreds of customer-specific adapters per replica this way.
Instant adapter switching. No need to reload a 70B weight file. The adapter load is microseconds.

Serving mode

Two serving modes. Dynamic (top button) keeps the base frozen and loads one of many small adapters per request. Merged (bottom button) bakes one adapter into the base and serves it as a single-tenant model. The merged form has zero overhead but locks in one tenant per replica. Try toggling modes to see a mismatched adapter hit the merged base.

The "dynamic" serving pattern got a proper research treatment with S-LoRA in 2023. Their contribution: unified paged memory for many adapters, heterogeneous batching across requests using different adapters, and some clever kernel tricks so the per-request overhead is under 5%. This is how modern multi-tenant LoRA serving works in production.

The broader PEFT family

LoRA is the method that won on adoption, but it's one of several parameter-efficient fine-tuning ideas that showed up in the same era. A few you'll run into:

PEFT family · what each method trains

BitFit

2021

frozen W, train only the small b vectors

trained: biases only

~0.08%

of base

Prefix Tuning

2021

learn virtual tokens that steer attention

trained: prepended KV states

~0.1%

of base

Adapter (Houlsby)

2019

bottleneck MLP added between layers

trained: inserted MLP

~2%

of base

LoRA

2021

additive low-rank update to linear layers

trained: BA on Q, V

~0.1–0.5%

of base

DoRA

2024

split W into magnitude · direction, adapt direction

trained: magnitude + LoRA direction

~0.2%

of base

QLoRA

2023

nf4-quantized frozen base + fp16 LoRA

trained: LoRA over 4-bit base

~0.2%

of base

Six PEFT methods and where each one places its trainable parameters. LoRA won on the serving side, but the others aren't dead — BitFit is popular for probing studies, DoRA is a 2024 upgrade to LoRA, and adapters still show up in research where architectural changes are needed.

A quick tour:

BitFit trains only the bias vectors of every linear layer. That's about 0.08% trainable parameters, and it's still competitive on some classification tasks. Mostly useful today for interpretability probes and as a baseline.
Prefix Tuning / Prompt Tuning prepend a handful of learnable "virtual tokens" to the input. The frozen model's attention learns to read them as soft instructions. Good for classification-style tasks; not as strong as LoRA on generation.
Adapters (Houlsby, Pfeiffer) insert a small bottleneck MLP between transformer blocks. Predates LoRA by two years. More trainable params than LoRA, adds real inference latency (extra layer, can't merge in), but the idea of "inject learnable capacity" is the direct intellectual ancestor.
LoRA is what this post is about. Low-rank, additive, merge-able.
DoRA (Weight-Decomposed Low-Rank Adaptation, 2024) splits each frozen weight matrix into magnitude and direction, adapts the direction with LoRA, and trains the magnitude directly. Usually matches or slightly beats LoRA, especially at low ranks.
QLoRA is the quantization-aware sibling. 4-bit base, 16-bit LoRA. Same math, drastically smaller memory bill.

If you're starting a new project today, LoRA (or its QLoRA variant for large models) is the default. DoRA is worth trying if you're tuning at very low ranks. The others are mostly historical or domain-specific.

Implementation sketch

Here's the smallest LoRA Linear layer that works. A wrapper around a frozen base layer, with a trainable $A$ and $B$ pair on the side.

import math
import torch
import torch.nn as nn
 
class LoRALinear(nn.Module):
    """
    Wraps an existing nn.Linear with a LoRA adapter.
    Freezes the base weight. Trains A and B only.
    """
 
    def __init__(self, base_linear: nn.Linear, r: int = 8, alpha: float = 16.0):
        super().__init__()
        self.base = base_linear
        self.r = r
        self.scale = alpha / r
 
        in_f = base_linear.in_features
        out_f = base_linear.out_features
 
        # A : (r, in_f), initialized small-random
        # B : (out_f, r), initialized to zero
        # At init, BA = 0, so the model behaves exactly like the base.
        self.A = nn.Parameter(torch.empty(r, in_f))
        self.B = nn.Parameter(torch.zeros(out_f, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
 
        # Freeze the base weight and bias — only A, B will get gradients.
        for p in self.base.parameters():
            p.requires_grad = False
 
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Original Linear: (..., in_f) -> (..., out_f)
        base_out = self.base(x)
 
        # LoRA branch: x @ A.T -> (..., r), then @ B.T -> (..., out_f)
        # This order keeps the intermediate tensor r-wide, which is cheap.
        lora_out = (x @ self.A.T) @ self.B.T
 
        return base_out + self.scale * lora_out
 
    @torch.no_grad()
    def merge(self) -> nn.Linear:
        """Fold the adapter into a new nn.Linear. One-time, for inference."""
        delta = self.scale * (self.B @ self.A)  # (out_f, in_f)
        merged = nn.Linear(
            self.base.in_features,
            self.base.out_features,
            bias=self.base.bias is not None,
        )
        merged.weight.copy_(self.base.weight + delta)
        if self.base.bias is not None:
            merged.bias.copy_(self.base.bias)
        return merged

First, $B$ initializes to zero and $A$ initializes to a small random matrix. At step 0, $BA = 0$ , which means the wrapped layer behaves exactly like the base. Training can only add to the base behavior, never yank it in a random direction at init. Second, the order of operations in forward keeps the intermediate tensor only $r$ -wide, which is a genuinely tiny amount of extra compute. If you'd computed $BA$ first as a full $d \times d$ matrix, you'd be doing pointless work.

Real libraries (Hugging Face's PEFT, LoRAX, FSDP-LoRA) have more machinery for state-dict handling, quantized-base support, target-module selection, and multi-adapter loading. But the core is this, in 40 lines.

Misconceptions

"LoRA is a compressed version of fine-tuning." Not quite. LoRA is a different hypothesis about where the adaptation lives. Full fine-tuning lets $\Delta W$ be any rank. LoRA constrains $\Delta W$ to rank $r$ . If the true update is low-rank (common for single-task adaptation), LoRA finds it. If the true update is genuinely high-rank (teaching new capabilities, new languages, new reasoning patterns), LoRA will underfit no matter how long you train. The "lossy fine-tuning" framing is right only in the sense that every constrained hypothesis class is lossy. Within its lane, LoRA is a structural bet that often happens to be correct.

"More rank is always better." Almost never. On most task-specific fine-tunes, quality plateaus somewhere between $r = 8$ and $r = 32$ , and going higher burns optimizer memory for no quality gain. The interesting failure mode is the opposite direction: a rank chosen too low for a genuinely complex task can hit a ceiling that looks like "LoRA doesn't work here" when the actual fix is a bigger $r$ or a different target-module set.

"QLoRA is just LoRA with quantized weights." That's the one-line pitch, and it's not wrong, but the interesting engineering is in the NF4 format, double quantization, and paged optimizer offload. Naive 4-bit quantization of the base weights would have made training unstable. What Dettmers et al. did was engineer a quantization scheme aware of how fine-tuning gradients interact with quantized weights, and prove that, when assembled correctly, the pipeline matched 16-bit training quality. That's a lot more than "slap int4 on it."

"You can't merge an adapter if you used QLoRA." You can, but the merge destroys the 4-bit savings. Merging means computing $W' = W + (\alpha / r) BA$ and storing $W'$ in whatever precision you want to serve in. If you dequantize $W$ to bf16, merge, and store bf16, you've spent your 4-bit budget. The usual QLoRA deployment path is to dequantize-merge-requantize, or to serve the base + adapter separately without merging. It's a real trade-off, not a blocker.

What's next

This is the final post of Arc 7. We've gone from pretraining data through distributed training, SFT, RLHF, DPO, constitutional AI, tool-use fine-tuning, long-context extension, synthetic data and distillation, and now parameter-efficient fine-tuning. The shape of post-training as a field, at least as of 2026, is roughly that stack.

Arc 8 turns to evaluation. The next post covers loss vs. benchmarks, why training loss and benchmark scores tell you different things about a model, and how to read both honestly. Everything we've trained in Arc 7 gets judged there.

Additional reading (and watching)

Rajbhandari, S., et al. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020. The paper that put clear per-parameter byte counts on Adam-style training and introduced the sharding tricks that made trillion-parameter training real.
Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. The canonical reference. Defines the $W + BA$ update, the $\alpha / r$ scaling, the merge trick, and the "adapt only Q and V" recommendation.
Kingma, D. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. Source of the $m$ and $v$ state that makes fine-tuning so memory-hungry in the first place.
Aghajanyan, A., et al. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL 2021. The paper that made "the update lives in a low-dimensional subspace" quantitative, which set up LoRA's existence.
Lialin, V., et al. (2023). Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning. The best single-survey starting point for the full PEFT landscape, including rank-vs-quality findings across many methods.
Dettmers, T., et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. Source of NF4, double quantization, paged optimizer offload, and the 65B-on-a-single-48GB-GPU result.
Sheng, Y., et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. How to batch requests across many adapters against a single frozen base model with under 5% overhead.
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2021). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. ACL 2022. Train only the biases; a strong minimalist baseline in the PEFT family.
Pfeiffer, J., et al. (2020). AdapterHub: A Framework for Adapting Transformers. EMNLP 2020 (System Demos). The adapters-as-insert-modules precursor to LoRA.
Liu, S.-Y., et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024. Splits $W$ into magnitude and direction, applies LoRA to the direction only. Consistently beats vanilla LoRA at low ranks.
Mangrulkar, S., et al. (2022). PEFT: State-of-the-Art Parameter-Efficient Fine-Tuning Methods. GitHub. The reference library for applying LoRA and friends to Hugging Face models.
Ryu, S. (2023). Low-rank Adaptation for Fast Text-to-Image Diffusion Fine-tuning. GitHub. The port of LoRA to Stable Diffusion that kicked off the SD LoRA ecosystem.
Lialin, V., et al. (2023). ReLoRA: High-Rank Training Through Low-Rank Updates. NeurIPS 2023 workshop. Stacks LoRA updates over training so the effective rank grows, bridging the gap between LoRA and full FT.