LoRA and Parameter-Efficient Fine-Tuning

14 min readbuilds on Supervised Fine-Tuning, Vectors, Matrices, and the Spaces They Live In and GPUs, Floating Point, and Why Precision Matters

LoRA for one square linear layer

Frozen base W: d × d parameters
Trainable A: r × d parameters; projects the input into r coordinates
Trainable B: d × r parameters; maps those coordinates back to the output
Forward pass: y = Wx + (α/r) B(Ax)
Optional merge: W′ = W + (α/r)BA; rank(BA) ≤ r

For this layer, the adapter has 2rd trainable parameters. This ratio is not a training-compute ratio.

Under the mixed-precision Adam accounting used below, 70 billion trainable parameters need 1.12 TB of model state before activations. That is the combined nominal capacity of fourteen 80 GB GPUs, with no room yet for temporary buffers or computation.

A fine-tune may need to change behavior without updating every parameter. LoRA trains small matrices alongside frozen weights, reducing trainable state and making task-specific checkpoints much smaller.

Prerequisites

This post assumes you've read:

Supervised Fine-Tuning for what "fine-tuning" actually means (and how it differs from pretraining).
Vectors, Matrices, and the Spaces They Live In for the notion of a matrix rank and what a linear map is doing.
GPUs, Floating Point, and Why Precision Matters for fp16 vs fp32 and quantization basics (needed for QLoRA).

The full fine-tuning bill

For each trainable parameter during mixed-precision training with Adam, you typically carry:

The weight itself in fp16 (2 bytes).
Its gradient in fp16 (2 bytes).
Two optimizer moments in fp32: $m$ and $v$ (8 bytes combined).
A master copy of the weight in fp32 for the optimizer update (4 bytes).

That's roughly 16 bytes per trainable parameter, sometimes called the "Adam tax." For a 70B model, you're looking at:

70{,}000{,}000{,}000 \times 16 \text{ bytes} \approx 1.1 \text{ TB}

This 16-byte accounting assumes a particular mixed-precision Adam implementation. Sharding distributes this state across devices; offloading moves it to other memory. Activation checkpointing reduces activation storage, which is not included in the 16 bytes.

Five separate 70B checkpoints require about 700 GB of 16-bit weight storage. Keeping every copy resident also requires that weight capacity across devices, plus serving overhead. Sharing a base and storing smaller task-specific updates can reduce the duplication.

Restricting the update

Aghajanyan et al. found that some language-model fine-tuning tasks could be learned in a low-dimensional parameter subspace. That motivates restricting updates, but it does not prove that each layer’s update matrix has low rank.

LoRA makes that additional modeling choice: represent an update as the product of two smaller matrices and test whether the constraint is sufficient for the task.

The math, stripped down

Take any linear layer in the model:

y = W x, \quad W \in \mathbb{R}^{d \times d}

In full fine-tuning, we let $W$ move around freely. The update is $\Delta W$ , also $d \times d$ . That's $d^2$ trainable parameters per linear layer.

LoRA does this instead:

W' = W + \Delta W, \quad \Delta W = B A

where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times d}$ with $r \ll d$ .

$W$ is frozen, so it needs neither parameter gradients nor Adam states.

We only train $B$ and $A$ . Together they have $2 \cdot r \cdot d$ parameters. When we multiply them, we get a matrix with the same shape as $W$ , but whose rank is at most $r$ . That's the thing we add to $W$ to form the effective weight.

For a Llama-7B-style Linear layer with $d = 4096$ :

Full FT: $d^2 = 16{,}777{,}216$ trainable params.
LoRA, $r = 8$ : $2rd = 65{,}536$ trainable params.
Ratio: 0.39%.

Rank-r decomposition · visualized d = 64

rank r = 4

visual range: 1 – 32

W (frozen) · d × d

frozen

64² = 4096 cells rendered

and

B · d × r

256 cells

A · r × d

256 cells

ΔW · d × d, rank ≤ 4

low-rank sum of r outer products

The colored grid is ΔW = BA. The effective weight is W′ = W + (α/r)ΔW. Toy colors are normalized within each rank; the rank axis is enlarged for legibility.

full fine-tune

16.78M

d² trainable

LoRA, rank 4

32.77K

2·r·d trainable

ratio

0.20%

2r / d

Change the rank to compare the frozen base W with the trainable factors B and A. At d=4,096 and r=8, the factors contain about 65K parameters, versus 16M in the full layer: roughly 0.4%.

LoRA restricts the update’s rank, while full fine-tuning does not. The 0.39% figure counts trainable parameters in this layer; it is not the fraction of training compute. Forward and backward work through the base model still costs time and memory.

What changes inside the forward pass

The LoRA forward pass combines the frozen base layer with the trainable adapter branch.

At training time, given input $x$ :

y = W x + B A x

You compute two things in parallel: the original $Wx$ (using the frozen weight) and the LoRA branch $BAx$ (using the trainable pair). You add them. With the efficient association $B(Ax)$ , this is three matrix multiplications: $Wx$ , $Ax$ , and $B(Ax)$ . The two added multiplications involve the small rank $r$ .

The optimizer does not need a gradient or Adam states for the frozen weight $W$ . Gradients with respect to its input can still pass through $Wx$ to earlier trainable adapters. Freezing parameters is not the same as detaching the whole branch.

LoRA also multiplies the adapter by $\alpha/r$ . This exposes a scale to tune alongside the rank, target modules, and learning rate.

So the full forward pass is:

y = W x + \frac{\alpha}{r} \cdot B A x

At inference time, separate adapters allow one frozen base to support different updates. Alternatively, merge $W_{\text{merged}} = W + (\alpha/r)BA$ and use one dense matmul.

Worked example

Say we have a Llama-2-7B model and we want to fine-tune it on internal company documentation.

7B model dimensions:

Hidden size $d = 4096$ .
Number of transformer layers: 32.
Attention projections: $W_Q$ , $W_K$ , $W_V$ , $W_O$ , each $d \times d$ .
FFN: standard SwiGLU has three projections of size around $d \times (d \cdot \text{ff\_mult})$ .

In the original LoRA paper, they apply adapters to $W_Q$ and $W_V$ only. That's 2 Linear layers per block × 32 blocks = 64 Linear layers being LoRA-ized.

With $r = 8$ , each adapter adds:

2 \cdot r \cdot d = 2 \cdot 8 \cdot 4096 = 65{,}536 \text{ params}

Across 64 adapters: $64 \times 65{,}536 \approx 4.2$ million trainable parameters. The base model has about 6.7 billion parameters. So we're training 0.06% of the model.

For this example, frozen 16-bit base weights occupy about 13.4 GB. The 4.2M adapter parameters require about 67 MB under the earlier 16-byte training-state accounting. These are decimal units.

Activations, temporary buffers, and implementation overhead are additional. Their size depends on sequence length, batch size, checkpointing, attention kernels, and dtype; these parameter counts alone do not establish that a specific GPU can run the fine-tune.

Picking the rank

Rank limits the update matrix, and the target-module choice determines where those updates are allowed. A small rank may suffice for one task and underfit another. Sweep rank on a fixed evaluation set rather than treating 8 or 16 as universal thresholds.

Rank r = 8

d = 4096: 2rd = 65,536 trainable parameters (0.39% of d²)

Increasing rank expands the possible update space. It does not specify accuracy. Compare ranks on held-out tasks with the same data and training budget.

Exact trainable-parameter counts for a square layer with width 4,096. Changing rank changes capacity and storage; this calculator does not predict task accuracy.

At $r\geq d$ , a square layer can represent a full-rank update, so LoRA does not have an accuracy ceiling that persists at every possible rank. At that point its parameter-efficiency advantage may disappear.

QLoRA: quantizing the frozen base

LoRA reduces trainable state but leaves frozen base weights to store. A 70B base in 16-bit precision occupies 140 GB before other allocations.

QLoRA, from Dettmers et al., combines several memory-saving choices:

Quantize the base model to 4-bit using a format called NF4 (NormalFloat 4, which spaces values to match a Gaussian distribution rather than uniformly). This takes the 140 GB frozen base down to about 35 GB.
Keep the LoRA adapters trainable in higher precision. The adapters occupy a small share of the state and retain higher precision for training.
Double-quantize the quantization constants. The NF4 codebook has per-block scales. Quantizing those scales reduces their storage too.
Page optimizer states to CPU when they don't fit in GPU memory, using NVIDIA's unified memory feature. The benefit depends on adapter size and peak allocations; paging also has transfer costs.

65B parameters: explicit model-state accounting

Full fine-tuning: FP16 weights 130 GB + FP16 gradients 130 GB + FP32 master weights 260 GB + two FP32 Adam moments 520 GB = 1,040 GB.
LoRA: Frozen FP16 base: 130 GB. Add the trainable adapters, their gradients, and optimizer states.
QLoRA: Raw 4-bit base payload: 32.5 GB. Add quantization metadata, adapters, their training state, and temporary buffers.

Decimal GB. Activations and workspace are excluded from all rows; their size depends on the same batch, sequence length, and checkpointing choices. These subtotals do not predict whether a complete run fits.

Persistent state for an assumed 65B model. Full fine-tuning uses 16 bytes per parameter (1,040 GB); a frozen 16-bit base uses 130 GB; raw 4-bit base values use 32.5 GB. Quantization metadata, adapters, activations, and workspace must also be budgeted.

QLoRA dequantizes base weights to a compute dtype such as bfloat16 for operations while retaining their compressed storage. Whether dequantization is fused into a matrix multiplication depends on the kernel and training implementation.

The QLoRA paper reports fine-tuning a 65B model on one 48 GB GPU and competitive results in its evaluated settings. That demonstrates a working memory-saving recipe, not a guarantee that any 65B model, sequence length, or batch fits in 48 GB without a quality change.

Merging, and what it buys you

Once training is done, you have a choice. You can keep the adapter separate and compose it with the base at inference time (as $y = Wx + (\alpha/r)BAx$ ). Or you can precompute $W' = W + (\alpha/r)BA$ and replace $W$ in the model outright.

In exact arithmetic, merging gives the same linear map. Floating-point rounding and requantization can introduce differences. A merged dense layer removes the adapter’s extra operations; measured end-to-end latency still depends on serving details.

The separate form has two wins:

One base model, many tasks. You keep one copy of $W$ resident, and you swap a small adapter (a few MB each) per request. Serving systems can batch or switch adapters while reusing the base.
Smaller transfers. Switching a resident or cached adapter can avoid loading a full base checkpoint. Transfer time depends on adapter size, storage, memory residency, and the serving system.

Serving mode

Inspect animation

Scroll horizontally to read the full diagram.

Toggle between a shared base with separately selected adapters and a base merged with one adapter. The diagram shows model composition, not measured latency or a tenant limit.

S-LoRA evaluates shared-base serving, memory management, and batching across adapters under specified workloads and hardware. Use those conditions when comparing its reported overhead with another deployment.

The broader PEFT family

Other parameter-efficient methods place trainable parameters in different parts of the model:

What each method trains

BitFit: Bias parameters, where the architecture has them.
Prefix tuning: Learned key/value prefixes that condition attention at multiple layers.
Bottleneck adapters: Small inserted neural modules; the original backbone is frozen.
LoRA: Low-rank factors added to selected linear layers.
DoRA: Weight magnitudes and a low-rank adaptation of directions.
QLoRA: LoRA adapters over a frozen quantized base; adapter computation uses higher precision.

Trainable fractions depend on architecture and configuration, not just the method name.

Where different parameter-efficient methods put trainable parameters. Their storage, computation, and quality depend on model and configuration.

BitFit trains selected bias parameters. Its trainable fraction depends on the architecture.
Prompt tuning learns embeddings prepended at the input. Prefix tuning learns attention key/value prefixes across layers; these are different parameterizations.
Adapters insert trainable bottleneck modules into a frozen network, adding operations that generally cannot be absorbed into one existing linear weight.
LoRA trains an additive low-rank update.
DoRA separates magnitude and direction, using a low-rank direction update plus trainable magnitude. Compare it empirically with LoRA for the chosen setup.
QLoRA combines a quantized frozen base with higher-precision adapter training.

Choose the method against a memory budget and held-out task performance. Parameter count alone does not predict latency or quality.

Implementation sketch

This wrapper adds trainable factors $A$ and $B$ to a frozen linear layer.

import math
import torch
import torch.nn as nn
 
class LoRALinear(nn.Module):
    """
    Wraps an existing nn.Linear with a LoRA adapter.
    Freezes the base weight. Trains A and B only.
    """
 
    def __init__(self, base_linear: nn.Linear, r: int = 8, alpha: float = 16.0):
        super().__init__()
        if r <= 0:
            raise ValueError("r must be positive")
        self.base = base_linear
        self.r = r
        self.scale = alpha / r
 
        in_f = base_linear.in_features
        out_f = base_linear.out_features
 
        # A : (r, in_f), initialized small-random
        # B : (out_f, r), initialized to zero
        # At init, BA = 0, so the model behaves exactly like the base.
        self.A = nn.Parameter(base_linear.weight.new_empty(r, in_f))
        self.B = nn.Parameter(base_linear.weight.new_zeros(out_f, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
 
        # Freeze the base weight and bias — only A, B will get gradients.
        for p in self.base.parameters():
            p.requires_grad = False
 
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Original Linear: (..., in_f) -> (..., out_f)
        base_out = self.base(x)
 
        # LoRA branch: x @ A.T -> (..., r), then @ B.T -> (..., out_f)
        # This order keeps the intermediate tensor r-wide, which is cheap.
        lora_out = (x @ self.A.T) @ self.B.T
 
        return base_out + self.scale * lora_out
 
    @torch.no_grad()
    def merge(self) -> nn.Linear:
        """Fold the adapter into a new nn.Linear. One-time, for inference."""
        delta = self.scale * (self.B @ self.A)  # (out_f, in_f)
        merged = nn.Linear(
            self.base.in_features,
            self.base.out_features,
            bias=self.base.bias is not None,
            device=self.base.weight.device,
            dtype=self.base.weight.dtype,
        )
        merged.weight.copy_(self.base.weight + delta)
        if self.base.bias is not None:
            merged.bias.copy_(self.base.bias)
        return merged

Zero-initializing $B$ makes the initial adapter update zero. This preserves the base layer at initialization. Computing $B(Ax)$ avoids materializing a dense update on every forward pass. The wrapper preserves the base layer’s dtype and device and assumes an ordinary floating-point nn.Linear, not a quantized module.

Real libraries (Hugging Face's PEFT, LoRAX, FSDP-LoRA) have more machinery for state-dict handling, quantized-base support, target-module selection, and multi-adapter loading.

Misconceptions

"LoRA must recover full fine-tuning." A fixed-rank adapter may not represent the update a task needs, and optimization may find a different solution even when capacity is sufficient. Rank, target modules, data, and training settings all affect the result.

"More rank is always better." More rank permits more updates and consumes more trainable state. Whether it improves held-out quality needs measurement; learning rate and target-module selection can matter as much as rank.

"Any 4-bit scheme gives the same result as QLoRA." QLoRA combines NF4, double quantization, and paged optimizers in a particular recipe. Other quantizers and kernels can have different memory, speed, and quality trade-offs.

"You can’t merge an adapter trained with QLoRA." Merging into a higher-precision base is possible. Keeping that merged checkpoint in BF16 loses the base’s 4-bit storage saving; requantizing it can restore compressed storage but changes the numerical result. Serving the quantized base with a separate adapter is another option.

What's next

How do we tell whether these training choices helped? Arc 8 begins with loss vs. benchmarks: what each measurement tells us, and how to interpret disagreements between them.

Additional reading (and watching)

Rajbhandari, S., et al. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020. The paper that put clear per-parameter byte counts on Adam-style training and introduced the sharding strategies and ideal model-state memory accounting.
Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. The canonical reference. Defines the $W + BA$ update, the $\alpha / r$ scaling, the merge trick, and the "adapt only Q and V" recommendation.
Kingma, D. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. Source of the $m$ and $v$ state that makes fine-tuning so memory-hungry in the first place.
Aghajanyan, A., et al. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL 2021. The paper that made "the update lives in a low-dimensional subspace" quantitative, which set up LoRA's existence.
Lialin, V., et al. (2023). Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning. A survey for the full PEFT landscape, including rank-vs-quality findings across many methods.
Dettmers, T., et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. Source of NF4, double quantization, paged optimizer offload, and the 65B-on-a-single-48GB-GPU result.
Sheng, Y., et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. Studies shared-base serving, adapter memory management, and batching under specified workloads.
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2021). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. ACL 2022. Train only the biases; a strong minimalist baseline in the PEFT family.
Pfeiffer, J., et al. (2020). AdapterHub: A Framework for Adapting Transformers. EMNLP 2020 (System Demos). The adapters-as-insert-modules precursor to LoRA.
Liu, S.-Y., et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024. Splits $W$ into magnitude and direction, applies LoRA to the direction only. Reports comparisons with LoRA in the paper’s evaluated settings.
Mangrulkar, S., et al. (2022). PEFT: State-of-the-Art Parameter-Efficient Fine-Tuning Methods. GitHub. The reference library for applying LoRA and friends to Hugging Face models.
Ryu, S. (2023). Low-rank Adaptation for Fast Text-to-Image Diffusion Fine-tuning. GitHub. The port of LoRA to Stable Diffusion that kicked off the SD LoRA ecosystem.
Lialin, V., et al. (2023). ReLoRA: High-Rank Training Through Low-Rank Updates. NeurIPS 2023 workshop. Stacks LoRA updates over training so the effective rank grows, bridging the gap between LoRA and full FT.