LoRA & QLoRA Architectural Policy
Low-Rank Adaptation (LoRA) freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, dramatically reducing the number of trainable parameters.
The platform explicitly separates controls that novices often confuse:
• LoRA Rank (r): Dimensionality of adapter weight decomposition (e.g. 16, 32).
• Retrieval Candidate top_k: Number of vector search items fetched from the index (e.g. 20-50).
• Reranker Final k: Top items preserved after cross-encoder scoring (e.g. 3-5).
• Generation Sampling top_k: Vocabulary distribution sampling truncation (e.g. 40).
Recommended Defaults
LoRA Rank (r) = 16
Captures rich domain adaptation while keeping adapter weights compact (~40MB-120MB per model).
LoRA Alpha (α) = 32 (2 * r)
Maintains stable scaling ratio (alpha / r = 2.0) preventing gradient explosion during backpropagation.
Target Modules: All Linear Layers
Adapting attention projections (q, k, v, o) and MLP feed-forward projections (gate, up, down) produces strictly superior generalization.
Learning Rate: 2e-4 (QLoRA) / 5e-5 (LoRA)
Cosine learning rate scheduler with 3% warmup steps to prevent loss spikes in early epochs.