Model Evaluation & Benchmarking Guide
Fine-tuning is only as effective as your evaluation methodology. This guide covers how to systematically verify that your LoRA adapter improves domain performance without catastrophic forgetting or latency regressions.
Verify instruction-following fidelity, output formatting (JSON/Markdown), and tone adherence on held-out test sets.
Evaluate convergence curve against your 15% validation split to confirm the model generalizes rather than memorizes.
Measure physical execution speed (tokens/sec) and time-to-first-token (TTFT) on your workstation GPU or Apple Silicon.
1. Side-by-Side Comparative Playground
The FineTuneMyAI Evaluation Playground (/app/inference) provides synchronized side-by-side inference. It executes identical test prompts simultaneously against:
- Base Foundational Model: The un-tuned model checkpoint (e.g. Qwen2.5-Coder 7B).
- Fine-Tuned Adapter: The base model with your newly trained low-rank adapter weights hot-loaded.
Each side outputs authentic hardware metrics including high-resolution latency (in milliseconds) and true generation speed (tokens per second).
2. Guarding Against Catastrophic Forgetting
When adapting an LLM to a specific domain (e.g. internal API syntax), it is critical to ensure general conversational and reasoning capabilities remain intact:
Always evaluate your adapter against 20 general reasoning prompts (logic, math, world knowledge) in addition to your domain-specific validation prompts. If general accuracy drops by more than 5%, consider lowering the LoRA rank ($r$) or increasing the learning rate weight decay.