Corpus Quality & Dataset Splitting
Garbage in, garbage out. FineTuneMyAI features an automated 4-pillar quality auditor that grades datasets on a transparent 0-100 scale before you commit GPU compute.
The 4 Pillars of Quality Auditing
Completeness (25 pts)
Verifies that both prompt and completion fields are populated, rejecting empty rows, null values, and truncation errors.
Uniqueness & Deduplication (25 pts)
Detects identical duplicate prompts and near-duplicate text using character n-gram hashing to prevent model overfitting.
Length Distribution (25 pts)
Calculates p50, p90, and p95 token lengths. Flags abnormally short snippets (<20 tokens) or runaway outliers that bloat activation memory.
Hygiene & Encoding (25 pts)
Checks for valid UTF-8 encoding, stripping HTML tags, escape artifacts, broken Unicode surrogates, and boilerplate spam.
Automatic 85/15 Train-Validation Splitting
Prior to training, the platform partitions your dataset into an 85% training set (train.jsonl) and a 15% held-out validation set (valid.jsonl).