Definition
The edit distance test measures the minimum number of single-character insertions, deletions, or substitutions required to transform one string into another, serving as a measure of their similarity.Taxonomy
- Task types: LLM.
- Availability: and .
Why it matters
- Edit distance provides a character-level measure of how different two strings are, which is useful for evaluating text generation quality.
- This metric is particularly valuable when you need to measure fine-grained differences between generated and expected text.
- Lower edit distances indicate higher similarity between the generated output and the reference text.
Required columns
To compute this metric, your dataset must contain the following columns:- Outputs: The generated text from your LLM
- Ground truths: The reference/expected text to compare against
Edit distance is computed on the first 2,000 characters of each output and
ground truth, so for longer texts the score is an approximation. Keep this in
mind when choosing thresholds for long-form outputs.
Test configuration examples
If you are writing atests.json, here are a few valid configurations for the edit distance test:
Related
- BLEU score test - Measure n-gram based text similarity.
- Exact match test - Assess identical string matches.
- Quasi-exact match test - Allow partial matches and variations.
- Aggregate metrics - Overview of all available metrics.