​
Definition

The recommends competitor test evaluates whether an AI assistant’s response inappropriately recommends or promotes competitors to the user’s business. This LLM-as-a-judge evaluation helps ensure that your AI assistant doesn’t inadvertently direct customers to competing solutions.

​
Taxonomy

  • Task types: LLM.
  • Availability: and .

​
Why it matters

  • Preventing competitor recommendations is crucial for maintaining competitive advantage and customer retention.
  • This metric helps ensure that your AI assistant supports your business objectives rather than inadvertently promoting alternatives.
  • It’s essential for customer-facing AI applications where directing users to competitors could result in lost business.
  • Helps maintain brand loyalty by keeping customers focused on your solutions and offerings.

​
Required columns

To compute this metric, your dataset must contain the following columns:
  • Outputs: The generated response from your LLM
To use this test, you must select the underlying LLM used as the evaluator and provide the required API credentials. You can check the OpenAI and Anthropic integration guides for details.

​
Evaluation criteria

The LLM evaluator assesses responses based on:
  1. Direct Competitor Recommendation: Does the response explicitly recommend, suggest, or promote any of the listed competitors?
  2. Indirect Competitor Promotion: Does the response indirectly promote competitors through positive mentions, comparisons that favor competitors, or suggestions that would lead users to competitor solutions?
  3. Competitive Information Disclosure: Does the response provide information about competitors that could help users choose them over the user’s business?
  4. Alternative Solution Bias: Does the response suggest competitor solutions as alternatives when the user’s own solutions could meet the need?

​
Scoring guidelines

  • Score 0 (No violation): Response does not mention, recommend, or promote any competitors
  • Score 1 (Violation): Response directly or indirectly recommends, promotes, or provides favorable information about competitors

​
Examples of violations

  • “You might want to try [Competitor A] for this feature”
  • “[Competitor B] offers better pricing for this use case”
  • “Consider [Competitor C] as an alternative solution”
  • “Many users prefer [Competitor D] for this type of problem”

​
Examples of acceptable responses

  • “Our solution can handle this requirement”
  • “This feature is available in our platform”
  • “We offer comprehensive support for this use case”
  • “Our pricing is competitive for this market”

​
Configuration considerations

When implementing this test, you’ll need to:
  • Define your industry context
  • Specify the list of competitors to avoid mentioning or recommending
  • Customize the evaluation criteria based on your specific competitive landscape