How to define a code evaluator

​

Step 1. Create the evaluator

1. Create an evaluator from one of the following pages in the LangSmith UI:
   • In the playground or from a dataset: Select the + Evaluator button.
   • From a tracing project: Select Add rules, configure your rule and select Apply evaluator.
2. Select Create custom code evaluator from the evaluator type options.

​

Step 2. Write your evaluator code

1. Accept run and example parameters.
2. Access data via run['inputs'], run['outputs'], and example['outputs'].
3. Return a dictionary with your metric name as the key.

​

Function signature

def perform_eval(run, example):
    # Access the data
    inputs = run['inputs']
    outputs = run['outputs']
    reference_outputs = example['outputs']  # Optional: reference/expected outputs

    # Your evaluation logic here
    score = ...

    # Return a dict with your metric name
    return {"metric_name": score}

​

Example: Exact match evaluator

def perform_eval(run, example):
    """Check if the answer exactly matches the expected answer."""
    actual = run['outputs']['answer']
    expected = example['outputs']['answer']

    is_correct = actual == expected
    return {"exact_match": is_correct}

​

Example: Concision evaluator

def perform_eval(run, example):
    """Score how concise the answer is. 1 is most concise, 5 is least concise."""
    answer = run['outputs']['answer']
    score = min(len(answer) // 1000, 4) + 1

    return {"concision_score": score}

​

Example: Input-based evaluator

def perform_eval(run, example):
    """Check if the input text contains toxic language."""
    text = run['inputs'].get('text', '').lower()
    toxic_words = ["idiot", "stupid", "hate", "awful"]

    is_toxic = any(word in text for word in toxic_words)
    return {"is_toxic": is_toxic}

​

Step 3. Configure the evaluator

​

Name and description

​

Feedback configuration

• Boolean: True/false feedback
• Categorical: String values representing categories
• Continuous: Numerical scoring within a range

​

Step 4. Test and save

1. Preview your evaluator on example data to ensure it works as expected
2. Click Save to make the evaluator available for use

​

Use your code evaluator

• When running evaluations from the playground
• As part of a dataset to automatically run evaluations on experiments
• When running online evaluations

​

Related

• LLM-as-a-judge evaluator (UI): Use an LLM to evaluate outputs
• Composite evaluators: Combine multiple evaluator scores