Evals

Measuring correctness with chatlas + Inspect AI

What are evals and why are they important?

• Generic term for measuring correctness and quality LLM responses.

• Many ways to measure, but “LLM as a grader” is the most common and scalable.

• Helpful for regression testing: you change your system (model, prompt, tools) and want to know things still work.

• Also useful for benchmarking: you want to know which model or prompt is best for your use case.

• The hard part: how do you get a representative dataset of inputs and expected outputs?
  • Next section we’ll learn about collecting user activity (OTel).

What’s in an eval?

• One popular eval framework is Inspect AI (chatlas integrates with it). There are three main components:

1. Dataset – test cases: realistic inputs + target responses.

2. Solver – your chat, turned into something that generates a response for each input.
   • chatlas makes this easy with .to_solver().

3. Scorer – grades the response against the target.
   • Main drivers are model_graded_fact() (fact-checking) and model_graded_qa() (general purpose QA).

Define a task

These three components are combined into a Task:

my_eval.py

from chatlas import ChatBedrockAnthropic
from inspect_ai import Task, task
from inspect_ai.dataset import csv_dataset
from inspect_ai.scorer import model_graded_qa

chat = ChatBedrockAnthropic()

@task
def my_eval():
    return Task(
        dataset=csv_dataset("my_eval_dataset.csv"),
        solver=chat.to_solver(),
        scorer=model_graded_qa(),
    )

Tasks can be run over a grid of models, parameters, etc.

A basic CSV dataset

my_eval_dataset.csv

input,           target
What is 2 + 2?,  4
What is 10 * 5?, 50

• input is what the solver sends.
• target is what the scorer checks it against.

Run and view

inspect eval my_eval.py
inspect view

• inspect view opens a browser UI: every sample, its score, and why the grader scored it that way.
• There’s also a VS Code extension if you’d rather stay in the editor.

Multi-turn evals

• If an eval involves multiple turns, a CSV isn’t enough to capture the conversation history.

• Instead, you can build the dataset from an actual chat with .export_eval().

chat.chat("My first name is Alice.")
chat.chat("My last name is Smith.")
chat.chat("What is my full name?")

chat.export_eval(
    "my_eval_dataset.jsonl",
    target="""
Response should include 'Alice Smith'.
""",
)

Each call appends one Sample:

• Prior turns become the input.
• The target is whatever grading guidance you give it.

Structured output evals

Recall the PaperResults extractor from the structured-data section. Let’s eval it properly:

chat = ctl.ChatBedrockAnthropic()

abstract = """ResNet-50 achieves 93.2% top-1 accuracy on CIFAR-10 and
76.1% on ImageNet. A fine-tuned ViT-B/16 reaches 98.1% on CIFAR-10."""

chat.chat_structured(abstract, data_model=PaperResults)

chat.export_eval(
    "papers_eval.jsonl",
    target="Should extract ResNet-50 (93.2% CIFAR-10, 76.1% ImageNet) "
           "and ViT-B/16 (98.1% CIFAR-10).",
)

• Repeat for a few more abstracts (ideally ones pulled from real papers :)

Scoring structured output

Pass data_model to .to_solver() and it calls .chat_structured() instead of .chat() – same trick, structured version:

from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_fact

@task
def papers_eval():
    return Task(
        dataset=json_dataset("papers_eval.jsonl"),
        solver=chat.to_solver(data_model=PaperResults),
        scorer=model_graded_fact(),
    )

• model_graded_fact() compares against a correct answer and ignores style/wording – a better fit here than model_graded_qa().

Keep in mind

• .export_eval() captures system prompt and conversation history by default
  • For this reason, .to_solver() doesn’t include them, but you should opt-in for hand-written datasets (like a CSV) that don’t carry system prompt / history themselves.

• The grader model (scoring) doesn’t need to be the same as the solver model (generating).
  • Good practice: use a stronger or different model to grade, which avoids a model marking its own homework.

Partial credit

• Most scorers support partial_credit=True – scores of 0, 0.5, or 1 instead of pass/fail.

• Useful when “mostly right” deserves some credit: a literature-mining extraction that gets 2 of 3 results correct, or code that runs but has a minor bug.

Your dataset is a sample – sample carefully

• The biggest threat to a useful eval isn’t the scorer, it’s an unrepresentative dataset.

• Use real inputs: actual questions, real abstracts, genuine edge cases – not cleaned-up examples written for a demo.

• Don’t have real users? Why not use AI to generate input for AI to then solve and score? 😜

Exercise

Starter: exercises/eval-starter.py – a PaperResults eval, already wired up with a Task, solver, and scorer, with 4 samples ready to go.

1. python exercises/eval-starter.py builds the dataset.
2. inspect eval exercises/eval-starter.py, then inspect view – see how each sample scored.
   • Tip: install “Inspect AI” VS Code extension for a better experience
3. Break it on purpose: edit one target to be wrong or vague, rerun steps 1-2, and watch the score drop.
4. Swap model_graded_fact for model_graded_qa – does grading get stricter or looser?