This guide explains how to create evaluation tests (.eval.ts files) for testing AI model interactions with specific tools or systems, such as Cloudflare Worker bindings or container environments.

Evals are automated tests designed to verify if an AI model correctly understands instructions and utilizes its available "tools" (functions, API calls, environment interactions) to achieve a desired outcome. They assess the model's ability to follow instructions, select appropriate tools, and provide correct arguments to those tools.

Evals are typically built using a testing framework like vitest combined with specialized evaluation libraries like vitest-evals. The main structure revolves around describeEval:

import { expect } from 'vitest'
import { describeEval } from 'vitest-evals'

import { checkFactuality } from '@repo/eval-tools/src/scorers'
import { eachModel } from '@repo/eval-tools/src/test-models'

import { initializeClient, runTask } from './utils' // Helper functions

eachModel('$modelName', ({ model }) => {
	// Optional: Run tests for multiple models
	describeEval('A descriptive name for the evaluation suite', {
		data: async () => [
			/* Test cases */
		],
		task: async (input) => {
			/* Test logic */
		},
		scorers: [
			/* Scoring functions */
		],
		threshold: 1, // Passing score threshold
		timeout: 60000, // Test timeout
	})
})

1. describeEval(name, options): Defines a suite of evaluation tests.

   • name: A string describing the purpose of the eval suite.
   • options: An object containing the configuration for the eval:
     • data: An async function returning an array of test case objects. Each object typically contains:
       • input: (string) The instruction given to the AI model.
       • expected: (string) A natural language description of the expected sequence of actions or outcome. This is used by scorers.
     • task: An async function that executes the actual test logic for a given input. It orchestrates the interaction with the AI/system and performs assertions.
     • scorers: An array of scoring functions (e.g., checkFactuality) that evaluate the test outcome based on the promptOutput from the task and the expected string from the data.
     • threshold: (number, usually between 0 and 1) The minimum score required from the scorers for the test case to pass. A threshold of 1 means a perfect score is required.
     • timeout: (number) Maximum time in milliseconds allowed for a single test case.

2. task(input) Function: The heart of the eval. It typically involves:

   • Setup: Initializing a client or test environment (initializeClient). This prepares the system for the test, configuring available tools or connections.
   • Execution: Running the actual interaction (runTask). This function sends the input instruction to the AI model via the client and captures the results, which usually include:
     • promptOutput: The textual response from the AI model.
     • toolCalls: A structured list of the tools the AI invoked, along with the arguments passed to each tool.
   • Assertions (expect): Using the testing framework's assertion library (vitest's expect in the examples) to verify that the correct tools were called with the correct arguments based on the toolCalls data. Sometimes, this involves direct interaction with the system state (e.g., reading a file created by a tool) to confirm the outcome.
   • Return Value: The task function usually returns the promptOutput to be evaluated by the scorers.

3. Scoring (checkFactuality, etc.): Automated functions that compare the actual outcome (represented by the promptOutput and implicitly by the assertions passed within the task) against the expected description.

4. Helper Utilities (./utils):

   • initializeClient(): Sets up the testing environment, connects to the system under test, and configures the available tools for the AI model.
   • runTask(client, model, input): Sends the input prompt to the specified AI model using the configured client, executes the model's reasoning and tool use, and returns the results (promptOutput, toolCalls).
   • eachModel(): (Optional) A utility to run the same evaluation suite against multiple different AI models.

1. Identify Tools: Define the specific actions or functions (the "tools") that the AI should be able to use within the system you're testing (e.g., kv_write, d1_query, container_exec).
2. Create Helper Functions: Implement your initializeClient and runTask (or similarly named) functions.
   • initializeClient: Should set up the necessary context, potentially using test environments like vitest-environment-miniflare for workers. It needs to make the defined tools available to the AI model simulation.
   • runTask: Needs to simulate the AI processing: take an input prompt, interact with an LLM (or a mock) configured with the tools, capture which tools are called and with what arguments, and capture the final text output.
3. Create Eval File (*.eval.ts): Create a new file (e.g., kv-operations.eval.ts).
4. Import Dependencies: Import describeEval, scorers, helpers, expect, etc.
5. Structure with describeEval: Define your evaluation suite.
6. Define Test Cases (data): Write specific test scenarios:
   • Provide clear, unambiguous input prompts that target the tools you want to test.
   • Write concise expected descriptions detailing the primary tool calls or outcomes anticipated.
7. Implement the task Function:
   • Call initializeClient.
   • Call runTask with the input.
   • Write expect assertions to rigorously check:
     • Were the correct tools called? (toolName)
     • Were they called in the expected order (if applicable)?
     • Were the arguments passed to the tools correct? (args)
     • (Optional) Interact with the system state if necessary to verify side effects.
   • Return the promptOutput.
8. Configure Scorers and Threshold: Choose appropriate scorers (often checkFactuality) and set a threshold.
9. Run Tests: Execute the evals using your test runner (e.g., vitest run).

// my-feature.eval.ts
import { expect } from 'vitest'
import { describeEval } from 'vitest-evals'

import { checkFactuality } from '@repo/eval-tools/src/scorers'

import { initializeClient, runTask } from './utils'

describeEval('Tests My Feature Tool Interactions', {
	data: async () => [
		{
			input: 'Use my_tool to process the data "example"',
			expected: 'The my_tool tool was called with data set to "example"',
		},
		// ... more test cases
	],
	task: async (input) => {
		const client = await initializeClient() // Sets up environment with my_tool
		const { promptOutput, toolCalls } = await runTask(client, 'your-model', input)

		// Check if my_tool was called
		const myToolCall = toolCalls.find((call) => call.toolName === 'my_tool')
		expect(myToolCall).toBeDefined()

		// Check arguments passed to my_tool
		expect(myToolCall?.args).toEqual(
			expect.objectContaining({
				data: 'example',
				// ... other expected args
			})
		)

		return promptOutput // Return AI output for scoring
	},
	scorers: [checkFactuality],
	threshold: 1,
})

• Clear Inputs: Write inputs as clear, actionable instructions.
• Specific Expected Outcomes: Make expected descriptions precise enough for scorers but focus on the key actions.
• Targeted Assertions: Use expect to verify the most critical aspects of tool calls (tool name, key arguments). Don't over-assert on trivial details unless necessary.
• Isolate Tests: Ensure each test case in data tests a specific interaction or a small sequence of interactions.
• Helper Functions: Keep initializeClient and runTask generic enough to be reused across different eval files for the same system.
• Use expect.objectContaining or expect.stringContaining: Often, you only need to verify parts of the arguments, not the entire structure, making tests less brittle.
• Descriptive Names: Use clear names for describeEval blocks and meaningful input/expected strings.