promptfoo Configuration

Skill for configuring promptfoo prompt evaluation and testing.

Overview

promptfoo enables:

• Prompt regression testing

• Multi-provider comparison

• Assertion-based validation

• HTML report generation

Use When

This skill is automatically invoked when:

• Setting up prompt testing

• Comparing LLM providers

• Creating evaluation pipelines

• Defining test assertions

Available Scripts

Script Description

scripts/init-promptfoo.sh

Initialize promptfoo project

scripts/run-eval.sh

Run evaluation with output

Available Templates

Template Description

templates/promptfooconfig.yaml

Basic configuration

templates/multi-provider.yaml

Provider comparison

templates/assertions.yaml

Assertion examples

Best Practices

• Use YAML anchors for repeated config

• Store prompts in separate files

• Use variables for dynamic tests

• Set appropriate thresholds