> ## Documentation Index
> Fetch the complete documentation index at: https://docs.golf.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Test config

> How to structure comprehensive test coverage with different test suite types and configurations

## Overview

Test suites are collections of related test cases that validate specific aspects of your MCP server. Each suite defines:

* Test type (conversational, compliance, security)
* Execution settings (parallelism, timeouts)
* Test cases with specific goals and success criteria

## Suite Types

### Conversational Test Suites

Test realistic user workflows through multi-turn conversations.

```json theme={null}
{
  "suite_id": "user_workflows",
  "name": "User Workflow Tests",
  "description": "Test common user interaction patterns",
  "suite_type": "conversational",
  "created_at": "2024-01-01T00:00:00Z",
  "test_cases": [
    {
      "test_id": "greeting_flow",
      "user_message": "Hello! What can you help me with?",
      "success_criteria": "Agent greets user politely and explains available capabilities",
      "max_turns": 5,
      "context_persistence": true,
      "metadata": {
        "category": "basic_interaction",
        "priority": "high"
      }
    }
  ],
  "user_patience_level": "medium",
  "conversation_style": "natural"
}
```

**Use cases:**

* New user onboarding flows
* Feature discovery and usage
* Complex multi-step tasks
* Error handling and recovery

### Compliance Test Suites

Validate MCP protocol conformance and technical requirements.

```json theme={null}
{
  "suite_id": "mcp_compliance",
  "name": "MCP Protocol Compliance",
  "description": "Validate MCP protocol implementation",
  "suite_type": "compliance",
  "created_at": "2024-01-01T00:00:00Z",
  "test_cases": [
    {
      "test_id": "handshake_validation",
      "protocol_version": "2025-06-18",
      "required_capabilities": [],
      "check_categories": ["handshake"]
    }
  ],
  "strict_mode": true
}
```

**Use cases:**

* Protocol compliance validation
* Capability negotiation testing
* Standard conformance verification

### Security Test Suites

Test authentication, authorization, and vulnerability resistance.

```json theme={null}
{
  "suite_id": "security_tests",
  "name": "Security Validation",
  "description": "Test authentication and security measures",
  "suite_type": "security",
  "created_at": "2024-01-01T00:00:00Z",
  "auth_required": true,
  "test_cases": [
    {
      "test_id": "auth_validation",
      "auth_method": "bearer_token",
      "rate_limit_threshold": 100,
      "vulnerability_checks": ["injection", "auth", "rate_limit"],
      "severity_threshold": "medium"
    }
  ],
  "include_penetration_tests": false
}
```

**Use cases:**

* Authentication mechanism validation
* Input sanitization testing
* Rate limiting verification
* Vulnerability scanning

## Test Suite Structure

### Core Fields

Every test suite includes these fields:

```json theme={null}
{
  "suite_id": "unique_identifier",
  "name": "Human-readable name",
  "description": "What this suite tests",
  "suite_type": "conversational|compliance|security",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 5,
  "test_cases": [...],
  "auth_required": false
}
```

**Field descriptions:**

* **`suite_id`** - Required unique identifier for referencing the suite
* **`name`** - Required descriptive name shown in reports
* **`description`** - Optional suite description (default: null)
* **`suite_type`** - Optional but recommended for proper execution behavior
* **`created_at`** - Creation timestamp (default: current UTC time)
* **`parallelism`** - Number of concurrent test executions (default: 5)
* **`auth_required`** - Whether server authentication is needed (default: false, but true for security suites)

### Test Case Structures

Each suite type has specific test case structures:

#### Conversational Test Cases

```json theme={null}
{
  "test_id": "unique_test_identifier",
  "user_message": "What the AI agent will say to the server",
  "success_criteria": "Natural language description of success",
  "max_turns": 10,
  "context_persistence": true,
  "metadata": {
    "category": "basic_interaction",
    "priority": "high"
  }
}
```

**Required fields:**

* `test_id` - Unique identifier for the test
* `user_message` - Initial message from user to start conversation
* `success_criteria` - Natural language description of what constitutes success

**Optional fields:**

* `max_turns` - Maximum conversation turns (default: 10)
* `context_persistence` - Whether to maintain context between turns (default: true)
* `metadata` - Additional test categorization and priority information

#### Compliance Test Cases

```json theme={null}
{
  "test_id": "protocol_check",
  "protocol_version": "2025-06-18",
  "required_capabilities": [],
  "check_categories": ["handshake", "capabilities", "tools", "resources"]
}
```

**Required fields:**

* `test_id` - Unique identifier for the test

**Optional fields:**

* `protocol_version` - MCP protocol version to test (default: "2025-06-18")
* `required_capabilities` - Capabilities that must be supported (default: \[])
* `check_categories` - Protocol aspects to validate (default: \["handshake", "capabilities", "tools", "resources"])

#### Security Test Cases

```json theme={null}
{
  "test_id": "security_check",
  "auth_method": "bearer_token",
  "rate_limit_threshold": 100,
  "vulnerability_checks": ["injection", "auth", "rate_limit"],
  "severity_threshold": "medium"
}
```

**Required fields:**

* `test_id` - Unique identifier for the test
* `auth_method` - Authentication method to test (e.g., "bearer\_token", "oauth")

**Optional fields:**

* `rate_limit_threshold` - Requests per minute before rate limiting (default: 100)
* `vulnerability_checks` - Security vulnerabilities to test (default: \["injection", "auth", "rate\_limit"])
* `severity_threshold` - Minimum severity level to report (default: "medium")

## Suite Configuration Options

### Execution Settings

Control how tests are executed:

```json theme={null}
{
  "parallelism": 3
}
```

**Available options:**

* **`parallelism`** - Concurrent test execution (default varies by suite type: compliance=2-3, security=2, conversational=1-3)

### User Simulation (Conversational Suites)

Configure how AI agents behave as users:

```json theme={null}
{
  "user_patience_level": "low|medium|high",
  "conversation_style": "natural"
}
```

**Simulation options:**

* **`user_patience_level`** - How quickly users get frustrated (default: "medium")
* **`conversation_style`** - Communication style (default: "natural")

### Type-Specific Options

```json theme={null}
{
  "strict_mode": true,
  "include_penetration_tests": false
}
```

**Compliance suites:**

* **`strict_mode`** - Strict protocol compliance checking (default: true)

**Security suites:**

* **`include_penetration_tests`** - Include penetration testing (default: false)

## Complete Suite Examples

### Basic User Interaction Suite

```json theme={null}
{
  "suite_id": "basic_interactions",
  "name": "Basic User Interactions",
  "description": "Test fundamental user workflows and interactions",
  "suite_type": "conversational",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 3,
  "test_cases": [
    {
      "test_id": "welcome_flow",
      "user_message": "Hi there! I'm new here. What can you help me with?",
      "success_criteria": "Agent provides friendly welcome and clear overview of capabilities",
      "max_turns": 5,
      "context_persistence": true,
      "metadata": {
        "category": "onboarding",
        "priority": "high"
      }
    }
  ],
  "user_patience_level": "medium",
  "conversation_style": "natural"
}
```

### Comprehensive Compliance Suite

```json theme={null}
{
  "suite_id": "mcp_protocol_validation",
  "name": "MCP Protocol Compliance Validation",
  "description": "Comprehensive validation of MCP protocol implementation",
  "suite_type": "compliance",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 2,
  "test_cases": [
    {
      "test_id": "protocol_handshake",
      "protocol_version": "2025-06-18",
      "check_categories": ["handshake"]
    },
    {
      "test_id": "capability_negotiation",
      "protocol_version": "2025-06-18",
      "required_capabilities": ["tools", "resources"],
      "check_categories": ["capabilities"]
    }
  ],
  "strict_mode": true
}
```

### Security Testing Suite

```json theme={null}
{
  "suite_id": "security_validation",
  "name": "Security and Authentication Tests",
  "description": "Validate security measures and authentication mechanisms",
  "suite_type": "security",
  "created_at": "2024-01-01T00:00:00Z",
  "auth_required": true,
  "parallelism": 2,
  "test_cases": [
    {
      "test_id": "bearer_token_auth",
      "auth_method": "bearer_token",
      "vulnerability_checks": ["auth"],
      "severity_threshold": "medium"
    }
  ],
  "include_penetration_tests": false
}
```

## Suite Management

### Creating Test Suites

**Interactive creation:**

```bash theme={null}
mcp-t create suite
```

**Template-based creation with built-in templates:**

```bash theme={null}
mcp-t create suite  # Interactive menu with template selection
```

**Direct type creation:**

```bash theme={null}
mcp-t create conversational
mcp-t create conversational --id my-suite --global
```

### Managing Test Cases

**Add test cases to existing suites:**

```bash theme={null}
mcp-t create test-case --suite-id my-suite
```

### Configuration Management

**List configurations:**

```bash theme={null}
mcp-t list              # List all configurations
mcp-t list servers      # List only servers
mcp-t list suites       # List only test suites
```

**Show specific configurations:**

```bash theme={null}
mcp-t show server SERVER_ID   # Show server details
mcp-t show suite SUITE_ID     # Show suite details
```

## Running Test Suites

### Basic Execution

```bash theme={null}
mcp-t run SUITE_ID SERVER_ID           # Basic test execution
mcp-t run SUITE_ID SERVER_ID --verbose # Detailed output
mcp-t run SUITE_ID SERVER_ID --global  # Save results globally
```

### Execution Behavior

* Tests run sequentially by default (parallelism=1 in execution)
* Suite parallelism setting controls internal test orchestration
* Results are saved to test\_results/runs/ directory
* Each test includes conversation transcript and LLM judge evaluation

## Built-in Templates

The framework includes four built-in templates with environment variable substitution:

### Basic Server Template

```json theme={null}
{
  "url": "${MCP_SERVER_URL}",
  "name": "${MCP_SERVER_NAME:-My MCP Server}",
  "authorization_token": "${MCP_AUTH_TOKEN:-}"
}
```

### Suite Templates

* **Compliance Template:** 3 test cases covering handshake, capabilities, and tools
* **Security Template:** 3 test cases covering auth validation, rate limiting, and injection testing
* **Conversational Template:** 2 test cases covering multi-turn conversations and error recovery

Templates support environment variable substitution with default values using `${VAR:-default}` syntax.

## Next Steps

Now that you understand test suites:

1. Create your first suite with `mcp-t create suite`
2. Run the quickstart with `mcp-t quickstart` for complete setup
3. Explore test results in the `test_results/` directory
4. Add servers with `mcp-t create server`
5. Generate test suites automatically with `mcp-t generate`