Test config

Overview

Test suites are collections of related test cases that validate specific aspects of your MCP server. Each suite defines:

Test type (conversational, compliance, security)
Execution settings (parallelism, timeouts)
Test cases with specific goals and success criteria

Suite Types

Conversational Test Suites

Test realistic user workflows through multi-turn conversations.

{
  "suite_id": "user_workflows",
  "name": "User Workflow Tests",
  "description": "Test common user interaction patterns",
  "suite_type": "conversational",
  "created_at": "2024-01-01T00:00:00Z",
  "test_cases": [
    {
      "test_id": "greeting_flow",
      "user_message": "Hello! What can you help me with?",
      "success_criteria": "Agent greets user politely and explains available capabilities",
      "max_turns": 5,
      "context_persistence": true,
      "metadata": {
        "category": "basic_interaction",
        "priority": "high"
      }
    }
  ],
  "user_patience_level": "medium",
  "conversation_style": "natural"
}

Use cases:

New user onboarding flows
Feature discovery and usage
Complex multi-step tasks
Error handling and recovery

Compliance Test Suites

Validate MCP protocol conformance and technical requirements.

{
  "suite_id": "mcp_compliance",
  "name": "MCP Protocol Compliance",
  "description": "Validate MCP protocol implementation",
  "suite_type": "compliance",
  "created_at": "2024-01-01T00:00:00Z",
  "test_cases": [
    {
      "test_id": "handshake_validation",
      "protocol_version": "2025-06-18",
      "required_capabilities": [],
      "check_categories": ["handshake"]
    }
  ],
  "strict_mode": true
}

Use cases:

Protocol compliance validation
Capability negotiation testing
Standard conformance verification

Security Test Suites

Test authentication, authorization, and vulnerability resistance.

{
  "suite_id": "security_tests",
  "name": "Security Validation",
  "description": "Test authentication and security measures",
  "suite_type": "security",
  "created_at": "2024-01-01T00:00:00Z",
  "auth_required": true,
  "test_cases": [
    {
      "test_id": "auth_validation",
      "auth_method": "bearer_token",
      "rate_limit_threshold": 100,
      "vulnerability_checks": ["injection", "auth", "rate_limit"],
      "severity_threshold": "medium"
    }
  ],
  "include_penetration_tests": false
}

Use cases:

Authentication mechanism validation
Input sanitization testing
Rate limiting verification
Vulnerability scanning

Test Suite Structure

Core Fields

Every test suite includes these fields:

{
  "suite_id": "unique_identifier",
  "name": "Human-readable name",
  "description": "What this suite tests",
  "suite_type": "conversational|compliance|security",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 5,
  "test_cases": [...],
  "auth_required": false
}

Field descriptions:

suite_id - Required unique identifier for referencing the suite
name - Required descriptive name shown in reports
description - Optional suite description (default: null)
suite_type - Optional but recommended for proper execution behavior
created_at - Creation timestamp (default: current UTC time)
parallelism - Number of concurrent test executions (default: 5)
auth_required - Whether server authentication is needed (default: false, but true for security suites)

Test Case Structures

Each suite type has specific test case structures:

Conversational Test Cases

{
  "test_id": "unique_test_identifier",
  "user_message": "What the AI agent will say to the server",
  "success_criteria": "Natural language description of success",
  "max_turns": 10,
  "context_persistence": true,
  "metadata": {
    "category": "basic_interaction",
    "priority": "high"
  }
}

Required fields:

test_id - Unique identifier for the test
user_message - Initial message from user to start conversation
success_criteria - Natural language description of what constitutes success

Optional fields:

max_turns - Maximum conversation turns (default: 10)
context_persistence - Whether to maintain context between turns (default: true)
metadata - Additional test categorization and priority information

Compliance Test Cases

{
  "test_id": "protocol_check",
  "protocol_version": "2025-06-18",
  "required_capabilities": [],
  "check_categories": ["handshake", "capabilities", "tools", "resources"]
}

Required fields:

test_id - Unique identifier for the test

Optional fields:

protocol_version - MCP protocol version to test (default: “2025-06-18”)
required_capabilities - Capabilities that must be supported (default: [])
check_categories - Protocol aspects to validate (default: [“handshake”, “capabilities”, “tools”, “resources”])

Security Test Cases

{
  "test_id": "security_check",
  "auth_method": "bearer_token",
  "rate_limit_threshold": 100,
  "vulnerability_checks": ["injection", "auth", "rate_limit"],
  "severity_threshold": "medium"
}

Required fields:

test_id - Unique identifier for the test
auth_method - Authentication method to test (e.g., “bearer_token”, “oauth”)

Optional fields:

rate_limit_threshold - Requests per minute before rate limiting (default: 100)
vulnerability_checks - Security vulnerabilities to test (default: [“injection”, “auth”, “rate_limit”])
severity_threshold - Minimum severity level to report (default: “medium”)

Suite Configuration Options

Execution Settings

Control how tests are executed:

{
  "parallelism": 3
}

Available options:

parallelism - Concurrent test execution (default varies by suite type: compliance=2-3, security=2, conversational=1-3)

User Simulation (Conversational Suites)

Configure how AI agents behave as users:

{
  "user_patience_level": "low|medium|high",
  "conversation_style": "natural"
}

Simulation options:

user_patience_level - How quickly users get frustrated (default: “medium”)
conversation_style - Communication style (default: “natural”)

Type-Specific Options

{
  "strict_mode": true,
  "include_penetration_tests": false
}

Compliance suites:

strict_mode - Strict protocol compliance checking (default: true)

Security suites:

include_penetration_tests - Include penetration testing (default: false)

Complete Suite Examples

Basic User Interaction Suite

{
  "suite_id": "basic_interactions",
  "name": "Basic User Interactions",
  "description": "Test fundamental user workflows and interactions",
  "suite_type": "conversational",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 3,
  "test_cases": [
    {
      "test_id": "welcome_flow",
      "user_message": "Hi there! I'm new here. What can you help me with?",
      "success_criteria": "Agent provides friendly welcome and clear overview of capabilities",
      "max_turns": 5,
      "context_persistence": true,
      "metadata": {
        "category": "onboarding",
        "priority": "high"
      }
    }
  ],
  "user_patience_level": "medium",
  "conversation_style": "natural"
}

Comprehensive Compliance Suite

{
  "suite_id": "mcp_protocol_validation",
  "name": "MCP Protocol Compliance Validation",
  "description": "Comprehensive validation of MCP protocol implementation",
  "suite_type": "compliance",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 2,
  "test_cases": [
    {
      "test_id": "protocol_handshake",
      "protocol_version": "2025-06-18",
      "check_categories": ["handshake"]
    },
    {
      "test_id": "capability_negotiation",
      "protocol_version": "2025-06-18",
      "required_capabilities": ["tools", "resources"],
      "check_categories": ["capabilities"]
    }
  ],
  "strict_mode": true
}

Security Testing Suite

{
  "suite_id": "security_validation",
  "name": "Security and Authentication Tests",
  "description": "Validate security measures and authentication mechanisms",
  "suite_type": "security",
  "created_at": "2024-01-01T00:00:00Z",
  "auth_required": true,
  "parallelism": 2,
  "test_cases": [
    {
      "test_id": "bearer_token_auth",
      "auth_method": "bearer_token",
      "vulnerability_checks": ["auth"],
      "severity_threshold": "medium"
    }
  ],
  "include_penetration_tests": false
}

Suite Management

Creating Test Suites

Interactive creation:

mcp-t create suite

Template-based creation with built-in templates:

mcp-t create suite  # Interactive menu with template selection

Direct type creation:

mcp-t create conversational
mcp-t create conversational --id my-suite --global

Managing Test Cases

Add test cases to existing suites:

mcp-t create test-case --suite-id my-suite

Configuration Management

List configurations:

mcp-t list              # List all configurations
mcp-t list servers      # List only servers
mcp-t list suites       # List only test suites

Show specific configurations:

mcp-t show server SERVER_ID   # Show server details
mcp-t show suite SUITE_ID     # Show suite details

Running Test Suites

Basic Execution

mcp-t run SUITE_ID SERVER_ID           # Basic test execution
mcp-t run SUITE_ID SERVER_ID --verbose # Detailed output
mcp-t run SUITE_ID SERVER_ID --global  # Save results globally

Execution Behavior

Tests run sequentially by default (parallelism=1 in execution)
Suite parallelism setting controls internal test orchestration
Results are saved to test_results/runs/ directory
Each test includes conversation transcript and LLM judge evaluation

Built-in Templates

The framework includes four built-in templates with environment variable substitution:

Basic Server Template

{
  "url": "${MCP_SERVER_URL}",
  "name": "${MCP_SERVER_NAME:-My MCP Server}",
  "authorization_token": "${MCP_AUTH_TOKEN:-}"
}

Suite Templates

Compliance Template: 3 test cases covering handshake, capabilities, and tools
Security Template: 3 test cases covering auth validation, rate limiting, and injection testing
Conversational Template: 2 test cases covering multi-turn conversations and error recovery

Templates support environment variable substitution with default values using ${VAR:-default} syntax.

Next Steps

Now that you understand test suites:

Create your first suite with mcp-t create suite
Run the quickstart with mcp-t quickstart for complete setup
Explore test results in the test_results/ directory
Add servers with mcp-t create server
Generate test suites automatically with mcp-t generate

Overview

Guides

Reference

Support

Overview

Suite Types

Conversational Test Suites

Compliance Test Suites

Security Test Suites

Test Suite Structure

Core Fields

Test Case Structures

Conversational Test Cases

Compliance Test Cases

Security Test Cases

Suite Configuration Options

Execution Settings

User Simulation (Conversational Suites)

Type-Specific Options

Complete Suite Examples

Basic User Interaction Suite

Comprehensive Compliance Suite

Security Testing Suite

Suite Management

Creating Test Suites

Managing Test Cases

Configuration Management

Running Test Suites

Basic Execution

Execution Behavior

Built-in Templates

Basic Server Template

Suite Templates

Next Steps

​Overview

​Suite Types

​Conversational Test Suites

​Compliance Test Suites

​Security Test Suites

​Test Suite Structure

​Core Fields

​Test Case Structures

​Conversational Test Cases

​Compliance Test Cases

​Security Test Cases

​Suite Configuration Options

​Execution Settings

​User Simulation (Conversational Suites)

​Type-Specific Options

​Complete Suite Examples

​Basic User Interaction Suite

​Comprehensive Compliance Suite

​Security Testing Suite

​Suite Management

​Creating Test Suites

​Managing Test Cases

​Configuration Management

​Running Test Suites

​Basic Execution

​Execution Behavior

​Built-in Templates

​Basic Server Template

​Suite Templates

​Next Steps

Overview

Suite Types

Conversational Test Suites

Compliance Test Suites

Security Test Suites

Test Suite Structure

Core Fields

Test Case Structures

Conversational Test Cases

Compliance Test Cases

Security Test Cases

Suite Configuration Options

Execution Settings

User Simulation (Conversational Suites)

Type-Specific Options

Complete Suite Examples

Basic User Interaction Suite

Comprehensive Compliance Suite

Security Testing Suite

Suite Management

Creating Test Suites

Managing Test Cases

Configuration Management

Running Test Suites

Basic Execution

Execution Behavior

Built-in Templates

Basic Server Template

Suite Templates

Next Steps