Skip to main content

​
Overview

Test suites are collections of related test cases that validate specific aspects of your MCP server. Each suite defines:
  • Test type (conversational, compliance, security)
  • Execution settings (parallelism, timeouts)
  • Test cases with specific goals and success criteria

​
Suite Types

​
Conversational Test Suites

Test realistic user workflows through multi-turn conversations. 
{
  "suite_id": "user_workflows",
  "name": "User Workflow Tests",
  "description": "Test common user interaction patterns",
  "suite_type": "conversational",
  "created_at": "2024-01-01T00:00:00Z",
  "test_cases": [
    {
      "test_id": "greeting_flow",
      "user_message": "Hello! What can you help me with?",
      "success_criteria": "Agent greets user politely and explains available capabilities",
      "max_turns": 5,
      "context_persistence": true,
      "metadata": {
        "category": "basic_interaction",
        "priority": "high"
      }
    }
  ],
  "user_patience_level": "medium",
  "conversation_style": "natural"
}
Use cases:
  • New user onboarding flows
  • Feature discovery and usage
  • Complex multi-step tasks
  • Error handling and recovery

​
Compliance Test Suites

Validate MCP protocol conformance and technical requirements. 
{
  "suite_id": "mcp_compliance",
  "name": "MCP Protocol Compliance",
  "description": "Validate MCP protocol implementation",
  "suite_type": "compliance",
  "created_at": "2024-01-01T00:00:00Z",
  "test_cases": [
    {
      "test_id": "handshake_validation",
      "protocol_version": "2025-06-18",
      "required_capabilities": [],
      "check_categories": ["handshake"]
    }
  ],
  "strict_mode": true
}
Use cases:
  • Protocol compliance validation
  • Capability negotiation testing
  • Standard conformance verification

​
Security Test Suites

Test authentication, authorization, and vulnerability resistance. 
{
  "suite_id": "security_tests",
  "name": "Security Validation",
  "description": "Test authentication and security measures",
  "suite_type": "security",
  "created_at": "2024-01-01T00:00:00Z",
  "auth_required": true,
  "test_cases": [
    {
      "test_id": "auth_validation",
      "auth_method": "bearer_token",
      "rate_limit_threshold": 100,
      "vulnerability_checks": ["injection", "auth", "rate_limit"],
      "severity_threshold": "medium"
    }
  ],
  "include_penetration_tests": false
}
Use cases:
  • Authentication mechanism validation
  • Input sanitization testing
  • Rate limiting verification
  • Vulnerability scanning

​
Test Suite Structure

​
Core Fields

Every test suite includes these fields: 
{
  "suite_id": "unique_identifier",
  "name": "Human-readable name",
  "description": "What this suite tests",
  "suite_type": "conversational|compliance|security",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 5,
  "test_cases": [...],
  "auth_required": false
}
Field descriptions:
  • suite_id - Required unique identifier for referencing the suite
  • name - Required descriptive name shown in reports
  • description - Optional suite description (default: null)
  • suite_type - Optional but recommended for proper execution behavior
  • created_at - Creation timestamp (default: current UTC time)
  • parallelism - Number of concurrent test executions (default: 5)
  • auth_required - Whether server authentication is needed (default: false, but true for security suites)

​
Test Case Structures

Each suite type has specific test case structures:

​
Conversational Test Cases

{
  "test_id": "unique_test_identifier",
  "user_message": "What the AI agent will say to the server",
  "success_criteria": "Natural language description of success",
  "max_turns": 10,
  "context_persistence": true,
  "metadata": {
    "category": "basic_interaction",
    "priority": "high"
  }
}
Required fields:
  • test_id - Unique identifier for the test
  • user_message - Initial message from user to start conversation
  • success_criteria - Natural language description of what constitutes success
Optional fields:
  • max_turns - Maximum conversation turns (default: 10)
  • context_persistence - Whether to maintain context between turns (default: true)
  • metadata - Additional test categorization and priority information

​
Compliance Test Cases

{
  "test_id": "protocol_check",
  "protocol_version": "2025-06-18",
  "required_capabilities": [],
  "check_categories": ["handshake", "capabilities", "tools", "resources"]
}
Required fields:
  • test_id - Unique identifier for the test
Optional fields:
  • protocol_version - MCP protocol version to test (default: “2025-06-18”)
  • required_capabilities - Capabilities that must be supported (default: [])
  • check_categories - Protocol aspects to validate (default: [“handshake”, “capabilities”, “tools”, “resources”])

​
Security Test Cases

{
  "test_id": "security_check",
  "auth_method": "bearer_token",
  "rate_limit_threshold": 100,
  "vulnerability_checks": ["injection", "auth", "rate_limit"],
  "severity_threshold": "medium"
}
Required fields:
  • test_id - Unique identifier for the test
  • auth_method - Authentication method to test (e.g., “bearer_token”, “oauth”)
Optional fields:
  • rate_limit_threshold - Requests per minute before rate limiting (default: 100)
  • vulnerability_checks - Security vulnerabilities to test (default: [“injection”, “auth”, “rate_limit”])
  • severity_threshold - Minimum severity level to report (default: “medium”)

​
Suite Configuration Options

​
Execution Settings

Control how tests are executed: 
{
  "parallelism": 3
}
Available options:
  • parallelism - Concurrent test execution (default varies by suite type: compliance=2-3, security=2, conversational=1-3)

​
User Simulation (Conversational Suites)

Configure how AI agents behave as users: 
{
  "user_patience_level": "low|medium|high",
  "conversation_style": "natural"
}
Simulation options:
  • user_patience_level - How quickly users get frustrated (default: “medium”)
  • conversation_style - Communication style (default: “natural”)

​
Type-Specific Options

{
  "strict_mode": true,
  "include_penetration_tests": false
}
Compliance suites:
  • strict_mode - Strict protocol compliance checking (default: true)
Security suites:
  • include_penetration_tests - Include penetration testing (default: false)

​
Complete Suite Examples

​
Basic User Interaction Suite

{
  "suite_id": "basic_interactions",
  "name": "Basic User Interactions",
  "description": "Test fundamental user workflows and interactions",
  "suite_type": "conversational",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 3,
  "test_cases": [
    {
      "test_id": "welcome_flow",
      "user_message": "Hi there! I'm new here. What can you help me with?",
      "success_criteria": "Agent provides friendly welcome and clear overview of capabilities",
      "max_turns": 5,
      "context_persistence": true,
      "metadata": {
        "category": "onboarding",
        "priority": "high"
      }
    }
  ],
  "user_patience_level": "medium",
  "conversation_style": "natural"
}

​
Comprehensive Compliance Suite

{
  "suite_id": "mcp_protocol_validation",
  "name": "MCP Protocol Compliance Validation",
  "description": "Comprehensive validation of MCP protocol implementation",
  "suite_type": "compliance",
  "created_at": "2024-01-01T00:00:00Z",
  "parallelism": 2,
  "test_cases": [
    {
      "test_id": "protocol_handshake",
      "protocol_version": "2025-06-18",
      "check_categories": ["handshake"]
    },
    {
      "test_id": "capability_negotiation",
      "protocol_version": "2025-06-18",
      "required_capabilities": ["tools", "resources"],
      "check_categories": ["capabilities"]
    }
  ],
  "strict_mode": true
}

​
Security Testing Suite

{
  "suite_id": "security_validation",
  "name": "Security and Authentication Tests",
  "description": "Validate security measures and authentication mechanisms",
  "suite_type": "security",
  "created_at": "2024-01-01T00:00:00Z",
  "auth_required": true,
  "parallelism": 2,
  "test_cases": [
    {
      "test_id": "bearer_token_auth",
      "auth_method": "bearer_token",
      "vulnerability_checks": ["auth"],
      "severity_threshold": "medium"
    }
  ],
  "include_penetration_tests": false
}

​
Suite Management

​
Creating Test Suites

Interactive creation: 
mcp-t create suite
Template-based creation with built-in templates: 
mcp-t create suite  # Interactive menu with template selection
Direct type creation: 
mcp-t create conversational
mcp-t create conversational --id my-suite --global

​
Managing Test Cases

Add test cases to existing suites: 
mcp-t create test-case --suite-id my-suite

​
Configuration Management

List configurations: 
mcp-t list              # List all configurations
mcp-t list servers      # List only servers
mcp-t list suites       # List only test suites
Show specific configurations: 
mcp-t show server SERVER_ID   # Show server details
mcp-t show suite SUITE_ID     # Show suite details

​
Running Test Suites

​
Basic Execution

mcp-t run SUITE_ID SERVER_ID           # Basic test execution
mcp-t run SUITE_ID SERVER_ID --verbose # Detailed output
mcp-t run SUITE_ID SERVER_ID --global  # Save results globally

​
Execution Behavior

  • Tests run sequentially by default (parallelism=1 in execution)
  • Suite parallelism setting controls internal test orchestration
  • Results are saved to test_results/runs/ directory
  • Each test includes conversation transcript and LLM judge evaluation

​
Built-in Templates

The framework includes four built-in templates with environment variable substitution:

​
Basic Server Template

{
  "url": "${MCP_SERVER_URL}",
  "name": "${MCP_SERVER_NAME:-My MCP Server}",
  "authorization_token": "${MCP_AUTH_TOKEN:-}"
}

​
Suite Templates

  • Compliance Template: 3 test cases covering handshake, capabilities, and tools
  • Security Template: 3 test cases covering auth validation, rate limiting, and injection testing
  • Conversational Template: 2 test cases covering multi-turn conversations and error recovery
Templates support environment variable substitution with default values using ${VAR:-default} syntax.

​
Next Steps

Now that you understand test suites:
  1. Create your first suite with mcp-t create suite
  2. Run the quickstart with mcp-t quickstart for complete setup
  3. Explore test results in the test_results/ directory
  4. Add servers with mcp-t create server
  5. Generate test suites automatically with mcp-t generate
⌘I