Testing philosophy

​

The Problem with Traditional Testing

• Natural language interactions instead of structured requests
• Multi-turn conversations that build context over time
• Dynamic tool discovery where capabilities are negotiated
• Human-like workflows that don’t follow predictable patterns

​

Unit Tests Don’t Capture Reality

# This tells you nothing about user experience:
assert get_stories() == [{"title": "Story 1", "id": 1}]

• Doesn’t test natural language understanding
• Misses conversation flow and context
• Ignores the actual user experience
• Can’t validate tool usage patterns

​

Integration Tests Are Too Brittle

curl -X POST /mcp -d '{"method": "get_stories"}' 
# Expected: {"stories": [...]}

• Breaks with minor response changes
• Doesn’t test through actual MCP protocol
• No validation of conversational behavior
• Requires maintaining rigid expectations

​

Manual Testing Doesn’t Scale

• Time-consuming - Each test takes 5-10 minutes
• Inconsistent - Different testers, different results
• Expensive - Requires dedicated QA resources
• Limited coverage - Can’t test edge cases systematically

​

The AI Agent Approach

​

Real User Simulation

User Simulator → AI Agent (Claude) → Your MCP Server
     ↓               ↓                    ↓
"Find me news"  Uses tools           Returns stories
     ↓               ↓                    ↓  
"About tech"    Refines search       Better results
     ↓               ↓                    ↓
"Perfect!"      Natural response     User satisfied

​

Intelligence-Based Evaluation

{
  "success_criteria": "Agent should help user find relevant tech news and confirm satisfaction",
  "judge_reasoning": "The agent successfully used search tools, found relevant tech stories, and confirmed the user was satisfied with the results. The conversation flow was natural and goal-oriented.",
  "verdict": "PASS"
}

​

Core Testing Principles

​

1. Test Behavior, Not Implementation

# Tests implementation details
assert server.search_stories("tech") == expected_json

{
  "user_message": "Find me tech news",
  "success_criteria": "User gets relevant tech news stories",
  "evaluation": "Judges whether goal was achieved"
}

​

2. Validate Complete User Journeys

{
  "test_id": "news_discovery_flow",
  "user_message": "I want to stay updated on AI developments",
  "success_criteria": "Agent helps user set up a reliable way to track AI news",
  "max_turns": 8,
  "expected_tools": ["search_stories", "subscribe_topic", "get_user_preferences"]
}

• Natural language understanding - Can the agent interpret “AI developments”?
• Tool orchestration - Does it use search, then subscribe appropriately?
• Conversation flow - Does it ask clarifying questions when needed?
• Goal completion - Is the user actually helped?

​

3. Embrace Multi-Turn Interactions

Turn 1: "Help me find news"
        → Agent asks: "What topics interest you?"

Turn 2: "Technology and startups" 
        → Agent searches and shows results

Turn 3: "These are too general"
        → Agent refines search criteria

Turn 4: "Perfect! Can I save this search?"
        → Agent explains save functionality

• Did the conversation feel natural?
• Was the user’s goal ultimately achieved?
• Did the agent handle clarifications well?

​

Types of Testing

​

Conversational Testing

• User goal achievement
• Conversation quality
• Context management
• Tool usage appropriateness

​

Compliance Testing

• Handshake negotiations
• Capability discovery
• Tool and resource availability
• Error handling patterns

​

Security Testing

• Access control enforcement
• Input validation effectiveness
• Rate limiting behavior
• Injection attack resistance

​

Next Steps

1. Learn about Servers - How to configure servers for testing
2. Understand Test Suites - Structure comprehensive test coverage
3. Learn about Test Generation - Automatically generate test suites