Conversational testing

Turn 1: "Help me find tech news"
        → Agent asks: "What specific topics interest you?"

Turn 2: "AI and startups"  
        → Agent uses search tools, shows results

Turn 3: "These are perfect! Can I save this search?"
        → Agent explains save functionality and helps set up preferences

Judge: ✅ PASS - User successfully found relevant content and learned about features

{
  "test_id": "user_onboarding_flow",
  "user_message": "I'm new here, what can you help me with?",
  "success_criteria": "Agent provides friendly greeting and clear overview of capabilities",
  "max_turns": 5,
  "metadata": {
    "category": "onboarding",
    "priority": "high"
  }
}

{
  "suite_id": "my_conversational_tests",
  "name": "User Journey Tests", 
  "suite_type": "conversational",
  "test_cases": [
    {
      "test_id": "greeting_test",
      "user_message": "Hello!",
      "success_criteria": "Friendly welcome with capability overview"
    }
  ]
}

{
  "suite_id": "advanced_conversations",
  "name": "Complex User Interactions",
  "suite_type": "conversational", 
  "user_patience_level": "low",
  "parallelism": 3,
  "test_cases": [...]
}

{
  "test_id": "new_user_onboarding",
  "user_message": "Hi, I just connected to this server. What can you do?",
  "success_criteria": "Agent provides clear overview of main features with examples",
  "max_turns": 5
}

{
  "test_id": "feature_exploration", 
  "user_message": "I heard you can help with data analysis. Show me how.",
  "success_criteria": "Agent demonstrates data analysis capabilities with practical examples",
  "max_turns": 8
}

{
  "test_id": "data_pipeline_setup",
  "user_message": "I need to set up automated reporting for my sales data",
  "success_criteria": "Agent guides user through complete pipeline setup with validation",
  "max_turns": 15
}

{
  "test_id": "confused_user_recovery",
  "user_message": "This isn't working, I'm confused",
  "success_criteria": "Agent asks clarifying questions and provides helpful guidance",
  "max_turns": 6
}

{
  "test_id": "impatient_user",
  "user_message": "Just give me the data already!",
  "success_criteria": "Agent handles impatience gracefully while collecting necessary details",
  "max_turns": 4,
  "metadata": {
    "category": "edge_cases",
    "priority": "medium"
  }
}

// Specific and measurable
"success_criteria": "Agent greets user warmly and lists at least 3 main capabilities with brief explanations"

// Focused on user value
"success_criteria": "User successfully creates their first report and understands how to modify it"

// Tests conversation quality  
"success_criteria": "Agent asks relevant follow-up questions and provides personalized recommendations"

// Too vague
"success_criteria": "Agent responds appropriately"

// Tests implementation details
"success_criteria": "Agent calls the get_reports() function"

// Unrealistic expectations
"success_criteria": "Agent perfectly anticipates every user need"

# Create conversational test suite (interactive menu)
mcp-t create suite

# Create conversational test suite directly
mcp-t create conversational
mcp-t create conversational --id my-chat-tests

# Run conversational tests
mcp-t run conversation-suite-id server-id --verbose

🤖 Starting conversational test: user_onboarding_flow

Turn 1/5
👤 User: Hi, I just connected to this server. What can you do?
🤖 Agent: Hello! I'm excited to help you get started...

Turn 2/5  
👤 User: That sounds great! Can you show me an example?
🤖 Agent: Absolutely! Let me demonstrate our search functionality...

⚖️  Judge Evaluation: ✅ PASS
   Reasoning: Agent provided warm greeting, clear capability overview,
   and practical demonstration. User expressed satisfaction and engagement.

✅ Test passed: user_onboarding_flow