Skip to main content

​
General Questions

​
What is MCP Testing Framework?

MCP Testing Framework is a Python CLI tool that uses AI agents to test Model Context Protocol (MCP) servers through realistic conversations. Instead of writing brittle unit tests, you define test scenarios in natural language and let AI agents interact with your MCP server to validate behavior.

​
How is this different from traditional testing?

Traditional testing requires writing specific assertions and mocking responses. MCP Testing Framework:
  • Uses real LLM clients (Claude, GPT) to interact with your server
  • Evaluates success using natural language criteria
  • Tests realistic user workflows and conversations
  • Has security and compliance testing built-in

​
What MCP servers can I test?

You can test any MCP server that implements the Model Context Protocol:
  • Streamable HTTP-based servers
  • Servers with OAuth authentication
  • Public and private MCP servers

​
Installation and Setup

​
How do I get API keys?

You’ll need API keys from AI providers: Set them as environment variables: 
export ANTHROPIC_API_KEY="sk-ant-your-key"
export OPENAI_API_KEY="sk-your-key"

​
Configuration Questions

​
Where are configurations stored?

The framework uses two locations:
  • Local: ./configs/ (project-specific)
  • System: ~/.config/mcp-t/ (user-wide)
Local configurations take precedence over system configurations.

​
Test Creation and Execution

​
How do I write good success criteria?

Make success criteria specific and measurable:
  • ❌ Bad: “Should work correctly”
  • ✅ Good: “Agent responds with a greeting and lists at least 3 available tools”
  • ❌ Bad: “Handles the request”
  • ✅ Good: “Successfully creates a new task and returns task ID in response”

​
What makes a good test case?

Good test cases are:
  • Specific: Clear user message and expected outcome
  • Realistic: Reflects actual user workflows
  • Measurable: Success criteria can be objectively evaluated
  • Independent: Doesn’t depend on other test results

​
How many test cases should I include?

Start small and grow incrementally:
  • Initial: 3-5 core functionality tests
  • Basic Suite: 10-15 tests covering main features
  • Comprehensive: 20+ tests including edge cases

​
Can I test error conditions?

Yes, create test cases that expect specific error handling: 
{
  "test_id": "invalid_input_handling",
  "user_message": "Process this invalid data: %%%invalid%%%",
  "success_criteria": "Agent gracefully handles invalid input and provides helpful error message"
}

​
Authentication and Security

​
How do I test authenticated servers?

The framework will automatically handle OAuth protected servers that comply with the MCP spec.

​
How do I test OAuth flows?

Use OAuth-specific test cases: 
{
  "test_id": "oauth_authentication",
  "auth_method": "oauth",
  "vulnerability_checks": ["auth"],
  "success_criteria": "Successfully authenticates via OAuth and maintains session"
}

​
Can I test security vulnerabilities?

Yes, use security test suites to check for common issues:
  • Authentication bypasses
  • Injection attacks
  • Rate limiting
  • Input validation

​
Debugging and Troubleshooting

​
My tests are failing but I don’t know why. How do I debug?

  1. Enable verbose output: mcp-t run suite server --verbose
  2. Test individually: Run one test at a time
  3. Review success criteria: Ensure they’re realistic and specific

​
How do I know if the problem is with my server or the test?

  1. Test server manually: Use curl or browser to verify functionality
  2. Check server logs: Look for errors during test execution
  3. Use health checks: Verify basic connectivity and authentication
  4. Test with minimal cases: Start with simple interactions

​
The AI judge seems wrong. How do I improve evaluation?

  1. Improve success criteria: Be more specific and measurable
  2. Provide more context: Include relevant background information
  3. Check response format: Ensure server responses are clear
  4. Send an issue: Use mcp-t report issue command to send us feedback

​
Advanced Usage

​
How do I test multiple servers with the same test suite?

Run the same test suite against different servers: 
mcp-t run my-suite dev-server
mcp-t run my-suite staging-server  
mcp-t run my-suite prod-server

​
Best Practices

​
How often should I run tests?

Recommended testing frequency:
  • Development: After each significant change
  • CI/CD: On every pull request

​
Should I test all MCP server features?

Focus testing on:
  • Core functionality: Essential features users rely on
  • Integration points: How your server interacts with clients
  • Error scenarios: How gracefully your server handles problems
  • Performance: Response times and resource usage

​
Getting Help

​
How do I report bugs or request features?

  • GitHub Issues: Report bugs and feature requests
  • Raport directly: Use mcp-t report issue command
  • Documentation: Check troubleshooting guides first

​
Can I contribute to the project?

Yes! Contributions are welcome:
  • Bug reports and feature requests
  • Documentation improvements
  • Code contributions
  • Community examples and templates
See the contributing guidelines in the repository for more details.
⌘I