> ## Documentation Index
> Fetch the complete documentation index at: https://docs.golf.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Your first test

> Step-by-step tutorial for creating and running your first MCP server test manually

This tutorial complements the [Quickstart Guide](quickstart) by showing manual configuration creation and detailed result analysis.

## Prerequisites

* MCP Testing Framework installed
* An MCP server to test (we'll use a demo server)

<Steps>
  <Step title="Create a Server Configuration">
    Use the CLI command to create your first server configuration:

    ```bash theme={null}
    mcp-t create server
    ```

    Follow the interactive prompts to configure your server. For this tutorial, you can also create the configuration manually:

    ```bash theme={null}
    mkdir -p configs/servers configs/suites
    ```

    **Example server configuration** (`configs/servers/my-first-server.json`):

    ```json theme={null}
    {
      "type": "url",
      "url": "https://no-auth-server.courier-mcp.authed-qukc4.ryvn.run/mcp/",
      "name": "hackernews_mcp_server"
    }
    ```

    This configures a test server that provides Hacker News functionality via MCP.
  </Step>

  <Step title="Create a Test Suite">
    Use the CLI command to create your first test suite:

    ```bash theme={null}
    mcp-t create suite
    ```

    Follow the interactive prompts to configure your test suite. For this tutorial, you can also create the configuration manually:

    **`configs/suites/my-first-suite.json`**:

    ```json theme={null}
    {
      "suite_id": "my_first_suite",
      "name": "My First Test Suite",
      "description": "Learning how to test MCP servers",
      "suite_type": "conversational",
      "created_at": "2024-01-01T00:00:00Z",
      "parallelism": 1,
      "test_cases": [
        {
          "test_id": "greeting_test",
          "user_message": "Hello! Can you help me browse Hacker News?",
          "success_criteria": "Agent should respond politely and explain Hacker News capabilities",
          "max_turns": 5,
          "context_persistence": true,
          "metadata": {
            "category": "basic_interaction",
            "priority": "high"
          }
        },
        {
          "test_id": "tool_discovery_test", 
          "user_message": "What Hacker News tools do you have available?",
          "success_criteria": "Agent should list available Hacker News MCP tools like get stories, search, etc.",
          "max_turns": 3,
          "context_persistence": true,
          "metadata": {
            "category": "tool_discovery",
            "priority": "high"
          }
        }
      ],
      "user_patience_level": "medium",
      "conversation_style": "natural"
    }
    ```

    ### Understanding Test Case Structure

    Each test case includes:

    <ResponseField name="test_id" type="string" required>
      Unique identifier for the test
    </ResponseField>

    <ResponseField name="user_message" type="string" required>
      What the AI agent will say to your server
    </ResponseField>

    <ResponseField name="success_criteria" type="string" required>
      Natural language description of what constitutes success
    </ResponseField>

    <ResponseField name="max_turns" type="number">
      Maximum conversation turns before timeout
    </ResponseField>

    <ResponseField name="context_persistence" type="boolean">
      Whether context carries between turns
    </ResponseField>

    <ResponseField name="metadata" type="object">
      Optional categorization and priority
    </ResponseField>

    ### Writing Good Success Criteria

    Success criteria should be specific but flexible:

    **Good examples:**

    * "Agent should respond politely and explain Hacker News capabilities"
    * "Agent should list at least 2 available tools and explain their purpose"
    * "Agent should successfully fetch and display story titles with URLs"

    **Avoid these patterns:**

    * **Too vague**: "Agent should work correctly"
    * **Too specific**: "Agent must respond with exactly 'Welcome to Hacker News!'"
    * **Technical**: "Agent should make HTTP GET request to /stories endpoint"
  </Step>

  <Step title="Run Your First Test">
    Execute the test suite:

    ```bash theme={null}
    mcp-t run my_first_suite my-first-server
    ```

    You'll see a summary of test results showing which tests passed or failed, along with confidence scores from the LLM judge.
  </Step>

  <Step title="Analyze Your Results">
    ### Test Verdict Meanings

    * **✅ PASS**: Test met the success criteria
    * **❌ FAIL**: Test did not meet the success criteria
    * **⚠️ TIMEOUT**: Test exceeded maximum turns or time limit
    * **🔥 ERROR**: Technical error occurred (server unreachable, API issues)

    ### Judge Reasoning

    The LLM judge analyzes each conversation and provides:

    * **Verdict**: Pass/fail decision
    * **Reasoning**: Detailed explanation of why the test passed or failed
    * **Confidence score**: How confident the judge is (0.0-1.0)
    * **Conversation quality**: How natural and helpful the interaction was

    ### Interpreting Confidence Scores

    * **0.9-1.0**: Very confident - clear pass/fail
    * **0.7-0.89**: Confident - good evidence for verdict
    * **0.5-0.69**: Somewhat confident - borderline cases
    * **Below 0.5**: Low confidence - may need better success criteria
  </Step>

  <Step title="View Detailed Results">
    For more detailed analysis:

    ```bash theme={null}
    # List recent test results
    mcp-t list results
    ```

    This shows recent test runs with their results and completion status.
  </Step>

  <Step title="Iterate and Improve">
    ### Adding More Test Cases

    Add more test cases to your suite to cover different scenarios:

    ```json theme={null}
    {
      "test_id": "story_fetching_test",
      "user_message": "Can you show me the top 3 stories from Hacker News?",
      "success_criteria": "Agent should successfully fetch and display at least 3 story titles with brief descriptions or URLs",
      "max_turns": 7,
      "context_persistence": true,
      "metadata": {
        "category": "core_functionality",
        "priority": "critical"
      }
    }
    ```

    ### Running with Verbose Output

    Get more detailed output during test execution:

    ```bash theme={null}
    mcp-t run my_first_suite my-first-server --verbose
    ```

    ### Testing Different Scenarios

    Create suites for different types of testing:

    * **Happy path**: Normal user workflows
    * **Edge cases**: Unusual requests or error conditions
    * **Security**: Authentication, input validation
    * **Performance**: Response times, handling multiple requests
  </Step>
</Steps>

## Common First-Time Issues

<Accordion title="Server Connection Failures">
  If your server is unreachable:

  ```
  🔥 ERROR - greeting_test
     Connection failed: Could not connect to MCP server at https://your-server.com/mcp
     Error: Connection timeout after 30 seconds
  ```

  **Solutions**:

  * Verify server URL is correct and accessible
  * Check server is running and responding to MCP protocol
  * Test server manually with `curl` or browser
</Accordion>

<Accordion title="Authentication Errors">
  If authentication fails:

  ```
  🔥 ERROR - greeting_test  
     Authentication failed: 401 Unauthorized
  ```

  **Solutions**:

  * Add authentication to server configuration
  * Verify API keys or credentials are correct
  * Check server authentication requirements
</Accordion>

## Next Steps

Now that you've created and run your first test:

1. **[Learn Core Concepts](../concepts/testing-philosophy)** - Understand why this testing approach works
2. **[Explore Test Types](../test-types/conversational)** - Learn about different testing strategies
3. **[Master the CLI](../cli/overview)** - Discover more powerful commands
4. **[Advanced Configuration](../configuration/files)** - Learn about templates and advanced options

## Creating More Tests

Use the interactive creation commands for faster development:

```bash theme={null}
# Create new server configurations
mcp-t create server

# Create new test suites with templates
mcp-t create suite

# Add test cases to existing suites  
mcp-t create test-case --suite-id my_first_suite
```

These interactive commands guide you through configuration creation and help avoid syntax errors.