How do I Use MCP Servers with Headless Browsers?

Model Context Protocol (MCP) servers can be seamlessly integrated with headless browsers like Puppeteer and Playwright to create powerful web scraping and browser automation workflows. This integration allows you to leverage MCP's structured tooling capabilities while performing complex browser-based data extraction tasks.

Understanding MCP and Headless Browser Integration

MCP servers provide a standardized protocol for tools and resources, while headless browsers enable programmatic control of web browsers without a graphical interface. Combining these technologies allows you to:

Automate browser interactions through MCP tools
Extract dynamic content from JavaScript-heavy websites
Handle authentication flows and session management
Capture screenshots and generate PDFs
Monitor network traffic and API calls

Setting Up MCP with Playwright

Playwright is one of the most popular headless browser frameworks, and it has native MCP server support. Here's how to set it up:

Installation

First, install the Playwright MCP server:

npm install @modelcontextprotocol/server-playwright

Configuration

Configure your MCP client (like Claude Desktop) to use the Playwright server. Add this to your MCP settings file:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-playwright"
      ]
    }
  }
}

Basic Usage with Python

Here's how to interact with the Playwright MCP server using Python:

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def scrape_with_playwright_mcp():
    server_params = StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-playwright"]
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Navigate to a page
            await session.call_tool(
                "browser_navigate",
                arguments={"url": "https://example.com"}
            )

            # Take a snapshot
            snapshot = await session.call_tool(
                "browser_snapshot",
                arguments={}
            )

            # Click an element
            await session.call_tool(
                "browser_click",
                arguments={
                    "element": "Login button",
                    "ref": "element-ref-from-snapshot"
                }
            )

            print(snapshot)

asyncio.run(scrape_with_playwright_mcp())

JavaScript Implementation

You can also use JavaScript to interact with the Playwright MCP server:

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

async function scrapeWithPlaywrightMCP() {
  const transport = new StdioClientTransport({
    command: "npx",
    args: ["-y", "@modelcontextprotocol/server-playwright"]
  });

  const client = new Client({
    name: "playwright-scraper",
    version: "1.0.0"
  }, {
    capabilities: {}
  });

  await client.connect(transport);

  // Navigate to page
  await client.callTool({
    name: "browser_navigate",
    arguments: {
      url: "https://example.com"
    }
  });

  // Get page snapshot
  const snapshot = await client.callTool({
    name: "browser_snapshot",
    arguments: {}
  });

  // Extract data
  const result = await client.callTool({
    name: "browser_evaluate",
    arguments: {
      function: "() => document.title"
    }
  });

  console.log("Page title:", result);

  await client.close();
}

scrapeWithPlaywrightMCP();

Working with Puppeteer MCP Server

While Playwright has official MCP support, you can also create custom MCP servers that wrap Puppeteer functionality for handling browser sessions and automation tasks.

Creating a Custom Puppeteer MCP Server

Here's an example of a basic Puppeteer MCP server:

import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import puppeteer from "puppeteer";

let browser = null;
let page = null;

const server = new Server({
  name: "puppeteer-mcp-server",
  version: "1.0.0"
}, {
  capabilities: {
    tools: {}
  }
});

// Tool: Launch browser
server.setRequestHandler("tools/call", async (request) => {
  const { name, arguments: args } = request.params;

  switch (name) {
    case "launch_browser":
      browser = await puppeteer.launch({
        headless: true,
        args: ['--no-sandbox', '--disable-setuid-sandbox']
      });
      page = await browser.newPage();
      return {
        content: [{ type: "text", text: "Browser launched successfully" }]
      };

    case "navigate":
      await page.goto(args.url, { waitUntil: 'networkidle2' });
      return {
        content: [{ type: "text", text: `Navigated to ${args.url}` }]
      };

    case "screenshot":
      const screenshot = await page.screenshot({ encoding: 'base64' });
      return {
        content: [{ type: "text", text: screenshot }]
      };

    case "extract_text":
      const text = await page.evaluate(() => document.body.innerText);
      return {
        content: [{ type: "text", text: text }]
      };

    case "close_browser":
      await browser.close();
      return {
        content: [{ type: "text", text: "Browser closed" }]
      };

    default:
      throw new Error(`Unknown tool: ${name}`);
  }
});

// List available tools
server.setRequestHandler("tools/list", async () => {
  return {
    tools: [
      {
        name: "launch_browser",
        description: "Launch a headless browser instance",
        inputSchema: { type: "object", properties: {} }
      },
      {
        name: "navigate",
        description: "Navigate to a URL",
        inputSchema: {
          type: "object",
          properties: {
            url: { type: "string", description: "The URL to navigate to" }
          },
          required: ["url"]
        }
      },
      {
        name: "screenshot",
        description: "Take a screenshot of the current page",
        inputSchema: { type: "object", properties: {} }
      },
      {
        name: "extract_text",
        description: "Extract all text from the page",
        inputSchema: { type: "object", properties: {} }
      },
      {
        name: "close_browser",
        description: "Close the browser",
        inputSchema: { type: "object", properties: {} }
      }
    ]
  };
});

const transport = new StdioServerTransport();
await server.connect(transport);

Using the Custom Puppeteer MCP Server

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def scrape_with_custom_puppeteer():
    server_params = StdioServerParameters(
        command="node",
        args=["path/to/puppeteer-mcp-server.js"]
    )

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Launch browser
            await session.call_tool("launch_browser", arguments={})

            # Navigate to page
            await session.call_tool("navigate", arguments={
                "url": "https://example.com"
            })

            # Extract text
            result = await session.call_tool("extract_text", arguments={})
            print("Extracted text:", result.content[0].text)

            # Close browser
            await session.call_tool("close_browser", arguments={})

asyncio.run(scrape_with_custom_puppeteer())

Advanced Techniques

Handling Dynamic Content

When working with JavaScript-heavy sites, you'll need to wait for content to load. Here's how to implement waitFor functionality:

// In your MCP server tool handler
case "wait_for_selector":
  await page.waitForSelector(args.selector, {
    timeout: args.timeout || 30000
  });
  return {
    content: [{ type: "text", text: `Element ${args.selector} found` }]
  };

Managing Multiple Pages

For scraping multiple pages in parallel:

case "open_new_page":
  const newPage = await browser.newPage();
  const pageId = generateUniqueId();
  pages[pageId] = newPage;
  return {
    content: [{ type: "text", text: pageId }]
  };

case "switch_page":
  page = pages[args.pageId];
  return {
    content: [{ type: "text", text: "Page switched" }]
  };

Network Monitoring

Monitor network requests to capture API calls:

case "monitor_requests":
  const requests = [];
  page.on('request', request => {
    requests.push({
      url: request.url(),
      method: request.method(),
      headers: request.headers()
    });
  });
  return {
    content: [{ type: "text", text: JSON.stringify(requests) }]
  };

Best Practices

Resource Management

Always clean up browser instances to prevent memory leaks:

try:
    await session.call_tool("launch_browser", arguments={})
    # Perform scraping operations
finally:
    await session.call_tool("close_browser", arguments={})

Error Handling

Implement robust error handling in your MCP server:

try {
  await page.goto(args.url, { waitUntil: 'networkidle2', timeout: 30000 });
} catch (error) {
  return {
    content: [{ type: "text", text: `Navigation failed: ${error.message}` }],
    isError: true
  };
}

Performance Optimization

Use headless mode for better performance
Disable unnecessary features like images and CSS when only extracting text
Implement connection pooling for multiple scraping tasks
Set appropriate timeouts to prevent hanging operations

browser = await puppeteer.launch({
  headless: true,
  args: [
    '--no-sandbox',
    '--disable-setuid-sandbox',
    '--disable-dev-shm-usage',
    '--disable-images',
    '--disable-css'
  ]
});

Alternative: Using WebScraping.AI API

If setting up and maintaining MCP servers with headless browsers seems complex, consider using a managed solution like WebScraping.AI. It provides JavaScript rendering capabilities without the need to manage browser instances:

curl -X GET "https://api.webscraping.ai/html?url=https://example.com&js=true" \
  -H "api_key: YOUR_API_KEY"

This approach eliminates the need for browser management while still accessing dynamic content through handling AJAX requests and JavaScript-rendered pages.

Conclusion

Integrating MCP servers with headless browsers provides a powerful framework for building sophisticated web scraping solutions. Whether you use the official Playwright MCP server or create custom Puppeteer-based servers, this architecture allows for modular, maintainable, and scalable scraping workflows. Start with the Playwright MCP server for quick implementation, then consider building custom servers when you need specialized functionality or integration with existing systems.

Remember to always respect website terms of service, implement rate limiting, and handle errors gracefully to build reliable scraping applications.

Table of contents