Web Search MCP Tool

This Model Context Protocol (MCP) tool enables LLMs to search the web using Google (via Serper API) and extract content from web pages with high-quality text conversion, including dynamic content that requires JavaScript rendering.

Features

• Search the web using Google through Serper API
• Retrieve and display multiple search results (top 3 by default)
• Extract and convert content from web pages using the html-to-text library
• Support for dynamic web pages with JavaScript content using Puppeteer
• Interactive search flow with options to explore additional results
• Cache search results for efficient follow-up queries
• Maintains proper formatting of headings, tables, and paragraphs
• Directly fetch and extract content from specific URLs

Installation

1. Clone this repository
2. Install dependencies:

npm install

3. Build the project:

npm run build

Note: This project includes Puppeteer for handling dynamic web pages.

Configuration

You need to set the SERPER_API_KEY environment variable with your Serper API key.

Get your API key by signing up at serper.dev.

Usage

Basic Search

// Example LLM client code
const result = await mcp.tools.invoke("websearch", "search", {
  query: "What is machine learning?",
});

Advanced Search Options

// Example LLM client code with all options
const result = await mcp.tools.invoke("websearch", "search", {
  query: "What is machine learning?",
  resultCount: 5, // Number of results to return (default: 3)
  fetchContent: true, // Whether to fetch the content of the first result
  useDynamicRendering: true, // Use Puppeteer for JavaScript-rendered content
});

Fetch Content from a Specific URL

// Example LLM client code
const result = await mcp.tools.invoke("websearch", "fetch-url", {
  url: "https://example.com/article",
  useDynamicRendering: false, // Use Puppeteer for JavaScript-rendered content
});

Fetch Content from a Specific Search Result

// Example LLM client code
const result = await mcp.tools.invoke("websearch", "fetch-search-result", {
  query: "What is machine learning?", // Must match the original search query
  resultIndex: 2, // Fetch the 2nd result (1-based index)
  useDynamicRendering: false, // Use Puppeteer for JavaScript-rendered content
});

Get More Search Results

// Example LLM client code
const result = await mcp.tools.invoke("websearch", "more-search-results", {
  query: "What is machine learning?", // Must match the original search query
  startIndex: 4, // Start from the 4th result (default)
  resultCount: 3, // Number of additional results to return
});

Development

To modify this tool:

1. Edit the source code in src/index.ts
2. Rebuild the project:

npm run build

Dynamic Web Page Rendering

The tool uses Puppeteer to handle dynamic web pages that require JavaScript rendering:

• The useDynamicRendering parameter can be set to true to enable this feature
• When enabled, Puppeteer launches a headless browser to render the page
• This allows the tool to extract content from SPAs (Single Page Applications) and other JavaScript-heavy sites
• Note that dynamic rendering is slower than standard fetching, so it should be used selectively

Search Result Caching

The tool implements a caching mechanism for search results:

• Results from each search are cached for 10 minutes
• This enables efficient follow-up queries using the fetch-search-result and more-search-results tools
• The cache is based on the exact search query, so changing the query will trigger a new search

Testing

Prerequisites

• Ensure you have successfully built your MCP project
• Have Claude Desktop installed from claude.ai
• Have Node.js installed on your system

Configuration Steps

1. Open Claude Desktop
2. Navigate to Settings
3. Go to the Developer tab
4. Locate and edit the configuration file

Configuration File Details

The configuration file is typically located at claude_desktop_config.json. You'll need to add a server configuration for your MCP project.

Example Configuration

{
  "mcpServers": {
    "websearch": {
      "command": "node",
      "args": ["/ABSOLUTE/PATH/TO/mcp-project/build/index.js"]
    }
  }
}

Important Notes:

• Replace /ABSOLUTE/PATH/TO/mcp-project/ with the full, absolute path to your MCP project directory
• Ensure the path points to the compiled index.js in your project's build directory
• Use forward slashes (/) even on Windows systems
• Double-check that the path is correct and the file exists

Verification

5. Reboot Claude Desktop
6. Check for the Hammer Tool in the interface

Common Issues

• Incorrect file paths
• Missing Node.js installation
• Compilation errors in the MCP project
• Insufficient permissions
• Puppeteer installation issues (might require additional dependencies on some Linux systems)

Interactive Search Workflow

The tool now supports an interactive search workflow:

1. User initiates a search with a query
2. Tool returns the top 3 results (by default) with content from the first result
3. User is prompted with options:
   • Fetch content from other results in the list
   • Continue searching with a refined query
   • Search for more results on the same topic
4. Based on the user's choice:
   • If requesting content from another result, use fetch-search-result
   • If refining the query, start a new search
   • If requesting more results, use more-search-results

This interactive workflow helps the LLM provide more comprehensive and relevant information to users.