Tavily Crawl
tavily_crawlCrawl a website starting from a URL. Extracts content from pages with configurable depth and breadth.
When to Use
Use this tool when you need to crawl a website starting from a url. extracts content from pages with configurable depth and breadth.. This is part of the Tavily Search provider on xpay✦.
MCP Connection
Connect to xpay✦ to access this tool (and 4+ others):
{
"mcpServers": {
"xpay": {
"url": "https://mcp.xpay.sh/mcp?key=YOUR_API_KEY"
}
}
}
For Claude Code:
claude mcp add --transport http xpay "https://mcp.xpay.sh/mcp?key=YOUR_API_KEY"
How to Execute
Use the xpay✦ meta-tools to run this tool:
xpay_details— Get full input schema:xpay_details("tavily/tavily_crawl")xpay_run— Execute:xpay_run("tavily/tavily_crawl", { ...inputs })
Input Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
instructions | string | No | Natural language instructions for the crawler. Instructions specify which types of pages the crawler should return. |
select_domains | array | No | Regex patterns to restrict crawling to specific domains or subdomains (e.g., ^docs.example.com$) |
include_favicon | boolean | No | Whether to include the favicon URL for each result |
select_paths | array | No | Regex patterns to select only URLs with specific path patterns (e.g., /docs/., /api/v1.) |
max_depth | integer | No | Max depth of the crawl. Defines how far from the base URL the crawler can explore. |
limit | integer | No | Total number of links the crawler will process before stopping |
format | string | No | The format of the extracted web page content. markdown returns content in markdown format. text returns plain text and may increase latency. |
extract_depth | string | No | Advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase latency |
allow_external | boolean | No | Whether to return external links in the final response |
url | string | Yes | The root URL to begin the crawl |
max_breadth | integer | No | Max number of links to follow per level of the tree (i.e., per page) |
Pricing
- Cost: $0.02/call
- Balance check: Use
xpay_balanceto check remaining credits - Get your API key at xpay.tools — $5 free credits included
Related Skills
- Tavily Search (all tools) — 5 tools
- Tavily Search — $0.02/call
- Tavily Extract — $0.02/call
- Tavily Map — $0.02/call
- Tavily Research — $0.02/call
Links
- Tool page: https://xpay.tools/tavily/tavily-crawl/
- Provider: https://xpay.tools/tavily/
- All tools: https://xpay.tools/explore
How to Execute
Input Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
instructions | string | No | Natural language instructions for the crawler. Instructions specify which types of pages the crawler should return. |
select_domains | array | No | Regex patterns to restrict crawling to specific domains or subdomains (e.g., ^docs\.example\.com$) |
include_favicon | boolean | No | Whether to include the favicon URL for each result |
select_paths | array | No | Regex patterns to select only URLs with specific path patterns (e.g., /docs/.*, /api/v1.*) |
max_depth | integer | No | Max depth of the crawl. Defines how far from the base URL the crawler can explore. |
limit | integer | No | Total number of links the crawler will process before stopping |
format | string | No | The format of the extracted web page content. markdown returns content in markdown format. text returns plain text and may increase latency. |
extract_depth | string | No | Advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase latency |
allow_external | boolean | No | Whether to return external links in the final response |
url | string | Yes | The root URL to begin the crawl |
max_breadth | integer | No | Max number of links to follow per level of the tree (i.e., per page) |
Related Tools from Tavily Search
Install Skill
Pricing
Cost
$0.02/call
Model
Flat rate
Provider
Tavily Search