Map

Map a website's URLs

POST/map
from $0.001/calljson responserest · mcp
01// about this endpoint

Map a website through superagnt to get a list of its URLs quickly. Optionally order results by relevance to a search term. Ideal for discovering a site's structure before scraping specific pages.

02// code samples

Call it over REST

Authenticate with a bearer token against https://api.superagnt.com. Swap the placeholders for your key and parameter values.

curlbash
curl -s -X POST "https://api.superagnt.com/map" \
  -H "Authorization: Bearer <YOUR_AGNTDATA_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{ "url": "<string>" }'
03// give it to your agent

Copy-paste prompt

Paste this into a coding agent. It connects the superagnt MCP server, finds the tool data_web_map, and runs a first call — with the REST fallback if MCP is unavailable.

agent prompttext
You are wiring up the superagnt "Web" API to use this endpoint: Map a website's URLs.

1. CONNECT over MCP (preferred). No token to paste either way.
   On Claude (claude.ai, the desktop app, or Claude Code signed in with a Claude
   account), the server is a connector: give the user this link, they click Add
   and approve:
     https://claude.ai/customize/connectors/yours?modal=add-custom-connector&connectorName=superagnt&connectorUrl=https%3A%2F%2Fmcp.superagnt.com%2Fmcp&open_in_browser=1
   Any other MCP client (or Claude Code signed in with an API key): add the server
   and let the client run the browser sign-in:
     claude mcp add --scope user --transport http superagnt https://mcp.superagnt.com/mcp
     claude mcp login superagnt
   MCP endpoint: https://mcp.superagnt.com/mcp
   After connecting, the tool you want is named exactly:
     data_web_map
   Call agnt_tools_search or agnt_tools_list_enabled to confirm it is available.

2. REST fallback (if you are not using MCP):
     POST https://api.superagnt.com/map
     Header: Authorization: Bearer <YOUR_AGNTDATA_API_KEY>
   Required parameters:
    - url (string, required) — The base URL of the site to map.
   Optional parameters:
    - search (string, optional) — Order returned URLs by relevance to this query.
    - sitemap (string, optional) — How to use the site's sitemap. Default 'include'.
    - includeSubdomains (boolean, optional) — Include URLs on subdomains.
    - ignoreQueryParameters (boolean, optional) — Treat URLs that differ only by query string as the same URL.
    - limit (integer, optional) — Maximum number of URLs to return. Default 5000, max 100000.
    - timeout (integer, optional) — Request timeout in milliseconds.
    - location (object, optional) — Geographic proxy location and language settings.

3. COST & KEY. This endpoint is metered — pricing varies by request, from $0.001 per call. Only successful calls are charged.
   Get a superagnt API key at https://app.superagnt.com.

4. TEST. Connect the server (or set the key), discover the tool
   (data_web_map), run one minimal call with just the required parameters,
   and report back the shape of the JSON response (top-level fields).

Reference (machine-readable variant): https://www.superagnt.com/docs/apis/web/endpoints/map.md
mcp tool definitionjson
{
  "name": "map",
  "description": "Returns a list of URLs found on a website, quickly. Useful for discovering the structure of a site before scraping specific pages. Optionally order results by relevance to a `search` term.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "format": "uri",
        "description": "The base URL of the site to map."
      },
      "search": {
        "type": "string",
        "description": "Order returned URLs by relevance to this query."
      },
      "sitemap": {
        "type": "string",
        "enum": [
          "skip",
          "include",
          "only"
        ],
        "description": "How to use the site's sitemap. Default 'include'.",
        "default": "include"
      },
      "includeSubdomains": {
        "type": "boolean",
        "description": "Include URLs on subdomains.",
        "default": true
      },
      "ignoreQueryParameters": {
        "type": "boolean",
        "description": "Treat URLs that differ only by query string as the same URL.",
        "default": true
      },
      "limit": {
        "type": "integer",
        "description": "Maximum number of URLs to return. Default 5000, max 100000.",
        "minimum": 1,
        "maximum": 100000,
        "default": 5000
      },
      "timeout": {
        "type": "integer",
        "description": "Request timeout in milliseconds.",
        "minimum": 1000
      },
      "location": {
        "type": "object",
        "description": "Geographic proxy location and language settings.",
        "properties": {
          "country": {
            "type": "string",
            "description": "ISO 3166-1 alpha-2 country code."
          },
          "languages": {
            "type": "array",
            "items": {
              "type": "string"
            }
          }
        }
      }
    },
    "required": [
      "url"
    ]
  }
}
04// parameters
NameInTypeRequiredDescription
urlbodystringrequiredThe base URL of the site to map.
searchbodystringoptionalOrder returned URLs by relevance to this query.
sitemapbodystringoptionalHow to use the site's sitemap. Default 'include'.
includeSubdomainsbodybooleanoptionalInclude URLs on subdomains.
ignoreQueryParametersbodybooleanoptionalTreat URLs that differ only by query string as the same URL.
limitbodyintegeroptionalMaximum number of URLs to return. Default 5000, max 100000.
timeoutbodyintegeroptionalRequest timeout in milliseconds.
locationbodyobjectoptionalGeographic proxy location and language settings.
05// responses

The list of URLs discovered on the site.

200json
{
  "type": "object"
}
06// pricing
from $0.001per successful request

Metered endpoint — the exact cost depends on the request. Credits are deducted per successful call; failed requests (4xx/5xx) are not charged.

Get started at app.superagnt.com.

08// more from Web

start calling

Point your client at https://mcp.superagnt.com/mcp and your agent has this endpoint, plus every other source on one balance.

Start free→
site map URL discovery APIwebsite URL list APIcrawl planning APIAI agents web data APIweb scraping APIweb search API for AI agentsscrape to markdown API