Map a website's URLs
/mapMap a website through superagnt to get a list of its URLs quickly. Optionally order results by relevance to a search term. Ideal for discovering a site's structure before scraping specific pages.
Call it over REST
Authenticate with a bearer token against https://api.superagnt.com. Swap the placeholders for your key and parameter values.
curl -s -X POST "https://api.superagnt.com/map" \
-H "Authorization: Bearer <YOUR_AGNTDATA_API_KEY>" \
-H "Content-Type: application/json" \
-d '{ "url": "<string>" }'Copy-paste prompt
Paste this into a coding agent. It connects the superagnt MCP server, finds the tool data_web_map, and runs a first call — with the REST fallback if MCP is unavailable.
You are wiring up the superagnt "Web" API to use this endpoint: Map a website's URLs.
1. CONNECT over MCP (preferred). No token to paste either way.
On Claude (claude.ai, the desktop app, or Claude Code signed in with a Claude
account), the server is a connector: give the user this link, they click Add
and approve:
https://claude.ai/customize/connectors/yours?modal=add-custom-connector&connectorName=superagnt&connectorUrl=https%3A%2F%2Fmcp.superagnt.com%2Fmcp&open_in_browser=1
Any other MCP client (or Claude Code signed in with an API key): add the server
and let the client run the browser sign-in:
claude mcp add --scope user --transport http superagnt https://mcp.superagnt.com/mcp
claude mcp login superagnt
MCP endpoint: https://mcp.superagnt.com/mcp
After connecting, the tool you want is named exactly:
data_web_map
Call agnt_tools_search or agnt_tools_list_enabled to confirm it is available.
2. REST fallback (if you are not using MCP):
POST https://api.superagnt.com/map
Header: Authorization: Bearer <YOUR_AGNTDATA_API_KEY>
Required parameters:
- url (string, required) — The base URL of the site to map.
Optional parameters:
- search (string, optional) — Order returned URLs by relevance to this query.
- sitemap (string, optional) — How to use the site's sitemap. Default 'include'.
- includeSubdomains (boolean, optional) — Include URLs on subdomains.
- ignoreQueryParameters (boolean, optional) — Treat URLs that differ only by query string as the same URL.
- limit (integer, optional) — Maximum number of URLs to return. Default 5000, max 100000.
- timeout (integer, optional) — Request timeout in milliseconds.
- location (object, optional) — Geographic proxy location and language settings.
3. COST & KEY. This endpoint is metered — pricing varies by request, from $0.001 per call. Only successful calls are charged.
Get a superagnt API key at https://app.superagnt.com.
4. TEST. Connect the server (or set the key), discover the tool
(data_web_map), run one minimal call with just the required parameters,
and report back the shape of the JSON response (top-level fields).
Reference (machine-readable variant): https://www.superagnt.com/docs/apis/web/endpoints/map.md{
"name": "map",
"description": "Returns a list of URLs found on a website, quickly. Useful for discovering the structure of a site before scraping specific pages. Optionally order results by relevance to a `search` term.",
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"format": "uri",
"description": "The base URL of the site to map."
},
"search": {
"type": "string",
"description": "Order returned URLs by relevance to this query."
},
"sitemap": {
"type": "string",
"enum": [
"skip",
"include",
"only"
],
"description": "How to use the site's sitemap. Default 'include'.",
"default": "include"
},
"includeSubdomains": {
"type": "boolean",
"description": "Include URLs on subdomains.",
"default": true
},
"ignoreQueryParameters": {
"type": "boolean",
"description": "Treat URLs that differ only by query string as the same URL.",
"default": true
},
"limit": {
"type": "integer",
"description": "Maximum number of URLs to return. Default 5000, max 100000.",
"minimum": 1,
"maximum": 100000,
"default": 5000
},
"timeout": {
"type": "integer",
"description": "Request timeout in milliseconds.",
"minimum": 1000
},
"location": {
"type": "object",
"description": "Geographic proxy location and language settings.",
"properties": {
"country": {
"type": "string",
"description": "ISO 3166-1 alpha-2 country code."
},
"languages": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
},
"required": [
"url"
]
}
}| Name | In | Type | Required | Description |
|---|---|---|---|---|
url | body | string | required | The base URL of the site to map. |
search | body | string | optional | Order returned URLs by relevance to this query. |
sitemap | body | string | optional | How to use the site's sitemap. Default 'include'. |
includeSubdomains | body | boolean | optional | Include URLs on subdomains. |
ignoreQueryParameters | body | boolean | optional | Treat URLs that differ only by query string as the same URL. |
limit | body | integer | optional | Maximum number of URLs to return. Default 5000, max 100000. |
timeout | body | integer | optional | Request timeout in milliseconds. |
location | body | object | optional | Geographic proxy location and language settings. |
The list of URLs discovered on the site.
{
"type": "object"
}Metered endpoint — the exact cost depends on the request. Credits are deducted per successful call; failed requests (4xx/5xx) are not charged.
Get started at app.superagnt.com.
start calling
Point your client at https://mcp.superagnt.com/mcp and your agent has this endpoint, plus every other source on one balance.