Web scraping for AI agents
Scrape any page to clean markdown or schema-shaped JSON, run a web search that returns full page content in one call, and map a site before you crawl it.
Copy the setup prompt.
Your agent does the rest.
Fetch and execute the appropriate instructions to set me up for superagnt_ from https://mcp.superagnt.com/agent-setup/prompt.md» once it is connected, run this so your agent can switch the source on itself.
agnt_tools_enable({ families: ['data:web'] })Everything your agent can pull
Any URL to clean markdown with navigation, cookie banners and boilerplate stripped, which is what a model can actually read.
Pass a JSON schema or a prompt and get typed fields back instead of prose, so the result drops straight into a table.
One call runs a web search and returns the page content for the results, so your agent does not need a second round trip per link.
Discover a site URLs quickly, then scrape only the pages that matter.
JavaScript-heavy pages are rendered before extraction, with waits and stronger anti-bot handling available per request.
Summary, HTML, raw HTML, links, images, full-page screenshots and change tracking, selected per call.
Cost tracks the work a request does, so a plain markdown scrape is cheaper than a rendered page with structured extraction.
tool names03as your agent sees them
data_web_scrapedata_web_searchdata_web_mapThree jobs it can start on today
Research that cites real pages
The agent searches, reads the page content that comes back with the results, and answers with links and quotes it actually retrieved rather than what it remembers.
Pricing and positioning watch
A schedule scrapes competitor pricing pages into a schema, writes each run to the workspace database, and reports what changed since last week.
Enrichment from a company site
Given a domain, the agent maps the site, scrapes the about and careers pages, and extracts locations, headcount signals and product lines into typed fields.
What comes back
Your agent reads this response directly, with no parsing layer of your own in between. The same JSON comes back over plain HTTP, so a cron job or a script sees exactly what the agent sees.
https://mcp.superagnt.com/mcptool calldata_web_scrape{
"url": "https://meridian-analytics.example/pricing",
"formats": ["markdown"]
}{
"markdown": "# Pricing\n\n## Starter\n$49 per month, up to 3 seats...",
"metadata": {
"title": "Pricing | Meridian Analytics",
"sourceURL": "https://meridian-analytics.example/pricing",
"statusCode": 200
},
"...": "links, warning when a format was skipped"
}What it costs to run
Calls in Web scraping are priced individually and deducted from your credit balance, and only a successful call is billed. The per-endpoint figures live in the reference docs, and every billable response reports its own cost in meta.costCents.
One balance covers this source and every other one, so there is no per-source subscription and no upstream account of your own to keep alive.
how credits work →Before you wire it up
01Can I get structured fields instead of markdown?
Yes. Pass a json format with a schema or a prompt on the scrape call and the response comes back as typed fields.
02How is it billed?
Per call, in credits, scaled to the work the request does. A plain markdown scrape costs less than a rendered page with structured extraction.
03How does my agent get these tools?
Connect to https://mcp.superagnt.com/mcp and call agnt_tools_enable with data:web.
04Do I need a separate scraping account or proxies?
No. agntdata abstracts the upstream web data infrastructure so you integrate once. You use one API key and one credit balance.
05Can it read JavaScript-heavy pages?
Yes. Pages are rendered before extraction, and you can wait for content or use enhanced anti-bot handling when needed.
Point your client at https://mcp.superagnt.com/mcp and your agent has this, plus everything else on one balance.