// data source

Web scraping for AI agents

Scrape any page to clean markdown or schema-shaped JSON, run a web search that returns full page content in one call, and map a site before you crawl it.

on by default on new serversover mcp or http

Copy the setup prompt.
Your agent does the rest.

works with any agent+ 8 more

Connect the superagnt MCP server at https://mcp.superagnt.com/mcp/web-scraping to this agent for me (I'll sign in through my browser). The setup guide for each kind of agent is at https://www.superagnt.com/install?facet=web-scraping

or follow the detailed setup instructions

» a new workspace starts with the Web scraping tools already on. On an existing workspace, your agent switches them on during setup.

01// what is inside

Everything your agent can pull

Scrape to markdown

Any URL to clean markdown with navigation, cookie banners and boilerplate stripped, which is what a model can actually read.

Structured extraction

Pass a JSON schema or a prompt and get typed fields back instead of prose, so the result drops straight into a table.

Search with content

One call runs a web search and returns the page content for the results, so your agent does not need a second round trip per link.

Site mapping

Discover a site URLs quickly, then scrape only the pages that matter.

Rendered pages

JavaScript-heavy pages are rendered before extraction, with waits and stronger anti-bot handling available per request.

Other output formats

Summary, HTML, raw HTML, links, images, full-page screenshots and change tracking, selected per call.

Usage-scaled pricing

Cost tracks the work a request does, so a plain markdown scrape is cheaper than a rendered page with structured extraction.

tool names03as your agent sees them

data_web_scrapedata_web_searchdata_web_map

02// what it gets used for

Three jobs it can start on today

Research that cites real pages

The agent searches, reads the page content that comes back with the results, and answers with links and quotes it actually retrieved rather than what it remembers.

Pricing and positioning watch

A schedule scrapes competitor pricing pages into a schema, writes each run to the workspace database, and reports what changed since last week.

Enrichment from a company site

Given a domain, the agent maps the site, scrapes the about and careers pages, and extracts locations, headcount signals and product lines into typed fields.

03// one call

What comes back

Your agent reads this response directly, with no parsing layer of your own in between. The same JSON comes back over plain HTTP, so a cron job or a script sees exactly what the agent sees.

https://mcp.superagnt.com/mcptool call

→ calldata_web_scrape

{
  "url": "https://meridian-analytics.example/pricing",
  "formats": ["markdown"]
}

← resultjson

{
  "markdown": "# Pricing\n\n## Starter\n$49 per month, up to 3 seats...",
  "metadata": {
    "title": "Pricing | Meridian Analytics",
    "sourceURL": "https://meridian-analytics.example/pricing",
    "statusCode": 200
  },
  "...": "links, warning when a format was skipped"
}

» trimmed for length. the shape is the shape your agent receives.

04// cost

What it costs to run

from / call

$0.001

deducted from your credit balance

Endpoints inside Web scraping are priced individually, so a heavier call costs more than a light one. The figure above is the cheapest call in the source.

One balance covers this source and every other one, so there is no per-source subscription and no upstream account of your own to keep alive.

how credits work →

05// questions

Before you wire it up

01Can I get structured fields instead of markdown?

Yes. Pass a json format with a schema or a prompt on the scrape call and the response comes back as typed fields.

02How is it billed?

Per call, in credits, scaled to the work the request does. A plain markdown scrape costs less than a rendered page with structured extraction.

03How does my agent get these tools?

Connect your MCP client to https://mcp.superagnt.com/mcp. A server created now serves these tools by default. On an older server, have the agent call agnt_tools_enable with the family id data:web and they appear in its next tools/list.

04Do I need a separate scraping account or proxies?

No. superagnt_ abstracts the upstream web data infrastructure so you integrate once. You use one API key and one credit balance.

05Can it read JavaScript-heavy pages?

Yes. Pages are rendered before extraction, and you can wait for content or use enhanced anti-bot handling when needed.

get your key

Point your client at https://mcp.superagnt.com/mcp and your agent has this, plus everything else on one balance.

Start free→

06// keep going

The rest of what your agent can reach

data sourceSEO dataLive Google results in classic and AI Mode, keyword research with volume and difficulty, competitor and backlink gap analysis, instant page audits, and AI answer visibility, all as agent tools.open →data sourceEmail finder and enrichmentFind and verify a professional email, find a mobile number, and enrich people and companies, behind one endpoint per job. Sources run cheapest first and stop as soon as the fields you asked for are filled.open →data sourceInstagramProfiles, reels, hashtag media, locations and explore, returned as structured JSON your agent can call directly. One key, one credit balance, no Instagram developer account.open →data sourceTikTokCreator profiles, video stats and keyword search across videos and accounts, as structured JSON. One key, no TikTok developer program to apply to.open →

» every tool on one page

Copy the setup prompt.Your agent does the rest.