Can Claude scrape websites?
Yes: Claude can scrape websites into rows in a table once superagnt is connected, from a plain chat with no Python. On its own, Claude's web fetch reads a page you link, but not pages built with JavaScript, and it leaves text in the chat rather than rows you can query next week.
Claude in Chrome, on paid plans, can read JavaScript pages in your own browser while Chrome is open. superagnt does the reading on its servers instead: in the run below, Claude scraped Canva's pricing page, which answered a plain request with a bot wall, and saved its four plans to a table, for 0.144 credits.
Connect superagnt to Claude
Copy the setup prompt.
Your agent does the rest.
Install superagnt for me. It's the superagnt plugin from the github.com/superagnt/plugins marketplace, plus the superagnt MCP server (on Claude it's a connector I add in my browser; elsewhere I sign in through my browser). The setup guide for each kind of agent is at https://www.superagnt.com/install» the base connection already serves web scraping and the workspace database. Nothing to switch on.
What Claude does alone, and what changes once it's connected
| job | Claude on its own | with superagnt |
|---|---|---|
| Read a page you link | Web fetch reads the page into the chat. It opens only URLs already in the conversation, and doesn't support pages built with JavaScript. | data_web_scrape returns the page's main content as clean markdown. JavaScript-heavy pages are rendered first, on superagnt's servers. |
| Pages built with JavaScript | Claude in Chrome (paid plans) reads them in your own browser, with your Chrome open. | Rendered on superagnt's servers. No browser of yours, no extension. |
| Keep what you scraped | Text in the chat, or a file you save. | Rows in your workspace's Postgres database (agnt_db_write), there to query next week or compare against. |
| Do it again every week | Claude in Chrome runs recurring browser tasks, as long as Chrome is open. | A hosted agent on a schedule (agnt_schedules_write) runs on superagnt's servers with your laptop closed. |
| What a page costs | Included in your Claude plan. | 0.144 credits a page (a credit is a cent) in the run below. The database calls cost nothing. |
The job, run on superagnt
Real tool calls on superagnt's staging environment, the same calls Claude makes once the server is connected. Each result is what the client received, trimmed where it says so and otherwise unchanged, and each wall time was measured around the call.
“Scrape Canva's pricing page, pull out each plan and its price, and save them to a table I can check again next week.”
- 01Scrape the pricing page805 ms wall time0.144 credits, $0.00144→ call
data_web_scrape{ "url": "https://www.canva.com/pricing/", "formats": ["markdown"], "onlyMainContent": true }← result# Plans and pricing ... ## Free Design anything and bring your ideas to life. No cost, just creativity. US$0 /year for one person ... ## Pro Unlock premium content, more powerful design tools, and AI features. US$144 /year for one person ... ## Business Create content faster, market smarter, and grow your business with advanced tools. US$250 /year per person ... ## Enterprise Empower your organization with end-to-end visual communication. Let’s talk» Trimmed: to the four plans' names, lines and prices out of a 24,311-character page, without its cookie banner, images, feature lists, comparison table and the response's metadata, which says this copy came from superagnt's cache, made at 20:50 UTC that day.
- 02Make a table for the snapshot3.5 s wall timenot charged→ call
agnt_db_sql{ "action": "apply_migration", "name": "create_pricing_snapshots", "query": "create table if not exists public.pricing_snapshots (\n id bigint generated always as identity primary key,\n company text not null,\n plan text not null,\n price_usd numeric,\n billed text,\n page_url text not null,\n scraped_at timestamptz not null default now()\n);" }← result{ "success": true } - 03Save the plans Claude read from the page331 ms wall timenot charged→ call
agnt_db_write{ "action": "insert", "table": "pricing_snapshots", "rows": [ { "company": "Canva", "plan": "Free", "price_usd": 0, "billed": "per year, one person", "page_url": "https://www.canva.com/pricing/" }, { "company": "Canva", "plan": "Pro", "price_usd": 144, "billed": "per year, one person", "page_url": "https://www.canva.com/pricing/" }, { "company": "Canva", "plan": "Business", "price_usd": 250, "billed": "per year, per person", "page_url": "https://www.canva.com/pricing/" }, { "company": "Canva", "plan": "Enterprise", "price_usd": null, "billed": "contact sales", "page_url": "https://www.canva.com/pricing/" } ] }← result{ "success": true, "affected_row_count": 4, "note": "Rows omitted to keep the response small. Pass `return_rows: true` if you need the inserted/updated row data." } - 04Read them back328 ms wall timenot charged→ call
agnt_db_read{ "action": "select", "table": "pricing_snapshots", "columns": ["company", "plan", "price_usd", "billed", "scraped_at"], "order": "id.asc" }← result{ "rows": [ { "company": "Canva", "plan": "Free", "price_usd": 0, "billed": "per year, one person", "scraped_at": "2026-11-18T22:36:49.310056+00:00" }, { "company": "Canva", "plan": "Pro", "price_usd": 144, "billed": "per year, one person", "scraped_at": "2026-11-18T22:36:49.310056+00:00" }, { "company": "Canva", "plan": "Business", "price_usd": 250, "billed": "per year, per person", "scraped_at": "2026-11-18T22:36:49.310056+00:00" }, { "company": "Canva", "plan": "Enterprise", "price_usd": null, "billed": "contact sales", "scraped_at": "2026-11-18T22:36:49.310056+00:00" } ] }
The same day, a plain HTTP request for canva.com/pricing got a 403 and Canva's "We’ll have you designing again soon" page, not the prices. Through superagnt the scrape came back in 0.8 seconds, from a copy cached at 20:50 UTC that day; fetched fresh the next day, the same page took 3.9 seconds for the same 0.144 credits. Claude read the plans out of it and the table took 4.1 seconds more. A weekly re-run is the same calls on a schedule, and each run adds dated rows next to the last ones. See Can Claude run tasks on a schedule?.
Set it up in your client
Claude
- 01Open the prefilled Add custom connector dialog on claude.ai. One connector covers Claude on the web, Claude Desktop and Cowork.
- 02Click Add, then approve on the superagnt consent screen. It creates your superagnt account if you don't have one.
- 03Ask: “Scrape <a pricing page> and save each plan and its price to a table.”
Free plans can add one custom connector. On Team and Enterprise plans, an owner adds it under Organization settings, Connectors (Anthropic).
Claude Code
- 01In Claude Code, run
/plugin marketplace add superagnt/plugins, then/plugin install superagnt@superagntfor the superagnt skills. - 02Add superagnt to Claude as a connector with the prefilled dialog and click Add. Claude Code signed in with your Claude account picks it up.
- 03Start a new session.
- 04Ask: “Scrape <a pricing page> and save each plan and its price to a table.”
ChatGPT
- 01In ChatGPT on the web, go to Plugins, select the plus button, then Add custom MCP server.
- 02Name it superagnt, enter the server URL under Connection, and set Authentication to OAuth. The server URL is
https://mcp.superagnt.com/mcp. - 03Tick "I understand and want to continue", then select Create as a plugin. Sign in to superagnt when asked. No token needed.
- 04Install the plugin, then start a new chat, type @ and select superagnt.
- 05Ask: “Scrape <a pricing page> and save each plan and its price to a table.”
ChatGPT takes custom MCP servers on the web, not on mobile, and only Business and Enterprise/Edu workspaces get write actions. On Pro, connected servers are read and fetch only, so saving rows to a table needs one of those plans in ChatGPT (OpenAI).
Codex
- 01In a terminal, run
codex mcp add superagnt --url https://mcp.superagnt.com/mcp, thencodex mcp login superagnt. The login opens a browser to approve. - 02Start a new Codex session.
- 03Ask: “Scrape <a pricing page> and save each plan and its price to a table.”
Claude's web fetch, Claude in Chrome and superagnt
| Claude web fetch | Claude in Chrome | superagnt | |
|---|---|---|---|
| Who can use it | Claude apps with web search, and the API | Paid plans: Pro, Max, Ultra, Team and Enterprise | Claude, ChatGPT, Claude Code, Codex and Cursor, once connected |
| Pages built with JavaScript | Not supported | Yes, in your own browser | Yes, rendered on superagnt's servers |
| Needs your computer on | No | Yes, with Chrome open | No |
| Where the results go | The chat | The chat, or a note, doc or form it fills | Rows in your workspace database |
| Runs on a schedule | No | Yes, while Chrome is open | Yes, as a hosted agent, laptop closed |
| Cost | Included (on the API, tokens only) | Included in the plan | 0.144 credits a page in our run |
Claude's columns are from Anthropic's help center and API docs, read Nov 18, 2026.
Pick by the job. To read one page while you browse, Claude in Chrome is already there. To pull the same fields from a list of pages every week, into a table you can sort, compare and hand to the next step (finding the companies, then their work emails), superagnt does it from the chat with nothing left running on your machine.
What a scrape costs and returns
- In superagnt's own test of 40 scrapes over 10 real pages, all 40 succeeded, from $0.00144 a page, and the markdown came back 25 times smaller than the raw HTML at the median: less for Claude to read, so less of your chat's room spent on page chrome (the measurements).
- A scrape can return a copy up to two days old, which is faster (0.8 seconds against 3.9 fresh, for this page) and costs the same. Ask for a fresh read when the price matters today (
maxAge: 0). - To find the pages first,
data_web_maplists a site's URLs, anddata_web_searchsearches the web with page content attached. - Pages behind a login stay out of reach: the tools read public pages only, and never fill a form, sign in or post.
Is web scraping legal?
It depends on the country, the data and the site's terms, and this isn't legal advice. In the US, after the hiQ v. LinkedIn rulings, a law firm's summary says collecting only public data, where you haven't agreed by contract not to, is very unlikely to break the federal anti-hacking law; contracts you signed and fake accounts for logged-in data are where the risk is, and it adds that the case law is still unsettled. Personal data brings privacy law too, such as GDPR in Europe.
- Read public pages only, and only the ones you need.
- Go slow, check robots.txt, and use a site's own API or export when it has one.
- Don't try to beat a site's anti-bot defenses or log in with throwaway accounts.
Common questions
01Can Claude scrape websites?
Yes, into rows in a table once superagnt is connected, from a plain chat with no Python. On its own, Claude's web fetch reads a page you link but doesn't support pages built with JavaScript, and Claude in Chrome reads them in your own browser while Chrome is open.
02Does Claude scrape data on its own?
It reads a page you give it when web search is on, and pulls details out of it into the chat. It opens only URLs already in the conversation, and web fetch skips pages built with JavaScript. With superagnt, it saves the fields as rows in your workspace database.
03Does Claude have a web search tool?
Yes. Claude searches the web when it helps, in the Claude apps; on the API, web search costs $10 per 1,000 searches plus tokens, and web fetch costs tokens only.
04Can Claude Code perform web searches?
Yes. Its WebSearch tool returns result titles and URLs, and WebFetch reads a page and returns a summary of what you asked for, up to 100,000 characters of the page per call.
05Can ChatGPT scrape websites?
ChatGPT searches the web on every plan, Free included, and ChatGPT Work's cloud browser, on paid plans other than Go, can read pages, click buttons and fill in forms. ChatGPT agent is no longer available. superagnt installs in ChatGPT too: Business and Enterprise workspaces can save rows, while Pro gets the read tools only.
06How do I scrape without getting banned?
Read public pages only, slowly, and back off when a site answers 429 or 503. Check robots.txt, use a site's own API when it has one, and never fight its anti-bot defenses or log in with throwaway accounts. superagnt's scrapes run on its servers and never sign in, so no account of yours is at stake.
07What are the top web scraping companies?
There is no neutral top ten: most lists are written by scraping vendors ranking themselves. The field splits into proxy networks and scraping APIs, no-code scraping apps, marketplaces of ready-made scrapers, browser services for AI agents, and open-source libraries you run yourself. Pick by the job: a one-off list, a weekly refresh, or millions of pages a day.
08What is the difference between web crawling and web scraping?
Crawling follows links to find pages, the way a search engine does; scraping pulls specific data out of the pages you found. In superagnt, data_web_map lists a site's URLs and data_web_scrape reads the ones that matter.
09What is web scraping in AI?
Usually it means letting the model do the extraction: you describe the fields you want in plain words, and Claude reads the page for meaning instead of you writing a selector for each field.
What else Claude can do with superagnt
- Can Claude search Twitter?
- Can Claude read Reddit threads?
- Reddit user analyzer: can Claude analyze any Reddit account?
- Can Claude look at LinkedIn profiles?
- Can Claude look at Instagram profiles?
- Can Claude watch YouTube videos?
- How do I get a YouTube transcript into Claude?
- Can Claude find email addresses?
- Can Claude verify email addresses?
- Can Claude see my competitors' Facebook ads?
- Can Claude save data between chats?
- Can Claude load a CSV into a database?
- Can Claude run tasks on a schedule?
- Can Claude make a dashboard?
- Why does Claude make up leads?
- Is there an Apollo alternative for Claude that shows the cost first?
- How do I set up signal-based prospecting with Claude?
- Why does my Reddit MCP return 403, and how do I give Claude access to Reddit now?
- How do I find buyers on Reddit with Claude?
- Which connectors should I install in Claude for marketing?
Go deeper: Web scraping MCP server · Web data for AI agents · Load a CSV into a database · Run tasks on a schedule
- Anthropic: Web fetch tool
- Anthropic: Enable and use web search
- Anthropic: Get started with Claude in Chrome
- Claude Code docs: tools reference
- OpenAI: Searching the web with ChatGPT
- OpenAI: Using cloud browser in ChatGPT
- OpenAI: ChatGPT agent
- Farella Braun + Martel: what the hiQ v. LinkedIn rulings say about scraping