Octocrawl
77777r7/octocrawl
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
- GitHub stars
- 6
- tools
- 2
- auth
- Secret needed
- transport
- STDIO, HTTP
- env vars
- 4
- last updated
- Nov 11, 2026
Install
From registry metadata$ npx add-mcp https://mcp.octocrawl.dev/mcp --name octocrawlRun in your terminal. add-mcp asks which agents to add it to (Claude Code, Codex, Cursor and more). Add -g to install it for every project.
Review the command, arguments and environment values before installing. MCP servers run with your local permissions.
Other ways
Claude Code
$ claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcpRun in your terminal. Add --scope user to make it available in every project.
Tools
Verified live · 2 toolsVerified live against the running server on Nov 6, 2026.
scrapeRead one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings n...19 params
Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings n...
Parameters* required
allowlistedDomainsarrayblockAdsbooleanBlock ad hosts, ads and cookie banners in the browser. Default true.
debugbooleanReturn the full audit: trace, ladderTrace and every attempt.
excludeTagsarrayCSS selectors removed with their contents.
fastModebooleanHTTP only: a page needing scripts gets the HTTP verdict (a shell: failed/empty_unverified). Default false.
formatsarrayWhat to return (default markdown): markdown, links, html, rawHtml, images, tables, json, attributes ({ selectors }), list ({ itemSelector?, fields? }: a record per repeated item, detected without i...
headersobjectExtra headers to the page's origin, recorded; User-Agent, client hints, authorization and cookie refused.
includeLinksbooleanAlso return the page's links. Default false.
includeTagsarrayCSS selectors of the only elements to keep, whatever onlyMainContent says (no sibling combinators, :nth-child or :has).
integrationstringYour workflow label; recorded, not sent to sites.
markdownOffsetintegerRead on from a cut markdown's truncatedAt.
maxFileBytesintegerLargest file to download; a larger one is failed/body_too_large.
mobilebooleanRead as declared mobile Chrome (Android UA, 412x915). Default false.
modestringThe declared identity the page is read under: standard (default) or research. A hosted server holds no saved logins, so it has no authed mode.one of
standard · researchonlyMainContentbooleanfalse also keeps header, navigation and footer. Default true.
removeBase64ImagesbooleanDrop data: URI images from the Markdown, keeping alt text. Default true.
timeoutintegerDeadline in milliseconds for the whole scrape. When it fires the result is partial with the content so far, or failed/timeout. Here by default 120000, max 180000; a longer one is lowered to it.
url*stringAn http(s) URL.
verifyobjectChecks on final delivery; passed covers configured checks only. Inspect failure/readiness before retrying. tableRowsMin needs tables format (zero-based columns); recordFields uniqueBy counts comple...
mapList a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs fold...15 params
List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs fold...
Parameters* required
crawlEntireDomainbooleanAdmit URLs anywhere on the start host, not only in the start URL's path subtree. Default false.
debugbooleanReturn the full map response instead of the compact one.
deduplicateSimilarURLsbooleanFold /a and /a/, / and /index.html, www and apex, http and https into one URL. Default true.
excludePathsarrayPathname regexes that leave a URL out; they win over includePaths.
ignoreQueryParametersbooleanFold URLs that differ only in their query string into the first one seen, returned without its query; each merge is counted (refused.collapsed, with samples under debug). Default false.
includePathsarrayPathname regexes a URL must match.
includeSubdomainsbooleanAdmit every host under the start URL's apex (the host with one leading www. removed; no public-suffix list). Default false. Each new host's robots.txt is read, for at most 20 hosts.
integrationstringYour workflow label; recorded, not sent to sites.
limitintegerLinks returned at most. Default 5000; a hosted server takes up to 5000. Reaching it is status completed with stoppedBy limit.
modestringThe declared identity robots.txt, the page and the sitemaps are read under. authed is not offered: a map reads public sitemaps and one public page.one of
standard · researchregexOnFullURLbooleanMatch includePaths and excludePaths against the canonical URL instead of its pathname. Default false.
searchstringKeep only the URLs in which every word (at most 10) appears, case-insensitively, in the decoded URL or its title. A filter, not a ranking: the order stays the discovery order, and limit counts the...
sitemapstringinclude (default): the start page's links and the sitemaps. skip: no sitemap is read. only: no page is read; the links are the sitemap entries in their listed order (the start URL only when a sitem...one of
include · skip · onlytimeoutintegerMilliseconds for the whole map. Default 60000; a hosted server takes up to 60000.
url*stringhttp(s) URL of the start page
Sponsors
Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthly
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any language
Agent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any languageConfiguration
W2L_API_URLThe Octocrawl API the server calls: npx octocrawl serve on this computer (default http://127.0.0.1:8787), or https://api.octocrawl.dev
W2L_API_TOKENsecretA bearer token for a hosted or self-hosted API; none for a local one
W2L_MCP_TOOLSETSThe tool groups to list, comma-separated: core, crawl, batch, products, logins, monitor, delivery, or all (default all); the same as --toolsets
W2L_MCP_READ_ONLY1 lists read-only tools alone, with a scrape that does not act as the person; the same as --read-only
Readme
Octocrawl
One link. Web data, ready.
Markdown and data from web pages, for AI agents and RAG.
Fewer tokens than raw HTML. Every result records how it was read.
Details
- Repository
- github.com/77777r7/octocrawl
- Website
- octocrawl.dev/docs/connect-mcp/?utm_source=mcp-registry&utm_medium=listing
- Categories
- Documents & Knowledge, Data & Analytics
- Package
@octocrawl/mcp- Transport
- STDIO, HTTP
- Auth
- Required
- Registry
- MCP registry, active
- Tools verified
- Nov 6, 2026
