Description
🍮 crawlbrulee for n8n
EU-native web scraping for AI agents & developers.
put crawlbrulee in your n8n workflows. this community node package turns any url into clean markdown, html, links, images, metadata or a screenshot, maps a site’s urls, and starts a workflow when a background scrape finishes. it also works as an AI Agent tool, so an agent in n8n can scrape a page on its own.
- everything runs in the EU. the fetch, the render, the cache and your result never leave EU servers. the proxy exit is the one hop you choose: pick an EU exit and nothing leaves at all. gdpr-aligned, with a data processing agreement.
- output made for models. markdown with ads, popups and cookie banners removed and the links kept, ready for the prompt. full-page screenshots can come back as tiles sized for an image model.
- the hard parts, handled. headless Chrome when a page needs it, rotating proxies with country selection, automatic retries, ad and cookie-banner removal, caching, background jobs and signed webhooks.
- start free. 750 credits, no credit card.
- Extract gives you exactly what you tick: the node sends every output you left unticked as an explicit “no”, so the api’s own defaults never add one.
- a Screenshot renders the page in a real browser and is billed at the screenshot rate, the priciest one — leave it on None when you don’t need the image.
- Scrape URL (Async) takes the same fields as Scrape URL, minus Download Screenshot (a submit gives you a job id, not an image), plus Webhook URL and Webhook Metadata.
get a free api key → dashboard.crawlbrulee.com
the package ships the crawlbrulee node (7 operations), the crawlbrulee Trigger node, and the crawlbrulee API credential. this readme covers them. for how the api behaves — endpoints, parameters, limits and errors — see the api docs.
—
install
n8n Cloud: once the node is verified, open the node panel, search for “crawlbrulee”, and pick the node. n8n installs it for you.
self-hosted: go to Settings → Community Nodes → Install, enter n8n-nodes-crawlbrulee, and confirm.
on self-hosted n8n, an AI Agent can only use a community node as a tool when the env var N8NCOMMUNITYPACKAGESALLOWTOOL_USAGE is true.
credentials
create an api key in the dashboard, then add a crawlbrulee API credential in n8n:
| field | what to put in it |
|—|—|
| API Key | your key, starting with cwbl_. required. |
| Base URL | leave as is. |
| Webhook Signing Secret | optional. starts with whsec_, from the dashboard under Account → Webhooks. |
the crawlbrulee Trigger checks deliveries against that secret; leave it empty and it takes them as they come. saving the credential makes a whoami call, so a bad key fails right there, at no credit cost.
the crawlbrulee node
pick a resource, then an operation:
| resource | operation | what it does |
|—|—|—|
| Scrape | Scrape URL | fetch one url, get back the content you asked for |
| Scrape | Scrape URL (Async) | start a background job, get back its job id |
| Scrape | Get Scrape Status | pending, running, done or failed |
| Scrape | Get Scrape Result | the content of a finished job, for 24 hours after it was submitted |
| Map | Map Website | a site’s urls, from its sitemap and its homepage links |
| Account | Get Credit Usage | credits used and left this billing cycle, plus your concurrency limit |
| Account | Get Account Info | the organization and token behind the api key |
scrape fields
every field has a hint in n8n. three things the hints don’t tell you:
elements
Elements reads named values from the page by CSS selector. add one row per value: a Name, a CSS Selector, the Output (Text, HTML or an Attribute such as href), and All Matches to get every match as a list. they come back in elements, under the names you gave. elements cost no extra credits. Extract starts on Cleaned HTML and Metadata, so clear it to get only the elements.
for a value per match, like a title, price and link per product card, put the request in Elements (JSON) under Options. it is merged with the list, and each name may be used once:
{
"books": {
"selector": "article.product_pod",
"all": true,
"fields": {
"title": { "selector": "h3 a", "output": "attribute", "attribute": "title" },
"price": ".price_color"
}
}
}
with a heading row (h1, Text), the item’s json gets this (the page has 20 books; two are shown here):
{
"elements": {
"heading": "All products",
"books": [
{ "title": "A Light in the Attic", "price": "£51.77" },
{ "title": "Tipping the Velvet", "price": "£53.74" }
]
}
}
a name with no match is null ([] with All Matches). selector rules and limits: see elements.
screenshots as files
turn on Download Screenshot — it’s on Scrape URL and on Get Scrape Result — and the node attaches the image as binary data under screenshot, plus one key per tile (screenshotslice0, screenshotslice1, …) when you set Slice Height. the urls stay in the json too, but they are signed links that expire 24 hours after the scrape (for Scrape URL (Async), 24 hours after the job was submitted), so turn this on when you need the image later in the workflow. a download can fail, since a signed url can expire; the item still comes through with its page content and the reason lands on the json as screenshotdownloaderror.
what comes back
one item in, one item out: the api’s json body becomes the item’s json, unchanged. Map Website adds pagination and truncation under response_meta, and its Limit starts at 50 in n8n, not the api’s 5000 — n8n’s rule for a field named Limit sets that, so raising it is usually the first thing you do. the api docs have the full response shape.
set the node’s “On Error” setting to continue and a failed item comes back as { "error": "…" } instead of stopping the run.
the page’s own status
Scrape URL and Get Scrape Result put the HTTP status the site answered with in pagestatuscode, next to url. it is the status of the final page, after redirects.
a page the site really served is a normal item, whatever its status. a 404, 410 or 503 page comes back with its content, like any other page. it is not a node error, so “On Error” never sees it. when the status matters to your workflow, check it before you use the content — for example an If node on {{ $json.pagestatuscode }} is less than 400.
zero data retention
turn on Zero Data Retention under Options of Scrape URL, Scrape URL (Async) or Map Website to keep the result out of the shared cache. anything stored to deliver it is kept for 24 hours, then deleted. it adds 1 credit and must be enabled for your organization. if it is not, the call fails with a clear message and costs no credits. see zero data retention.
what a call cost
scrape, map and finished-job responses carry response_meta.usage, so the cost of each call is in front of you:
| field | what it is |
|—|—|
| totalcreditcost | credits this call cost |
| enginecreditcost | the engine’s base price: 1 for http, 3 for browser, 5 for screenshot, 0 for cache |
| proxy_multiplier | 1 for the Basic proxy tier, 5 for Advanced |
| screenshotslicingcredit_cost | 1 when the screenshot was cut into slices on this call, else 0 (reusing slices from the cache costs 0). not on Map Website |
| zerodataretentioncreditcost | 1 when Zero Data Retention added its credit to this call, else 0 |
| engine, proxy | the engine and the proxy tier that did the work |
totalcreditcost is always enginecreditcost × proxymultiplier + screenshotslicingcreditcost + zerodataretentioncreditcost (map has no slicing part). a page we don’t bill, such as one the site answered with a 5xx status, has 0 in every *creditcost field. the credits and pricing page lists what is billed.
the crawlbrulee Trigger
use it to react to a background scrape instead of polling for it:
1. add a crawlbrulee Trigger, give it the same credential, and copy the webhook url it shows.
2. paste that url into the Webhook URL field of a Scrape URL (Async) operation.
3. when the job finishes, we post one scrape.complete event there and the trigger fires.
the item carries the event id, the timestamp and the job data. a job that got a page has status success and the page’s HTTP status in pagestatuscode, so a 404 page arrives as success too. turn on Fetch Result and, for a job that succeeded, the trigger fetches the page content into result; if that fetch fails, the reason lands in result_error and the item still comes through.
with the secret set, the trigger checks the HMAC-SHA256 signature on every delivery. it drops one that fails the check, one that isn’t a scrape.complete event, and one whose event id is among the last 500 it saw: nothing runs, and we still answer 200 so it is not retried. that id list lives in the node’s workflow data, so a fresh copy of the workflow starts empty.
errors
the node turns api errors into messages you can act on: a rate limit says how long to wait, a Zero Data Retention request your organization cannot make yet says to turn the option off or ask us to enable it, out of credits and concurrency limit are each named, an anti-bot block says so with no retry hint, a scrape error points you at the Advanced proxy tier and at Require JS, and a site we could not reach at all (for example it timed out, or its certificate is bad) tells you to check the url and try again later. the raw response stays on the error for the n8n error view, and every error name is in the error reference.
development
pnpm install
pnpm lint # the same rule set n8n's verification scan runs
pnpm verify # lint, format, typecheck, tests, build
pnpm dev # local n8n with this node linked, rebuilt on every change
pnpm dev links the package into its own n8n under ~/.n8n-node-cli and starts it on localhost:5678 — the first run downloads n8n, so give it a while; press o to open it. add --external-n8n when you run n8n yourself, with N8NDEVRELOAD=true set there.
the package has zero runtime dependencies, because a verified n8n node may not ship any: @crawlbrulee/sdk is a types-only dev dependency, and the signature check uses the Web Crypto global.
part of the crawlbrulee toolkit
one api, many ways to call it:
@crawlbrulee/sdkcrawlbrulee on pypinpx crawlbrulee@crawlbrulee/mcpdocs: crawlbrulee.com/docs · dashboard: dashboard.crawlbrulee.com
