Description

@gluecrawl/n8n-nodes-gluecrawl

This is an n8n community node. It lets you use Gluecrawl in your
n8n workflows.

> Point Gluecrawl at a URL and describe what you want in plain English. Its mapper agents work
> out the selectors and pagination once, then every later run replays that config
> deterministically and returns structured rows.

n8n is a fair-code licensed
workflow automation platform.

Installation
Operations
Credentials
Compatibility
Usage
AI Agent Tool Usage
Development
Resources
Version history

Installation

Follow the installation guide
in the n8n community nodes documentation, using the package name
@gluecrawl/n8n-nodes-gluecrawl.

Manual install, from your n8n user folder:

cd ~/.n8n
npm install @gluecrawl/n8n-nodes-gluecrawl

Restart n8n afterwards.

> [!NOTE]
> n8n Cloud installs verified community nodes only. This package is built against the
> verification rules and is submitted through the n8n Creator Hub. Until verification is granted,
> use self-hosted n8n.

Operations

The package ships two nodes.

Gluecrawl

| Resource | Operation | Endpoint | Notes |
| ——– | ———— | —————————– | —————————————————————————- |
| Job | Create | POST /v1/jobs | URL plus a Goal or explicit Columns. Charges credits upfront. |
| Job | Get | GET /v1/jobs/{id} | Status, and once ready the columns the scraper will produce |
| Job | Get Many | GET /v1/jobs | Paginated, with Return All |
| Job | Delete | DELETE /v1/jobs/{id} | Cascade-deletes the job’s schedule. Irreversible. |
| Run | Start | POST /v1/jobs/{id}/runs | Reruns a ready job on its cached config. No LLM work, no upfront charge. |
| Run | Get | GET /v1/runs/{id} | Status, item and page counts, credits used, billing breakdown |
| Run | Get Many | GET /v1/jobs/{id}/runs | Run history for one job, paginated |
| Run | Get Items | GET /v1/runs/{id}/items | One n8n item per scraped row. Return All auto-paginates at 500 rows/request. |
| Run | Download CSV | GET /v1/runs/{id}/items/csv | The run’s rows as a CSV file on the item’s binary property |

Each operation is its own entry in the node panel — search “Gluecrawl” and pick the action
directly.

The rows a scrape produced are operations on Run, not a resource of their own: /v1 has no
item entity to address, and every row endpoint is keyed on a run.

Gluecrawl Trigger

| Event | Fires when |
| ————— | —————————————————— |
| run.completed | A run reached completed. The usual entry point. |
| run.failed | A run reached failed |
| job.ready | The mapper finished and the job is scrapeable |
| job.failed | The mapper failed; the job is dead and needs replacing |

An optional Job filter drops events for other jobs. Leave it empty to receive events for
every job on the account.

Credentials

API keys are available on every plan, including Free — one key each. Every operation in this
node works on a Free key, including the webhook trigger; the one exception is enabling a schedule
on a job, which returns a 403 saying your plan does not permit scheduling. That gate is lifted on
the pricing page.

1. Sign up at gluecrawl.ai and verify your email address.
2. Mint a key in the dashboard under Settings → API keys.
3. In n8n, create a Gluecrawl API credential and paste the key.

| Field | Notes |
| ———— | ——————————————————————————————————- |
| API Key | Stored encrypted by n8n, sent as Authorization: Bearer |
| Base URL | Defaults to https://api.gluecrawl.ai. A trailing slash, or a URL already ending in /v1, is handled. |

Press Test after saving. The test issues a cheap authenticated read (GET /v1/jobs?limit=1)
because /v1 has no dedicated auth-check endpoint, and it distinguishes a bad key from an
unverified email address and from a plan without API access.

> [!IMPORTANT]
> There is one active key per account. Minting a new key revokes the previous one
> immediately, and every workflow still using it starts failing with invalidapikey.

Compatibility

• Node.js 22 or higher.
• n8n 1.79.0 or higher for AI tool support. Earlier versions work for everything else.
• Developed and tested against n8n-workflow 2.x.
• Zero runtime dependencies, by design — an n8n Cloud verification requirement.

Usage

The Job → Run → rows model

A job holds the mapper config for one URL. Creating one runs the LLM mapper and charges
credits upfront. A run executes that config and produces rows. Reruns skip the mapper
entirely, so create the job once and rerun it.

Job statuses:

| Status | Meaning |
| ————- | ————————————————————— |
| in_progress | Mapping is still running |
| ready | Mapper config exists; the job can be rerun cheaply |
| failed | Mapping failed. Terminal. |
| stale | A scrape found zero items, usually a site change. Terminal. |

Gluecrawl never re-maps an existing job, so retrying a run against a failed or stale one
keeps returning 409 jobnotready. Create a new job instead.

Picking jobs and runs

Job and Run fields are pickers with two modes: From List (browse or search your
account — select the credential first) and By ID (paste a UUID, or use an expression such as
{{ $json.id }}).

• Jobs are listed as host - status, runs as timestamp - status. Two jobs on the same host

with the same status look identical — pick by ID if you keep several.

• Run: Get, Run: Get Items and Run: Download CSV ask for a Job as well as a Run.

The job only scopes the run list; it is not sent to the API. Every Gluecrawl output carrying a
run ID carries the job ID beside it, so both fields fill from the same upstream record.

• Changing the Job does not clear a run already chosen — that is n8n’s behaviour for any

dependent picker. The node checks the two agree before reading anything, so a stale selection
fails loudly instead of returning another job’s data.

Waiting for a scrape to finish

Job: Create and Run: Start both offer Wait for Completion, which polls /v1 in-process until
the work is done. It is off by default.

> [!WARNING]
> The poll holds the n8n execution for the whole scrape — on n8n Cloud, one of your plan’s
> concurrency slots (5 on Starter, 20 on Pro). Job: Create is the expensive one: it waits out the
> LLM mapper and the first scrape.

• Webhook-triggered workflows should leave it off. n8n fails a webhook request that has not

answered within 100 seconds with a 524, well under the wait’s own 300-second default.

• A timeout does not cancel anything. /v1 has no cancel endpoint. On expiry the node fails

with an error naming the run id, but the run keeps executing and is still charged. Recover
it with Run: Get / Run: Get Items, or let the trigger pick it up. Retrying starts a second
billable run.

• Stopping the execution does stop the polling. “Stop execution”, a workflow timeout or a

worker shutdown ends the wait on the next tick, and the node makes no further API calls. The
same caveat as a timeout applies to the work itself: the run continues on Gluecrawl’s side and
is still charged, so collect it by run id rather than starting another.

• Turn it on for short interactive scrapes and the AI-agent path. For anything scheduled or

high-volume, leave it off and let the Gluecrawl Trigger wake the workflow on run.completed.

Output shapes

• Scraped rows. Job: Create (with row output on) and Run: Start emit identical keys: the

row’s extracted columns plus runid, pagenumber and item_index. Those three are written
last, so they win a name clash with a column of the same name. Use Run: Get Items with
Simplify on for the raw, un-merged row.

• Simplify. Job: Get and Run: Get return the raw record by default. The toggle flattens

nested fields — a job’s input and columns to the goal text and column names, a run’s
billing to credits_settled. Job: Delete confirms with {deleted: true, id}.

• Continue On Fail. Every operation emits the same error item: error holds the summary,

errorDescription the actionable half. Run-scoped operations also carry run_id. One Switch
expression covers the whole package.

Trigger behaviour

Each trigger workflow registers its own endpoint. An account holds up to 5. If it is at the
cap, activation fails with an error naming the limit; the node never deletes one of your
endpoints to make room.

Adopt, don’t clobber. The trigger only touches an endpoint pointing at itself. On activation
it adopts an endpoint already pointing at this workflow’s URL (updating the event list if it
drifted) or registers a new one; endpoints pointing elsewhere are never repointed or deleted. On
deactivation it deletes the endpoint only if it created it and it still points here, so
deactivating a workflow can never destroy a webhook you configured yourself.

Deliveries are signed with HMAC-SHA256 over the exact bytes sent:

X-Gluecrawl-Signature: t=,v1=

signed as {timestamp}.{raw body} with a per-endpoint secret. **A delivery that does not verify
is rejected** and never reaches your workflow. The timestamp is inside the signed material and
checked against a 5-minute tolerance, so a captured delivery cannot be replayed.

> [!NOTE]
> Gluecrawl discloses the secret once, when the endpoint is created. An endpoint the node
> adopted therefore has a secret it never saw, and its deliveries are refused. Deactivate and
> reactivate to register an endpoint the node owns.

After verifying, the node fetches the referenced run or job and emits the API record rather than
the bare payload, since the next node usually wants it. A webhook.test delivery is emitted as a
marked test item (test: true) — trigger one from the dashboard to confirm wiring without
running a scrape.

Delivery is at-least-once. Gluecrawl attempts each delivery up to 5 times, backing off
roughly 10s, 30s, 2m and 10m, with a 10-second timeout per attempt. A brief n8n outage costs
latency rather than the event — but your workflow can see the same event twice. Key
de-duplication on the payload’s event_id.

Webhook targets must be https and resolve to a public IP; http://localhost and private
ranges are rejected with 422 invalidwebhookurl.

Costs and rate limits

Every Job: Create mints a persistent job and runs the LLM mapper. The creation cost is
charged upfront and there is no dedupe by URL, so:

• Give AI agents a small Max Pages, and steer them toward Run: Start against an existing

job.

• For anything recurring, create the job once and rerun it.
• Jobs persist until deleted. Housekeeping is Job: Get Many + Job: Delete.

Runs that fail through the platform’s dead-letter path are refunded automatically, as is
502 enqueue_failed, which is safe to retry.

| Endpoint group | Limit |
| ————————————— | —————- |
| Job: Create | 60 / minute |
| Run: Start | 10 / minute |
| Webhook create / update / delete / test | 10 / minute |
| All GET operations, including CSV | not rate limited |

Exceeding a limit returns 429; the node surfaces Retry-After in the error so a Wait node or
n8n’s retry setting can honour it.

Example workflows

| Goal | Workflow |
| —————– | —————————————————————————— |
| Scheduled refresh | Schedule Trigger → Gluecrawl (Run: Start, wait on) → Run: Get Items → Sheets |
| Event-driven | Gluecrawl Trigger (run.completed) → Run: Get Items → transform → destination |
| On-demand scrape | Form / Chat trigger → Gluecrawl (Job: Create, goal mode, wait on) → reply |

The event-driven shape is the one to prefer at any volume: the workflow wakes only when a run
actually finishes, and costs no execution time while the scrape is running.

AI Agent Tool Usage

The action node sets usableAsTool, so an AI Agent can call it directly. Gluecrawl returns
tabular JSON rather than a Markdown blob, which is what makes the result usable by the next tool
in the chain without a parsing step.

• Self-hosted n8n needs N8NCOMMUNITYPACKAGESALLOWTOOL_USAGE=true to allow community

nodes as AI tools.

• The canonical agent call is Job: Create in goal mode with Wait for Completion and row

output on — an agent has nowhere to put a job id it cannot resolve in the same call. This is
the one place the inline wait is clearly worth it.

• Job and Run fields are resource locators, so $fromAI() can fill them. Point agents at

Run: Start against an existing job wherever one exists — it skips the mapper and the
upfront charge.

• Keep Max Pages small. Every Job: Create an agent makes is a new billable job.

Development

Node 22 and npm. No runtime dependencies, so npm ci installs only the toolchain.

npm ci
npm run lint        # eslint, including the n8n community-node ruleset
npm run typecheck
npm test            # jest, /v1 mocked with nock
npm run build       # tsc to dist/ plus icon assets
npm run scan        # the n8n community-package scan, source and packed tarball

To try the nodes in a real editor:

npm run build
mkdir -p ~/.n8n/nodes
cd ~/.n8n/nodes && npm install /absolute/path/to/gluecrawl-n8n

Testing the trigger locally needs a public URL. Gluecrawl refuses non-https and private-IP
targets, so http://localhost:5678/... fails with 422 invalidwebhookurl — including on “Test
workflow”, which calls the same registration hook. n8n start --tunnel no longer works: n8n 2.0
removed the flag and ignores it silently, leaving the webhook base URL on localhost. Run your
own tunnel and hand n8n its address:

cloudflared tunnel --url http://localhost:5678   # prints https://.trycloudflare.com
N8NWEBHOOKURL="https://.trycloudflare.com" npx n8n start

N8NWEBHOOKURL is the n8n 2.x name for the old WEBHOOK_URL. n8n echoes the value on startup
(“Editor is now accessible via: …”), the quickest check that the node will register a public
URL. Quick tunnels get a fresh hostname every restart, so expect to re-register the trigger.

Releasing is tag-driven and CI-only. Land on main with CI green, bump version, then
git tag vX.Y.Z && git push origin vX.Y.Z. The publish workflow builds, tests and publishes with
npm provenance, which verified
community nodes must carry. Never run npm publish locally — a local publish produces an
unattested tarball.

Resources

• n8n community nodes documentation
• Gluecrawl — product, pricing and dashboard
• Issues and feature requests

Version history

1.0.5

• API access is on every Gluecrawl plan, free included, and the node now says so. The

credential field and the 403 hint both told users that API access requires a Starter plan or
above. That was true when it was written and is not any more, so a free user reading it would
buy a plan to get a key they already had. Enabling a schedule on a job is the one operation a
paid plan gates; every other operation in this node, the trigger included, works on a free key.

• The 403 hint no longer names a plan at all. It covers two different failures — an unverified

email address and a plan without API access — and it is now almost always the former, so
pointing at an upgrade sent people to fix the wrong thing.

1.0.4

• The webhook signing secret no longer outlives the endpoint it belongs to. Deactivating a

trigger workflow cleared the endpoint ownership record but kept the secret. If the endpoint was
then re-created — in the dashboard, or by a later activation — the same n8n URL answered to a
new endpoint with a new secret, and the stored one rejected every delivery while the workflow
looked correctly configured. The secret is now tied to the endpoint id: it is discarded when
that id changes or the endpoint is deleted, and kept when the endpoint is unchanged.

• The “no signing secret” message now gives advice that works. It told users to deactivate and

reactivate, which cannot help when another endpoint already points at this workflow’s URL — the
workflow simply adopts it again, still without a secret. It now says to delete that endpoint in
the dashboard first.

• Removed the subcategories field from both node codex files, as requested by the n8n node

review. n8n reads it only for the AI category, so it never had an effect here.

1.0.3

• Wait for Completion now honours execution cancellation. Stopping a workflow — from the

editor, a workflow timeout, or a queue-mode worker shutting down — previously left the poll
sleeping out its current tick and then issuing more API calls for the rest of its timeout
budget. It now stops on the next tick and makes no further calls. A cancellation is also no
longer reported as a Gluecrawl failure, and is no longer converted into an error item when
Continue On Fail is on.

1.0.2

Why it changed: the wait is an in-process poll that holds the n8n execution — and a Cloud
concurrency slot — for the whole scrape, and it timed out on the default path. A timeout there
cancels nothing, because /v1 has no cancel endpoint, so the run kept going and stayed billable.

• The Timeout field is unchanged at 300s and still unbounded.
• README restructured onto n8n’s community-node template.

1.0.1

• Dropped the endpoint notice from the Gluecrawl Trigger parameters
• Published under the @gluecrawl npm org

1.0.0

• Initial release: Job and Run operations, the Gluecrawl Trigger, and AI tool support

License

MIT