Description

n8n Community Node for datHere QSV (@dartfx/n8n-nodes-qsv)


> [!NOTE]
> Node APIs, parameters, and operation interfaces in @dartfx/n8n-nodes-qsv are actively expanding. Please report any issues or feature requests on GitHub Issues.

A custom n8n community node for ultra-fast, zero-copy tabular data wrangling, transformation, validation, and analytics powered by datHere QSV.

—

⚡ Key Features

• File Path-First (Zero-Copy Architecture): Pass filesystem paths (inputPath, outputPath) between nodes to stream multi-gigabyte datasets directly through native Rust engines without overwhelming n8n memory. Downstream nodes can chain output directly with {{ $json.outputPath }}.
• First-Class Positional & Typed Parameters: Intuitive UI form fields for command-specific arguments (e.g. selection for select, regex for search, sampleSize for sample, sql for sqlp, column/row/value for edit) with precise docopt CLI argument ordering.
• 73+ Automated QSV Operations: Full suite of high-performance tabular operations:

– Profiling & Analysis: stats, frequency, schema, sniff, count, moarstats, pragmastat, profile
– Transformation & Cleaning: apply, behead, dedup, denull, fill, flatten, fmt, replace, safenames, rename, pseudo, edit
– Slicing, Search & Sampling: index, slice, search, searchset, select, sample, split, partition
– High-Speed SQL & Joins: sqlp (Polars SQL engine), join, joinp, pivotp, diff, explode, implode
– Format Conversions: to (Parquet, XLSX, SQLite, Postgres, ODS, DataPackage), excel, json, jsonl, tojsonl, fixedwidth, geoconvert
– AI & Web Services: describegpt (LLM metadata/chat), fetch, fetchpost, geocode
– Advanced Scripting & Validation: luau embedded scripting, validate (JSON Schema / RFC4180), blake3 cryptographic hashing, synthesize (statistical test data generation)

• Feature-Gated Command Diagnostics: Tagged feature requirements in UI dropdowns ([Feature: polars], [Feature: synthesize], [Feature: luau]) with actionable NodeOperationError guidance if a host binary lacks a compiled feature.
• Dynamic CLI Synchronizer: pnpm run generate:qsv automatically queries qsv --list and keeps all node definitions and parameter forms synchronized with your installed QSV binary.

—

📦 Installation

In self-hosted n8n instances:

Follow the n8n Community Nodes installation guide:

1. Go to Settings > Community Nodes.
2. Select Install.
3. Enter @dartfx/n8n-nodes-qsv.
4. Agree to the risks and select Install.

—

🛠 System Prerequisites

This community node executes the high-performance QSV CLI under the hood. The qsv binary must be installed and accessible in the system $PATH where n8n is running.

> [!IMPORTANT]
> n8n 3.0+ Docker Deployment Requirement:
> Starting with n8n 3.0, self-hosted n8n requires a Docker-based deployment (bare npm / npx n8n installations are deprecated/removed). Because the official n8n Docker image does not include the native qsv binary, you must either:
>
> 1. Use a custom multi-stage Dockerfile (recommended below) to bake qsv into your runtime container.
> 2. Mount a host-compiled qsv binary into /usr/local/bin/qsv in your container.

1. Host Installation (n8n v1 / v2 or Local Development)

• macOS (Homebrew):

  brew install qsv
  

• Debian / Ubuntu / Linux:

Download pre-compiled binaries from QSV GitHub Releases or build via Cargo:

  cargo install qsv --locked --bin qsv --features all_features
  

• Windows:

  scoop install qsv
  # or
  choco install qsv
  

2. Docker / Self-Hosted n8n Container (Required for n8n 3.0+)

If you run n8n using Docker (standard for n8n 3.0+), create a custom multi-stage Dockerfile that fetches the versioned qsv binary:

Stage 1: Fetch and unpack the QSV binary
FROM alpine:latest AS qsv-fetcher
See https://github.com/dathere/qsv/releases for latest
ARG QSV_VERSION=23.0.1
RUN apk add --no-cache curl unzip 
    && ARCH=$(uname -m) 
    && if [ "$ARCH" = "x86_64" ]; then 
         QSVARCH="x8664-unknown-linux-musl"; 
       elif [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then 
         QSV_ARCH="aarch64-unknown-linux-gnu"; 
       else 
         QSVARCH="x8664-unknown-linux-musl"; 
       fi 
    && curl -fsSL "https://github.com/dathere/qsv/releases/download/${QSVVERSION}/qsv-${QSVVERSION}-${QSV_ARCH}.zip" -o /tmp/qsv.zip 
    && unzip -q -o /tmp/qsv.zip qsv -d /usr/local/bin/ 
    && chmod +x /usr/local/bin/qsv
Stage 2: n8n runtime image
FROM docker.n8n.io/n8nio/n8n:latest
USER root
COPY --from=qsv-fetcher /usr/local/bin/qsv /usr/local/bin/qsv
USER node

#### Docker Compose Example (docker-compose.yml)

services:
  n8n:
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8NCOMMUNITYPACKAGES_ENABLED=true
      - DARTFXQSVBIN_PATH=/usr/local/bin/qsv
    volumes:
      - n8n_data:/home/node/.n8n
      - ./data:/data
volumes:
  n8n_data:

> [!TIP]
> Pre-Baking @dartfx/n8n-nodes-qsv (Immutable Containers):
> If you want the community node package pre-installed inside the image (no in-app installation step required), add this to the end of your Dockerfile:
>

> USER node
> RUN cd ~/.n8n && npm install @dartfx/n8n-nodes-qsv
> 

3. Environment Variables (Custom Binary Path)

If qsv is installed in a non-standard location or outside your default $PATH, configure one of the following environment variables on your n8n instance:

| Variable | Description | Default |
| :————————– | :———————————————————— | :—— |
| DARTFXQSVBIN_PATH | Primary override for the QSV binary path used by DartFX nodes | — |
| QSVBINPATH / QSV_PATH | Generic fallback QSV binary path | — |
| (Fallback) | System $PATH resolution | qsv |

Example in .env / Docker Compose:

DARTFXQSVBIN_PATH=/opt/custom/bin/qsv

• Node.js: v22+ (for local development & building)

—

📐 Versioning & Compatibility Strategy

This package follows a decoupled versioning model to support multiple independent node collections (e.g. QSV, Data Artifex FAIRification, Harvester):

1. Package Semantic Versioning (@dartfx/n8n-nodes-qsv)

• The package follows standard Semantic Versioning (starting at 0.1.0).
• Package releases are independent of external CLI versioning numbers.

– Patch (0.1.x): Bug fixes, parameter corrections, and docs across any node collection.
– Minor (0.x.0): Adding new node collections, new operations, or non-breaking features.
– Major (1.0.0): Breaking changes in workflow node interfaces or execution contracts.

2. External Dependency Compatibility Matrix

Node collections that wrap host binaries declare and verify their target CLI versions:

| Node Collection | Host Requirement | Tested / Target CLI Version | Notes |
| :—————————- | :———————- | :————————– | :———————————————— |
| QSV Data Wrangler (qsv) | qsv binary in $PATH | v23.0.1 (>= 23.0.0) | Generated dynamically via pnpm run generate:qsv |
| DartFx FAIR Nodes | None (pure JS/TS) | N/A | Fully self-contained |

3. Workflow Upgrade Behavior in n8n

• Non-breaking releases (Same typeVersion: 1): When an n8n instance installs a package update, existing workflows immediately execute the latest code without manual node re-configuration.
• Breaking schema changes (typeVersion: 2+): If a node introduces breaking parameter schema changes, existing workflows remain safely pinned to typeVersion: 1 until manually upgraded by the user in the n8n canvas.

—

💻 Development & Local Testing

1. Setup & Build

Clone repository
git clone https://github.com/DataArtifex/dartfx-n8n-nodes-qsv.git
cd dartfx-n8n-nodes-qsv
Use supported Node LTS (20/22)
nvm use
Install dependencies
pnpm install
Regenerate QSV node definitions from latest CLI help (if updating QSV commands)
pnpm run generate:qsv
Build package
pnpm run build
Watch mode for active development
pnpm run dev

—

2. Testing with a Local n8n Instance

#### Method A: Docker Volume Mount (Standard & Required for n8n 3.0+)

For n8n 3.0+ or any Docker-based instance, mount your local built repository into the container’s custom nodes folder:

docker run -it --rm 
  --name n8n 
  -p 5678:5678 
  -v ~/.n8n:/home/node/.n8n 
  -v $(pwd):/home/node/.n8n/custom/node_modules/@dartfx/n8n-nodes-qsv:ro 
  -v /path/to/local/qsv:/usr/local/bin/qsv:ro 
  docker.n8n.io/n8nio/n8n:latest

(Note: qsv CLI must be mounted or baked into the Docker container’s $PATH for QSV nodes to execute).

#### Method B: Direct Local Link with pnpm link

(n8n v1 / v2 Bare CLI)

If running an older n8n v1 or v2 instance directly via Node / npm on your host machine:

mkdir -p ~/.n8n/custom
cd ~/.n8n/custom
pnpm link /path/to/dartfx-n8n-nodes-qsv
Start n8n
n8n start

> 🔄 Do changes auto-refresh?
>
> – Recompilation: Running pnpm run dev automatically recompiles TypeScript into dist/ on save.
> – n8n Process: No, n8n caches loaded node modules in memory on startup. You must stop and restart n8n (and reload your browser tab) whenever you update node code for changes to take effect.

#### Method C: Traditional 2-Step npm link (n8n v1 / v2 Bare CLI)

Step 1: Register package globally
cd /path/to/dartfx-n8n-nodes-qsv
npm link
Step 2: Link into n8n custom directory
mkdir -p ~/.n8n/custom
cd ~/.n8n/custom
npm link @dartfx/n8n-nodes-qsv
Start n8n
n8n start

—

3. Troubleshooting & Permissions

#### macOS External Drive Permissions (/Volumes/... / os error 2 / os error 1)

If you see errors like:

Failed executing 'qsv count': io error: No such file or directory (os error 2)
or
io error: Operation not permitted (os error 1)

when accessing files on external drives (under /Volumes//...):

1. macOS restricts child processes spawned by Node from accessing external drives without user consent.
2. Go to System Settings > Privacy & Security > Files and Folders (or Full Disk Access).
3. Locate the application where you run n8n (e.g. Terminal, iTerm2, VS Code, or Ghostty).
4. Ensure Removable Volumes (and/or Full Disk Access) is toggled ON.
5. Restart your terminal / n8n session.

#### Docker Filesystem Paths

If n8n is running inside Docker, host filesystem paths like /Volumes/... or /Users/... are not visible inside the container by default. You must volume-mount the data directory into the container:

-v /Volumes/MyExternalDrive/data:/data:ro

and then reference the path inside n8n as /data/myfile.csv.

—

🚀 Deployment & Publishing (Staging & Production)

🔐 1. Authentication Setup

Publishing scoped packages under the @dartfx organization requires an npm access token:

• In your local environment (user-level ~/.npmrc):

  pnpm config set "//registry.npmjs.org/:authToken" "YOURNPM_TOKEN" --location=global
  

• In GitHub Actions Secrets (CI/CD):

Set DARTFXNODEAUTHTOKEN (or NODEAUTHTOKEN / NPMTOKEN) in Settings > Secrets and variables > Actions.

—

🧪 2. Staging Deployment (Isolated Pre-Release)

To test the package in a live n8n instance without replacing or affecting the production latest release:

1. Simulate package build & inspect tarball (Zero Risk):

   pnpm publish --dry-run --no-git-checks
   

2. Set a pre-release version without creating a Git tag:

   pnpm version 0.2.0-staging.0 --no-git-tag-version
   

3. Publish to npm with the staging dist-tag:

   pnpm publish --tag staging --access public --no-git-checks
   

4. Install and verify in n8n:
In n8n (Settings > Community Nodes > Install), specify:

   @dartfx/n8n-nodes-qsv@staging
   

(or exact version @dartfx/n8n-nodes-qsv@0.2.0-staging.0)

—

🚢 3. Production Deployment

#### Option A: Automated via Git Tag & GitHub Actions (Recommended)

Pushing an annotated version tag triggers the .github/workflows/publish.yml workflow:

1. Clean working tree and run checks
pnpm run lint && pnpm run test && pnpm run build
2. Bump production version and create git tag
pnpm version patch -m "chore(release): %s"  # or minor / major
3. Push commit and tag to GitHub
git push origin main --follow-tags

#### Option B: Promote Existing Staged Version to Production

Promote a tested staging release to latest without rebuilding:

npm dist-tag add @dartfx/n8n-nodes-qsv@0.2.0-staging.0 latest

#### Option C: Manual CLI Production Publish

pnpm publish --tag latest --access public

—

See AGENT.md for full architecture and development workflows, and RELEASING.md for detailed release policies and semver guide.