web-automation

Automated web browsing and scraping using Playwright with Camoufox anti-detection browser.

What this skill is for

Automating web workflows
Authenticated session flows (logins/cookies)
Extracting page content to markdown
Working with bot-protected or dynamic pages

Requirements

Node.js 20+
pnpm
Network access to download browser binaries

First-time setup

cd ~/.openclaw/workspace/skills/web-automation/scripts
pnpm install
npx camoufox-js fetch
pnpm approve-builds
pnpm rebuild better-sqlite3 esbuild

System libraries (for OpenClaw Docker builds)

export OPENCLAW_DOCKER_APT_PACKAGES="ffmpeg jq curl libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 libasound2"

Native module note

If pnpm install warns that build scripts were ignored for native modules such as better-sqlite3 or esbuild, run:

pnpm approve-builds
pnpm rebuild better-sqlite3 esbuild

Without this, browse.ts and scrape.ts may fail before launch because the native bindings are missing.

Common commands

# Browse a page
npx tsx browse.ts --url "https://example.com"

# Scrape markdown
npx tsx scrape.ts --url "https://example.com" --mode main --output page.md

# Authenticate flow
npx tsx auth.ts --url "https://example.com/login"

# General natural-language browser flow
npx tsx flow.ts --instruction 'go to https://search.fiorinis.com then type "pippo" then press enter then wait 2s'

Natural-language flow runner (`flow.ts`)

Use flow.ts when you want a general command style like:

"go to this site"
"find this button and click it"
"type this and press enter"

Example

npx tsx flow.ts --instruction 'go to https://example.com then click on "Sign in" then type "stef@example.com" in #email then press enter'

You can also use JSON steps for deterministic runs:

npx tsx flow.ts --steps '[{"action":"goto","url":"https://example.com"},{"action":"click","text":"Sign in"}]'

2.0 KiB Raw Blame History