refactor: consolidate web scraping into web-automation
This commit is contained in:
Stefano Fiorini
@@ -1,20 +1,20 @@
|
||||
---
|
||||
name: web-automation
|
||||
description: Browse and scrape web pages using Playwright with Camoufox anti-detection browser. Use when automating web workflows, extracting page content to markdown, handling authenticated sessions, or scraping websites with bot protection.
|
||||
description: Browse and scrape web pages using Playwright with Camoufox anti-detection browser. Use when automating web workflows, extracting rendered page content, handling authenticated sessions, or scraping websites with bot protection.
|
||||
---
|
||||
|
||||
# Web Automation with Camoufox (Codex)
|
||||
|
||||
Automated web browsing and scraping using Playwright with Camoufox anti-detection browser.
|
||||
Automated web browsing and scraping using Playwright with two execution paths under one skill:
|
||||
|
||||
## Routing Rule
|
||||
- one-shot extraction via `extract.js`
|
||||
- broader stateful automation via Camoufox and the existing `auth.ts`, `browse.ts`, `flow.ts`, and `scrape.ts`
|
||||
|
||||
Before using this skill, classify the task:
|
||||
## When To Use Which Command
|
||||
|
||||
- If the task is one-shot page extraction for title/content from a single URL, use `~/.openclaw/workspace/skills/playwright-safe/SKILL.md` instead.
|
||||
- If the task needs a multi-step browser flow, authenticated session handling, or interactive navigation/click/type behavior, use this `web-automation` skill.
|
||||
|
||||
Do not use `web-automation` for simple single-page extraction when `playwright-safe` is available.
|
||||
- Use `node scripts/extract.js "<URL>"` for one-shot extraction from a single URL when you need rendered content, bounded stealth behavior, and JSON output.
|
||||
- Use `npx tsx scrape.ts ...` when you need markdown output, Readability extraction, full-page cleanup, or selector-based scraping.
|
||||
- Use `npx tsx browse.ts ...`, `auth.ts`, or `flow.ts` when the task needs interactive navigation, persistent sessions, login handling, click/type actions, or multi-step workflows.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -27,6 +27,7 @@ Do not use `web-automation` for simple single-page extraction when `playwright-s
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/skills/web-automation/scripts
|
||||
pnpm install
|
||||
npx playwright install chromium
|
||||
npx camoufox-js fetch
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
@@ -38,13 +39,13 @@ Before running any automation, verify Playwright + Camoufox dependencies are ins
|
||||
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/skills/web-automation/scripts
|
||||
node -e "require.resolve('playwright-core/package.json');require.resolve('camoufox-js/package.json');console.log('OK: playwright-core + camoufox-js installed')"
|
||||
node -e "require.resolve('playwright/package.json');require.resolve('playwright-core/package.json');require.resolve('camoufox-js/package.json');console.log('OK: playwright + playwright-core + camoufox-js installed')"
|
||||
node -e "const fs=require('fs');const t=fs.readFileSync('browse.ts','utf8');if(!/camoufox-js/.test(t)){throw new Error('browse.ts is not configured for Camoufox')}console.log('OK: Camoufox integration detected in browse.ts')"
|
||||
```
|
||||
|
||||
If any check fails, stop and return:
|
||||
|
||||
"Missing dependency/config: web-automation requires `playwright-core` + `camoufox-js` and Camoufox-based scripts. Run setup in this skill, then retry."
|
||||
"Missing dependency/config: web-automation requires `playwright`, `playwright-core`, and `camoufox-js` with Camoufox-based scripts. Run setup in this skill, then retry."
|
||||
|
||||
If runtime fails with missing native bindings for `better-sqlite3` or `esbuild`, run:
|
||||
|
||||
@@ -56,11 +57,35 @@ pnpm rebuild better-sqlite3 esbuild
|
||||
|
||||
## Quick Reference
|
||||
|
||||
- One-shot JSON extract: `node scripts/extract.js "https://example.com"`
|
||||
- Browse page: `npx tsx browse.ts --url "https://example.com"`
|
||||
- Scrape markdown: `npx tsx scrape.ts --url "https://example.com" --mode main --output page.md`
|
||||
- Authenticate: `npx tsx auth.ts --url "https://example.com/login"`
|
||||
- Natural-language flow: `npx tsx flow.ts --instruction 'go to https://example.com then click on "Login" then type "user@example.com" in #email then press enter'`
|
||||
|
||||
## One-shot extraction
|
||||
|
||||
Use `extract.js` when you need a single page fetch with JavaScript rendering and lightweight anti-bot shaping, but not a full automation session.
|
||||
|
||||
```bash
|
||||
node scripts/extract.js "https://example.com"
|
||||
WAIT_TIME=5000 node scripts/extract.js "https://example.com"
|
||||
SCREENSHOT_PATH=/tmp/page.png SAVE_HTML=true node scripts/extract.js "https://example.com"
|
||||
```
|
||||
|
||||
Output is JSON only and includes fields such as:
|
||||
|
||||
- `requestedUrl`
|
||||
- `finalUrl`
|
||||
- `title`
|
||||
- `content`
|
||||
- `metaDescription`
|
||||
- `status`
|
||||
- `elapsedSeconds`
|
||||
- `challengeDetected`
|
||||
- optional `screenshot`
|
||||
- optional `htmlFile`
|
||||
|
||||
## General flow runner
|
||||
|
||||
Use `flow.ts` for multi-step commands in plain language (go/click/type/press/wait/screenshot).
|
||||
@@ -76,3 +101,4 @@ npx tsx flow.ts --instruction 'go to https://search.fiorinis.com then type "pipp
|
||||
- Sessions persist in Camoufox profile storage.
|
||||
- Use `--wait` for dynamic pages.
|
||||
- Use `--mode selector --selector "..."` for targeted extraction.
|
||||
- `extract.js` keeps stealth and bounded anti-bot shaping while keeping the Chromium sandbox enabled.
|
||||
|
||||
Reference in New Issue
Block a user