feat(web-automation): implement milestone M2 mirror and docs
This commit is contained in:
@@ -2,15 +2,25 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
Automate browsing and scraping with Playwright + Camoufox.
|
||||
Automate rendered browsing, scraping, authentication, and multi-step browser flows with Playwright-compatible CloakBrowser.
|
||||
|
||||
## What Ships In Every Variant
|
||||
|
||||
- `browse.ts` for direct navigation and screenshots
|
||||
- `auth.ts` for form and Microsoft SSO login flows
|
||||
- `scrape.ts` for markdown extraction
|
||||
- `flow.ts` for natural-language or JSON browser steps
|
||||
- `extract.js` for one-shot rendered JSON extraction
|
||||
- `check-install.js` for install and wiring validation
|
||||
- `scan-local-app.ts` for configurable local-app smoke scans
|
||||
|
||||
## Requirements
|
||||
|
||||
- Node.js 20+
|
||||
- pnpm
|
||||
- `cloakbrowser`
|
||||
- `playwright-core`
|
||||
- `camoufox-js`
|
||||
- Network access to download Camoufox browser artifacts
|
||||
- Network access to download the CloakBrowser binary on first use
|
||||
|
||||
## Install
|
||||
|
||||
@@ -21,8 +31,9 @@ mkdir -p ~/.codex/skills/web-automation
|
||||
cp -R skills/web-automation/codex/* ~/.codex/skills/web-automation/
|
||||
cd ~/.codex/skills/web-automation/scripts
|
||||
pnpm install
|
||||
pnpm add playwright-core camoufox-js
|
||||
npx camoufox-js fetch
|
||||
npx cloakbrowser install
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
```
|
||||
|
||||
### Claude Code
|
||||
@@ -32,8 +43,9 @@ mkdir -p ~/.claude/skills/web-automation
|
||||
cp -R skills/web-automation/claude-code/* ~/.claude/skills/web-automation/
|
||||
cd ~/.claude/skills/web-automation/scripts
|
||||
pnpm install
|
||||
pnpm add playwright-core camoufox-js
|
||||
npx camoufox-js fetch
|
||||
npx cloakbrowser install
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
```
|
||||
|
||||
### OpenCode
|
||||
@@ -43,25 +55,80 @@ mkdir -p ~/.config/opencode/skills/web-automation
|
||||
cp -R skills/web-automation/opencode/* ~/.config/opencode/skills/web-automation/
|
||||
cd ~/.config/opencode/skills/web-automation/scripts
|
||||
pnpm install
|
||||
pnpm add playwright-core camoufox-js
|
||||
npx camoufox-js fetch
|
||||
npx cloakbrowser install
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
```
|
||||
|
||||
## Verify Installation & Dependencies
|
||||
## Update To The Latest CloakBrowser
|
||||
|
||||
Run inside the installed `scripts/` directory for the variant you are using:
|
||||
|
||||
```bash
|
||||
pnpm up cloakbrowser playwright-core
|
||||
npx cloakbrowser install
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
```
|
||||
|
||||
This repo intentionally treats `cloakbrowser` as a refreshable dependency: update to the latest available compatible release, then regenerate the lockfile from that resolved set.
|
||||
|
||||
## Verify Installation & Wiring
|
||||
|
||||
Run in the installed `scripts/` folder:
|
||||
|
||||
```bash
|
||||
node -e "require.resolve('playwright-core/package.json');require.resolve('camoufox-js/package.json');console.log('OK: playwright-core + camoufox-js installed')"
|
||||
node -e "const fs=require('fs');const t=fs.readFileSync('browse.ts','utf8');if(!/camoufox-js/.test(t)){throw new Error('browse.ts is not configured for Camoufox')}console.log('OK: Camoufox integration detected in browse.ts')"
|
||||
node check-install.js
|
||||
```
|
||||
|
||||
If checks fail, stop and return:
|
||||
Expected checks:
|
||||
|
||||
"Missing dependency/config: web-automation requires `playwright-core` + `camoufox-js` and Camoufox-based scripts. Run setup in this skill, then retry."
|
||||
- `cloakbrowser` and `playwright-core` resolve correctly
|
||||
- `browse.ts` is wired to CloakBrowser
|
||||
- the frozen reference repo + commit recorded in `reference-source.json` are visible to the operator
|
||||
|
||||
If the check fails, stop and return:
|
||||
|
||||
"Missing dependency/config: web-automation requires `cloakbrowser` and `playwright-core` with CloakBrowser-based scripts. Run setup in this skill, then retry."
|
||||
|
||||
If runtime later fails with native-binding issues, run:
|
||||
|
||||
```bash
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `CLOAKBROWSER_PROFILE_PATH`
|
||||
- `CLOAKBROWSER_HEADLESS`
|
||||
- `CLOAKBROWSER_USERNAME`
|
||||
- `CLOAKBROWSER_PASSWORD`
|
||||
|
||||
There are no `CAMOUFOX_*` compatibility aliases in this migration.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
- Browse: `npx tsx browse.ts --url "https://example.com"`
|
||||
- Scrape: `npx tsx scrape.ts --url "https://example.com" --mode main --output page.md`
|
||||
- Auth: `npx tsx auth.ts --url "https://example.com/login"`
|
||||
- Scrape markdown: `npx tsx scrape.ts --url "https://example.com" --mode main --output page.md`
|
||||
- Authenticate: `npx tsx auth.ts --url "https://example.com/login"`
|
||||
- Natural-language flow: `npx tsx flow.ts --instruction 'go to https://example.com then click on "Login"'`
|
||||
- JSON extract: `node extract.js "https://example.com"`
|
||||
- Local smoke scan: `SCAN_BASE_URL=http://localhost:3000 SCAN_ROUTES=/,/dashboard npx tsx scan-local-app.ts`
|
||||
|
||||
## Local App Smoke Scan
|
||||
|
||||
`scan-local-app.ts` is generic. Configure it with:
|
||||
|
||||
- `SCAN_BASE_URL`
|
||||
- `SCAN_LOGIN_PATH`
|
||||
- `SCAN_USERNAME`
|
||||
- `SCAN_PASSWORD`
|
||||
- `SCAN_USERNAME_SELECTOR`
|
||||
- `SCAN_PASSWORD_SELECTOR`
|
||||
- `SCAN_SUBMIT_SELECTOR`
|
||||
- `SCAN_ROUTES`
|
||||
- `SCAN_REPORT_PATH`
|
||||
- `SCAN_HEADLESS`
|
||||
|
||||
If `SCAN_USERNAME` or `SCAN_PASSWORD` are omitted, the script falls back to `CLOAKBROWSER_USERNAME` and `CLOAKBROWSER_PASSWORD`.
|
||||
|
||||
Reference in New Issue
Block a user