feat(web-automation): implement milestone M1 canonical codex migration
This commit is contained in:
Stefano Fiorini
@@ -1,48 +1,101 @@
|
||||
---
|
||||
name: web-automation
|
||||
description: Browse and scrape web pages using Playwright with Camoufox anti-detection browser. Use when automating web workflows, extracting page content to markdown, handling authenticated sessions, or scraping websites with bot protection.
|
||||
description: Browse and scrape web pages using Playwright-compatible CloakBrowser. Use when automating web workflows, extracting rendered page content, handling authenticated sessions, or running multi-step browser flows.
|
||||
---
|
||||
|
||||
# Web Automation with Camoufox (Codex)
|
||||
# Web Automation with CloakBrowser (Codex)
|
||||
|
||||
Automated web browsing and scraping using Playwright with Camoufox anti-detection browser.
|
||||
Automated web browsing and scraping using Playwright-compatible CloakBrowser with two execution paths:
|
||||
|
||||
- one-shot extraction via `extract.js`
|
||||
- broader stateful automation via `auth.ts`, `browse.ts`, `flow.ts`, `scan-local-app.ts`, and `scrape.ts`
|
||||
|
||||
## Requirements
|
||||
|
||||
- Node.js 20+
|
||||
- pnpm
|
||||
- Network access to download browser binaries
|
||||
- Network access to download the CloakBrowser binary on first use
|
||||
|
||||
## First-Time Setup
|
||||
|
||||
```bash
|
||||
cd ~/.codex/skills/web-automation/scripts
|
||||
pnpm install
|
||||
npx camoufox-js fetch
|
||||
npx cloakbrowser install
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
```
|
||||
|
||||
## Updating CloakBrowser
|
||||
|
||||
```bash
|
||||
cd ~/.codex/skills/web-automation/scripts
|
||||
pnpm up cloakbrowser playwright-core
|
||||
npx cloakbrowser install
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
```
|
||||
|
||||
## Prerequisite Check (MANDATORY)
|
||||
|
||||
Before running any automation, verify Playwright + Camoufox dependencies are installed and scripts are configured to use Camoufox.
|
||||
Before running automation, verify CloakBrowser and Playwright Core are installed and wired correctly.
|
||||
|
||||
```bash
|
||||
cd ~/.codex/skills/web-automation/scripts
|
||||
node -e "require.resolve('playwright-core/package.json');require.resolve('camoufox-js/package.json');console.log('OK: playwright-core + camoufox-js installed')"
|
||||
node -e "const fs=require('fs');const t=fs.readFileSync('browse.ts','utf8');if(!/camoufox-js/.test(t)){throw new Error('browse.ts is not configured for Camoufox')}console.log('OK: Camoufox integration detected in browse.ts')"
|
||||
node check-install.js
|
||||
```
|
||||
|
||||
If any check fails, stop and return:
|
||||
`check-install.js` also prints the frozen reference repo + commit recorded in `reference-source.json`, so operators can confirm the canonical import source before using the skill.
|
||||
|
||||
"Missing dependency/config: web-automation requires `playwright-core` + `camoufox-js` and Camoufox-based scripts. Run setup in this skill, then retry."
|
||||
If the check fails, stop and return:
|
||||
|
||||
"Missing dependency/config: web-automation requires `cloakbrowser` and `playwright-core` with CloakBrowser-based scripts. Run setup in this skill, then retry."
|
||||
|
||||
If runtime fails with missing native bindings for `better-sqlite3` or `esbuild`, run:
|
||||
|
||||
```bash
|
||||
cd ~/.codex/skills/web-automation/scripts
|
||||
pnpm approve-builds
|
||||
pnpm rebuild better-sqlite3 esbuild
|
||||
```
|
||||
|
||||
## When To Use Which Command
|
||||
|
||||
- Use `node extract.js "<URL>"` for a one-shot rendered fetch with JSON output.
|
||||
- Use `npx tsx scrape.ts ...` when you need markdown extraction, Readability cleanup, or selector-based scraping.
|
||||
- Use `npx tsx browse.ts ...`, `auth.ts`, or `flow.ts` when the task needs login handling, persistent sessions, clicks, typing, screenshots, or multi-step navigation.
|
||||
- Use `npx tsx scan-local-app.ts` when you need a configurable local-app smoke pass driven by `SCAN_*` and `CLOAKBROWSER_*` environment variables.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
- Install check: `node check-install.js`
|
||||
- One-shot JSON extract: `node extract.js "https://example.com"`
|
||||
- Browse page: `npx tsx browse.ts --url "https://example.com"`
|
||||
- Scrape markdown: `npx tsx scrape.ts --url "https://example.com" --mode main --output page.md`
|
||||
- Authenticate: `npx tsx auth.ts --url "https://example.com/login"`
|
||||
- Natural-language flow: `npx tsx flow.ts --instruction 'go to https://example.com then click on "Login" then type "user@example.com" in #email then press enter'`
|
||||
- Local app smoke scan: `SCAN_BASE_URL=http://localhost:3000 SCAN_ROUTES=/,/dashboard npx tsx scan-local-app.ts`
|
||||
|
||||
## Local App Smoke Scan
|
||||
|
||||
`scan-local-app.ts` is intentionally generic. Configure it with environment variables instead of editing the file:
|
||||
|
||||
- `SCAN_BASE_URL`
|
||||
- `SCAN_LOGIN_PATH`
|
||||
- `SCAN_USERNAME`
|
||||
- `SCAN_PASSWORD`
|
||||
- `SCAN_USERNAME_SELECTOR`
|
||||
- `SCAN_PASSWORD_SELECTOR`
|
||||
- `SCAN_SUBMIT_SELECTOR`
|
||||
- `SCAN_ROUTES`
|
||||
- `SCAN_REPORT_PATH`
|
||||
- `SCAN_HEADLESS`
|
||||
|
||||
If `SCAN_USERNAME` or `SCAN_PASSWORD` are omitted, the script falls back to `CLOAKBROWSER_USERNAME` and `CLOAKBROWSER_PASSWORD`.
|
||||
|
||||
## Notes
|
||||
|
||||
- Sessions persist in Camoufox profile storage.
|
||||
- Sessions persist in CloakBrowser profile storage.
|
||||
- Use `--wait` for dynamic pages.
|
||||
- Use `--mode selector --selector "..."` for targeted extraction.
|
||||
- `extract.js` keeps a bounded stealth/rendered fetch path without needing a long-lived automation session.
|
||||
|
||||
Reference in New Issue
Block a user