stefano
251148c3ff
## Summary - add repository-wide quality tooling and verification scaffolding, including CI workflows, pnpm workspace setup, ESLint/Prettier/markdown checks, and generated-output verification helpers - reorganize skill sources and generation flow by introducing canonical `_source` variants, generator/manifests, reusable helper abstractions, and shared web-automation/browser utilities - clean up and expand documentation so the root README flows into docs and skill docs, with clearer development, reviewer, installer, and workflow guidance ## Notable changes - docs flow and consistency cleanup across `README.md`, `docs/README.md`, and related docs - new scripts for `check`, docs verification, generated-file verification, shell portability, and safe directory replacement - refactors in Atlassian and web-automation skill runtimes to reduce duplication and centralize reusable code - changelog, development documentation, and CI surface updates ## Test Plan - [ ] `pnpm run check` - [ ] review generated/manifests and skill sync outputs - [ ] smoke-check docs flow from `README.md` to `docs/README.md` to skill docs ## Notes - this branch currently includes tracked `skills/web-automation/shared/node_modules` content that should be reviewed carefully as potentially noisy/accidental committed artifacts Co-authored-by: Stefano Fiorini <stefano.fiorini@firsthorizon.com> Reviewed-on: #1
177 lines
4.5 KiB
Markdown
177 lines
4.5 KiB
Markdown
# WEB-AUTOMATION
|
|
|
|
## Purpose
|
|
|
|
Automate rendered browsing, scraping, authentication, and multi-step browser flows with Playwright-compatible CloakBrowser.
|
|
|
|
## What Ships In Every Variant
|
|
|
|
- `browse.ts` for direct navigation and screenshots
|
|
- `auth.ts` for form and Microsoft SSO login flows
|
|
- `scrape.ts` for markdown extraction
|
|
- `flow.ts` for natural-language or JSON browser steps
|
|
- `extract.js` for one-shot rendered JSON extraction
|
|
- `check-install.js` for install and wiring validation
|
|
- `scan-local-app.ts` for configurable local-app smoke scans
|
|
|
|
## Requirements
|
|
|
|
- Node.js 20+
|
|
- pnpm
|
|
- `cloakbrowser`
|
|
- `playwright-core`
|
|
- Network access to download the CloakBrowser binary on first use
|
|
|
|
## Install
|
|
|
|
### Codex
|
|
|
|
```bash
|
|
mkdir -p ~/.codex/skills/web-automation
|
|
cp -R skills/web-automation/codex/* ~/.codex/skills/web-automation/
|
|
cd ~/.codex/skills/web-automation/scripts
|
|
pnpm install
|
|
npx cloakbrowser install
|
|
pnpm approve-builds
|
|
pnpm rebuild better-sqlite3 esbuild
|
|
```
|
|
|
|
### Claude Code
|
|
|
|
```bash
|
|
mkdir -p ~/.claude/skills/web-automation
|
|
cp -R skills/web-automation/claude-code/* ~/.claude/skills/web-automation/
|
|
cd ~/.claude/skills/web-automation/scripts
|
|
pnpm install
|
|
npx cloakbrowser install
|
|
pnpm approve-builds
|
|
pnpm rebuild better-sqlite3 esbuild
|
|
```
|
|
|
|
### Cursor
|
|
|
|
Repo-local install:
|
|
|
|
```bash
|
|
mkdir -p .cursor/skills/web-automation
|
|
cp -R skills/web-automation/cursor/* .cursor/skills/web-automation/
|
|
cd .cursor/skills/web-automation/scripts
|
|
pnpm install
|
|
npx cloakbrowser install
|
|
pnpm approve-builds
|
|
pnpm rebuild better-sqlite3 esbuild
|
|
```
|
|
|
|
Global installs use `~/.cursor/skills/web-automation/` instead.
|
|
|
|
### OpenCode
|
|
|
|
```bash
|
|
mkdir -p ~/.config/opencode/skills/web-automation
|
|
cp -R skills/web-automation/opencode/* ~/.config/opencode/skills/web-automation/
|
|
cd ~/.config/opencode/skills/web-automation/scripts
|
|
pnpm install
|
|
npx cloakbrowser install
|
|
pnpm approve-builds
|
|
pnpm rebuild better-sqlite3 esbuild
|
|
```
|
|
|
|
### Pi
|
|
|
|
Recommended full Pi package install:
|
|
|
|
```bash
|
|
./scripts/install-pi-package.sh --global
|
|
# or, for project-local Pi package install
|
|
./scripts/install-pi-package.sh --local
|
|
```
|
|
|
|
Manual single-skill Pi install from the package mirror:
|
|
|
|
```bash
|
|
pnpm run sync:pi
|
|
mkdir -p .pi/skills/web-automation
|
|
cp -R pi-package/skills/web-automation/* .pi/skills/web-automation/
|
|
cd .pi/skills/web-automation/scripts
|
|
pnpm install --frozen-lockfile
|
|
npx cloakbrowser install
|
|
pnpm approve-builds
|
|
pnpm rebuild better-sqlite3 esbuild
|
|
```
|
|
|
|
Global manual installs use `~/.pi/agent/skills/web-automation/` instead of `.pi/skills/web-automation/`.
|
|
|
|
## Update To The Latest CloakBrowser
|
|
|
|
Run inside the installed `scripts/` directory for the variant you are using:
|
|
|
|
```bash
|
|
pnpm up cloakbrowser playwright-core
|
|
npx cloakbrowser install
|
|
pnpm approve-builds
|
|
pnpm rebuild better-sqlite3 esbuild
|
|
```
|
|
|
|
This repo intentionally treats `cloakbrowser` as a refreshable dependency: update to the latest
|
|
available compatible release, then regenerate the lockfile from that resolved set.
|
|
|
|
## Verify Installation & Wiring
|
|
|
|
Run in the installed `scripts/` folder:
|
|
|
|
```bash
|
|
node check-install.js
|
|
```
|
|
|
|
Expected checks:
|
|
|
|
- `cloakbrowser` and `playwright-core` resolve correctly
|
|
- `browse.ts` is wired to CloakBrowser
|
|
|
|
If the check fails, stop and return:
|
|
|
|
"Missing dependency/config: web-automation requires `cloakbrowser` and `playwright-core` with
|
|
CloakBrowser-based scripts. Run setup in this skill, then retry."
|
|
|
|
If runtime later fails with native-binding issues, run:
|
|
|
|
```bash
|
|
pnpm approve-builds
|
|
pnpm rebuild better-sqlite3 esbuild
|
|
```
|
|
|
|
## Environment Variables
|
|
|
|
- `CLOAKBROWSER_PROFILE_PATH`
|
|
- `CLOAKBROWSER_HEADLESS`
|
|
- `CLOAKBROWSER_USERNAME`
|
|
- `CLOAKBROWSER_PASSWORD`
|
|
|
|
There are no `CAMOUFOX_*` compatibility aliases in this migration.
|
|
|
|
## Usage Examples
|
|
|
|
- Browse: `npx tsx browse.ts --url "https://example.com"`
|
|
- Scrape markdown: `npx tsx scrape.ts --url "https://example.com" --mode main --output page.md`
|
|
- Authenticate: `npx tsx auth.ts --url "https://example.com/login"`
|
|
- Natural-language flow: `npx tsx flow.ts --instruction 'go to https://example.com then click on "Login"'`
|
|
- JSON extract: `node extract.js "https://example.com"`
|
|
- Local smoke scan: `SCAN_BASE_URL=http://localhost:3000 SCAN_ROUTES=/,/dashboard npx tsx scan-local-app.ts`
|
|
|
|
## Local App Smoke Scan
|
|
|
|
`scan-local-app.ts` is generic. Configure it with:
|
|
|
|
- `SCAN_BASE_URL`
|
|
- `SCAN_LOGIN_PATH`
|
|
- `SCAN_USERNAME`
|
|
- `SCAN_PASSWORD`
|
|
- `SCAN_USERNAME_SELECTOR`
|
|
- `SCAN_PASSWORD_SELECTOR`
|
|
- `SCAN_SUBMIT_SELECTOR`
|
|
- `SCAN_ROUTES`
|
|
- `SCAN_REPORT_PATH`
|
|
- `SCAN_HEADLESS`
|
|
|
|
If `SCAN_USERNAME` or `SCAN_PASSWORD` are omitted, the script falls back to `CLOAKBROWSER_USERNAME` and `CLOAKBROWSER_PASSWORD`.
|