docs: add web automation consolidation plans
This commit is contained in:
21
docs/plans/2026-03-10-web-automation-consolidation-design.md
Normal file
21
docs/plans/2026-03-10-web-automation-consolidation-design.md
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
# Web Automation Consolidation Design
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
Consolidate `playwright-safe` into `web-automation` so the repo exposes a single web skill. Keep the proven one-shot extractor behavior, rename it to `extract.js`, and remove the separate `playwright-safe` skill and docs.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
`web-automation` remains the only published skill. It will expose two capability bands under one skill: one-shot extraction via `scripts/extract.js`, and broader stateful automation via the existing `auth.ts`, `browse.ts`, `flow.ts`, and `scrape.ts` commands. The one-shot extractor will keep the current safe Playwright behavior: single URL, JSON output, bounded stealth/anti-bot handling, and no sandbox-disabling Chromium flags.
|
||||||
|
|
||||||
|
## Migration
|
||||||
|
- Copy the working extractor into `skills/web-automation/scripts/extract.js`
|
||||||
|
- Update `skills/web-automation/SKILL.md` and `docs/web-automation.md` to describe both one-shot extraction and full automation
|
||||||
|
- Remove `skills/playwright-safe/`
|
||||||
|
- Remove `docs/playwright-safe.md`
|
||||||
|
- Remove README/doc index references to `playwright-safe`
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- `node skills/web-automation/scripts/extract.js` -> JSON error for missing URL
|
||||||
|
- `node skills/web-automation/scripts/extract.js ftp://example.com` -> JSON error for invalid scheme
|
||||||
|
- `node skills/web-automation/scripts/extract.js https://example.com` -> valid JSON result with title/status
|
||||||
|
- Repo text scan confirms no remaining published references directing users to `playwright-safe`
|
||||||
|
- Commit, push, and clean up the worktree
|
||||||
132
docs/plans/2026-03-10-web-automation-consolidation.md
Normal file
132
docs/plans/2026-03-10-web-automation-consolidation.md
Normal file
@@ -0,0 +1,132 @@
|
|||||||
|
# Web Automation Consolidation Implementation Plan
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||||
|
|
||||||
|
**Goal:** Consolidate the separate `playwright-safe` skill into `web-automation` and publish a single web skill with both one-shot extraction and broader automation.
|
||||||
|
|
||||||
|
**Architecture:** Move the proven safe one-shot extractor into `skills/web-automation/scripts/extract.js`, update `web-automation` docs to expose it as the simple path, and remove the separate `playwright-safe` skill and docs. Keep the extractor behavior unchanged except for its new location/name.
|
||||||
|
|
||||||
|
**Tech Stack:** Node.js, Playwright, Camoufox skill docs, git
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Create isolated worktree
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: repo git metadata only
|
||||||
|
|
||||||
|
**Step 1: Create worktree**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
git -C /Users/stefano/.openclaw/workspace/projects/stef-openclaw-skills worktree add /Users/stefano/.openclaw/workspace/projects/stef-openclaw-skills/.worktrees/web-automation-consolidation -b feature/web-automation-consolidation
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Verify baseline**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
git -C /Users/stefano/.openclaw/workspace/projects/stef-openclaw-skills/.worktrees/web-automation-consolidation status --short --branch
|
||||||
|
```
|
||||||
|
Expected: clean feature branch
|
||||||
|
|
||||||
|
### Task 2: Move the extractor into web-automation
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `skills/web-automation/scripts/extract.js`
|
||||||
|
- Read: `skills/playwright-safe/scripts/playwright-safe.js`
|
||||||
|
|
||||||
|
**Step 1: Copy the extractor**
|
||||||
|
- Copy the proven script content into `skills/web-automation/scripts/extract.js`
|
||||||
|
- Adjust only relative paths/messages if needed
|
||||||
|
|
||||||
|
**Step 2: Preserve behavior**
|
||||||
|
- Keep JSON-only output
|
||||||
|
- Keep URL validation
|
||||||
|
- Keep stealth/anti-bot behavior
|
||||||
|
- Keep sandbox enabled
|
||||||
|
|
||||||
|
### Task 3: Update skill and docs
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `skills/web-automation/SKILL.md`
|
||||||
|
- Modify: `docs/web-automation.md`
|
||||||
|
- Modify: `README.md`
|
||||||
|
- Modify: `docs/README.md`
|
||||||
|
- Delete: `skills/playwright-safe/SKILL.md`
|
||||||
|
- Delete: `skills/playwright-safe/package.json`
|
||||||
|
- Delete: `skills/playwright-safe/package-lock.json`
|
||||||
|
- Delete: `skills/playwright-safe/.gitignore`
|
||||||
|
- Delete: `skills/playwright-safe/scripts/playwright-safe.js`
|
||||||
|
- Delete: `docs/playwright-safe.md`
|
||||||
|
|
||||||
|
**Step 1: Update docs**
|
||||||
|
- Make `web-automation` the only published web skill
|
||||||
|
- Document `extract.js` as the one-shot extraction path
|
||||||
|
- Remove published references to `playwright-safe`
|
||||||
|
|
||||||
|
**Step 2: Remove redundant skill**
|
||||||
|
- Delete the separate `playwright-safe` skill files and doc
|
||||||
|
|
||||||
|
### Task 4: Verify behavior
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Test: `skills/web-automation/scripts/extract.js`
|
||||||
|
|
||||||
|
**Step 1: Missing URL check**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
cd /Users/stefano/.openclaw/workspace/projects/stef-openclaw-skills/.worktrees/web-automation-consolidation && node skills/web-automation/scripts/extract.js
|
||||||
|
```
|
||||||
|
Expected: JSON error about missing URL
|
||||||
|
|
||||||
|
**Step 2: Invalid scheme check**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
cd /Users/stefano/.openclaw/workspace/projects/stef-openclaw-skills/.worktrees/web-automation-consolidation && node skills/web-automation/scripts/extract.js ftp://example.com
|
||||||
|
```
|
||||||
|
Expected: JSON error about only http/https URLs allowed
|
||||||
|
|
||||||
|
**Step 3: Smoke test**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
cd /Users/stefano/.openclaw/workspace/projects/stef-openclaw-skills/.worktrees/web-automation-consolidation && node skills/web-automation/scripts/extract.js https://example.com
|
||||||
|
```
|
||||||
|
Expected: JSON with title `Example Domain`, status `200`, and no sandbox-disabling flags in code
|
||||||
|
|
||||||
|
**Step 4: Reference scan**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
cd /Users/stefano/.openclaw/workspace/projects/stef-openclaw-skills/.worktrees/web-automation-consolidation && rg -n "playwright-safe" README.md docs skills
|
||||||
|
```
|
||||||
|
Expected: no remaining published references, or only intentional historical plan docs
|
||||||
|
|
||||||
|
### Task 5: Commit, push, and clean up
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: git history only
|
||||||
|
|
||||||
|
**Step 1: Commit**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
git add skills/web-automation docs README.md
|
||||||
|
git commit -m "refactor: consolidate web scraping into web-automation"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Push**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
git push -u origin feature/web-automation-consolidation
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3: Merge and cleanup**
|
||||||
|
- Fast-forward or merge to `main`
|
||||||
|
- Push `main`
|
||||||
|
- Remove the worktree
|
||||||
|
- Delete the feature branch
|
||||||
Reference in New Issue
Block a user