Compare commits

...

2 Commits

51 changed files with 10592 additions and 1 deletions

View File

@@ -1,3 +1,54 @@
# ai-coding-skills # ai-coding-skills
Cross-agent coding skills for Codex, Claude Code, and OpenCode Cross-agent skill collection for **Codex**, **Claude Code**, and **OpenCode**.
This repo is organized similarly to `obra/superpowers` and is designed to scale to many skills over time.
## Goals
- Keep skills portable across coding agents.
- Keep each skill self-contained (`skills/<skill-name>/<agent-variant>/SKILL.md` + resources).
## Repository Layout
```text
ai-coding-skills/
├── README.md
├── docs/
│ └── install.md
├── skills/
│ ├── _template/
│ │ └── SKILL.md
│ ├── create-plan/
│ │ ├── codex/
│ │ ├── claude-code/
│ │ └── opencode/
│ └── web-automation/
│ ├── codex/
│ ├── claude-code/
│ └── opencode/
├── .codex/
├── .claude-plugin/
├── .opencode/
│ └── plugins/
├── commands/
├── hooks/
└── tests/
```
## Skills
| Skill | Agent Variant | Purpose | Status |
|---|---|---|---|
| create-plan | codex | Structured planning with milestones + runbook-first execution workflow | Ready |
| create-plan | claude-code | Structured planning with milestones + runbook-first execution workflow | Ready |
| create-plan | opencode | Structured planning with milestones + runbook-first execution workflow | Ready |
| web-automation | codex | Playwright + Camoufox browsing/scraping/auth automation | Ready |
| web-automation | claude-code | Playwright + Camoufox browsing/scraping/auth automation | Ready |
| web-automation | opencode | Playwright + Camoufox browsing/scraping/auth automation | Ready |
See install instructions: `docs/install.md`
## Compatibility Policy
Each skill should explicitly document agent compatibility and any prerequisites directly in its own `SKILL.md`.

86
docs/install.md Normal file
View File

@@ -0,0 +1,86 @@
# Installation
Install from this repo similarly to superpowers: clone once, then copy/symlink agent-specific skills.
## 1) Clone
```bash
git clone https://git.fiorinis.com/Home/ai-coding-skills.git
cd ai-coding-skills
```
## 2) Install `create-plan`
### Codex
```bash
mkdir -p ~/.codex/skills/create-plan
cp -R skills/create-plan/codex/* ~/.codex/skills/create-plan/
```
### Claude Code
```bash
mkdir -p ~/.claude/skills/create-plan
cp -R skills/create-plan/claude-code/* ~/.claude/skills/create-plan/
```
### OpenCode
Use your OpenCode skills/plugin path and copy the opencode variant.
```bash
# Example path (adjust for your environment)
mkdir -p ~/.opencode/skills/create-plan
cp -R skills/create-plan/opencode/* ~/.opencode/skills/create-plan/
```
## 3) Install `web-automation`
### Codex
```bash
mkdir -p ~/.codex/skills/web-automation
cp -R skills/web-automation/codex/* ~/.codex/skills/web-automation/
cd ~/.codex/skills/web-automation/scripts
pnpm install
npx camoufox-js fetch
```
### Claude Code
```bash
mkdir -p ~/.claude/skills/web-automation
cp -R skills/web-automation/claude-code/* ~/.claude/skills/web-automation/
cd ~/.claude/skills/web-automation/scripts
pnpm install
npx camoufox-js fetch
```
### OpenCode
```bash
mkdir -p ~/.opencode/skills/web-automation
cp -R skills/web-automation/opencode/* ~/.opencode/skills/web-automation/
cd ~/.opencode/skills/web-automation/scripts
pnpm install
npx camoufox-js fetch
```
Requirements for `web-automation`:
- Node.js 20+
- pnpm
- Browser download/network access for Camoufox
## 4) Verify
- Ensure `SKILL.md` exists in the destination folder.
- Ensure `templates/` files (create-plan) or `scripts/` files (web-automation) are present.
- Start a new agent session and ask:
- "create a plan for ..." (create-plan)
- "scrape this page ..." (web-automation)
## Notes
- `docs/plans` / `docs/plan` can be gitignored by design.
- Plan artifacts are intentionally local.

View File

@@ -0,0 +1,81 @@
---
name: create-plan
description: Use when starting a new feature, project, or complex task that needs structured planning with milestones, bite-sized stories, and resumable execution context. Trigger for "create a plan", "make a plan", "plan this", "start planning", and similar requests.
---
# Create Plan (Claude Code)
Create and maintain a local plan folder under `docs/plans/` (or `docs/plan/` if that repository convention is already in use).
## Prerequisite Check (MANDATORY)
This skill depends on Superpowers planning skills. Before proceeding, verify required dependencies exist.
Required:
- Superpowers repo: `https://github.com/obra/superpowers`
- `brainstorming` skill
- `writing-plans` skill
If any dependency is missing, stop immediately and return:
"Missing dependency: Superpowers planning skills are required (`brainstorming`, `writing-plans`). Install from https://github.com/obra/superpowers, then retry."
## Process
### Phase 1: Analyze
- Explore the codebase and existing patterns.
### Phase 2: Gather Requirements
- Ask questions one at a time until user says ready.
- Confirm scope, constraints, success criteria, dependencies.
### Phase 3: Design
- Invoke `superpowers:brainstorming`.
- Propose 2-3 approaches and recommend one.
### Phase 4: Plan
- Invoke `superpowers:writing-plans`.
- Break into milestones and bite-sized stories.
### Phase 5: Generate Plan Files (MANDATORY)
Create `docs/plans/YYYY-MM-DD-<short-title>/` with all files below:
1. `original-plan.md` - copy of original planner-generated plan.
2. `final-transcript.md` - copy of final planning transcript used to reach approved plan.
3. `milestone-plan.md` - full implementation spec (from template).
4. `story-tracker.md` - story/milestone status tracker (from template).
5. `continuation-runbook.md` - execution instructions and context (from template).
Use templates from this skill's `templates/` folder.
### Phase 6: Handoff
Always instruct the executing agent:
> Read `docs/plans/YYYY-MM-DD-<short-title>/continuation-runbook.md` first, then execute from that folder.
Do not rely on planner-private files during implementation.
## Execution Rules to Include in Plan (MANDATORY)
- Run lint/typecheck/tests after each milestone.
- Prefer linting changed files only for speed.
- Commit locally after each completed milestone (**do not push**).
- Stop and ask user for feedback.
- Apply feedback, rerun checks, and commit again.
- Move to next milestone only after user approval.
- After all milestones are completed and approved, ask permission to push.
- Only after approved push: mark plan as completed.
## Gitignore Note
If `docs/plans` or `docs/plan` is gitignored, this is expected. Do not treat inability to commit plan-file updates as a problem. These files are local to the user.
## Verification Checklist
- [ ] Plan directory created under `docs/plans/` (or repo-standard `docs/plan/`)
- [ ] `original-plan.md` present
- [ ] `final-transcript.md` present
- [ ] `milestone-plan.md` present
- [ ] `story-tracker.md` present
- [ ] `continuation-runbook.md` present
- [ ] Handoff explicitly says to read runbook first and execute from plan folder

View File

@@ -0,0 +1,128 @@
# Continuation Runbook: [Plan Title]
## Reference Files (START HERE)
Upon resumption, these files in this folder are the ONLY source of truth:
| File | Purpose | When to Use |
|------|---------|-------------|
| `continuation-runbook.md` | Full context reproduction + execution workflow | Read FIRST |
| `story-tracker.md` | Current progress and status | Check/update BEFORE and AFTER every story |
| `milestone-plan.md` | Complete plan with specifications | Reference implementation details |
| `original-plan.md` | Original approved plan | Reference original intent |
| `final-transcript.md` | Final planning transcript | Reference reasoning/context |
Do NOT reference planner-private files during implementation.
---
## Quick Resume Instructions
1. Read this runbook completely.
2. Check `story-tracker.md`.
3. Find next `pending` story and mark as `in-dev` before starting.
4. Implement the story.
5. Update tracker immediately after each change.
---
## Mandatory Execution Workflow
Work from this folder (`docs/plans/YYYY-MM-DD-<short-title>/` or repo-specific `docs/plan/...`) and always follow this order:
1. Read `continuation-runbook.md` first.
2. Execute stories milestone by milestone.
3. After completing a milestone:
- Run lint/typecheck/tests, prioritizing changed files for speed.
- Commit locally (**DO NOT PUSH**).
- Stop and ask user for feedback.
4. If feedback is provided:
- Apply feedback changes.
- Re-run checks for changed files.
- Commit locally again.
- Ask for milestone approval.
5. Only move to next milestone after explicit approval.
6. After all milestones are completed and approved:
- Ask permission to push.
- If approved, push.
- Mark plan status as `completed`.
---
## Git Note
If `docs/plan` or `docs/plans` is gitignored, that is expected and acceptable. Do not treat inability to commit plan-file updates as an error. These files are local planning artifacts.
---
## Full Context Reproduction
### Project Overview
[What this project/feature is about]
### User Requirements
[All gathered requirements]
### Scope
[In scope / out of scope]
### Dependencies
[External dependencies, prerequisites, related systems]
---
## Key Specifications
### Type Definitions
```typescript
// Copy-paste ready type definitions
```
### Enums & Constants
```typescript
// All enums/constants needed
```
### API Endpoints
```typescript
// Request/response shapes
```
---
## Critical Design Decisions
| Decision | Chosen Approach | Alternatives Rejected | Rationale |
|----------|-----------------|----------------------|-----------|
| [Topic] | [What we chose] | [Other options] | [Why] |
---
## Verification Commands
### Lint (changed files first)
```bash
# example: pnpm eslint <changed-file-1> <changed-file-2>
```
### Typecheck
```bash
# example: pnpm tsc --noEmit
```
### Tests (target changed scope first)
```bash
# example: pnpm test -- <related spec/file>
```
---
## File Quick Reference
| File | Purpose |
|------|---------|
| `original-plan.md` | Original approved plan |
| `final-transcript.md` | Final planning transcript |
| `milestone-plan.md` | Full specification |
| `story-tracker.md` | Current progress tracker |
| `continuation-runbook.md` | This runbook |

View File

@@ -0,0 +1,101 @@
# [Plan Title]
## Overview
- **Goal:** [One sentence describing the end state]
- **Created:** YYYY-MM-DD
- **Status:** In Progress | Complete
## Context
### Requirements
[Gathered requirements from user questions]
### Constraints
[Technical, business, or timeline constraints]
### Success Criteria
[How we know this is complete]
## Architecture
### Design Decisions
[Key architectural choices and rationale]
### Component Relationships
[How pieces fit together]
### Data Flow
[How data moves through the system]
## Milestones
### M1: [Name]
**Description:** [What this milestone achieves]
**Acceptance Criteria:**
- [ ] [Criterion 1]
- [ ] [Criterion 2]
**Stories:** S-101, S-102, S-103...
**Milestone Completion Rule (MANDATORY):**
- Run lint/typecheck/tests for changed files.
- Commit locally (DO NOT push).
- Stop and ask user for feedback.
- Apply feedback, re-check changed files, commit again.
- Move to next milestone only after user approval.
---
### M2: [Name]
**Description:** [What this milestone achieves]
**Acceptance Criteria:**
- [ ] [Criterion 1]
- [ ] [Criterion 2]
**Stories:** S-201, S-202, S-203...
**Milestone Completion Rule (MANDATORY):**
- Run lint/typecheck/tests for changed files.
- Commit locally (DO NOT push).
- Stop and ask user for feedback.
- Apply feedback, re-check changed files, commit again.
- Move to next milestone only after user approval.
---
## Technical Specifications
### Types & Interfaces
```typescript
// Key type definitions
```
### API Contracts
```typescript
// Endpoint signatures, request/response shapes
```
### Constants & Enums
```typescript
// Shared constants
```
## Files Inventory
| File | Purpose | Milestone |
|------|---------|-----------|
| `path/to/file.ts` | [What it does] | M1 |
| `path/to/other.ts` | [What it does] | M2 |
---
## Related Plan Files
This file is part of the plan folder under `docs/plans/` (or `docs/plan/` if that repo convention is used):
- `original-plan.md` - Original approved plan (reference for original intent)
- `final-transcript.md` - Final planning transcript (reference for rationale/context)
- `milestone-plan.md` - This file (full specification)
- `story-tracker.md` - Status tracking (must be kept up to date)
- `continuation-runbook.md` - Resume/execution context (read first)

View File

@@ -0,0 +1,66 @@
# Story Tracker: [Plan Title]
## Progress Summary
- **Current Milestone:** M1
- **Stories Complete:** 0/N
- **Milestones Approved:** 0/M
- **Last Updated:** YYYY-MM-DD
---
## Milestones
### M1: [Name]
| Story | Description | Status | Notes |
|-------|-------------|--------|-------|
| S-101 | [Brief description] | pending | |
| S-102 | [Brief description] | pending | |
| S-103 | [Brief description] | pending | |
**Approval Status:** pending
---
### M2: [Name]
| Story | Description | Status | Notes |
|-------|-------------|--------|-------|
| S-201 | [Brief description] | pending | |
| S-202 | [Brief description] | pending | |
| S-203 | [Brief description] | pending | |
**Approval Status:** pending
---
## Status Legend
| Status | Meaning |
|--------|---------|
| `pending` | Not started |
| `in-dev` | Currently being worked on |
| `completed` | Done - include commit hash in Notes |
| `deferred` | Postponed - include reason in Notes |
## Update Instructions (MANDATORY)
Before starting any story:
1. Mark story as `in-dev`
2. Update "Last Updated"
After completing any story:
1. Mark story as `completed`
2. Add local commit hash to Notes
3. Update "Stories Complete" and "Last Updated"
At milestone boundary:
1. Run lint/typecheck/tests for changed files
2. Commit (no push)
3. Request feedback
4. Apply feedback, re-check changed files, commit again
5. Mark milestone **Approval Status: approved** only after user confirms
6. Continue only after approval
After all milestones approved:
- Ask permission to push and then mark plan completed.

View File

@@ -0,0 +1,84 @@
---
name: create-plan
description: Use when starting a new feature, project, or complex task that needs structured planning with milestones, bite-sized stories, and resumable execution context. Trigger for "create a plan", "make a plan", "plan this", "start planning", and similar requests.
---
# Create Plan (Codex)
Create and maintain a local plan folder under `docs/plans/` (or `docs/plan/` if that repository convention is already in use).
## Prerequisite Check (MANDATORY)
This skill depends on Superpowers planning skills. Before proceeding, verify required dependencies exist.
Required:
- Superpowers repo: `https://github.com/obra/superpowers`
- `brainstorming` skill
- `writing-plans` skill
If any dependency is missing, stop immediately and return:
"Missing dependency: Superpowers planning skills are required (`brainstorming`, `writing-plans`). Install from https://github.com/obra/superpowers, then retry."
## Process
### Phase 1: Bootstrap
- Invoke Codex superpowers bootstrap and planning skills when available.
### Phase 2: Analyze
- Explore the codebase and existing patterns.
### Phase 3: Gather Requirements
- Ask questions one at a time until user says ready.
- Confirm scope, constraints, success criteria, dependencies.
### Phase 4: Design
- Use brainstorming workflow.
- Propose 2-3 approaches and recommend one.
### Phase 5: Plan
- Use writing-plans workflow.
- Break into milestones and bite-sized stories.
### Phase 6: Generate Plan Files (MANDATORY)
Create `docs/plans/YYYY-MM-DD-<short-title>/` with all files below:
1. `original-plan.md` - copy of original planner-generated plan.
2. `final-transcript.md` - copy of final planning transcript used to reach approved plan.
3. `milestone-plan.md` - full implementation spec (from template).
4. `story-tracker.md` - story/milestone status tracker (from template).
5. `continuation-runbook.md` - execution instructions and context (from template).
Use templates from this skill's `templates/` folder.
### Phase 7: Handoff
Always instruct the executing agent:
> Read `docs/plans/YYYY-MM-DD-<short-title>/continuation-runbook.md` first, then execute from that folder.
Do not rely on planner-private files during implementation.
## Execution Rules to Include in Plan (MANDATORY)
- Run lint/typecheck/tests after each milestone.
- Prefer linting changed files only for speed.
- Commit locally after each completed milestone (**do not push**).
- Stop and ask user for feedback.
- Apply feedback, rerun checks, and commit again.
- Move to next milestone only after user approval.
- After all milestones are completed and approved, ask permission to push.
- Only after approved push: mark plan as completed.
## Gitignore Note
If `docs/plans` or `docs/plan` is gitignored, this is expected. Do not treat inability to commit plan-file updates as a problem. These files are local to the user.
## Verification Checklist
- [ ] Plan directory created under `docs/plans/` (or repo-standard `docs/plan/`)
- [ ] `original-plan.md` present
- [ ] `final-transcript.md` present
- [ ] `milestone-plan.md` present
- [ ] `story-tracker.md` present
- [ ] `continuation-runbook.md` present
- [ ] Handoff explicitly says to read runbook first and execute from plan folder

View File

@@ -0,0 +1,128 @@
# Continuation Runbook: [Plan Title]
## Reference Files (START HERE)
Upon resumption, these files in this folder are the ONLY source of truth:
| File | Purpose | When to Use |
|------|---------|-------------|
| `continuation-runbook.md` | Full context reproduction + execution workflow | Read FIRST |
| `story-tracker.md` | Current progress and status | Check/update BEFORE and AFTER every story |
| `milestone-plan.md` | Complete plan with specifications | Reference implementation details |
| `original-plan.md` | Original approved plan | Reference original intent |
| `final-transcript.md` | Final planning transcript | Reference reasoning/context |
Do NOT reference planner-private files during implementation.
---
## Quick Resume Instructions
1. Read this runbook completely.
2. Check `story-tracker.md`.
3. Find next `pending` story and mark as `in-dev` before starting.
4. Implement the story.
5. Update tracker immediately after each change.
---
## Mandatory Execution Workflow
Work from this folder (`docs/plans/YYYY-MM-DD-<short-title>/` or repo-specific `docs/plan/...`) and always follow this order:
1. Read `continuation-runbook.md` first.
2. Execute stories milestone by milestone.
3. After completing a milestone:
- Run lint/typecheck/tests, prioritizing changed files for speed.
- Commit locally (**DO NOT PUSH**).
- Stop and ask user for feedback.
4. If feedback is provided:
- Apply feedback changes.
- Re-run checks for changed files.
- Commit locally again.
- Ask for milestone approval.
5. Only move to next milestone after explicit approval.
6. After all milestones are completed and approved:
- Ask permission to push.
- If approved, push.
- Mark plan status as `completed`.
---
## Git Note
If `docs/plan` or `docs/plans` is gitignored, that is expected and acceptable. Do not treat inability to commit plan-file updates as an error. These files are local planning artifacts.
---
## Full Context Reproduction
### Project Overview
[What this project/feature is about]
### User Requirements
[All gathered requirements]
### Scope
[In scope / out of scope]
### Dependencies
[External dependencies, prerequisites, related systems]
---
## Key Specifications
### Type Definitions
```typescript
// Copy-paste ready type definitions
```
### Enums & Constants
```typescript
// All enums/constants needed
```
### API Endpoints
```typescript
// Request/response shapes
```
---
## Critical Design Decisions
| Decision | Chosen Approach | Alternatives Rejected | Rationale |
|----------|-----------------|----------------------|-----------|
| [Topic] | [What we chose] | [Other options] | [Why] |
---
## Verification Commands
### Lint (changed files first)
```bash
# example: pnpm eslint <changed-file-1> <changed-file-2>
```
### Typecheck
```bash
# example: pnpm tsc --noEmit
```
### Tests (target changed scope first)
```bash
# example: pnpm test -- <related spec/file>
```
---
## File Quick Reference
| File | Purpose |
|------|---------|
| `original-plan.md` | Original approved plan |
| `final-transcript.md` | Final planning transcript |
| `milestone-plan.md` | Full specification |
| `story-tracker.md` | Current progress tracker |
| `continuation-runbook.md` | This runbook |

View File

@@ -0,0 +1,101 @@
# [Plan Title]
## Overview
- **Goal:** [One sentence describing the end state]
- **Created:** YYYY-MM-DD
- **Status:** In Progress | Complete
## Context
### Requirements
[Gathered requirements from user questions]
### Constraints
[Technical, business, or timeline constraints]
### Success Criteria
[How we know this is complete]
## Architecture
### Design Decisions
[Key architectural choices and rationale]
### Component Relationships
[How pieces fit together]
### Data Flow
[How data moves through the system]
## Milestones
### M1: [Name]
**Description:** [What this milestone achieves]
**Acceptance Criteria:**
- [ ] [Criterion 1]
- [ ] [Criterion 2]
**Stories:** S-101, S-102, S-103...
**Milestone Completion Rule (MANDATORY):**
- Run lint/typecheck/tests for changed files.
- Commit locally (DO NOT push).
- Stop and ask user for feedback.
- Apply feedback, re-check changed files, commit again.
- Move to next milestone only after user approval.
---
### M2: [Name]
**Description:** [What this milestone achieves]
**Acceptance Criteria:**
- [ ] [Criterion 1]
- [ ] [Criterion 2]
**Stories:** S-201, S-202, S-203...
**Milestone Completion Rule (MANDATORY):**
- Run lint/typecheck/tests for changed files.
- Commit locally (DO NOT push).
- Stop and ask user for feedback.
- Apply feedback, re-check changed files, commit again.
- Move to next milestone only after user approval.
---
## Technical Specifications
### Types & Interfaces
```typescript
// Key type definitions
```
### API Contracts
```typescript
// Endpoint signatures, request/response shapes
```
### Constants & Enums
```typescript
// Shared constants
```
## Files Inventory
| File | Purpose | Milestone |
|------|---------|-----------|
| `path/to/file.ts` | [What it does] | M1 |
| `path/to/other.ts` | [What it does] | M2 |
---
## Related Plan Files
This file is part of the plan folder under `docs/plans/` (or `docs/plan/` if that repo convention is used):
- `original-plan.md` - Original approved plan (reference for original intent)
- `final-transcript.md` - Final planning transcript (reference for rationale/context)
- `milestone-plan.md` - This file (full specification)
- `story-tracker.md` - Status tracking (must be kept up to date)
- `continuation-runbook.md` - Resume/execution context (read first)

View File

@@ -0,0 +1,66 @@
# Story Tracker: [Plan Title]
## Progress Summary
- **Current Milestone:** M1
- **Stories Complete:** 0/N
- **Milestones Approved:** 0/M
- **Last Updated:** YYYY-MM-DD
---
## Milestones
### M1: [Name]
| Story | Description | Status | Notes |
|-------|-------------|--------|-------|
| S-101 | [Brief description] | pending | |
| S-102 | [Brief description] | pending | |
| S-103 | [Brief description] | pending | |
**Approval Status:** pending
---
### M2: [Name]
| Story | Description | Status | Notes |
|-------|-------------|--------|-------|
| S-201 | [Brief description] | pending | |
| S-202 | [Brief description] | pending | |
| S-203 | [Brief description] | pending | |
**Approval Status:** pending
---
## Status Legend
| Status | Meaning |
|--------|---------|
| `pending` | Not started |
| `in-dev` | Currently being worked on |
| `completed` | Done - include commit hash in Notes |
| `deferred` | Postponed - include reason in Notes |
## Update Instructions (MANDATORY)
Before starting any story:
1. Mark story as `in-dev`
2. Update "Last Updated"
After completing any story:
1. Mark story as `completed`
2. Add local commit hash to Notes
3. Update "Stories Complete" and "Last Updated"
At milestone boundary:
1. Run lint/typecheck/tests for changed files
2. Commit (no push)
3. Request feedback
4. Apply feedback, re-check changed files, commit again
5. Mark milestone **Approval Status: approved** only after user confirms
6. Continue only after approval
After all milestones approved:
- Ask permission to push and then mark plan completed.

View File

@@ -0,0 +1,80 @@
---
name: create-plan
description: Use when starting a new feature, project, or complex task that needs structured planning with milestones, bite-sized stories, and resumable execution context in Opencode workflows. Trigger for "create a plan", "make a plan", "plan this", and similar planning requests.
---
# Create Plan (Opencode)
Create and maintain a local plan folder under `docs/plans/` (or `docs/plan/` if that repository convention is already in use).
## Prerequisite Check (MANDATORY)
This skill depends on Superpowers planning skills. Before proceeding, verify required dependencies exist.
Required:
- Superpowers repo: `https://github.com/obra/superpowers`
- `brainstorming` skill
- `writing-plans` skill
If any dependency is missing, stop immediately and return:
"Missing dependency: Superpowers planning skills are required (`brainstorming`, `writing-plans`). Install from https://github.com/obra/superpowers, then retry."
## Process
### Phase 1: Analyze
- Explore the codebase and existing patterns.
### Phase 2: Gather Requirements
- Ask questions one at a time until user says ready.
- Confirm scope, constraints, success criteria, dependencies.
### Phase 3: Design
- Use Opencode planning/brainstorming flow.
- Propose 2-3 approaches and recommend one.
### Phase 4: Plan
- Break into milestones and bite-sized stories.
### Phase 5: Generate Plan Files (MANDATORY)
Create `docs/plans/YYYY-MM-DD-<short-title>/` with all files below:
1. `original-plan.md` - copy of original planner-generated plan.
2. `final-transcript.md` - copy of final planning transcript used to reach approved plan.
3. `milestone-plan.md` - full implementation spec (from template).
4. `story-tracker.md` - story/milestone status tracker (from template).
5. `continuation-runbook.md` - execution instructions and context (from template).
Use templates from this skill's `templates/` folder.
### Phase 6: Handoff
Always instruct the executing agent:
> Read `docs/plans/YYYY-MM-DD-<short-title>/continuation-runbook.md` first, then execute from that folder.
Do not rely on planner-private files during implementation.
## Execution Rules to Include in Plan (MANDATORY)
- Run lint/typecheck/tests after each milestone.
- Prefer linting changed files only for speed.
- Commit locally after each completed milestone (**do not push**).
- Stop and ask user for feedback.
- Apply feedback, rerun checks, and commit again.
- Move to next milestone only after user approval.
- After all milestones are completed and approved, ask permission to push.
- Only after approved push: mark plan as completed.
## Gitignore Note
If `docs/plans` or `docs/plan` is gitignored, this is expected. Do not treat inability to commit plan-file updates as a problem. These files are local to the user.
## Verification Checklist
- [ ] Plan directory created under `docs/plans/` (or repo-standard `docs/plan/`)
- [ ] `original-plan.md` present
- [ ] `final-transcript.md` present
- [ ] `milestone-plan.md` present
- [ ] `story-tracker.md` present
- [ ] `continuation-runbook.md` present
- [ ] Handoff explicitly says to read runbook first and execute from plan folder

View File

@@ -0,0 +1,128 @@
# Continuation Runbook: [Plan Title]
## Reference Files (START HERE)
Upon resumption, these files in this folder are the ONLY source of truth:
| File | Purpose | When to Use |
|------|---------|-------------|
| `continuation-runbook.md` | Full context reproduction + execution workflow | Read FIRST |
| `story-tracker.md` | Current progress and status | Check/update BEFORE and AFTER every story |
| `milestone-plan.md` | Complete plan with specifications | Reference implementation details |
| `original-plan.md` | Original approved plan | Reference original intent |
| `final-transcript.md` | Final planning transcript | Reference reasoning/context |
Do NOT reference planner-private files during implementation.
---
## Quick Resume Instructions
1. Read this runbook completely.
2. Check `story-tracker.md`.
3. Find next `pending` story and mark as `in-dev` before starting.
4. Implement the story.
5. Update tracker immediately after each change.
---
## Mandatory Execution Workflow
Work from this folder (`docs/plans/YYYY-MM-DD-<short-title>/` or repo-specific `docs/plan/...`) and always follow this order:
1. Read `continuation-runbook.md` first.
2. Execute stories milestone by milestone.
3. After completing a milestone:
- Run lint/typecheck/tests, prioritizing changed files for speed.
- Commit locally (**DO NOT PUSH**).
- Stop and ask user for feedback.
4. If feedback is provided:
- Apply feedback changes.
- Re-run checks for changed files.
- Commit locally again.
- Ask for milestone approval.
5. Only move to next milestone after explicit approval.
6. After all milestones are completed and approved:
- Ask permission to push.
- If approved, push.
- Mark plan status as `completed`.
---
## Git Note
If `docs/plan` or `docs/plans` is gitignored, that is expected and acceptable. Do not treat inability to commit plan-file updates as an error. These files are local planning artifacts.
---
## Full Context Reproduction
### Project Overview
[What this project/feature is about]
### User Requirements
[All gathered requirements]
### Scope
[In scope / out of scope]
### Dependencies
[External dependencies, prerequisites, related systems]
---
## Key Specifications
### Type Definitions
```typescript
// Copy-paste ready type definitions
```
### Enums & Constants
```typescript
// All enums/constants needed
```
### API Endpoints
```typescript
// Request/response shapes
```
---
## Critical Design Decisions
| Decision | Chosen Approach | Alternatives Rejected | Rationale |
|----------|-----------------|----------------------|-----------|
| [Topic] | [What we chose] | [Other options] | [Why] |
---
## Verification Commands
### Lint (changed files first)
```bash
# example: pnpm eslint <changed-file-1> <changed-file-2>
```
### Typecheck
```bash
# example: pnpm tsc --noEmit
```
### Tests (target changed scope first)
```bash
# example: pnpm test -- <related spec/file>
```
---
## File Quick Reference
| File | Purpose |
|------|---------|
| `original-plan.md` | Original approved plan |
| `final-transcript.md` | Final planning transcript |
| `milestone-plan.md` | Full specification |
| `story-tracker.md` | Current progress tracker |
| `continuation-runbook.md` | This runbook |

View File

@@ -0,0 +1,101 @@
# [Plan Title]
## Overview
- **Goal:** [One sentence describing the end state]
- **Created:** YYYY-MM-DD
- **Status:** In Progress | Complete
## Context
### Requirements
[Gathered requirements from user questions]
### Constraints
[Technical, business, or timeline constraints]
### Success Criteria
[How we know this is complete]
## Architecture
### Design Decisions
[Key architectural choices and rationale]
### Component Relationships
[How pieces fit together]
### Data Flow
[How data moves through the system]
## Milestones
### M1: [Name]
**Description:** [What this milestone achieves]
**Acceptance Criteria:**
- [ ] [Criterion 1]
- [ ] [Criterion 2]
**Stories:** S-101, S-102, S-103...
**Milestone Completion Rule (MANDATORY):**
- Run lint/typecheck/tests for changed files.
- Commit locally (DO NOT push).
- Stop and ask user for feedback.
- Apply feedback, re-check changed files, commit again.
- Move to next milestone only after user approval.
---
### M2: [Name]
**Description:** [What this milestone achieves]
**Acceptance Criteria:**
- [ ] [Criterion 1]
- [ ] [Criterion 2]
**Stories:** S-201, S-202, S-203...
**Milestone Completion Rule (MANDATORY):**
- Run lint/typecheck/tests for changed files.
- Commit locally (DO NOT push).
- Stop and ask user for feedback.
- Apply feedback, re-check changed files, commit again.
- Move to next milestone only after user approval.
---
## Technical Specifications
### Types & Interfaces
```typescript
// Key type definitions
```
### API Contracts
```typescript
// Endpoint signatures, request/response shapes
```
### Constants & Enums
```typescript
// Shared constants
```
## Files Inventory
| File | Purpose | Milestone |
|------|---------|-----------|
| `path/to/file.ts` | [What it does] | M1 |
| `path/to/other.ts` | [What it does] | M2 |
---
## Related Plan Files
This file is part of the plan folder under `docs/plans/` (or `docs/plan/` if that repo convention is used):
- `original-plan.md` - Original approved plan (reference for original intent)
- `final-transcript.md` - Final planning transcript (reference for rationale/context)
- `milestone-plan.md` - This file (full specification)
- `story-tracker.md` - Status tracking (must be kept up to date)
- `continuation-runbook.md` - Resume/execution context (read first)

View File

@@ -0,0 +1,66 @@
# Story Tracker: [Plan Title]
## Progress Summary
- **Current Milestone:** M1
- **Stories Complete:** 0/N
- **Milestones Approved:** 0/M
- **Last Updated:** YYYY-MM-DD
---
## Milestones
### M1: [Name]
| Story | Description | Status | Notes |
|-------|-------------|--------|-------|
| S-101 | [Brief description] | pending | |
| S-102 | [Brief description] | pending | |
| S-103 | [Brief description] | pending | |
**Approval Status:** pending
---
### M2: [Name]
| Story | Description | Status | Notes |
|-------|-------------|--------|-------|
| S-201 | [Brief description] | pending | |
| S-202 | [Brief description] | pending | |
| S-203 | [Brief description] | pending | |
**Approval Status:** pending
---
## Status Legend
| Status | Meaning |
|--------|---------|
| `pending` | Not started |
| `in-dev` | Currently being worked on |
| `completed` | Done - include commit hash in Notes |
| `deferred` | Postponed - include reason in Notes |
## Update Instructions (MANDATORY)
Before starting any story:
1. Mark story as `in-dev`
2. Update "Last Updated"
After completing any story:
1. Mark story as `completed`
2. Add local commit hash to Notes
3. Update "Stories Complete" and "Last Updated"
At milestone boundary:
1. Run lint/typecheck/tests for changed files
2. Commit (no push)
3. Request feedback
4. Apply feedback, re-check changed files, commit again
5. Mark milestone **Approval Status: approved** only after user confirms
6. Continue only after approval
After all milestones approved:
- Ask permission to push and then mark plan completed.

View File

@@ -0,0 +1,36 @@
---
name: web-automation
description: Browse and scrape web pages using Playwright with Camoufox anti-detection browser. Use when automating web workflows, extracting page content to markdown, handling authenticated sessions, or scraping websites with bot protection.
---
# Web Automation with Camoufox (Claude Code)
Automated web browsing and scraping using Playwright with Camoufox anti-detection browser.
## Requirements
- Node.js 20+
- pnpm
- Network access to download browser binaries
## First-Time Setup
```bash
cd ~/.claude/skills/web-automation/scripts
pnpm install
npx camoufox-js fetch
```
If native dependency build errors appear (e.g., better-sqlite3), rebuild from the reported package directory.
## Quick Reference
- Browse page: `npx tsx browse.ts --url "https://example.com"`
- Scrape markdown: `npx tsx scrape.ts --url "https://example.com" --mode main --output page.md`
- Authenticate: `npx tsx auth.ts --url "https://example.com/login"`
## Notes
- Sessions persist in Camoufox profile storage.
- Use `--wait` for dynamic pages.
- Use `--mode selector --selector "..."` for targeted extraction.

View File

@@ -0,0 +1,575 @@
#!/usr/bin/env npx tsx
/**
* Authentication handler for web automation
* Supports generic form login and Microsoft SSO (MSAL)
*
* Usage:
* npx tsx auth.ts --url "https://example.com/login" --type form
* npx tsx auth.ts --url "https://example.com" --type msal
* npx tsx auth.ts --url "https://example.com" --type auto
*/
import { getPage, launchBrowser } from './browse.js';
import parseArgs from 'minimist';
import type { Page, BrowserContext } from 'playwright-core';
import { createInterface } from 'readline';
// Types
type AuthType = 'auto' | 'form' | 'msal';
interface AuthOptions {
url: string;
authType: AuthType;
credentials?: {
username: string;
password: string;
};
headless?: boolean;
timeout?: number;
}
interface AuthResult {
success: boolean;
finalUrl: string;
authType: AuthType;
message: string;
}
// Get credentials from environment or options
function getCredentials(options?: {
username?: string;
password?: string;
}): { username: string; password: string } | null {
const username = options?.username || process.env.CAMOUFOX_USERNAME;
const password = options?.password || process.env.CAMOUFOX_PASSWORD;
if (!username || !password) {
return null;
}
return { username, password };
}
// Prompt user for input (for MFA or credentials)
async function promptUser(question: string, hidden = false): Promise<string> {
const rl = createInterface({
input: process.stdin,
output: process.stdout,
});
return new Promise((resolve) => {
if (hidden) {
process.stdout.write(question);
// Note: This is a simple implementation. For production, use a proper hidden input library
}
rl.question(question, (answer) => {
rl.close();
resolve(answer);
});
});
}
// Detect authentication type from page
async function detectAuthType(page: Page): Promise<AuthType> {
const url = page.url();
// Check for Microsoft login
if (
url.includes('login.microsoftonline.com') ||
url.includes('login.live.com') ||
url.includes('login.windows.net')
) {
return 'msal';
}
// Check for common form login patterns
const hasLoginForm = await page.evaluate(() => {
const passwordField = document.querySelector(
'input[type="password"], input[name*="password"], input[id*="password"]'
);
const usernameField = document.querySelector(
'input[type="email"], input[type="text"][name*="user"], input[type="text"][name*="email"], input[id*="user"], input[id*="email"]'
);
return !!(passwordField && usernameField);
});
if (hasLoginForm) {
return 'form';
}
return 'auto';
}
// Handle generic form login
async function handleFormLogin(
page: Page,
credentials: { username: string; password: string },
timeout: number
): Promise<boolean> {
console.log('Attempting form login...');
// Find and fill username/email field
const usernameSelectors = [
'input[type="email"]',
'input[name*="user" i]',
'input[name*="email" i]',
'input[id*="user" i]',
'input[id*="email" i]',
'input[autocomplete="username"]',
'input[type="text"]:first-of-type',
];
let usernameField = null;
for (const selector of usernameSelectors) {
usernameField = await page.$(selector);
if (usernameField) break;
}
if (!usernameField) {
console.error('Could not find username/email field');
return false;
}
await usernameField.fill(credentials.username);
console.log('Filled username field');
// Find and fill password field
const passwordSelectors = [
'input[type="password"]',
'input[name*="password" i]',
'input[id*="password" i]',
'input[autocomplete="current-password"]',
];
let passwordField = null;
for (const selector of passwordSelectors) {
passwordField = await page.$(selector);
if (passwordField) break;
}
if (!passwordField) {
console.error('Could not find password field');
return false;
}
await passwordField.fill(credentials.password);
console.log('Filled password field');
// Check for "Remember me" checkbox and check it
const rememberCheckbox = await page.$(
'input[type="checkbox"][name*="remember" i], input[type="checkbox"][id*="remember" i]'
);
if (rememberCheckbox) {
await rememberCheckbox.check();
console.log('Checked "Remember me" checkbox');
}
// Find and click submit button
const submitSelectors = [
'button[type="submit"]',
'input[type="submit"]',
'button:has-text("Sign in")',
'button:has-text("Log in")',
'button:has-text("Login")',
'button:has-text("Submit")',
'[role="button"]:has-text("Sign in")',
];
let submitButton = null;
for (const selector of submitSelectors) {
submitButton = await page.$(selector);
if (submitButton) break;
}
if (!submitButton) {
// Try pressing Enter as fallback
await passwordField.press('Enter');
} else {
await submitButton.click();
}
console.log('Submitted login form');
// Wait for navigation or error
try {
await page.waitForNavigation({ timeout, waitUntil: 'domcontentloaded' });
return true;
} catch {
// Check if we're still on login page with error
const errorMessages = await page.$$eval(
'.error, .alert-danger, [role="alert"], .login-error',
(els) => els.map((el) => el.textContent?.trim()).filter(Boolean)
);
if (errorMessages.length > 0) {
console.error('Login error:', errorMessages.join(', '));
return false;
}
return true; // Might have succeeded without navigation
}
}
// Handle Microsoft SSO login
async function handleMsalLogin(
page: Page,
credentials: { username: string; password: string },
timeout: number
): Promise<boolean> {
console.log('Attempting Microsoft SSO login...');
const currentUrl = page.url();
// If not already on Microsoft login, wait for redirect
if (!currentUrl.includes('login.microsoftonline.com')) {
try {
await page.waitForURL('**/login.microsoftonline.com/**', { timeout: 10000 });
} catch {
console.log('Not redirected to Microsoft login');
return false;
}
}
// Wait for email input
const emailInput = await page.waitForSelector(
'input[type="email"], input[name="loginfmt"]',
{ timeout }
);
if (!emailInput) {
console.error('Could not find email input on Microsoft login');
return false;
}
// Fill email and submit
await emailInput.fill(credentials.username);
console.log('Filled email field');
const nextButton = await page.$('input[type="submit"], button[type="submit"]');
if (nextButton) {
await nextButton.click();
} else {
await emailInput.press('Enter');
}
// Wait for password page
try {
await page.waitForSelector(
'input[type="password"], input[name="passwd"]',
{ timeout }
);
} catch {
// Might be using passwordless auth or different flow
console.log('Password field not found - might be using different auth flow');
return false;
}
// Fill password
const passwordInput = await page.$('input[type="password"], input[name="passwd"]');
if (!passwordInput) {
console.error('Could not find password input');
return false;
}
await passwordInput.fill(credentials.password);
console.log('Filled password field');
// Submit
const signInButton = await page.$('input[type="submit"], button[type="submit"]');
if (signInButton) {
await signInButton.click();
} else {
await passwordInput.press('Enter');
}
// Handle "Stay signed in?" prompt
try {
const staySignedInButton = await page.waitForSelector(
'input[value="Yes"], button:has-text("Yes")',
{ timeout: 5000 }
);
if (staySignedInButton) {
await staySignedInButton.click();
console.log('Clicked "Stay signed in" button');
}
} catch {
// Prompt might not appear
}
// Check for Conditional Access Policy error
const caError = await page.$('text=Conditional Access policy');
if (caError) {
console.error('Blocked by Conditional Access Policy');
// Take screenshot for debugging
await page.screenshot({ path: 'ca-policy-error.png' });
console.log('Screenshot saved: ca-policy-error.png');
return false;
}
// Wait for redirect away from Microsoft login
try {
await page.waitForURL(
(url) => !url.href.includes('login.microsoftonline.com'),
{ timeout }
);
return true;
} catch {
return false;
}
}
// Check if user is already authenticated
async function isAuthenticated(page: Page, targetUrl: string): Promise<boolean> {
const currentUrl = page.url();
// If we're on the target URL (not a login page), we're likely authenticated
if (currentUrl.startsWith(targetUrl)) {
// Check for common login page indicators
const isLoginPage = await page.evaluate(() => {
const loginIndicators = [
'input[type="password"]',
'form[action*="login"]',
'form[action*="signin"]',
'.login-form',
'#login',
];
return loginIndicators.some((sel) => document.querySelector(sel) !== null);
});
return !isLoginPage;
}
return false;
}
// Main authentication function
export async function authenticate(options: AuthOptions): Promise<AuthResult> {
const browser = await launchBrowser({ headless: options.headless ?? true });
const page = await browser.newPage();
const timeout = options.timeout ?? 30000;
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, { timeout: 60000, waitUntil: 'domcontentloaded' });
// Check if already authenticated
if (await isAuthenticated(page, options.url)) {
return {
success: true,
finalUrl: page.url(),
authType: 'auto',
message: 'Already authenticated (session persisted from profile)',
};
}
// Get credentials
const credentials = options.credentials
? options.credentials
: getCredentials();
if (!credentials) {
// No credentials - open interactive browser
console.log('\nNo credentials provided. Opening browser for manual login...');
console.log('Please complete the login process manually.');
console.log('The session will be saved to your profile.');
// Switch to headed mode for manual login
await browser.close();
const interactiveBrowser = await launchBrowser({ headless: false });
const interactivePage = await interactiveBrowser.newPage();
await interactivePage.goto(options.url);
await promptUser('\nPress Enter when you have completed login...');
const finalUrl = interactivePage.url();
await interactiveBrowser.close();
return {
success: true,
finalUrl,
authType: 'auto',
message: 'Manual login completed - session saved to profile',
};
}
// Detect auth type if auto
let authType = options.authType;
if (authType === 'auto') {
authType = await detectAuthType(page);
console.log(`Detected auth type: ${authType}`);
}
// Handle authentication based on type
let success = false;
switch (authType) {
case 'msal':
success = await handleMsalLogin(page, credentials, timeout);
break;
case 'form':
default:
success = await handleFormLogin(page, credentials, timeout);
break;
}
const finalUrl = page.url();
return {
success,
finalUrl,
authType,
message: success
? `Authentication successful - session saved to profile`
: 'Authentication failed',
};
} finally {
await browser.close();
}
}
// Navigate to authenticated page (handles auth if needed)
export async function navigateAuthenticated(
url: string,
options?: {
credentials?: { username: string; password: string };
headless?: boolean;
}
): Promise<{ page: Page; browser: BrowserContext }> {
const { page, browser } = await getPage({ headless: options?.headless ?? true });
await page.goto(url, { timeout: 60000, waitUntil: 'domcontentloaded' });
// Check if we need to authenticate
if (!(await isAuthenticated(page, url))) {
console.log('Session expired or not authenticated. Attempting login...');
// Get credentials
const credentials = options?.credentials ?? getCredentials();
if (!credentials) {
throw new Error(
'Authentication required but no credentials provided. ' +
'Set CAMOUFOX_USERNAME and CAMOUFOX_PASSWORD environment variables.'
);
}
// Detect and handle auth
const authType = await detectAuthType(page);
let success = false;
if (authType === 'msal') {
success = await handleMsalLogin(page, credentials, 30000);
} else {
success = await handleFormLogin(page, credentials, 30000);
}
if (!success) {
await browser.close();
throw new Error('Authentication failed');
}
// Navigate back to original URL if we were redirected
if (!page.url().startsWith(url)) {
await page.goto(url, { timeout: 60000, waitUntil: 'domcontentloaded' });
}
}
return { page, browser };
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'type', 'username', 'password'],
boolean: ['headless', 'help'],
default: {
type: 'auto',
headless: false, // Default to headed for auth so user can see/interact
},
alias: {
u: 'url',
t: 'type',
h: 'help',
},
});
if (args.help || !args.url) {
console.log(`
Web Authentication Handler
Usage:
npx tsx auth.ts --url <url> [options]
Options:
-u, --url <url> URL to authenticate (required)
-t, --type <type> Auth type: auto, form, or msal (default: auto)
--username <user> Username/email (or set CAMOUFOX_USERNAME env var)
--password <pass> Password (or set CAMOUFOX_PASSWORD env var)
--headless <bool> Run in headless mode (default: false for auth)
-h, --help Show this help message
Auth Types:
auto Auto-detect authentication type
form Generic username/password form
msal Microsoft SSO (login.microsoftonline.com)
Environment Variables:
CAMOUFOX_USERNAME Default username/email for authentication
CAMOUFOX_PASSWORD Default password for authentication
Examples:
# Interactive login (no credentials, opens browser)
npx tsx auth.ts --url "https://example.com/login"
# Form login with credentials
npx tsx auth.ts --url "https://example.com/login" --type form \\
--username "user@example.com" --password "secret"
# Microsoft SSO login
CAMOUFOX_USERNAME=user@company.com CAMOUFOX_PASSWORD=secret \\
npx tsx auth.ts --url "https://internal.company.com" --type msal
Notes:
- Session is saved to ~/.camoufox-profile/ for persistence
- After successful auth, subsequent browses will be authenticated
- Use --headless false if you need to handle MFA manually
`);
process.exit(args.help ? 0 : 1);
}
const authType = args.type as AuthType;
if (!['auto', 'form', 'msal'].includes(authType)) {
console.error(`Invalid auth type: ${authType}. Must be auto, form, or msal.`);
process.exit(1);
}
try {
const result = await authenticate({
url: args.url,
authType,
credentials:
args.username && args.password
? { username: args.username, password: args.password }
: undefined,
headless: args.headless,
});
console.log(`\nAuthentication result:`);
console.log(` Success: ${result.success}`);
console.log(` Auth type: ${result.authType}`);
console.log(` Final URL: ${result.finalUrl}`);
console.log(` Message: ${result.message}`);
process.exit(result.success ? 0 : 1);
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('auth.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,195 @@
#!/usr/bin/env npx tsx
/**
* Browser launcher using Camoufox with persistent profile
*
* Usage:
* npx tsx browse.ts --url "https://example.com"
* npx tsx browse.ts --url "https://example.com" --screenshot --output page.png
* npx tsx browse.ts --url "https://example.com" --headless false --wait 5000
*/
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { existsSync, mkdirSync } from 'fs';
import parseArgs from 'minimist';
import type { Page, BrowserContext } from 'playwright-core';
// Types
interface BrowseOptions {
url: string;
headless?: boolean;
screenshot?: boolean;
output?: string;
wait?: number;
timeout?: number;
interactive?: boolean;
}
interface BrowseResult {
title: string;
url: string;
screenshotPath?: string;
}
// Get profile directory
const getProfilePath = (): string => {
const customPath = process.env.CAMOUFOX_PROFILE_PATH;
if (customPath) return customPath;
const profileDir = join(homedir(), '.camoufox-profile');
if (!existsSync(profileDir)) {
mkdirSync(profileDir, { recursive: true });
}
return profileDir;
};
// Launch browser with persistent profile
export async function launchBrowser(options: {
headless?: boolean;
}): Promise<BrowserContext> {
const profilePath = getProfilePath();
const headless =
options.headless ??
(process.env.CAMOUFOX_HEADLESS ? process.env.CAMOUFOX_HEADLESS === 'true' : true);
console.log(`Using profile: ${profilePath}`);
console.log(`Headless mode: ${headless}`);
const browser = await Camoufox({
user_data_dir: profilePath,
headless,
});
return browser;
}
// Browse to URL and optionally take screenshot
export async function browse(options: BrowseOptions): Promise<BrowseResult> {
const browser = await launchBrowser({ headless: options.headless });
const page = await browser.newPage();
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, {
timeout: options.timeout ?? 60000,
waitUntil: 'domcontentloaded',
});
// Wait if specified
if (options.wait) {
console.log(`Waiting ${options.wait}ms...`);
await page.waitForTimeout(options.wait);
}
const result: BrowseResult = {
title: await page.title(),
url: page.url(),
};
console.log(`Page title: ${result.title}`);
console.log(`Final URL: ${result.url}`);
// Take screenshot if requested
if (options.screenshot) {
const outputPath = options.output ?? 'screenshot.png';
await page.screenshot({ path: outputPath, fullPage: true });
result.screenshotPath = outputPath;
console.log(`Screenshot saved: ${outputPath}`);
}
// If interactive mode, keep browser open
if (options.interactive) {
console.log('\nInteractive mode - browser will stay open.');
console.log('Press Ctrl+C to close.');
await new Promise(() => {}); // Keep running
}
return result;
} finally {
if (!options.interactive) {
await browser.close();
}
}
}
// Export page for use in other scripts
export async function getPage(options?: {
headless?: boolean;
}): Promise<{ page: Page; browser: BrowserContext }> {
const browser = await launchBrowser({ headless: options?.headless });
const page = await browser.newPage();
return { page, browser };
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'output'],
boolean: ['screenshot', 'headless', 'interactive', 'help'],
default: {
headless: true,
screenshot: false,
interactive: false,
},
alias: {
u: 'url',
o: 'output',
s: 'screenshot',
h: 'help',
i: 'interactive',
},
});
if (args.help || !args.url) {
console.log(`
Web Browser with Camoufox
Usage:
npx tsx browse.ts --url <url> [options]
Options:
-u, --url <url> URL to navigate to (required)
-s, --screenshot Take a screenshot of the page
-o, --output <path> Output path for screenshot (default: screenshot.png)
--headless <bool> Run in headless mode (default: true)
--wait <ms> Wait time after page load in milliseconds
--timeout <ms> Navigation timeout (default: 60000)
-i, --interactive Keep browser open for manual interaction
-h, --help Show this help message
Examples:
npx tsx browse.ts --url "https://example.com"
npx tsx browse.ts --url "https://example.com" --screenshot --output page.png
npx tsx browse.ts --url "https://example.com" --headless false --interactive
Environment Variables:
CAMOUFOX_PROFILE_PATH Custom profile directory (default: ~/.camoufox-profile/)
CAMOUFOX_HEADLESS Default headless mode (true/false)
`);
process.exit(args.help ? 0 : 1);
}
try {
await browse({
url: args.url,
headless: args.headless,
screenshot: args.screenshot,
output: args.output,
wait: args.wait ? parseInt(args.wait, 10) : undefined,
timeout: args.timeout ? parseInt(args.timeout, 10) : undefined,
interactive: args.interactive,
});
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('browse.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,27 @@
{
"name": "web-automation-scripts",
"version": "1.0.0",
"description": "Web browsing and scraping scripts using Camoufox",
"type": "module",
"scripts": {
"browse": "tsx browse.ts",
"scrape": "tsx scrape.ts",
"fetch-browser": "npx camoufox-js fetch"
},
"dependencies": {
"camoufox-js": "^0.8.5",
"playwright-core": "^1.40.0",
"turndown": "^7.1.2",
"turndown-plugin-gfm": "^1.0.2",
"@mozilla/readability": "^0.5.0",
"jsdom": "^24.0.0",
"minimist": "^1.2.8"
},
"devDependencies": {
"typescript": "^5.3.0",
"@types/turndown": "^5.0.4",
"@types/jsdom": "^21.1.6",
"@types/minimist": "^1.2.5",
"tsx": "^4.7.0"
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,351 @@
#!/usr/bin/env npx tsx
/**
* Web scraper that extracts content to markdown
*
* Usage:
* npx tsx scrape.ts --url "https://example.com" --mode main
* npx tsx scrape.ts --url "https://example.com" --mode full --output page.md
* npx tsx scrape.ts --url "https://example.com" --mode selector --selector ".content"
*/
import TurndownService from 'turndown';
import * as turndownPluginGfm from 'turndown-plugin-gfm';
import { Readability } from '@mozilla/readability';
import { JSDOM } from 'jsdom';
import { writeFileSync } from 'fs';
import parseArgs from 'minimist';
import { getPage } from './browse.js';
// Types
type ScrapeMode = 'main' | 'full' | 'selector';
interface ScrapeOptions {
url: string;
mode: ScrapeMode;
selector?: string;
output?: string;
includeLinks?: boolean;
includeTables?: boolean;
includeImages?: boolean;
headless?: boolean;
wait?: number;
}
interface ScrapeResult {
title: string;
url: string;
markdown: string;
byline?: string;
excerpt?: string;
}
// Configure Turndown for markdown conversion
function createTurndownService(options: {
includeLinks?: boolean;
includeTables?: boolean;
includeImages?: boolean;
}): TurndownService {
const turndown = new TurndownService({
headingStyle: 'atx',
hr: '---',
bulletListMarker: '-',
codeBlockStyle: 'fenced',
fence: '```',
emDelimiter: '*',
strongDelimiter: '**',
linkStyle: 'inlined',
});
// Add GFM support (tables, strikethrough, task lists)
turndown.use(turndownPluginGfm.gfm);
// Custom rule for code blocks with language detection
turndown.addRule('codeBlockWithLanguage', {
filter: (node) => {
return (
node.nodeName === 'PRE' &&
node.firstChild?.nodeName === 'CODE'
);
},
replacement: (_content, node) => {
const codeNode = node.firstChild as HTMLElement;
const className = codeNode.getAttribute('class') || '';
const langMatch = className.match(/language-(\w+)/);
const lang = langMatch ? langMatch[1] : '';
const code = codeNode.textContent || '';
return `\n\n\`\`\`${lang}\n${code}\n\`\`\`\n\n`;
},
});
// Remove images if not included
if (!options.includeImages) {
turndown.addRule('removeImages', {
filter: 'img',
replacement: () => '',
});
}
// Remove links but keep text if not included
if (!options.includeLinks) {
turndown.addRule('removeLinks', {
filter: 'a',
replacement: (content) => content,
});
}
// Remove script, style, nav, footer, aside elements
turndown.remove(['script', 'style', 'nav', 'footer', 'aside', 'noscript']);
return turndown;
}
// Extract main content using Readability
function extractMainContent(html: string, url: string): {
content: string;
title: string;
byline?: string;
excerpt?: string;
} {
const dom = new JSDOM(html, { url });
const reader = new Readability(dom.window.document);
const article = reader.parse();
if (!article) {
throw new Error('Could not extract main content from page');
}
return {
content: article.content,
title: article.title,
byline: article.byline || undefined,
excerpt: article.excerpt || undefined,
};
}
// Scrape a URL and return markdown
export async function scrape(options: ScrapeOptions): Promise<ScrapeResult> {
const { page, browser } = await getPage({ headless: options.headless ?? true });
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, {
timeout: 60000,
waitUntil: 'domcontentloaded',
});
// Wait if specified
if (options.wait) {
console.log(`Waiting ${options.wait}ms for dynamic content...`);
await page.waitForTimeout(options.wait);
}
const pageTitle = await page.title();
const pageUrl = page.url();
let html: string;
let title = pageTitle;
let byline: string | undefined;
let excerpt: string | undefined;
// Get HTML based on mode
switch (options.mode) {
case 'main': {
// Get full page HTML and extract with Readability
const fullHtml = await page.content();
const extracted = extractMainContent(fullHtml, pageUrl);
html = extracted.content;
title = extracted.title || pageTitle;
byline = extracted.byline;
excerpt = extracted.excerpt;
break;
}
case 'selector': {
if (!options.selector) {
throw new Error('Selector mode requires --selector option');
}
const element = await page.$(options.selector);
if (!element) {
throw new Error(`Selector not found: ${options.selector}`);
}
html = await element.innerHTML();
break;
}
case 'full':
default: {
// Get body content, excluding common non-content elements
html = await page.evaluate(() => {
// Remove common non-content elements
const selectorsToRemove = [
'script', 'style', 'noscript', 'iframe',
'nav', 'header', 'footer', '.cookie-banner',
'.advertisement', '.ads', '#ads', '.social-share',
'.comments', '#comments', '.sidebar'
];
selectorsToRemove.forEach(selector => {
document.querySelectorAll(selector).forEach(el => el.remove());
});
return document.body.innerHTML;
});
break;
}
}
// Convert to markdown
const turndown = createTurndownService({
includeLinks: options.includeLinks ?? true,
includeTables: options.includeTables ?? true,
includeImages: options.includeImages ?? false,
});
let markdown = turndown.turndown(html);
// Add title as H1 if not already present
if (!markdown.startsWith('# ')) {
markdown = `# ${title}\n\n${markdown}`;
}
// Add metadata header
const metadataLines = [
`<!-- Scraped from: ${pageUrl} -->`,
byline ? `<!-- Author: ${byline} -->` : null,
excerpt ? `<!-- Excerpt: ${excerpt} -->` : null,
`<!-- Scraped at: ${new Date().toISOString()} -->`,
'',
].filter(Boolean);
markdown = metadataLines.join('\n') + '\n' + markdown;
// Clean up excessive whitespace
markdown = markdown
.replace(/\n{4,}/g, '\n\n\n')
.replace(/[ \t]+$/gm, '')
.trim();
const result: ScrapeResult = {
title,
url: pageUrl,
markdown,
byline,
excerpt,
};
// Save to file if output specified
if (options.output) {
writeFileSync(options.output, markdown, 'utf-8');
console.log(`Markdown saved to: ${options.output}`);
}
return result;
} finally {
await browser.close();
}
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'mode', 'selector', 'output'],
boolean: ['headless', 'links', 'tables', 'images', 'help'],
default: {
mode: 'main',
headless: true,
links: true,
tables: true,
images: false,
},
alias: {
u: 'url',
m: 'mode',
s: 'selector',
o: 'output',
h: 'help',
},
});
if (args.help || !args.url) {
console.log(`
Web Scraper - Extract content to Markdown
Usage:
npx tsx scrape.ts --url <url> [options]
Options:
-u, --url <url> URL to scrape (required)
-m, --mode <mode> Scrape mode: main, full, or selector (default: main)
-s, --selector <sel> CSS selector for selector mode
-o, --output <path> Output file path for markdown
--headless <bool> Run in headless mode (default: true)
--wait <ms> Wait time for dynamic content
--links Include links in output (default: true)
--tables Include tables in output (default: true)
--images Include images in output (default: false)
-h, --help Show this help message
Scrape Modes:
main Extract main article content using Readability (best for articles)
full Full page content with common elements removed
selector Extract specific element by CSS selector
Examples:
npx tsx scrape.ts --url "https://docs.example.com/guide" --mode main
npx tsx scrape.ts --url "https://example.com" --mode full --output page.md
npx tsx scrape.ts --url "https://example.com" --mode selector --selector ".api-docs"
npx tsx scrape.ts --url "https://example.com" --mode main --no-links --output clean.md
Output Format:
- GitHub Flavored Markdown (tables, strikethrough, task lists)
- Proper heading hierarchy
- Code blocks with language detection
- Metadata comments at top (source URL, date)
`);
process.exit(args.help ? 0 : 1);
}
const mode = args.mode as ScrapeMode;
if (!['main', 'full', 'selector'].includes(mode)) {
console.error(`Invalid mode: ${mode}. Must be main, full, or selector.`);
process.exit(1);
}
try {
const result = await scrape({
url: args.url,
mode,
selector: args.selector,
output: args.output,
includeLinks: args.links,
includeTables: args.tables,
includeImages: args.images,
headless: args.headless,
wait: args.wait ? parseInt(args.wait, 10) : undefined,
});
// Print result summary
console.log(`\nScrape complete:`);
console.log(` Title: ${result.title}`);
console.log(` URL: ${result.url}`);
if (result.byline) console.log(` Author: ${result.byline}`);
console.log(` Markdown length: ${result.markdown.length} chars`);
// Print markdown if not saved to file
if (!args.output) {
console.log('\n--- Markdown Output ---\n');
console.log(result.markdown);
}
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('scrape.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,39 @@
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { mkdirSync, existsSync } from 'fs';
async function test() {
const profilePath = join(homedir(), '.camoufox-profile');
if (!existsSync(profilePath)) {
mkdirSync(profilePath, { recursive: true });
}
console.log('Profile path:', profilePath);
console.log('Launching with full options...');
const browser = await Camoufox({
headless: true,
user_data_dir: profilePath,
// humanize: 1.5, // Test without this first
// geoip: true, // Test without this first
// enable_cache: true,
// block_webrtc: false,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://github.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await page.screenshot({ path: '/tmp/github-test.png' });
console.log('Screenshot saved');
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,22 @@
import { Camoufox } from 'camoufox-js';
async function test() {
console.log('Launching Camoufox with minimal config...');
const browser = await Camoufox({
headless: true,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://example.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,32 @@
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { mkdirSync, existsSync } from 'fs';
async function test() {
const profilePath = join(homedir(), '.camoufox-profile');
if (!existsSync(profilePath)) {
mkdirSync(profilePath, { recursive: true });
}
console.log('Profile path:', profilePath);
console.log('Launching with user_data_dir...');
const browser = await Camoufox({
headless: true,
user_data_dir: profilePath,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://example.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,16 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"strict": true,
"skipLibCheck": true,
"resolveJsonModule": true,
"outDir": "./dist",
"rootDir": "."
},
"include": ["*.ts"],
"exclude": ["node_modules", "dist"]
}

View File

@@ -0,0 +1,8 @@
declare module 'turndown-plugin-gfm' {
import TurndownService from 'turndown';
export function gfm(turndownService: TurndownService): void;
export function strikethrough(turndownService: TurndownService): void;
export function tables(turndownService: TurndownService): void;
export function taskListItems(turndownService: TurndownService): void;
}

View File

@@ -0,0 +1,36 @@
---
name: web-automation
description: Browse and scrape web pages using Playwright with Camoufox anti-detection browser. Use when automating web workflows, extracting page content to markdown, handling authenticated sessions, or scraping websites with bot protection.
---
# Web Automation with Camoufox (Codex)
Automated web browsing and scraping using Playwright with Camoufox anti-detection browser.
## Requirements
- Node.js 20+
- pnpm
- Network access to download browser binaries
## First-Time Setup
```bash
cd ~/.codex/skills/web-automation/scripts
pnpm install
npx camoufox-js fetch
```
If native dependency build errors appear (e.g., better-sqlite3), rebuild from the reported package directory.
## Quick Reference
- Browse page: `npx tsx browse.ts --url "https://example.com"`
- Scrape markdown: `npx tsx scrape.ts --url "https://example.com" --mode main --output page.md`
- Authenticate: `npx tsx auth.ts --url "https://example.com/login"`
## Notes
- Sessions persist in Camoufox profile storage.
- Use `--wait` for dynamic pages.
- Use `--mode selector --selector "..."` for targeted extraction.

View File

@@ -0,0 +1,575 @@
#!/usr/bin/env npx tsx
/**
* Authentication handler for web automation
* Supports generic form login and Microsoft SSO (MSAL)
*
* Usage:
* npx tsx auth.ts --url "https://example.com/login" --type form
* npx tsx auth.ts --url "https://example.com" --type msal
* npx tsx auth.ts --url "https://example.com" --type auto
*/
import { getPage, launchBrowser } from './browse.js';
import parseArgs from 'minimist';
import type { Page, BrowserContext } from 'playwright-core';
import { createInterface } from 'readline';
// Types
type AuthType = 'auto' | 'form' | 'msal';
interface AuthOptions {
url: string;
authType: AuthType;
credentials?: {
username: string;
password: string;
};
headless?: boolean;
timeout?: number;
}
interface AuthResult {
success: boolean;
finalUrl: string;
authType: AuthType;
message: string;
}
// Get credentials from environment or options
function getCredentials(options?: {
username?: string;
password?: string;
}): { username: string; password: string } | null {
const username = options?.username || process.env.CAMOUFOX_USERNAME;
const password = options?.password || process.env.CAMOUFOX_PASSWORD;
if (!username || !password) {
return null;
}
return { username, password };
}
// Prompt user for input (for MFA or credentials)
async function promptUser(question: string, hidden = false): Promise<string> {
const rl = createInterface({
input: process.stdin,
output: process.stdout,
});
return new Promise((resolve) => {
if (hidden) {
process.stdout.write(question);
// Note: This is a simple implementation. For production, use a proper hidden input library
}
rl.question(question, (answer) => {
rl.close();
resolve(answer);
});
});
}
// Detect authentication type from page
async function detectAuthType(page: Page): Promise<AuthType> {
const url = page.url();
// Check for Microsoft login
if (
url.includes('login.microsoftonline.com') ||
url.includes('login.live.com') ||
url.includes('login.windows.net')
) {
return 'msal';
}
// Check for common form login patterns
const hasLoginForm = await page.evaluate(() => {
const passwordField = document.querySelector(
'input[type="password"], input[name*="password"], input[id*="password"]'
);
const usernameField = document.querySelector(
'input[type="email"], input[type="text"][name*="user"], input[type="text"][name*="email"], input[id*="user"], input[id*="email"]'
);
return !!(passwordField && usernameField);
});
if (hasLoginForm) {
return 'form';
}
return 'auto';
}
// Handle generic form login
async function handleFormLogin(
page: Page,
credentials: { username: string; password: string },
timeout: number
): Promise<boolean> {
console.log('Attempting form login...');
// Find and fill username/email field
const usernameSelectors = [
'input[type="email"]',
'input[name*="user" i]',
'input[name*="email" i]',
'input[id*="user" i]',
'input[id*="email" i]',
'input[autocomplete="username"]',
'input[type="text"]:first-of-type',
];
let usernameField = null;
for (const selector of usernameSelectors) {
usernameField = await page.$(selector);
if (usernameField) break;
}
if (!usernameField) {
console.error('Could not find username/email field');
return false;
}
await usernameField.fill(credentials.username);
console.log('Filled username field');
// Find and fill password field
const passwordSelectors = [
'input[type="password"]',
'input[name*="password" i]',
'input[id*="password" i]',
'input[autocomplete="current-password"]',
];
let passwordField = null;
for (const selector of passwordSelectors) {
passwordField = await page.$(selector);
if (passwordField) break;
}
if (!passwordField) {
console.error('Could not find password field');
return false;
}
await passwordField.fill(credentials.password);
console.log('Filled password field');
// Check for "Remember me" checkbox and check it
const rememberCheckbox = await page.$(
'input[type="checkbox"][name*="remember" i], input[type="checkbox"][id*="remember" i]'
);
if (rememberCheckbox) {
await rememberCheckbox.check();
console.log('Checked "Remember me" checkbox');
}
// Find and click submit button
const submitSelectors = [
'button[type="submit"]',
'input[type="submit"]',
'button:has-text("Sign in")',
'button:has-text("Log in")',
'button:has-text("Login")',
'button:has-text("Submit")',
'[role="button"]:has-text("Sign in")',
];
let submitButton = null;
for (const selector of submitSelectors) {
submitButton = await page.$(selector);
if (submitButton) break;
}
if (!submitButton) {
// Try pressing Enter as fallback
await passwordField.press('Enter');
} else {
await submitButton.click();
}
console.log('Submitted login form');
// Wait for navigation or error
try {
await page.waitForNavigation({ timeout, waitUntil: 'domcontentloaded' });
return true;
} catch {
// Check if we're still on login page with error
const errorMessages = await page.$$eval(
'.error, .alert-danger, [role="alert"], .login-error',
(els) => els.map((el) => el.textContent?.trim()).filter(Boolean)
);
if (errorMessages.length > 0) {
console.error('Login error:', errorMessages.join(', '));
return false;
}
return true; // Might have succeeded without navigation
}
}
// Handle Microsoft SSO login
async function handleMsalLogin(
page: Page,
credentials: { username: string; password: string },
timeout: number
): Promise<boolean> {
console.log('Attempting Microsoft SSO login...');
const currentUrl = page.url();
// If not already on Microsoft login, wait for redirect
if (!currentUrl.includes('login.microsoftonline.com')) {
try {
await page.waitForURL('**/login.microsoftonline.com/**', { timeout: 10000 });
} catch {
console.log('Not redirected to Microsoft login');
return false;
}
}
// Wait for email input
const emailInput = await page.waitForSelector(
'input[type="email"], input[name="loginfmt"]',
{ timeout }
);
if (!emailInput) {
console.error('Could not find email input on Microsoft login');
return false;
}
// Fill email and submit
await emailInput.fill(credentials.username);
console.log('Filled email field');
const nextButton = await page.$('input[type="submit"], button[type="submit"]');
if (nextButton) {
await nextButton.click();
} else {
await emailInput.press('Enter');
}
// Wait for password page
try {
await page.waitForSelector(
'input[type="password"], input[name="passwd"]',
{ timeout }
);
} catch {
// Might be using passwordless auth or different flow
console.log('Password field not found - might be using different auth flow');
return false;
}
// Fill password
const passwordInput = await page.$('input[type="password"], input[name="passwd"]');
if (!passwordInput) {
console.error('Could not find password input');
return false;
}
await passwordInput.fill(credentials.password);
console.log('Filled password field');
// Submit
const signInButton = await page.$('input[type="submit"], button[type="submit"]');
if (signInButton) {
await signInButton.click();
} else {
await passwordInput.press('Enter');
}
// Handle "Stay signed in?" prompt
try {
const staySignedInButton = await page.waitForSelector(
'input[value="Yes"], button:has-text("Yes")',
{ timeout: 5000 }
);
if (staySignedInButton) {
await staySignedInButton.click();
console.log('Clicked "Stay signed in" button');
}
} catch {
// Prompt might not appear
}
// Check for Conditional Access Policy error
const caError = await page.$('text=Conditional Access policy');
if (caError) {
console.error('Blocked by Conditional Access Policy');
// Take screenshot for debugging
await page.screenshot({ path: 'ca-policy-error.png' });
console.log('Screenshot saved: ca-policy-error.png');
return false;
}
// Wait for redirect away from Microsoft login
try {
await page.waitForURL(
(url) => !url.href.includes('login.microsoftonline.com'),
{ timeout }
);
return true;
} catch {
return false;
}
}
// Check if user is already authenticated
async function isAuthenticated(page: Page, targetUrl: string): Promise<boolean> {
const currentUrl = page.url();
// If we're on the target URL (not a login page), we're likely authenticated
if (currentUrl.startsWith(targetUrl)) {
// Check for common login page indicators
const isLoginPage = await page.evaluate(() => {
const loginIndicators = [
'input[type="password"]',
'form[action*="login"]',
'form[action*="signin"]',
'.login-form',
'#login',
];
return loginIndicators.some((sel) => document.querySelector(sel) !== null);
});
return !isLoginPage;
}
return false;
}
// Main authentication function
export async function authenticate(options: AuthOptions): Promise<AuthResult> {
const browser = await launchBrowser({ headless: options.headless ?? true });
const page = await browser.newPage();
const timeout = options.timeout ?? 30000;
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, { timeout: 60000, waitUntil: 'domcontentloaded' });
// Check if already authenticated
if (await isAuthenticated(page, options.url)) {
return {
success: true,
finalUrl: page.url(),
authType: 'auto',
message: 'Already authenticated (session persisted from profile)',
};
}
// Get credentials
const credentials = options.credentials
? options.credentials
: getCredentials();
if (!credentials) {
// No credentials - open interactive browser
console.log('\nNo credentials provided. Opening browser for manual login...');
console.log('Please complete the login process manually.');
console.log('The session will be saved to your profile.');
// Switch to headed mode for manual login
await browser.close();
const interactiveBrowser = await launchBrowser({ headless: false });
const interactivePage = await interactiveBrowser.newPage();
await interactivePage.goto(options.url);
await promptUser('\nPress Enter when you have completed login...');
const finalUrl = interactivePage.url();
await interactiveBrowser.close();
return {
success: true,
finalUrl,
authType: 'auto',
message: 'Manual login completed - session saved to profile',
};
}
// Detect auth type if auto
let authType = options.authType;
if (authType === 'auto') {
authType = await detectAuthType(page);
console.log(`Detected auth type: ${authType}`);
}
// Handle authentication based on type
let success = false;
switch (authType) {
case 'msal':
success = await handleMsalLogin(page, credentials, timeout);
break;
case 'form':
default:
success = await handleFormLogin(page, credentials, timeout);
break;
}
const finalUrl = page.url();
return {
success,
finalUrl,
authType,
message: success
? `Authentication successful - session saved to profile`
: 'Authentication failed',
};
} finally {
await browser.close();
}
}
// Navigate to authenticated page (handles auth if needed)
export async function navigateAuthenticated(
url: string,
options?: {
credentials?: { username: string; password: string };
headless?: boolean;
}
): Promise<{ page: Page; browser: BrowserContext }> {
const { page, browser } = await getPage({ headless: options?.headless ?? true });
await page.goto(url, { timeout: 60000, waitUntil: 'domcontentloaded' });
// Check if we need to authenticate
if (!(await isAuthenticated(page, url))) {
console.log('Session expired or not authenticated. Attempting login...');
// Get credentials
const credentials = options?.credentials ?? getCredentials();
if (!credentials) {
throw new Error(
'Authentication required but no credentials provided. ' +
'Set CAMOUFOX_USERNAME and CAMOUFOX_PASSWORD environment variables.'
);
}
// Detect and handle auth
const authType = await detectAuthType(page);
let success = false;
if (authType === 'msal') {
success = await handleMsalLogin(page, credentials, 30000);
} else {
success = await handleFormLogin(page, credentials, 30000);
}
if (!success) {
await browser.close();
throw new Error('Authentication failed');
}
// Navigate back to original URL if we were redirected
if (!page.url().startsWith(url)) {
await page.goto(url, { timeout: 60000, waitUntil: 'domcontentloaded' });
}
}
return { page, browser };
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'type', 'username', 'password'],
boolean: ['headless', 'help'],
default: {
type: 'auto',
headless: false, // Default to headed for auth so user can see/interact
},
alias: {
u: 'url',
t: 'type',
h: 'help',
},
});
if (args.help || !args.url) {
console.log(`
Web Authentication Handler
Usage:
npx tsx auth.ts --url <url> [options]
Options:
-u, --url <url> URL to authenticate (required)
-t, --type <type> Auth type: auto, form, or msal (default: auto)
--username <user> Username/email (or set CAMOUFOX_USERNAME env var)
--password <pass> Password (or set CAMOUFOX_PASSWORD env var)
--headless <bool> Run in headless mode (default: false for auth)
-h, --help Show this help message
Auth Types:
auto Auto-detect authentication type
form Generic username/password form
msal Microsoft SSO (login.microsoftonline.com)
Environment Variables:
CAMOUFOX_USERNAME Default username/email for authentication
CAMOUFOX_PASSWORD Default password for authentication
Examples:
# Interactive login (no credentials, opens browser)
npx tsx auth.ts --url "https://example.com/login"
# Form login with credentials
npx tsx auth.ts --url "https://example.com/login" --type form \\
--username "user@example.com" --password "secret"
# Microsoft SSO login
CAMOUFOX_USERNAME=user@company.com CAMOUFOX_PASSWORD=secret \\
npx tsx auth.ts --url "https://internal.company.com" --type msal
Notes:
- Session is saved to ~/.camoufox-profile/ for persistence
- After successful auth, subsequent browses will be authenticated
- Use --headless false if you need to handle MFA manually
`);
process.exit(args.help ? 0 : 1);
}
const authType = args.type as AuthType;
if (!['auto', 'form', 'msal'].includes(authType)) {
console.error(`Invalid auth type: ${authType}. Must be auto, form, or msal.`);
process.exit(1);
}
try {
const result = await authenticate({
url: args.url,
authType,
credentials:
args.username && args.password
? { username: args.username, password: args.password }
: undefined,
headless: args.headless,
});
console.log(`\nAuthentication result:`);
console.log(` Success: ${result.success}`);
console.log(` Auth type: ${result.authType}`);
console.log(` Final URL: ${result.finalUrl}`);
console.log(` Message: ${result.message}`);
process.exit(result.success ? 0 : 1);
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('auth.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,195 @@
#!/usr/bin/env npx tsx
/**
* Browser launcher using Camoufox with persistent profile
*
* Usage:
* npx tsx browse.ts --url "https://example.com"
* npx tsx browse.ts --url "https://example.com" --screenshot --output page.png
* npx tsx browse.ts --url "https://example.com" --headless false --wait 5000
*/
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { existsSync, mkdirSync } from 'fs';
import parseArgs from 'minimist';
import type { Page, BrowserContext } from 'playwright-core';
// Types
interface BrowseOptions {
url: string;
headless?: boolean;
screenshot?: boolean;
output?: string;
wait?: number;
timeout?: number;
interactive?: boolean;
}
interface BrowseResult {
title: string;
url: string;
screenshotPath?: string;
}
// Get profile directory
const getProfilePath = (): string => {
const customPath = process.env.CAMOUFOX_PROFILE_PATH;
if (customPath) return customPath;
const profileDir = join(homedir(), '.camoufox-profile');
if (!existsSync(profileDir)) {
mkdirSync(profileDir, { recursive: true });
}
return profileDir;
};
// Launch browser with persistent profile
export async function launchBrowser(options: {
headless?: boolean;
}): Promise<BrowserContext> {
const profilePath = getProfilePath();
const headless =
options.headless ??
(process.env.CAMOUFOX_HEADLESS ? process.env.CAMOUFOX_HEADLESS === 'true' : true);
console.log(`Using profile: ${profilePath}`);
console.log(`Headless mode: ${headless}`);
const browser = await Camoufox({
user_data_dir: profilePath,
headless,
});
return browser;
}
// Browse to URL and optionally take screenshot
export async function browse(options: BrowseOptions): Promise<BrowseResult> {
const browser = await launchBrowser({ headless: options.headless });
const page = await browser.newPage();
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, {
timeout: options.timeout ?? 60000,
waitUntil: 'domcontentloaded',
});
// Wait if specified
if (options.wait) {
console.log(`Waiting ${options.wait}ms...`);
await page.waitForTimeout(options.wait);
}
const result: BrowseResult = {
title: await page.title(),
url: page.url(),
};
console.log(`Page title: ${result.title}`);
console.log(`Final URL: ${result.url}`);
// Take screenshot if requested
if (options.screenshot) {
const outputPath = options.output ?? 'screenshot.png';
await page.screenshot({ path: outputPath, fullPage: true });
result.screenshotPath = outputPath;
console.log(`Screenshot saved: ${outputPath}`);
}
// If interactive mode, keep browser open
if (options.interactive) {
console.log('\nInteractive mode - browser will stay open.');
console.log('Press Ctrl+C to close.');
await new Promise(() => {}); // Keep running
}
return result;
} finally {
if (!options.interactive) {
await browser.close();
}
}
}
// Export page for use in other scripts
export async function getPage(options?: {
headless?: boolean;
}): Promise<{ page: Page; browser: BrowserContext }> {
const browser = await launchBrowser({ headless: options?.headless });
const page = await browser.newPage();
return { page, browser };
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'output'],
boolean: ['screenshot', 'headless', 'interactive', 'help'],
default: {
headless: true,
screenshot: false,
interactive: false,
},
alias: {
u: 'url',
o: 'output',
s: 'screenshot',
h: 'help',
i: 'interactive',
},
});
if (args.help || !args.url) {
console.log(`
Web Browser with Camoufox
Usage:
npx tsx browse.ts --url <url> [options]
Options:
-u, --url <url> URL to navigate to (required)
-s, --screenshot Take a screenshot of the page
-o, --output <path> Output path for screenshot (default: screenshot.png)
--headless <bool> Run in headless mode (default: true)
--wait <ms> Wait time after page load in milliseconds
--timeout <ms> Navigation timeout (default: 60000)
-i, --interactive Keep browser open for manual interaction
-h, --help Show this help message
Examples:
npx tsx browse.ts --url "https://example.com"
npx tsx browse.ts --url "https://example.com" --screenshot --output page.png
npx tsx browse.ts --url "https://example.com" --headless false --interactive
Environment Variables:
CAMOUFOX_PROFILE_PATH Custom profile directory (default: ~/.camoufox-profile/)
CAMOUFOX_HEADLESS Default headless mode (true/false)
`);
process.exit(args.help ? 0 : 1);
}
try {
await browse({
url: args.url,
headless: args.headless,
screenshot: args.screenshot,
output: args.output,
wait: args.wait ? parseInt(args.wait, 10) : undefined,
timeout: args.timeout ? parseInt(args.timeout, 10) : undefined,
interactive: args.interactive,
});
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('browse.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,29 @@
{
"name": "web-automation-scripts",
"version": "1.0.0",
"description": "Web browsing and scraping scripts using Camoufox",
"type": "module",
"scripts": {
"browse": "tsx browse.ts",
"scrape": "tsx scrape.ts",
"fetch-browser": "npx camoufox-js fetch"
},
"dependencies": {
"@mozilla/readability": "^0.5.0",
"camoufox-js": "^0.8.5",
"jsdom": "^24.0.0",
"minimist": "^1.2.8",
"playwright-core": "^1.40.0",
"turndown": "^7.1.2",
"turndown-plugin-gfm": "^1.0.2"
},
"devDependencies": {
"@types/jsdom": "^21.1.6",
"@types/minimist": "^1.2.5",
"@types/turndown": "^5.0.4",
"esbuild": "0.27.0",
"tsx": "^4.7.0",
"typescript": "^5.3.0"
},
"packageManager": "pnpm@10.18.1+sha512.77a884a165cbba2d8d1c19e3b4880eee6d2fcabd0d879121e282196b80042351d5eb3ca0935fa599da1dc51265cc68816ad2bddd2a2de5ea9fdf92adbec7cd34"
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,212 @@
import { writeFileSync } from 'fs';
import { getPage } from './browse.js';
const baseUrl = 'http://localhost:3000';
const username = 'analyst@fhb.local';
const password = process.env.CAMOUFOX_PASSWORD ?? '';
const reportPath = '/Users/stefano.fiorini/Documents/projects/fhb-loan-spreading-pilot-a/docs/plans/2026-01-24-financials-analysis-redesign/web-automation-scan.md';
type NavResult = {
requestedUrl: string;
url: string;
status: number | null;
title: string;
error?: string;
};
async function gotoWithStatus(page: any, url: string): Promise<NavResult> {
const resp = await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 60000 }).catch((e: unknown) => ({ error: e }));
if (resp?.error) {
return {
requestedUrl: url,
url: page.url(),
status: null,
title: await page.title().catch(() => ''),
error: String(resp.error),
};
}
return {
requestedUrl: url,
url: page.url(),
status: resp ? resp.status() : null,
title: await page.title().catch(() => ''),
};
}
async function textOrNull(page: any, selector: string): Promise<string | null> {
const loc = page.locator(selector).first();
try {
if ((await loc.count()) === 0) return null;
const txt = await loc.textContent();
return txt ? txt.trim().replace(/\s+/g, ' ') : null;
} catch {
return null;
}
}
async function main() {
const { page, browser } = await getPage({ headless: true });
const lines: string[] = [];
lines.push('# Web Automation Scan (local)');
lines.push('');
lines.push(`- Base URL: ${baseUrl}`);
lines.push(`- Timestamp: ${new Date().toISOString()}`);
lines.push('');
try {
lines.push('## Login');
await gotoWithStatus(page, `${baseUrl}/login`);
await page.locator('input[name="email"]').fill(username);
await page.locator('input[name="password"]').fill(password);
await page.locator('button[type="submit"]').click();
await page.waitForTimeout(2500);
const cookies = await page.context().cookies();
const sessionCookie = cookies.find((c: any) => c.name === 'fhb_session');
lines.push(`- After submit URL: ${page.url()}`);
lines.push(`- Has session cookie (fhb_session): ${Boolean(sessionCookie)}`);
lines.push('');
lines.push('## Demo Case');
const casesNav = await gotoWithStatus(page, `${baseUrl}/cases`);
lines.push(`- GET /cases → status ${casesNav.status ?? 'ERR'}, final ${casesNav.url}`);
const envCaseId = process.env.SCAN_CASE_ID?.trim() || null;
let selectedCaseId: string | null = envCaseId;
if (!selectedCaseId) {
const caseLinks = await page.$$eval('a[href^="/cases/"]', (as) =>
as
.map((a) => ({
href: (a as HTMLAnchorElement).getAttribute('href') || '',
text: (a.textContent || '').trim(),
}))
.filter((x) => x.href.includes('/cases/'))
);
const preferredTitles = ['Demo - Strong Borrower', 'Demo - Weak Borrower', 'Demo - Incomplete'];
for (const title of preferredTitles) {
const match = caseLinks.find((l) => l.text.includes(title) && l.href.includes('/cases/'));
const href = match?.href ?? '';
const m = href.match(/\/cases\/([0-9a-f-]{36})/i);
if (m) {
selectedCaseId = m[1];
break;
}
}
if (!selectedCaseId) {
const firstHref =
caseLinks.map((l) => l.href).find((h) => /\/cases\/[0-9a-f-]{36}/i.test(h)) ?? null;
const m = firstHref?.match(/\/cases\/([0-9a-f-]{36})/i) ?? null;
selectedCaseId = m?.[1] ?? null;
}
}
lines.push(`- Selected caseId: ${selectedCaseId ?? '(none found)'}`);
if (!selectedCaseId) {
lines.push('');
lines.push('⚠️ Could not find a demo case link on /cases.');
writeFileSync(reportPath, lines.join('\n') + '\n', 'utf-8');
return;
}
const caseBase = `${baseUrl}/cases/${selectedCaseId}/journey`;
lines.push('');
lines.push('## Route Checks');
const routesToCheck = [
`${caseBase}`,
`${caseBase}/financials`,
`${caseBase}/financials/income`,
`${caseBase}/analysis`,
`${caseBase}/analysis/configure`,
`${caseBase}/analysis/ai`,
`${caseBase}/analysis/ai/detail`,
`${caseBase}/spreads`,
];
for (const url of routesToCheck) {
const r = await gotoWithStatus(page, url);
const h1 = await textOrNull(page, 'h1');
const finalPath = r.url.startsWith(baseUrl) ? r.url.slice(baseUrl.length) : r.url;
lines.push(`- ${url.slice(baseUrl.length)} → status ${r.status ?? 'ERR'} (final ${finalPath})${h1 ? `, h1="${h1}"` : ''}`);
}
lines.push('');
lines.push('## Spreadsheet Analysis (UI)');
await gotoWithStatus(page, `${caseBase}/analysis/configure`);
const runButton = page.locator('button:has-text("Run Analysis")').first();
const disabled = await runButton.isDisabled().catch(() => true);
lines.push(`- Run button disabled: ${disabled}`);
if (!disabled) {
await runButton.click();
const resultsWait = page
.waitForURL('**/journey/analysis/results**', { timeout: 180000 })
.then(() => 'results' as const);
const errorWait = page
.locator('[role="alert"]')
.filter({ hasText: 'Error' })
.first()
.waitFor({ timeout: 180000 })
.then(() => 'error' as const);
const outcome = await Promise.race([resultsWait, errorWait]).catch(() => 'timeout' as const);
if (outcome === 'results') {
await page.waitForTimeout(1500);
lines.push(`- Results URL: ${page.url().replace(baseUrl, '')}`);
const downloadHref = await page
.locator('a[href*="/journey/analysis/download"]')
.first()
.getAttribute('href')
.catch(() => null);
if (downloadHref) {
const dlUrl = downloadHref.startsWith('http') ? downloadHref : `${baseUrl}${downloadHref}`;
const dlResp = await page.goto(dlUrl, { waitUntil: 'commit', timeout: 60000 }).catch(() => null);
lines.push(
`- Download route status: ${dlResp?.status() ?? 'ERR'} (Content-Type: ${dlResp?.headers()?.['content-type'] ?? 'n/a'})`
);
} else {
lines.push('- Download link not found on results page');
}
} else if (outcome === 'error') {
const errorText = await page
.locator('[role="alert"]')
.first()
.textContent()
.then((t: string | null) => (t ? t.trim().replace(/\\s+/g, ' ') : null))
.catch(() => null);
lines.push(`- Stayed on configure page; saw error callout: ${errorText ?? '(unable to read)'}`);
lines.push('- Skipping download check because analysis did not complete.');
} else {
lines.push('- Timed out waiting for results or error after clicking Run Analysis.');
}
} else {
lines.push('- Skipped running analysis because Run button was disabled.');
}
lines.push('');
lines.push('## Notes');
lines.push('- This scan avoids scraping financial values; it records route availability and basic headings.');
writeFileSync(reportPath, lines.join('\n') + '\n', 'utf-8');
} finally {
await browser.close();
}
}
main().catch((err) => {
console.error(err);
process.exitCode = 1;
});

View File

@@ -0,0 +1,351 @@
#!/usr/bin/env npx tsx
/**
* Web scraper that extracts content to markdown
*
* Usage:
* npx tsx scrape.ts --url "https://example.com" --mode main
* npx tsx scrape.ts --url "https://example.com" --mode full --output page.md
* npx tsx scrape.ts --url "https://example.com" --mode selector --selector ".content"
*/
import TurndownService from 'turndown';
import * as turndownPluginGfm from 'turndown-plugin-gfm';
import { Readability } from '@mozilla/readability';
import { JSDOM } from 'jsdom';
import { writeFileSync } from 'fs';
import parseArgs from 'minimist';
import { getPage } from './browse.js';
// Types
type ScrapeMode = 'main' | 'full' | 'selector';
interface ScrapeOptions {
url: string;
mode: ScrapeMode;
selector?: string;
output?: string;
includeLinks?: boolean;
includeTables?: boolean;
includeImages?: boolean;
headless?: boolean;
wait?: number;
}
interface ScrapeResult {
title: string;
url: string;
markdown: string;
byline?: string;
excerpt?: string;
}
// Configure Turndown for markdown conversion
function createTurndownService(options: {
includeLinks?: boolean;
includeTables?: boolean;
includeImages?: boolean;
}): TurndownService {
const turndown = new TurndownService({
headingStyle: 'atx',
hr: '---',
bulletListMarker: '-',
codeBlockStyle: 'fenced',
fence: '```',
emDelimiter: '*',
strongDelimiter: '**',
linkStyle: 'inlined',
});
// Add GFM support (tables, strikethrough, task lists)
turndown.use(turndownPluginGfm.gfm);
// Custom rule for code blocks with language detection
turndown.addRule('codeBlockWithLanguage', {
filter: (node) => {
return (
node.nodeName === 'PRE' &&
node.firstChild?.nodeName === 'CODE'
);
},
replacement: (_content, node) => {
const codeNode = node.firstChild as HTMLElement;
const className = codeNode.getAttribute('class') || '';
const langMatch = className.match(/language-(\w+)/);
const lang = langMatch ? langMatch[1] : '';
const code = codeNode.textContent || '';
return `\n\n\`\`\`${lang}\n${code}\n\`\`\`\n\n`;
},
});
// Remove images if not included
if (!options.includeImages) {
turndown.addRule('removeImages', {
filter: 'img',
replacement: () => '',
});
}
// Remove links but keep text if not included
if (!options.includeLinks) {
turndown.addRule('removeLinks', {
filter: 'a',
replacement: (content) => content,
});
}
// Remove script, style, nav, footer, aside elements
turndown.remove(['script', 'style', 'nav', 'footer', 'aside', 'noscript']);
return turndown;
}
// Extract main content using Readability
function extractMainContent(html: string, url: string): {
content: string;
title: string;
byline?: string;
excerpt?: string;
} {
const dom = new JSDOM(html, { url });
const reader = new Readability(dom.window.document);
const article = reader.parse();
if (!article) {
throw new Error('Could not extract main content from page');
}
return {
content: article.content,
title: article.title,
byline: article.byline || undefined,
excerpt: article.excerpt || undefined,
};
}
// Scrape a URL and return markdown
export async function scrape(options: ScrapeOptions): Promise<ScrapeResult> {
const { page, browser } = await getPage({ headless: options.headless ?? true });
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, {
timeout: 60000,
waitUntil: 'domcontentloaded',
});
// Wait if specified
if (options.wait) {
console.log(`Waiting ${options.wait}ms for dynamic content...`);
await page.waitForTimeout(options.wait);
}
const pageTitle = await page.title();
const pageUrl = page.url();
let html: string;
let title = pageTitle;
let byline: string | undefined;
let excerpt: string | undefined;
// Get HTML based on mode
switch (options.mode) {
case 'main': {
// Get full page HTML and extract with Readability
const fullHtml = await page.content();
const extracted = extractMainContent(fullHtml, pageUrl);
html = extracted.content;
title = extracted.title || pageTitle;
byline = extracted.byline;
excerpt = extracted.excerpt;
break;
}
case 'selector': {
if (!options.selector) {
throw new Error('Selector mode requires --selector option');
}
const element = await page.$(options.selector);
if (!element) {
throw new Error(`Selector not found: ${options.selector}`);
}
html = await element.innerHTML();
break;
}
case 'full':
default: {
// Get body content, excluding common non-content elements
html = await page.evaluate(() => {
// Remove common non-content elements
const selectorsToRemove = [
'script', 'style', 'noscript', 'iframe',
'nav', 'header', 'footer', '.cookie-banner',
'.advertisement', '.ads', '#ads', '.social-share',
'.comments', '#comments', '.sidebar'
];
selectorsToRemove.forEach(selector => {
document.querySelectorAll(selector).forEach(el => el.remove());
});
return document.body.innerHTML;
});
break;
}
}
// Convert to markdown
const turndown = createTurndownService({
includeLinks: options.includeLinks ?? true,
includeTables: options.includeTables ?? true,
includeImages: options.includeImages ?? false,
});
let markdown = turndown.turndown(html);
// Add title as H1 if not already present
if (!markdown.startsWith('# ')) {
markdown = `# ${title}\n\n${markdown}`;
}
// Add metadata header
const metadataLines = [
`<!-- Scraped from: ${pageUrl} -->`,
byline ? `<!-- Author: ${byline} -->` : null,
excerpt ? `<!-- Excerpt: ${excerpt} -->` : null,
`<!-- Scraped at: ${new Date().toISOString()} -->`,
'',
].filter(Boolean);
markdown = metadataLines.join('\n') + '\n' + markdown;
// Clean up excessive whitespace
markdown = markdown
.replace(/\n{4,}/g, '\n\n\n')
.replace(/[ \t]+$/gm, '')
.trim();
const result: ScrapeResult = {
title,
url: pageUrl,
markdown,
byline,
excerpt,
};
// Save to file if output specified
if (options.output) {
writeFileSync(options.output, markdown, 'utf-8');
console.log(`Markdown saved to: ${options.output}`);
}
return result;
} finally {
await browser.close();
}
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'mode', 'selector', 'output'],
boolean: ['headless', 'links', 'tables', 'images', 'help'],
default: {
mode: 'main',
headless: true,
links: true,
tables: true,
images: false,
},
alias: {
u: 'url',
m: 'mode',
s: 'selector',
o: 'output',
h: 'help',
},
});
if (args.help || !args.url) {
console.log(`
Web Scraper - Extract content to Markdown
Usage:
npx tsx scrape.ts --url <url> [options]
Options:
-u, --url <url> URL to scrape (required)
-m, --mode <mode> Scrape mode: main, full, or selector (default: main)
-s, --selector <sel> CSS selector for selector mode
-o, --output <path> Output file path for markdown
--headless <bool> Run in headless mode (default: true)
--wait <ms> Wait time for dynamic content
--links Include links in output (default: true)
--tables Include tables in output (default: true)
--images Include images in output (default: false)
-h, --help Show this help message
Scrape Modes:
main Extract main article content using Readability (best for articles)
full Full page content with common elements removed
selector Extract specific element by CSS selector
Examples:
npx tsx scrape.ts --url "https://docs.example.com/guide" --mode main
npx tsx scrape.ts --url "https://example.com" --mode full --output page.md
npx tsx scrape.ts --url "https://example.com" --mode selector --selector ".api-docs"
npx tsx scrape.ts --url "https://example.com" --mode main --no-links --output clean.md
Output Format:
- GitHub Flavored Markdown (tables, strikethrough, task lists)
- Proper heading hierarchy
- Code blocks with language detection
- Metadata comments at top (source URL, date)
`);
process.exit(args.help ? 0 : 1);
}
const mode = args.mode as ScrapeMode;
if (!['main', 'full', 'selector'].includes(mode)) {
console.error(`Invalid mode: ${mode}. Must be main, full, or selector.`);
process.exit(1);
}
try {
const result = await scrape({
url: args.url,
mode,
selector: args.selector,
output: args.output,
includeLinks: args.links,
includeTables: args.tables,
includeImages: args.images,
headless: args.headless,
wait: args.wait ? parseInt(args.wait, 10) : undefined,
});
// Print result summary
console.log(`\nScrape complete:`);
console.log(` Title: ${result.title}`);
console.log(` URL: ${result.url}`);
if (result.byline) console.log(` Author: ${result.byline}`);
console.log(` Markdown length: ${result.markdown.length} chars`);
// Print markdown if not saved to file
if (!args.output) {
console.log('\n--- Markdown Output ---\n');
console.log(result.markdown);
}
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('scrape.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,39 @@
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { mkdirSync, existsSync } from 'fs';
async function test() {
const profilePath = join(homedir(), '.camoufox-profile');
if (!existsSync(profilePath)) {
mkdirSync(profilePath, { recursive: true });
}
console.log('Profile path:', profilePath);
console.log('Launching with full options...');
const browser = await Camoufox({
headless: true,
user_data_dir: profilePath,
// humanize: 1.5, // Test without this first
// geoip: true, // Test without this first
// enable_cache: true,
// block_webrtc: false,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://github.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await page.screenshot({ path: '/tmp/github-test.png' });
console.log('Screenshot saved');
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,22 @@
import { Camoufox } from 'camoufox-js';
async function test() {
console.log('Launching Camoufox with minimal config...');
const browser = await Camoufox({
headless: true,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://example.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,32 @@
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { mkdirSync, existsSync } from 'fs';
async function test() {
const profilePath = join(homedir(), '.camoufox-profile');
if (!existsSync(profilePath)) {
mkdirSync(profilePath, { recursive: true });
}
console.log('Profile path:', profilePath);
console.log('Launching with user_data_dir...');
const browser = await Camoufox({
headless: true,
user_data_dir: profilePath,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://example.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,78 @@
import { getPage } from './browse.js';
type Extracted = {
title: string;
url: string;
colorVars: Array<[string, string]>;
samples: Record<string, null | { background: string; color: string; border: string }>;
};
function isColorValue(value: string) {
return /#([0-9a-f]{3,4}|[0-9a-f]{6}|[0-9a-f]{8})\b/i.test(value) || /\brgb\(|\bhsl\(/i.test(value);
}
async function main() {
const url = process.argv[2] ?? 'https://www.firsthorizon.com';
const { page, browser } = await getPage({ headless: true });
try {
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 60000 });
await page.waitForTimeout(5000);
const data = await page.evaluate(`(() => {
const rootStyles = getComputedStyle(document.documentElement);
const vars = {};
for (let i = 0; i < rootStyles.length; i++) {
const prop = rootStyles[i];
if (prop && prop.startsWith('--')) {
vars[prop] = rootStyles.getPropertyValue(prop).trim();
}
}
const pick = (selector) => {
const el = document.querySelector(selector);
if (!el) return null;
const cs = getComputedStyle(el);
return {
background: cs.backgroundColor,
color: cs.color,
border: cs.borderColor,
};
};
return {
title: document.title,
url: location.href,
vars,
samples: {
body: pick('body'),
header: pick('header'),
nav: pick('nav'),
primaryButton: pick('button, [role="button"], a[role="button"], a.button, .button'),
link: pick('a'),
},
};
})()`);
const entries = Object.entries(data.vars) as Array<[string, string]>;
const colorVars = entries
.filter(([, v]) => v && isColorValue(v))
.sort((a, b) => a[0].localeCompare(b[0]));
const out: Extracted = {
title: data.title,
url: data.url,
colorVars,
samples: data.samples,
};
process.stdout.write(JSON.stringify(out, null, 2));
} finally {
await browser.close();
}
}
main().catch((error) => {
console.error(error);
process.exit(1);
});

View File

@@ -0,0 +1,16 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"strict": true,
"skipLibCheck": true,
"resolveJsonModule": true,
"outDir": "./dist",
"rootDir": "."
},
"include": ["*.ts"],
"exclude": ["node_modules", "dist"]
}

View File

@@ -0,0 +1,8 @@
declare module 'turndown-plugin-gfm' {
import TurndownService from 'turndown';
export function gfm(turndownService: TurndownService): void;
export function strikethrough(turndownService: TurndownService): void;
export function tables(turndownService: TurndownService): void;
export function taskListItems(turndownService: TurndownService): void;
}

View File

@@ -0,0 +1,37 @@
---
name: web-automation
description: Browse and scrape web pages using Playwright with Camoufox anti-detection browser. Use when automating web workflows, extracting page content to markdown, handling authenticated sessions, or scraping websites with bot protection.
---
# Web Automation with Camoufox (OpenCode)
Automated web browsing and scraping using Playwright with Camoufox anti-detection browser.
## Requirements
- Node.js 20+
- pnpm
- Network access to download browser binaries
## First-Time Setup
```bash
# Use your OpenCode skills path
cd ~/.opencode/skills/web-automation/scripts
pnpm install
npx camoufox-js fetch
```
If native dependency build errors appear (e.g., better-sqlite3), rebuild from the reported package directory.
## Quick Reference
- Browse page: `npx tsx browse.ts --url "https://example.com"`
- Scrape markdown: `npx tsx scrape.ts --url "https://example.com" --mode main --output page.md`
- Authenticate: `npx tsx auth.ts --url "https://example.com/login"`
## Notes
- Sessions persist in Camoufox profile storage.
- Use `--wait` for dynamic pages.
- Use `--mode selector --selector "..."` for targeted extraction.

View File

@@ -0,0 +1,575 @@
#!/usr/bin/env npx tsx
/**
* Authentication handler for web automation
* Supports generic form login and Microsoft SSO (MSAL)
*
* Usage:
* npx tsx auth.ts --url "https://example.com/login" --type form
* npx tsx auth.ts --url "https://example.com" --type msal
* npx tsx auth.ts --url "https://example.com" --type auto
*/
import { getPage, launchBrowser } from './browse.js';
import parseArgs from 'minimist';
import type { Page, BrowserContext } from 'playwright-core';
import { createInterface } from 'readline';
// Types
type AuthType = 'auto' | 'form' | 'msal';
interface AuthOptions {
url: string;
authType: AuthType;
credentials?: {
username: string;
password: string;
};
headless?: boolean;
timeout?: number;
}
interface AuthResult {
success: boolean;
finalUrl: string;
authType: AuthType;
message: string;
}
// Get credentials from environment or options
function getCredentials(options?: {
username?: string;
password?: string;
}): { username: string; password: string } | null {
const username = options?.username || process.env.CAMOUFOX_USERNAME;
const password = options?.password || process.env.CAMOUFOX_PASSWORD;
if (!username || !password) {
return null;
}
return { username, password };
}
// Prompt user for input (for MFA or credentials)
async function promptUser(question: string, hidden = false): Promise<string> {
const rl = createInterface({
input: process.stdin,
output: process.stdout,
});
return new Promise((resolve) => {
if (hidden) {
process.stdout.write(question);
// Note: This is a simple implementation. For production, use a proper hidden input library
}
rl.question(question, (answer) => {
rl.close();
resolve(answer);
});
});
}
// Detect authentication type from page
async function detectAuthType(page: Page): Promise<AuthType> {
const url = page.url();
// Check for Microsoft login
if (
url.includes('login.microsoftonline.com') ||
url.includes('login.live.com') ||
url.includes('login.windows.net')
) {
return 'msal';
}
// Check for common form login patterns
const hasLoginForm = await page.evaluate(() => {
const passwordField = document.querySelector(
'input[type="password"], input[name*="password"], input[id*="password"]'
);
const usernameField = document.querySelector(
'input[type="email"], input[type="text"][name*="user"], input[type="text"][name*="email"], input[id*="user"], input[id*="email"]'
);
return !!(passwordField && usernameField);
});
if (hasLoginForm) {
return 'form';
}
return 'auto';
}
// Handle generic form login
async function handleFormLogin(
page: Page,
credentials: { username: string; password: string },
timeout: number
): Promise<boolean> {
console.log('Attempting form login...');
// Find and fill username/email field
const usernameSelectors = [
'input[type="email"]',
'input[name*="user" i]',
'input[name*="email" i]',
'input[id*="user" i]',
'input[id*="email" i]',
'input[autocomplete="username"]',
'input[type="text"]:first-of-type',
];
let usernameField = null;
for (const selector of usernameSelectors) {
usernameField = await page.$(selector);
if (usernameField) break;
}
if (!usernameField) {
console.error('Could not find username/email field');
return false;
}
await usernameField.fill(credentials.username);
console.log('Filled username field');
// Find and fill password field
const passwordSelectors = [
'input[type="password"]',
'input[name*="password" i]',
'input[id*="password" i]',
'input[autocomplete="current-password"]',
];
let passwordField = null;
for (const selector of passwordSelectors) {
passwordField = await page.$(selector);
if (passwordField) break;
}
if (!passwordField) {
console.error('Could not find password field');
return false;
}
await passwordField.fill(credentials.password);
console.log('Filled password field');
// Check for "Remember me" checkbox and check it
const rememberCheckbox = await page.$(
'input[type="checkbox"][name*="remember" i], input[type="checkbox"][id*="remember" i]'
);
if (rememberCheckbox) {
await rememberCheckbox.check();
console.log('Checked "Remember me" checkbox');
}
// Find and click submit button
const submitSelectors = [
'button[type="submit"]',
'input[type="submit"]',
'button:has-text("Sign in")',
'button:has-text("Log in")',
'button:has-text("Login")',
'button:has-text("Submit")',
'[role="button"]:has-text("Sign in")',
];
let submitButton = null;
for (const selector of submitSelectors) {
submitButton = await page.$(selector);
if (submitButton) break;
}
if (!submitButton) {
// Try pressing Enter as fallback
await passwordField.press('Enter');
} else {
await submitButton.click();
}
console.log('Submitted login form');
// Wait for navigation or error
try {
await page.waitForNavigation({ timeout, waitUntil: 'domcontentloaded' });
return true;
} catch {
// Check if we're still on login page with error
const errorMessages = await page.$$eval(
'.error, .alert-danger, [role="alert"], .login-error',
(els) => els.map((el) => el.textContent?.trim()).filter(Boolean)
);
if (errorMessages.length > 0) {
console.error('Login error:', errorMessages.join(', '));
return false;
}
return true; // Might have succeeded without navigation
}
}
// Handle Microsoft SSO login
async function handleMsalLogin(
page: Page,
credentials: { username: string; password: string },
timeout: number
): Promise<boolean> {
console.log('Attempting Microsoft SSO login...');
const currentUrl = page.url();
// If not already on Microsoft login, wait for redirect
if (!currentUrl.includes('login.microsoftonline.com')) {
try {
await page.waitForURL('**/login.microsoftonline.com/**', { timeout: 10000 });
} catch {
console.log('Not redirected to Microsoft login');
return false;
}
}
// Wait for email input
const emailInput = await page.waitForSelector(
'input[type="email"], input[name="loginfmt"]',
{ timeout }
);
if (!emailInput) {
console.error('Could not find email input on Microsoft login');
return false;
}
// Fill email and submit
await emailInput.fill(credentials.username);
console.log('Filled email field');
const nextButton = await page.$('input[type="submit"], button[type="submit"]');
if (nextButton) {
await nextButton.click();
} else {
await emailInput.press('Enter');
}
// Wait for password page
try {
await page.waitForSelector(
'input[type="password"], input[name="passwd"]',
{ timeout }
);
} catch {
// Might be using passwordless auth or different flow
console.log('Password field not found - might be using different auth flow');
return false;
}
// Fill password
const passwordInput = await page.$('input[type="password"], input[name="passwd"]');
if (!passwordInput) {
console.error('Could not find password input');
return false;
}
await passwordInput.fill(credentials.password);
console.log('Filled password field');
// Submit
const signInButton = await page.$('input[type="submit"], button[type="submit"]');
if (signInButton) {
await signInButton.click();
} else {
await passwordInput.press('Enter');
}
// Handle "Stay signed in?" prompt
try {
const staySignedInButton = await page.waitForSelector(
'input[value="Yes"], button:has-text("Yes")',
{ timeout: 5000 }
);
if (staySignedInButton) {
await staySignedInButton.click();
console.log('Clicked "Stay signed in" button');
}
} catch {
// Prompt might not appear
}
// Check for Conditional Access Policy error
const caError = await page.$('text=Conditional Access policy');
if (caError) {
console.error('Blocked by Conditional Access Policy');
// Take screenshot for debugging
await page.screenshot({ path: 'ca-policy-error.png' });
console.log('Screenshot saved: ca-policy-error.png');
return false;
}
// Wait for redirect away from Microsoft login
try {
await page.waitForURL(
(url) => !url.href.includes('login.microsoftonline.com'),
{ timeout }
);
return true;
} catch {
return false;
}
}
// Check if user is already authenticated
async function isAuthenticated(page: Page, targetUrl: string): Promise<boolean> {
const currentUrl = page.url();
// If we're on the target URL (not a login page), we're likely authenticated
if (currentUrl.startsWith(targetUrl)) {
// Check for common login page indicators
const isLoginPage = await page.evaluate(() => {
const loginIndicators = [
'input[type="password"]',
'form[action*="login"]',
'form[action*="signin"]',
'.login-form',
'#login',
];
return loginIndicators.some((sel) => document.querySelector(sel) !== null);
});
return !isLoginPage;
}
return false;
}
// Main authentication function
export async function authenticate(options: AuthOptions): Promise<AuthResult> {
const browser = await launchBrowser({ headless: options.headless ?? true });
const page = await browser.newPage();
const timeout = options.timeout ?? 30000;
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, { timeout: 60000, waitUntil: 'domcontentloaded' });
// Check if already authenticated
if (await isAuthenticated(page, options.url)) {
return {
success: true,
finalUrl: page.url(),
authType: 'auto',
message: 'Already authenticated (session persisted from profile)',
};
}
// Get credentials
const credentials = options.credentials
? options.credentials
: getCredentials();
if (!credentials) {
// No credentials - open interactive browser
console.log('\nNo credentials provided. Opening browser for manual login...');
console.log('Please complete the login process manually.');
console.log('The session will be saved to your profile.');
// Switch to headed mode for manual login
await browser.close();
const interactiveBrowser = await launchBrowser({ headless: false });
const interactivePage = await interactiveBrowser.newPage();
await interactivePage.goto(options.url);
await promptUser('\nPress Enter when you have completed login...');
const finalUrl = interactivePage.url();
await interactiveBrowser.close();
return {
success: true,
finalUrl,
authType: 'auto',
message: 'Manual login completed - session saved to profile',
};
}
// Detect auth type if auto
let authType = options.authType;
if (authType === 'auto') {
authType = await detectAuthType(page);
console.log(`Detected auth type: ${authType}`);
}
// Handle authentication based on type
let success = false;
switch (authType) {
case 'msal':
success = await handleMsalLogin(page, credentials, timeout);
break;
case 'form':
default:
success = await handleFormLogin(page, credentials, timeout);
break;
}
const finalUrl = page.url();
return {
success,
finalUrl,
authType,
message: success
? `Authentication successful - session saved to profile`
: 'Authentication failed',
};
} finally {
await browser.close();
}
}
// Navigate to authenticated page (handles auth if needed)
export async function navigateAuthenticated(
url: string,
options?: {
credentials?: { username: string; password: string };
headless?: boolean;
}
): Promise<{ page: Page; browser: BrowserContext }> {
const { page, browser } = await getPage({ headless: options?.headless ?? true });
await page.goto(url, { timeout: 60000, waitUntil: 'domcontentloaded' });
// Check if we need to authenticate
if (!(await isAuthenticated(page, url))) {
console.log('Session expired or not authenticated. Attempting login...');
// Get credentials
const credentials = options?.credentials ?? getCredentials();
if (!credentials) {
throw new Error(
'Authentication required but no credentials provided. ' +
'Set CAMOUFOX_USERNAME and CAMOUFOX_PASSWORD environment variables.'
);
}
// Detect and handle auth
const authType = await detectAuthType(page);
let success = false;
if (authType === 'msal') {
success = await handleMsalLogin(page, credentials, 30000);
} else {
success = await handleFormLogin(page, credentials, 30000);
}
if (!success) {
await browser.close();
throw new Error('Authentication failed');
}
// Navigate back to original URL if we were redirected
if (!page.url().startsWith(url)) {
await page.goto(url, { timeout: 60000, waitUntil: 'domcontentloaded' });
}
}
return { page, browser };
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'type', 'username', 'password'],
boolean: ['headless', 'help'],
default: {
type: 'auto',
headless: false, // Default to headed for auth so user can see/interact
},
alias: {
u: 'url',
t: 'type',
h: 'help',
},
});
if (args.help || !args.url) {
console.log(`
Web Authentication Handler
Usage:
npx tsx auth.ts --url <url> [options]
Options:
-u, --url <url> URL to authenticate (required)
-t, --type <type> Auth type: auto, form, or msal (default: auto)
--username <user> Username/email (or set CAMOUFOX_USERNAME env var)
--password <pass> Password (or set CAMOUFOX_PASSWORD env var)
--headless <bool> Run in headless mode (default: false for auth)
-h, --help Show this help message
Auth Types:
auto Auto-detect authentication type
form Generic username/password form
msal Microsoft SSO (login.microsoftonline.com)
Environment Variables:
CAMOUFOX_USERNAME Default username/email for authentication
CAMOUFOX_PASSWORD Default password for authentication
Examples:
# Interactive login (no credentials, opens browser)
npx tsx auth.ts --url "https://example.com/login"
# Form login with credentials
npx tsx auth.ts --url "https://example.com/login" --type form \\
--username "user@example.com" --password "secret"
# Microsoft SSO login
CAMOUFOX_USERNAME=user@company.com CAMOUFOX_PASSWORD=secret \\
npx tsx auth.ts --url "https://internal.company.com" --type msal
Notes:
- Session is saved to ~/.camoufox-profile/ for persistence
- After successful auth, subsequent browses will be authenticated
- Use --headless false if you need to handle MFA manually
`);
process.exit(args.help ? 0 : 1);
}
const authType = args.type as AuthType;
if (!['auto', 'form', 'msal'].includes(authType)) {
console.error(`Invalid auth type: ${authType}. Must be auto, form, or msal.`);
process.exit(1);
}
try {
const result = await authenticate({
url: args.url,
authType,
credentials:
args.username && args.password
? { username: args.username, password: args.password }
: undefined,
headless: args.headless,
});
console.log(`\nAuthentication result:`);
console.log(` Success: ${result.success}`);
console.log(` Auth type: ${result.authType}`);
console.log(` Final URL: ${result.finalUrl}`);
console.log(` Message: ${result.message}`);
process.exit(result.success ? 0 : 1);
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('auth.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,195 @@
#!/usr/bin/env npx tsx
/**
* Browser launcher using Camoufox with persistent profile
*
* Usage:
* npx tsx browse.ts --url "https://example.com"
* npx tsx browse.ts --url "https://example.com" --screenshot --output page.png
* npx tsx browse.ts --url "https://example.com" --headless false --wait 5000
*/
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { existsSync, mkdirSync } from 'fs';
import parseArgs from 'minimist';
import type { Page, BrowserContext } from 'playwright-core';
// Types
interface BrowseOptions {
url: string;
headless?: boolean;
screenshot?: boolean;
output?: string;
wait?: number;
timeout?: number;
interactive?: boolean;
}
interface BrowseResult {
title: string;
url: string;
screenshotPath?: string;
}
// Get profile directory
const getProfilePath = (): string => {
const customPath = process.env.CAMOUFOX_PROFILE_PATH;
if (customPath) return customPath;
const profileDir = join(homedir(), '.camoufox-profile');
if (!existsSync(profileDir)) {
mkdirSync(profileDir, { recursive: true });
}
return profileDir;
};
// Launch browser with persistent profile
export async function launchBrowser(options: {
headless?: boolean;
}): Promise<BrowserContext> {
const profilePath = getProfilePath();
const headless =
options.headless ??
(process.env.CAMOUFOX_HEADLESS ? process.env.CAMOUFOX_HEADLESS === 'true' : true);
console.log(`Using profile: ${profilePath}`);
console.log(`Headless mode: ${headless}`);
const browser = await Camoufox({
user_data_dir: profilePath,
headless,
});
return browser;
}
// Browse to URL and optionally take screenshot
export async function browse(options: BrowseOptions): Promise<BrowseResult> {
const browser = await launchBrowser({ headless: options.headless });
const page = await browser.newPage();
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, {
timeout: options.timeout ?? 60000,
waitUntil: 'domcontentloaded',
});
// Wait if specified
if (options.wait) {
console.log(`Waiting ${options.wait}ms...`);
await page.waitForTimeout(options.wait);
}
const result: BrowseResult = {
title: await page.title(),
url: page.url(),
};
console.log(`Page title: ${result.title}`);
console.log(`Final URL: ${result.url}`);
// Take screenshot if requested
if (options.screenshot) {
const outputPath = options.output ?? 'screenshot.png';
await page.screenshot({ path: outputPath, fullPage: true });
result.screenshotPath = outputPath;
console.log(`Screenshot saved: ${outputPath}`);
}
// If interactive mode, keep browser open
if (options.interactive) {
console.log('\nInteractive mode - browser will stay open.');
console.log('Press Ctrl+C to close.');
await new Promise(() => {}); // Keep running
}
return result;
} finally {
if (!options.interactive) {
await browser.close();
}
}
}
// Export page for use in other scripts
export async function getPage(options?: {
headless?: boolean;
}): Promise<{ page: Page; browser: BrowserContext }> {
const browser = await launchBrowser({ headless: options?.headless });
const page = await browser.newPage();
return { page, browser };
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'output'],
boolean: ['screenshot', 'headless', 'interactive', 'help'],
default: {
headless: true,
screenshot: false,
interactive: false,
},
alias: {
u: 'url',
o: 'output',
s: 'screenshot',
h: 'help',
i: 'interactive',
},
});
if (args.help || !args.url) {
console.log(`
Web Browser with Camoufox
Usage:
npx tsx browse.ts --url <url> [options]
Options:
-u, --url <url> URL to navigate to (required)
-s, --screenshot Take a screenshot of the page
-o, --output <path> Output path for screenshot (default: screenshot.png)
--headless <bool> Run in headless mode (default: true)
--wait <ms> Wait time after page load in milliseconds
--timeout <ms> Navigation timeout (default: 60000)
-i, --interactive Keep browser open for manual interaction
-h, --help Show this help message
Examples:
npx tsx browse.ts --url "https://example.com"
npx tsx browse.ts --url "https://example.com" --screenshot --output page.png
npx tsx browse.ts --url "https://example.com" --headless false --interactive
Environment Variables:
CAMOUFOX_PROFILE_PATH Custom profile directory (default: ~/.camoufox-profile/)
CAMOUFOX_HEADLESS Default headless mode (true/false)
`);
process.exit(args.help ? 0 : 1);
}
try {
await browse({
url: args.url,
headless: args.headless,
screenshot: args.screenshot,
output: args.output,
wait: args.wait ? parseInt(args.wait, 10) : undefined,
timeout: args.timeout ? parseInt(args.timeout, 10) : undefined,
interactive: args.interactive,
});
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('browse.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,29 @@
{
"name": "web-automation-scripts",
"version": "1.0.0",
"description": "Web browsing and scraping scripts using Camoufox",
"type": "module",
"scripts": {
"browse": "tsx browse.ts",
"scrape": "tsx scrape.ts",
"fetch-browser": "npx camoufox-js fetch"
},
"dependencies": {
"@mozilla/readability": "^0.5.0",
"camoufox-js": "^0.8.5",
"jsdom": "^24.0.0",
"minimist": "^1.2.8",
"playwright-core": "^1.40.0",
"turndown": "^7.1.2",
"turndown-plugin-gfm": "^1.0.2"
},
"devDependencies": {
"@types/jsdom": "^21.1.6",
"@types/minimist": "^1.2.5",
"@types/turndown": "^5.0.4",
"esbuild": "0.27.0",
"tsx": "^4.7.0",
"typescript": "^5.3.0"
},
"packageManager": "pnpm@10.18.1+sha512.77a884a165cbba2d8d1c19e3b4880eee6d2fcabd0d879121e282196b80042351d5eb3ca0935fa599da1dc51265cc68816ad2bddd2a2de5ea9fdf92adbec7cd34"
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,212 @@
import { writeFileSync } from 'fs';
import { getPage } from './browse.js';
const baseUrl = 'http://localhost:3000';
const username = 'analyst@fhb.local';
const password = process.env.CAMOUFOX_PASSWORD ?? '';
const reportPath = '/Users/stefano.fiorini/Documents/projects/fhb-loan-spreading-pilot-a/docs/plans/2026-01-24-financials-analysis-redesign/web-automation-scan.md';
type NavResult = {
requestedUrl: string;
url: string;
status: number | null;
title: string;
error?: string;
};
async function gotoWithStatus(page: any, url: string): Promise<NavResult> {
const resp = await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 60000 }).catch((e: unknown) => ({ error: e }));
if (resp?.error) {
return {
requestedUrl: url,
url: page.url(),
status: null,
title: await page.title().catch(() => ''),
error: String(resp.error),
};
}
return {
requestedUrl: url,
url: page.url(),
status: resp ? resp.status() : null,
title: await page.title().catch(() => ''),
};
}
async function textOrNull(page: any, selector: string): Promise<string | null> {
const loc = page.locator(selector).first();
try {
if ((await loc.count()) === 0) return null;
const txt = await loc.textContent();
return txt ? txt.trim().replace(/\s+/g, ' ') : null;
} catch {
return null;
}
}
async function main() {
const { page, browser } = await getPage({ headless: true });
const lines: string[] = [];
lines.push('# Web Automation Scan (local)');
lines.push('');
lines.push(`- Base URL: ${baseUrl}`);
lines.push(`- Timestamp: ${new Date().toISOString()}`);
lines.push('');
try {
lines.push('## Login');
await gotoWithStatus(page, `${baseUrl}/login`);
await page.locator('input[name="email"]').fill(username);
await page.locator('input[name="password"]').fill(password);
await page.locator('button[type="submit"]').click();
await page.waitForTimeout(2500);
const cookies = await page.context().cookies();
const sessionCookie = cookies.find((c: any) => c.name === 'fhb_session');
lines.push(`- After submit URL: ${page.url()}`);
lines.push(`- Has session cookie (fhb_session): ${Boolean(sessionCookie)}`);
lines.push('');
lines.push('## Demo Case');
const casesNav = await gotoWithStatus(page, `${baseUrl}/cases`);
lines.push(`- GET /cases → status ${casesNav.status ?? 'ERR'}, final ${casesNav.url}`);
const envCaseId = process.env.SCAN_CASE_ID?.trim() || null;
let selectedCaseId: string | null = envCaseId;
if (!selectedCaseId) {
const caseLinks = await page.$$eval('a[href^="/cases/"]', (as) =>
as
.map((a) => ({
href: (a as HTMLAnchorElement).getAttribute('href') || '',
text: (a.textContent || '').trim(),
}))
.filter((x) => x.href.includes('/cases/'))
);
const preferredTitles = ['Demo - Strong Borrower', 'Demo - Weak Borrower', 'Demo - Incomplete'];
for (const title of preferredTitles) {
const match = caseLinks.find((l) => l.text.includes(title) && l.href.includes('/cases/'));
const href = match?.href ?? '';
const m = href.match(/\/cases\/([0-9a-f-]{36})/i);
if (m) {
selectedCaseId = m[1];
break;
}
}
if (!selectedCaseId) {
const firstHref =
caseLinks.map((l) => l.href).find((h) => /\/cases\/[0-9a-f-]{36}/i.test(h)) ?? null;
const m = firstHref?.match(/\/cases\/([0-9a-f-]{36})/i) ?? null;
selectedCaseId = m?.[1] ?? null;
}
}
lines.push(`- Selected caseId: ${selectedCaseId ?? '(none found)'}`);
if (!selectedCaseId) {
lines.push('');
lines.push('⚠️ Could not find a demo case link on /cases.');
writeFileSync(reportPath, lines.join('\n') + '\n', 'utf-8');
return;
}
const caseBase = `${baseUrl}/cases/${selectedCaseId}/journey`;
lines.push('');
lines.push('## Route Checks');
const routesToCheck = [
`${caseBase}`,
`${caseBase}/financials`,
`${caseBase}/financials/income`,
`${caseBase}/analysis`,
`${caseBase}/analysis/configure`,
`${caseBase}/analysis/ai`,
`${caseBase}/analysis/ai/detail`,
`${caseBase}/spreads`,
];
for (const url of routesToCheck) {
const r = await gotoWithStatus(page, url);
const h1 = await textOrNull(page, 'h1');
const finalPath = r.url.startsWith(baseUrl) ? r.url.slice(baseUrl.length) : r.url;
lines.push(`- ${url.slice(baseUrl.length)} → status ${r.status ?? 'ERR'} (final ${finalPath})${h1 ? `, h1="${h1}"` : ''}`);
}
lines.push('');
lines.push('## Spreadsheet Analysis (UI)');
await gotoWithStatus(page, `${caseBase}/analysis/configure`);
const runButton = page.locator('button:has-text("Run Analysis")').first();
const disabled = await runButton.isDisabled().catch(() => true);
lines.push(`- Run button disabled: ${disabled}`);
if (!disabled) {
await runButton.click();
const resultsWait = page
.waitForURL('**/journey/analysis/results**', { timeout: 180000 })
.then(() => 'results' as const);
const errorWait = page
.locator('[role="alert"]')
.filter({ hasText: 'Error' })
.first()
.waitFor({ timeout: 180000 })
.then(() => 'error' as const);
const outcome = await Promise.race([resultsWait, errorWait]).catch(() => 'timeout' as const);
if (outcome === 'results') {
await page.waitForTimeout(1500);
lines.push(`- Results URL: ${page.url().replace(baseUrl, '')}`);
const downloadHref = await page
.locator('a[href*="/journey/analysis/download"]')
.first()
.getAttribute('href')
.catch(() => null);
if (downloadHref) {
const dlUrl = downloadHref.startsWith('http') ? downloadHref : `${baseUrl}${downloadHref}`;
const dlResp = await page.goto(dlUrl, { waitUntil: 'commit', timeout: 60000 }).catch(() => null);
lines.push(
`- Download route status: ${dlResp?.status() ?? 'ERR'} (Content-Type: ${dlResp?.headers()?.['content-type'] ?? 'n/a'})`
);
} else {
lines.push('- Download link not found on results page');
}
} else if (outcome === 'error') {
const errorText = await page
.locator('[role="alert"]')
.first()
.textContent()
.then((t: string | null) => (t ? t.trim().replace(/\\s+/g, ' ') : null))
.catch(() => null);
lines.push(`- Stayed on configure page; saw error callout: ${errorText ?? '(unable to read)'}`);
lines.push('- Skipping download check because analysis did not complete.');
} else {
lines.push('- Timed out waiting for results or error after clicking Run Analysis.');
}
} else {
lines.push('- Skipped running analysis because Run button was disabled.');
}
lines.push('');
lines.push('## Notes');
lines.push('- This scan avoids scraping financial values; it records route availability and basic headings.');
writeFileSync(reportPath, lines.join('\n') + '\n', 'utf-8');
} finally {
await browser.close();
}
}
main().catch((err) => {
console.error(err);
process.exitCode = 1;
});

View File

@@ -0,0 +1,351 @@
#!/usr/bin/env npx tsx
/**
* Web scraper that extracts content to markdown
*
* Usage:
* npx tsx scrape.ts --url "https://example.com" --mode main
* npx tsx scrape.ts --url "https://example.com" --mode full --output page.md
* npx tsx scrape.ts --url "https://example.com" --mode selector --selector ".content"
*/
import TurndownService from 'turndown';
import * as turndownPluginGfm from 'turndown-plugin-gfm';
import { Readability } from '@mozilla/readability';
import { JSDOM } from 'jsdom';
import { writeFileSync } from 'fs';
import parseArgs from 'minimist';
import { getPage } from './browse.js';
// Types
type ScrapeMode = 'main' | 'full' | 'selector';
interface ScrapeOptions {
url: string;
mode: ScrapeMode;
selector?: string;
output?: string;
includeLinks?: boolean;
includeTables?: boolean;
includeImages?: boolean;
headless?: boolean;
wait?: number;
}
interface ScrapeResult {
title: string;
url: string;
markdown: string;
byline?: string;
excerpt?: string;
}
// Configure Turndown for markdown conversion
function createTurndownService(options: {
includeLinks?: boolean;
includeTables?: boolean;
includeImages?: boolean;
}): TurndownService {
const turndown = new TurndownService({
headingStyle: 'atx',
hr: '---',
bulletListMarker: '-',
codeBlockStyle: 'fenced',
fence: '```',
emDelimiter: '*',
strongDelimiter: '**',
linkStyle: 'inlined',
});
// Add GFM support (tables, strikethrough, task lists)
turndown.use(turndownPluginGfm.gfm);
// Custom rule for code blocks with language detection
turndown.addRule('codeBlockWithLanguage', {
filter: (node) => {
return (
node.nodeName === 'PRE' &&
node.firstChild?.nodeName === 'CODE'
);
},
replacement: (_content, node) => {
const codeNode = node.firstChild as HTMLElement;
const className = codeNode.getAttribute('class') || '';
const langMatch = className.match(/language-(\w+)/);
const lang = langMatch ? langMatch[1] : '';
const code = codeNode.textContent || '';
return `\n\n\`\`\`${lang}\n${code}\n\`\`\`\n\n`;
},
});
// Remove images if not included
if (!options.includeImages) {
turndown.addRule('removeImages', {
filter: 'img',
replacement: () => '',
});
}
// Remove links but keep text if not included
if (!options.includeLinks) {
turndown.addRule('removeLinks', {
filter: 'a',
replacement: (content) => content,
});
}
// Remove script, style, nav, footer, aside elements
turndown.remove(['script', 'style', 'nav', 'footer', 'aside', 'noscript']);
return turndown;
}
// Extract main content using Readability
function extractMainContent(html: string, url: string): {
content: string;
title: string;
byline?: string;
excerpt?: string;
} {
const dom = new JSDOM(html, { url });
const reader = new Readability(dom.window.document);
const article = reader.parse();
if (!article) {
throw new Error('Could not extract main content from page');
}
return {
content: article.content,
title: article.title,
byline: article.byline || undefined,
excerpt: article.excerpt || undefined,
};
}
// Scrape a URL and return markdown
export async function scrape(options: ScrapeOptions): Promise<ScrapeResult> {
const { page, browser } = await getPage({ headless: options.headless ?? true });
try {
// Navigate to URL
console.log(`Navigating to: ${options.url}`);
await page.goto(options.url, {
timeout: 60000,
waitUntil: 'domcontentloaded',
});
// Wait if specified
if (options.wait) {
console.log(`Waiting ${options.wait}ms for dynamic content...`);
await page.waitForTimeout(options.wait);
}
const pageTitle = await page.title();
const pageUrl = page.url();
let html: string;
let title = pageTitle;
let byline: string | undefined;
let excerpt: string | undefined;
// Get HTML based on mode
switch (options.mode) {
case 'main': {
// Get full page HTML and extract with Readability
const fullHtml = await page.content();
const extracted = extractMainContent(fullHtml, pageUrl);
html = extracted.content;
title = extracted.title || pageTitle;
byline = extracted.byline;
excerpt = extracted.excerpt;
break;
}
case 'selector': {
if (!options.selector) {
throw new Error('Selector mode requires --selector option');
}
const element = await page.$(options.selector);
if (!element) {
throw new Error(`Selector not found: ${options.selector}`);
}
html = await element.innerHTML();
break;
}
case 'full':
default: {
// Get body content, excluding common non-content elements
html = await page.evaluate(() => {
// Remove common non-content elements
const selectorsToRemove = [
'script', 'style', 'noscript', 'iframe',
'nav', 'header', 'footer', '.cookie-banner',
'.advertisement', '.ads', '#ads', '.social-share',
'.comments', '#comments', '.sidebar'
];
selectorsToRemove.forEach(selector => {
document.querySelectorAll(selector).forEach(el => el.remove());
});
return document.body.innerHTML;
});
break;
}
}
// Convert to markdown
const turndown = createTurndownService({
includeLinks: options.includeLinks ?? true,
includeTables: options.includeTables ?? true,
includeImages: options.includeImages ?? false,
});
let markdown = turndown.turndown(html);
// Add title as H1 if not already present
if (!markdown.startsWith('# ')) {
markdown = `# ${title}\n\n${markdown}`;
}
// Add metadata header
const metadataLines = [
`<!-- Scraped from: ${pageUrl} -->`,
byline ? `<!-- Author: ${byline} -->` : null,
excerpt ? `<!-- Excerpt: ${excerpt} -->` : null,
`<!-- Scraped at: ${new Date().toISOString()} -->`,
'',
].filter(Boolean);
markdown = metadataLines.join('\n') + '\n' + markdown;
// Clean up excessive whitespace
markdown = markdown
.replace(/\n{4,}/g, '\n\n\n')
.replace(/[ \t]+$/gm, '')
.trim();
const result: ScrapeResult = {
title,
url: pageUrl,
markdown,
byline,
excerpt,
};
// Save to file if output specified
if (options.output) {
writeFileSync(options.output, markdown, 'utf-8');
console.log(`Markdown saved to: ${options.output}`);
}
return result;
} finally {
await browser.close();
}
}
// CLI entry point
async function main() {
const args = parseArgs(process.argv.slice(2), {
string: ['url', 'mode', 'selector', 'output'],
boolean: ['headless', 'links', 'tables', 'images', 'help'],
default: {
mode: 'main',
headless: true,
links: true,
tables: true,
images: false,
},
alias: {
u: 'url',
m: 'mode',
s: 'selector',
o: 'output',
h: 'help',
},
});
if (args.help || !args.url) {
console.log(`
Web Scraper - Extract content to Markdown
Usage:
npx tsx scrape.ts --url <url> [options]
Options:
-u, --url <url> URL to scrape (required)
-m, --mode <mode> Scrape mode: main, full, or selector (default: main)
-s, --selector <sel> CSS selector for selector mode
-o, --output <path> Output file path for markdown
--headless <bool> Run in headless mode (default: true)
--wait <ms> Wait time for dynamic content
--links Include links in output (default: true)
--tables Include tables in output (default: true)
--images Include images in output (default: false)
-h, --help Show this help message
Scrape Modes:
main Extract main article content using Readability (best for articles)
full Full page content with common elements removed
selector Extract specific element by CSS selector
Examples:
npx tsx scrape.ts --url "https://docs.example.com/guide" --mode main
npx tsx scrape.ts --url "https://example.com" --mode full --output page.md
npx tsx scrape.ts --url "https://example.com" --mode selector --selector ".api-docs"
npx tsx scrape.ts --url "https://example.com" --mode main --no-links --output clean.md
Output Format:
- GitHub Flavored Markdown (tables, strikethrough, task lists)
- Proper heading hierarchy
- Code blocks with language detection
- Metadata comments at top (source URL, date)
`);
process.exit(args.help ? 0 : 1);
}
const mode = args.mode as ScrapeMode;
if (!['main', 'full', 'selector'].includes(mode)) {
console.error(`Invalid mode: ${mode}. Must be main, full, or selector.`);
process.exit(1);
}
try {
const result = await scrape({
url: args.url,
mode,
selector: args.selector,
output: args.output,
includeLinks: args.links,
includeTables: args.tables,
includeImages: args.images,
headless: args.headless,
wait: args.wait ? parseInt(args.wait, 10) : undefined,
});
// Print result summary
console.log(`\nScrape complete:`);
console.log(` Title: ${result.title}`);
console.log(` URL: ${result.url}`);
if (result.byline) console.log(` Author: ${result.byline}`);
console.log(` Markdown length: ${result.markdown.length} chars`);
// Print markdown if not saved to file
if (!args.output) {
console.log('\n--- Markdown Output ---\n');
console.log(result.markdown);
}
} catch (error) {
console.error('Error:', error instanceof Error ? error.message : error);
process.exit(1);
}
}
// Run if executed directly
const isMainModule = process.argv[1]?.includes('scrape.ts');
if (isMainModule) {
main();
}

View File

@@ -0,0 +1,39 @@
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { mkdirSync, existsSync } from 'fs';
async function test() {
const profilePath = join(homedir(), '.camoufox-profile');
if (!existsSync(profilePath)) {
mkdirSync(profilePath, { recursive: true });
}
console.log('Profile path:', profilePath);
console.log('Launching with full options...');
const browser = await Camoufox({
headless: true,
user_data_dir: profilePath,
// humanize: 1.5, // Test without this first
// geoip: true, // Test without this first
// enable_cache: true,
// block_webrtc: false,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://github.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await page.screenshot({ path: '/tmp/github-test.png' });
console.log('Screenshot saved');
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,22 @@
import { Camoufox } from 'camoufox-js';
async function test() {
console.log('Launching Camoufox with minimal config...');
const browser = await Camoufox({
headless: true,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://example.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,32 @@
import { Camoufox } from 'camoufox-js';
import { homedir } from 'os';
import { join } from 'path';
import { mkdirSync, existsSync } from 'fs';
async function test() {
const profilePath = join(homedir(), '.camoufox-profile');
if (!existsSync(profilePath)) {
mkdirSync(profilePath, { recursive: true });
}
console.log('Profile path:', profilePath);
console.log('Launching with user_data_dir...');
const browser = await Camoufox({
headless: true,
user_data_dir: profilePath,
});
console.log('Browser launched');
const page = await browser.newPage();
console.log('Page created');
await page.goto('https://example.com', { timeout: 30000 });
console.log('Navigated to:', page.url());
console.log('Title:', await page.title());
await browser.close();
console.log('Done');
}
test().catch(console.error);

View File

@@ -0,0 +1,78 @@
import { getPage } from './browse.js';
type Extracted = {
title: string;
url: string;
colorVars: Array<[string, string]>;
samples: Record<string, null | { background: string; color: string; border: string }>;
};
function isColorValue(value: string) {
return /#([0-9a-f]{3,4}|[0-9a-f]{6}|[0-9a-f]{8})\b/i.test(value) || /\brgb\(|\bhsl\(/i.test(value);
}
async function main() {
const url = process.argv[2] ?? 'https://www.firsthorizon.com';
const { page, browser } = await getPage({ headless: true });
try {
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 60000 });
await page.waitForTimeout(5000);
const data = await page.evaluate(`(() => {
const rootStyles = getComputedStyle(document.documentElement);
const vars = {};
for (let i = 0; i < rootStyles.length; i++) {
const prop = rootStyles[i];
if (prop && prop.startsWith('--')) {
vars[prop] = rootStyles.getPropertyValue(prop).trim();
}
}
const pick = (selector) => {
const el = document.querySelector(selector);
if (!el) return null;
const cs = getComputedStyle(el);
return {
background: cs.backgroundColor,
color: cs.color,
border: cs.borderColor,
};
};
return {
title: document.title,
url: location.href,
vars,
samples: {
body: pick('body'),
header: pick('header'),
nav: pick('nav'),
primaryButton: pick('button, [role="button"], a[role="button"], a.button, .button'),
link: pick('a'),
},
};
})()`);
const entries = Object.entries(data.vars) as Array<[string, string]>;
const colorVars = entries
.filter(([, v]) => v && isColorValue(v))
.sort((a, b) => a[0].localeCompare(b[0]));
const out: Extracted = {
title: data.title,
url: data.url,
colorVars,
samples: data.samples,
};
process.stdout.write(JSON.stringify(out, null, 2));
} finally {
await browser.close();
}
}
main().catch((error) => {
console.error(error);
process.exit(1);
});

View File

@@ -0,0 +1,16 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"strict": true,
"skipLibCheck": true,
"resolveJsonModule": true,
"outDir": "./dist",
"rootDir": "."
},
"include": ["*.ts"],
"exclude": ["node_modules", "dist"]
}

View File

@@ -0,0 +1,8 @@
declare module 'turndown-plugin-gfm' {
import TurndownService from 'turndown';
export function gfm(turndownService: TurndownService): void;
export function strikethrough(turndownService: TurndownService): void;
export function tables(turndownService: TurndownService): void;
export function taskListItems(turndownService: TurndownService): void;
}