Compare commits
5 Commits
26a968797c
...
4204d28077
| Author | SHA1 | Date | |
|---|---|---|---|
 stefano
|
4204d28077 | ||
|
stefano
|
c1286e9c42 | ||
|
stefano
|
1e0e265f1e | ||
|
stefano
|
ef326896f4 | ||
|
stefano
|
8ad532545d |
@@ -16,6 +16,7 @@ This repository contains practical OpenClaw skills and companion integrations. I
|
|||||||
|
|
||||||
| Skill | What it does | Path |
|
| Skill | What it does | Path |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
|
| `amazon-shopping` | Search Amazon.com product results with bounded web-automation, price/unit-price filters, delivery summaries, specs, ratings, and review metadata. | `skills/amazon-shopping` |
|
||||||
| `flight-finder` | Collect structured flight-search inputs, run bounded multi-source travel searches, rank results in USD, and gate fixed-template PDF/email delivery through Luke’s sender path. | `skills/flight-finder` |
|
| `flight-finder` | Collect structured flight-search inputs, run bounded multi-source travel searches, rank results in USD, and gate fixed-template PDF/email delivery through Luke’s sender path. | `skills/flight-finder` |
|
||||||
| `gitea-api` | Interact with Gitea via REST API (repos, issues, PRs, releases, branches, user info). | `skills/gitea-api` |
|
| `gitea-api` | Interact with Gitea via REST API (repos, issues, PRs, releases, branches, user info). | `skills/gitea-api` |
|
||||||
| `nordvpn-client` | Install, log in to, connect, disconnect, and verify NordVPN sessions across Linux CLI and macOS NordLynx/WireGuard backends. | `skills/nordvpn-client` |
|
| `nordvpn-client` | Install, log in to, connect, disconnect, and verify NordVPN sessions across Linux CLI and macOS NordLynx/WireGuard backends. | `skills/nordvpn-client` |
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ This folder contains detailed docs for each skill in this repository.
|
|||||||
|
|
||||||
## Skills
|
## Skills
|
||||||
|
|
||||||
|
- [`amazon-shopping`](amazon-shopping.md) — Amazon.com product search with local web-automation, product filters, pricing, delivery, specs, and review metadata
|
||||||
- [`flight-finder`](flight-finder.md) — Typed flight-search intake, bounded source orchestration, PDF report rendering, and Luke-sender email delivery
|
- [`flight-finder`](flight-finder.md) — Typed flight-search intake, bounded source orchestration, PDF report rendering, and Luke-sender email delivery
|
||||||
- [`gitea-api`](gitea-api.md) — REST-based Gitea automation (no `tea` CLI required)
|
- [`gitea-api`](gitea-api.md) — REST-based Gitea automation (no `tea` CLI required)
|
||||||
- [`nordvpn-client`](nordvpn-client.md) — Cross-platform NordVPN install, login, connect, disconnect, and verification with Linux CLI and macOS NordLynx/WireGuard support
|
- [`nordvpn-client`](nordvpn-client.md) — Cross-platform NordVPN install, login, connect, disconnect, and verification with Linux CLI and macOS NordLynx/WireGuard support
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
# Amazon Shopping Skill
|
||||||
|
|
||||||
|
`amazon-shopping` searches Amazon.com product results with bounded, read-only web automation and deterministic local filtering.
|
||||||
|
|
||||||
|
## Example Invocation
|
||||||
|
|
||||||
|
```text
|
||||||
|
use amazon-shopping to search for 100w led bulbs that cost less than $4 each and have over 200 reviews with a review score of more than 4.5 stars
|
||||||
|
```
|
||||||
|
|
||||||
|
## Helper Commands
|
||||||
|
|
||||||
|
Run from the installed skill:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.openclaw/workspace/skills/amazon-shopping
|
||||||
|
scripts/search-products 'USB-C charger under $30' --limit 10 --json
|
||||||
|
scripts/search-products '100w led bulbs that cost less than $4 each and have over 200 reviews with a review score of more than 4.5 stars' --limit 5 --markdown
|
||||||
|
scripts/search-products 'USB-C cable with over 1000 reviews and rating over 4 stars' --limit 3 --json --skip-details
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `--dry-run` to parse a request and show planned filters without navigating to Amazon:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/search-products 'USB-C charger under $30' --dry-run --json
|
||||||
|
```
|
||||||
|
|
||||||
|
Use single quotes when a request contains dollar amounts so the shell does not expand `$4`. `--max N` is accepted as a compatibility alias for `--limit N`.
|
||||||
|
|
||||||
|
## Dependency
|
||||||
|
|
||||||
|
This skill depends on the workspace `web-automation` skill and its CloakBrowser runtime.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
openclaw skills info web-automation
|
||||||
|
node "$HOME/.openclaw/workspace/skills/web-automation/scripts/check-install.js"
|
||||||
|
```
|
||||||
|
|
||||||
|
Setup or update the active `amazon-shopping` helper:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.openclaw/workspace/skills/amazon-shopping
|
||||||
|
npm ci
|
||||||
|
npm run lint
|
||||||
|
npm run typecheck
|
||||||
|
npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fields
|
||||||
|
|
||||||
|
Each result includes the product ASIN, title, source URL, price, unit price when visible, rating, review count, delivery summary, specs, feature bullets, seller, availability, sponsored marker, matched filters, missing fields, and extraction notes.
|
||||||
|
|
||||||
|
Unknown or hidden fields stay unknown. The skill does not invent delivery dates, star histograms, prices, or review counts.
|
||||||
|
|
||||||
|
## Filters
|
||||||
|
|
||||||
|
Supported request filters include:
|
||||||
|
|
||||||
|
- minimum rating
|
||||||
|
- minimum review count
|
||||||
|
- maximum product price
|
||||||
|
- maximum unit price
|
||||||
|
- result limit
|
||||||
|
- maximum search pages
|
||||||
|
|
||||||
|
`over 200 reviews` and `more than 4.5 stars` are strict comparisons. `at least 200 reviews` and `4.5 stars or better` are inclusive comparisons.
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
|
||||||
|
This skill is for operator-directed product research, not purchasing automation.
|
||||||
|
|
||||||
|
- It checks Amazon robots directives before live navigation.
|
||||||
|
- It does not sign in, add to cart, purchase, access wishlists, submit reviews, crawl review pages, or bypass CAPTCHA/block pages.
|
||||||
|
- It stops and reports a warning when Amazon returns a challenge, block, or disallowed robots path.
|
||||||
|
- It uses default bounded operation: 15 results, 2 search pages, detail pages one at a time.
|
||||||
|
|
||||||
|
## Live Smoke
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.openclaw/workspace/skills/amazon-shopping
|
||||||
|
scripts/search-products '100w led bulbs that cost less than $4 each and have over 200 reviews with a review score of more than 4.5 stars' --limit 5 --json
|
||||||
|
```
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
coverage/
|
||||||
|
*.log
|
||||||
|
tmp/
|
||||||
|
out/
|
||||||
|
*.real.html
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
---
|
||||||
|
name: amazon-shopping
|
||||||
|
description: Search amazon.com shopping results with product filters using the local web-automation skill. Use when the user asks to find, compare, filter, or summarize Amazon products by description, price, delivery, specs, review count, star rating, or star distribution.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Amazon Shopping
|
||||||
|
|
||||||
|
Use this skill for read-only Amazon product discovery and comparison.
|
||||||
|
|
||||||
|
## First Checks
|
||||||
|
|
||||||
|
Verify the browser dependency before live use:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
openclaw skills info web-automation
|
||||||
|
node "$HOME/.openclaw/workspace/skills/web-automation/scripts/check-install.js"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Search Products
|
||||||
|
|
||||||
|
Run the helper from the installed skill directory:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$HOME/.openclaw/workspace/skills/amazon-shopping/scripts/search-products" '<user request>' --json
|
||||||
|
```
|
||||||
|
|
||||||
|
Use single quotes around product requests that contain dollar amounts so the shell does not expand `$4` or similar text. Use `--limit N`; `--max N` is accepted as a compatibility alias. If your execution tool supports a timeout option, set it to at least 600 seconds for live runs with detail enrichment. Use `--skip-details` only for a quick preview or when the user does not need specifications and delivery details.
|
||||||
|
|
||||||
|
Default to at most 15 products unless the user asks for a different count. For requested counts above 30, ask before continuing or split the request into batches. Always include source URLs, report missing fields explicitly, and do not claim review histogram data unless it was visible and extracted.
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
|
||||||
|
This skill is for operator-directed, read-only product research. Before live scraping, the helper checks Amazon robots directives for planned paths. Do not automate sign-in, cart, purchase, wishlist, review submission, review-page crawling, CAPTCHA bypass, or blocked-page bypass. If Amazon returns a challenge or block page, stop and report that status.
|
||||||
|
|
||||||
|
Read references when needed:
|
||||||
|
- `references/amazon-data-map.md` for fields and selectors.
|
||||||
|
- `references/web-automation-prompts.md` for browser extraction prompts.
|
||||||
|
- `references/compliance-and-failure-modes.md` for blocked-page and unknown-field behavior.
|
||||||
Generated
+743
@@ -0,0 +1,743 @@
|
|||||||
|
{
|
||||||
|
"name": "amazon-shopping-scripts",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"lockfileVersion": 3,
|
||||||
|
"requires": true,
|
||||||
|
"packages": {
|
||||||
|
"": {
|
||||||
|
"name": "amazon-shopping-scripts",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"dependencies": {
|
||||||
|
"minimist": "^1.2.8",
|
||||||
|
"node-html-parser": "^7.1.0"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/minimist": "^1.2.5",
|
||||||
|
"@types/node": "^24.8.1",
|
||||||
|
"tsx": "^4.20.6",
|
||||||
|
"typescript": "^5.9.3"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/aix-ppc64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg==",
|
||||||
|
"cpu": [
|
||||||
|
"ppc64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"aix"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/android-arm": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-jbPXvB4Yj2yBV7HUfE2KHe4GJX51QplCN1pGbYjvsyCZbQmies29EoJbkEc+vYuU5o45AfQn37vZlyXy4YJ8RQ==",
|
||||||
|
"cpu": [
|
||||||
|
"arm"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"android"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/android-arm64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-62dPZHpIXzvChfvfLJow3q5dDtiNMkwiRzPylSCfriLvZeq0a1bWChrGx/BbUbPwOrsWKMn8idSllklzBy+dgQ==",
|
||||||
|
"cpu": [
|
||||||
|
"arm64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"android"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/android-x64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-x5VpMODneVDb70PYV2VQOmIUUiBtY3D3mPBG8NxVk5CogneYhkR7MmM3yR/uMdITLrC1ml/NV1rj4bMJuy9MCg==",
|
||||||
|
"cpu": [
|
||||||
|
"x64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"android"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/darwin-arm64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-5lckdqeuBPlKUwvoCXIgI2D9/ABmPq3Rdp7IfL70393YgaASt7tbju3Ac+ePVi3KDH6N2RqePfHnXkaDtY9fkw==",
|
||||||
|
"cpu": [
|
||||||
|
"arm64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"darwin"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/darwin-x64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-rYnXrKcXuT7Z+WL5K980jVFdvVKhCHhUwid+dDYQpH+qu+TefcomiMAJpIiC2EM3Rjtq0sO3StMV/+3w3MyyqQ==",
|
||||||
|
"cpu": [
|
||||||
|
"x64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"darwin"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/freebsd-arm64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-B48PqeCsEgOtzME2GbNM2roU29AMTuOIN91dsMO30t+Ydis3z/3Ngoj5hhnsOSSwNzS+6JppqWsuhTp6E82l2w==",
|
||||||
|
"cpu": [
|
||||||
|
"arm64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"freebsd"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/freebsd-x64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-jOBDK5XEjA4m5IJK3bpAQF9/Lelu/Z9ZcdhTRLf4cajlB+8VEhFFRjWgfy3M1O4rO2GQ/b2dLwCUGpiF/eATNQ==",
|
||||||
|
"cpu": [
|
||||||
|
"x64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"freebsd"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-arm": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-RkT/YXYBTSULo3+af8Ib0ykH8u2MBh57o7q/DAs3lTJlyVQkgQvlrPTnjIzzRPQyavxtPtfg0EopvDyIt0j1rA==",
|
||||||
|
"cpu": [
|
||||||
|
"arm"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-arm64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-RZPHBoxXuNnPQO9rvjh5jdkRmVizktkT7TCDkDmQ0W2SwHInKCAV95GRuvdSvA7w4VMwfCjUiPwDi0ZO6Nfe9A==",
|
||||||
|
"cpu": [
|
||||||
|
"arm64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-ia32": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-GA48aKNkyQDbd3KtkplYWT102C5sn/EZTY4XROkxONgruHPU72l+gW+FfF8tf2cFjeHaRbWpOYa/uRBz/Xq1Pg==",
|
||||||
|
"cpu": [
|
||||||
|
"ia32"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-loong64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-a4POruNM2oWsD4WKvBSEKGIiWQF8fZOAsycHOt6JBpZ+JN2n2JH9WAv56SOyu9X5IqAjqSIPTaJkqN8F7XOQ5Q==",
|
||||||
|
"cpu": [
|
||||||
|
"loong64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-mips64el": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-KabT5I6StirGfIz0FMgl1I+R1H73Gp0ofL9A3nG3i/cYFJzKHhouBV5VWK1CSgKvVaG4q1RNpCTR2LuTVB3fIw==",
|
||||||
|
"cpu": [
|
||||||
|
"mips64el"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-ppc64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-gRsL4x6wsGHGRqhtI+ifpN/vpOFTQtnbsupUF5R5YTAg+y/lKelYR1hXbnBdzDjGbMYjVJLJTd2OFmMewAgwlQ==",
|
||||||
|
"cpu": [
|
||||||
|
"ppc64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-riscv64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-hL25LbxO1QOngGzu2U5xeXtxXcW+/GvMN3ejANqXkxZ/opySAZMrc+9LY/WyjAan41unrR3YrmtTsUpwT66InQ==",
|
||||||
|
"cpu": [
|
||||||
|
"riscv64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-s390x": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-2k8go8Ycu1Kb46vEelhu1vqEP+UeRVj2zY1pSuPdgvbd5ykAw82Lrro28vXUrRmzEsUV0NzCf54yARIK8r0fdw==",
|
||||||
|
"cpu": [
|
||||||
|
"s390x"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/linux-x64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-hzznmADPt+OmsYzw1EE33ccA+HPdIqiCRq7cQeL1Jlq2gb1+OyWBkMCrYGBJ+sxVzve2ZJEVeePbLM2iEIZSxA==",
|
||||||
|
"cpu": [
|
||||||
|
"x64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"linux"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/netbsd-arm64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-b6pqtrQdigZBwZxAn1UpazEisvwaIDvdbMbmrly7cDTMFnw/+3lVxxCTGOrkPVnsYIosJJXAsILG9XcQS+Yu6w==",
|
||||||
|
"cpu": [
|
||||||
|
"arm64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"netbsd"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/netbsd-x64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-OfatkLojr6U+WN5EDYuoQhtM+1xco+/6FSzJJnuWiUw5eVcicbyK3dq5EeV/QHT1uy6GoDhGbFpprUiHUYggrw==",
|
||||||
|
"cpu": [
|
||||||
|
"x64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"netbsd"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/openbsd-arm64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-AFuojMQTxAz75Fo8idVcqoQWEHIXFRbOc1TrVcFSgCZtQfSdc1RXgB3tjOn/krRHENUB4j00bfGjyl2mJrU37A==",
|
||||||
|
"cpu": [
|
||||||
|
"arm64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"openbsd"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/openbsd-x64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-+A1NJmfM8WNDv5CLVQYJ5PshuRm/4cI6WMZRg1by1GwPIQPCTs1GLEUHwiiQGT5zDdyLiRM/l1G0Pv54gvtKIg==",
|
||||||
|
"cpu": [
|
||||||
|
"x64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"openbsd"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/openharmony-arm64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-+KrvYb/C8zA9CU/g0sR6w2RBw7IGc5J2BPnc3dYc5VJxHCSF1yNMxTV5LQ7GuKteQXZtspjFbiuW5/dOj7H4Yw==",
|
||||||
|
"cpu": [
|
||||||
|
"arm64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"openharmony"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/sunos-x64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-ikktIhFBzQNt/QDyOL580ti9+5mL/YZeUPKU2ivGtGjdTYoqz6jObj6nOMfhASpS4GU4Q/Clh1QtxWAvcYKamA==",
|
||||||
|
"cpu": [
|
||||||
|
"x64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"sunos"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/win32-arm64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-7yRhbHvPqSpRUV7Q20VuDwbjW5kIMwTHpptuUzV+AA46kiPze5Z7qgt6CLCK3pWFrHeNfDd1VKgyP4O+ng17CA==",
|
||||||
|
"cpu": [
|
||||||
|
"arm64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"win32"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/win32-ia32": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-SmwKXe6VHIyZYbBLJrhOoCJRB/Z1tckzmgTLfFYOfpMAx63BJEaL9ExI8x7v0oAO3Zh6D/Oi1gVxEYr5oUCFhw==",
|
||||||
|
"cpu": [
|
||||||
|
"ia32"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"win32"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@esbuild/win32-x64": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-56hiAJPhwQ1R4i+21FVF7V8kSD5zZTdHcVuRFMW0hn753vVfQN8xlx4uOPT4xoGH0Z/oVATuR82AiqSTDIpaHg==",
|
||||||
|
"cpu": [
|
||||||
|
"x64"
|
||||||
|
],
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"win32"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/@types/minimist": {
|
||||||
|
"version": "1.2.5",
|
||||||
|
"resolved": "https://registry.npmjs.org/@types/minimist/-/minimist-1.2.5.tgz",
|
||||||
|
"integrity": "sha512-hov8bUuiLiyFPGyFPE1lwWhmzYbirOXQNNo40+y3zow8aFVTeyn3VWL0VFFfdNddA8S4Vf0Tc062rzyNr7Paag==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT"
|
||||||
|
},
|
||||||
|
"node_modules/@types/node": {
|
||||||
|
"version": "24.12.2",
|
||||||
|
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.12.2.tgz",
|
||||||
|
"integrity": "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"dependencies": {
|
||||||
|
"undici-types": "~7.16.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/boolbase": {
|
||||||
|
"version": "1.0.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/boolbase/-/boolbase-1.0.0.tgz",
|
||||||
|
"integrity": "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==",
|
||||||
|
"license": "ISC"
|
||||||
|
},
|
||||||
|
"node_modules/css-select": {
|
||||||
|
"version": "5.2.2",
|
||||||
|
"resolved": "https://registry.npmjs.org/css-select/-/css-select-5.2.2.tgz",
|
||||||
|
"integrity": "sha512-TizTzUddG/xYLA3NXodFM0fSbNizXjOKhqiQQwvhlspadZokn1KDy0NZFS0wuEubIYAV5/c1/lAr0TaaFXEXzw==",
|
||||||
|
"license": "BSD-2-Clause",
|
||||||
|
"dependencies": {
|
||||||
|
"boolbase": "^1.0.0",
|
||||||
|
"css-what": "^6.1.0",
|
||||||
|
"domhandler": "^5.0.2",
|
||||||
|
"domutils": "^3.0.1",
|
||||||
|
"nth-check": "^2.0.1"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/sponsors/fb55"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/css-what": {
|
||||||
|
"version": "6.2.2",
|
||||||
|
"resolved": "https://registry.npmjs.org/css-what/-/css-what-6.2.2.tgz",
|
||||||
|
"integrity": "sha512-u/O3vwbptzhMs3L1fQE82ZSLHQQfto5gyZzwteVIEyeaY5Fc7R4dapF/BvRoSYFeqfBk4m0V1Vafq5Pjv25wvA==",
|
||||||
|
"license": "BSD-2-Clause",
|
||||||
|
"engines": {
|
||||||
|
"node": ">= 6"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/sponsors/fb55"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/dom-serializer": {
|
||||||
|
"version": "2.0.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/dom-serializer/-/dom-serializer-2.0.0.tgz",
|
||||||
|
"integrity": "sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg==",
|
||||||
|
"license": "MIT",
|
||||||
|
"dependencies": {
|
||||||
|
"domelementtype": "^2.3.0",
|
||||||
|
"domhandler": "^5.0.2",
|
||||||
|
"entities": "^4.2.0"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/cheeriojs/dom-serializer?sponsor=1"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/domelementtype": {
|
||||||
|
"version": "2.3.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/domelementtype/-/domelementtype-2.3.0.tgz",
|
||||||
|
"integrity": "sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw==",
|
||||||
|
"funding": [
|
||||||
|
{
|
||||||
|
"type": "github",
|
||||||
|
"url": "https://github.com/sponsors/fb55"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"license": "BSD-2-Clause"
|
||||||
|
},
|
||||||
|
"node_modules/domhandler": {
|
||||||
|
"version": "5.0.3",
|
||||||
|
"resolved": "https://registry.npmjs.org/domhandler/-/domhandler-5.0.3.tgz",
|
||||||
|
"integrity": "sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w==",
|
||||||
|
"license": "BSD-2-Clause",
|
||||||
|
"dependencies": {
|
||||||
|
"domelementtype": "^2.3.0"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">= 4"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/fb55/domhandler?sponsor=1"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/domutils": {
|
||||||
|
"version": "3.2.2",
|
||||||
|
"resolved": "https://registry.npmjs.org/domutils/-/domutils-3.2.2.tgz",
|
||||||
|
"integrity": "sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw==",
|
||||||
|
"license": "BSD-2-Clause",
|
||||||
|
"dependencies": {
|
||||||
|
"dom-serializer": "^2.0.0",
|
||||||
|
"domelementtype": "^2.3.0",
|
||||||
|
"domhandler": "^5.0.3"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/fb55/domutils?sponsor=1"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/entities": {
|
||||||
|
"version": "4.5.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/entities/-/entities-4.5.0.tgz",
|
||||||
|
"integrity": "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==",
|
||||||
|
"license": "BSD-2-Clause",
|
||||||
|
"engines": {
|
||||||
|
"node": ">=0.12"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/fb55/entities?sponsor=1"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/esbuild": {
|
||||||
|
"version": "0.27.7",
|
||||||
|
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz",
|
||||||
|
"integrity": "sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w==",
|
||||||
|
"dev": true,
|
||||||
|
"hasInstallScript": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"bin": {
|
||||||
|
"esbuild": "bin/esbuild"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
},
|
||||||
|
"optionalDependencies": {
|
||||||
|
"@esbuild/aix-ppc64": "0.27.7",
|
||||||
|
"@esbuild/android-arm": "0.27.7",
|
||||||
|
"@esbuild/android-arm64": "0.27.7",
|
||||||
|
"@esbuild/android-x64": "0.27.7",
|
||||||
|
"@esbuild/darwin-arm64": "0.27.7",
|
||||||
|
"@esbuild/darwin-x64": "0.27.7",
|
||||||
|
"@esbuild/freebsd-arm64": "0.27.7",
|
||||||
|
"@esbuild/freebsd-x64": "0.27.7",
|
||||||
|
"@esbuild/linux-arm": "0.27.7",
|
||||||
|
"@esbuild/linux-arm64": "0.27.7",
|
||||||
|
"@esbuild/linux-ia32": "0.27.7",
|
||||||
|
"@esbuild/linux-loong64": "0.27.7",
|
||||||
|
"@esbuild/linux-mips64el": "0.27.7",
|
||||||
|
"@esbuild/linux-ppc64": "0.27.7",
|
||||||
|
"@esbuild/linux-riscv64": "0.27.7",
|
||||||
|
"@esbuild/linux-s390x": "0.27.7",
|
||||||
|
"@esbuild/linux-x64": "0.27.7",
|
||||||
|
"@esbuild/netbsd-arm64": "0.27.7",
|
||||||
|
"@esbuild/netbsd-x64": "0.27.7",
|
||||||
|
"@esbuild/openbsd-arm64": "0.27.7",
|
||||||
|
"@esbuild/openbsd-x64": "0.27.7",
|
||||||
|
"@esbuild/openharmony-arm64": "0.27.7",
|
||||||
|
"@esbuild/sunos-x64": "0.27.7",
|
||||||
|
"@esbuild/win32-arm64": "0.27.7",
|
||||||
|
"@esbuild/win32-ia32": "0.27.7",
|
||||||
|
"@esbuild/win32-x64": "0.27.7"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/fsevents": {
|
||||||
|
"version": "2.3.3",
|
||||||
|
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
|
||||||
|
"integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
|
||||||
|
"dev": true,
|
||||||
|
"hasInstallScript": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"darwin"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/get-tsconfig": {
|
||||||
|
"version": "4.14.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-4.14.0.tgz",
|
||||||
|
"integrity": "sha512-yTb+8DXzDREzgvYmh6s9vHsSVCHeC0G3PI5bEXNBHtmshPnO+S5O7qgLEOn0I5QvMy6kpZN8K1NKGyilLb93wA==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"dependencies": {
|
||||||
|
"resolve-pkg-maps": "^1.0.0"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/privatenumber/get-tsconfig?sponsor=1"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/he": {
|
||||||
|
"version": "1.2.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/he/-/he-1.2.0.tgz",
|
||||||
|
"integrity": "sha512-F/1DnUGPopORZi0ni+CvrCgHQ5FyEAHRLSApuYWMmrbSwoN2Mn/7k+Gl38gJnR7yyDZk6WLXwiGod1JOWNDKGw==",
|
||||||
|
"license": "MIT",
|
||||||
|
"bin": {
|
||||||
|
"he": "bin/he"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/minimist": {
|
||||||
|
"version": "1.2.8",
|
||||||
|
"resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",
|
||||||
|
"integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",
|
||||||
|
"license": "MIT",
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/sponsors/ljharb"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/node-html-parser": {
|
||||||
|
"version": "7.1.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/node-html-parser/-/node-html-parser-7.1.0.tgz",
|
||||||
|
"integrity": "sha512-iJo8b2uYGT40Y8BTyy5ufL6IVbN8rbm/1QK2xffXU/1a/v3AAa0d1YAoqBNYqaS4R/HajkWIpIfdE6KcyFh1AQ==",
|
||||||
|
"license": "MIT",
|
||||||
|
"dependencies": {
|
||||||
|
"css-select": "^5.1.0",
|
||||||
|
"he": "1.2.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/nth-check": {
|
||||||
|
"version": "2.1.1",
|
||||||
|
"resolved": "https://registry.npmjs.org/nth-check/-/nth-check-2.1.1.tgz",
|
||||||
|
"integrity": "sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w==",
|
||||||
|
"license": "BSD-2-Clause",
|
||||||
|
"dependencies": {
|
||||||
|
"boolbase": "^1.0.0"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/fb55/nth-check?sponsor=1"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/resolve-pkg-maps": {
|
||||||
|
"version": "1.0.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/resolve-pkg-maps/-/resolve-pkg-maps-1.0.0.tgz",
|
||||||
|
"integrity": "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"funding": {
|
||||||
|
"url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/tsx": {
|
||||||
|
"version": "4.21.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.21.0.tgz",
|
||||||
|
"integrity": "sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"dependencies": {
|
||||||
|
"esbuild": "~0.27.0",
|
||||||
|
"get-tsconfig": "^4.7.5"
|
||||||
|
},
|
||||||
|
"bin": {
|
||||||
|
"tsx": "dist/cli.mjs"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18.0.0"
|
||||||
|
},
|
||||||
|
"optionalDependencies": {
|
||||||
|
"fsevents": "~2.3.3"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/typescript": {
|
||||||
|
"version": "5.9.3",
|
||||||
|
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
|
||||||
|
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"bin": {
|
||||||
|
"tsc": "bin/tsc",
|
||||||
|
"tsserver": "bin/tsserver"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=14.17"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/undici-types": {
|
||||||
|
"version": "7.16.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.16.0.tgz",
|
||||||
|
"integrity": "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"name": "amazon-shopping-scripts",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"description": "Amazon shopping helper CLI for OpenClaw skills",
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {
|
||||||
|
"amazon-shopping": "tsx src/cli.ts",
|
||||||
|
"lint": "node scripts/lint.mjs",
|
||||||
|
"test": "node --import tsx --test tests/*.test.ts",
|
||||||
|
"typecheck": "tsc --noEmit"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"minimist": "^1.2.8",
|
||||||
|
"node-html-parser": "^7.1.0"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/minimist": "^1.2.5",
|
||||||
|
"@types/node": "^24.8.1",
|
||||||
|
"tsx": "^4.20.6",
|
||||||
|
"typescript": "^5.9.3"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Amazon Data Map
|
||||||
|
|
||||||
|
Use this reference when deciding which visible Amazon fields can be reported by `amazon-shopping`.
|
||||||
|
|
||||||
|
## Product Search Fields
|
||||||
|
|
||||||
|
Search result cards should be treated as candidates, not final truth. Prefer cards with a non-empty `data-asin` value. Extract only visible data from the rendered search page:
|
||||||
|
|
||||||
|
| Output field | Search-page source | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `asin` | `data-asin` on result card | Required for normalized detail links. |
|
||||||
|
| `title` | product heading or product link text | Trim sponsored/accessibility boilerplate. |
|
||||||
|
| `url` | product link | Normalize to `https://www.amazon.com/dp/<ASIN>` when safe. |
|
||||||
|
| `imageUrl` | visible product image `src` | Optional. |
|
||||||
|
| `price` | visible `.a-price` text | Do not infer absent prices from snippets. |
|
||||||
|
| `rating` | `aria-label` or visible text like `4.6 out of 5 stars` | Optional. |
|
||||||
|
| `reviewCount` | text near rating like `1,234 ratings` | Optional. |
|
||||||
|
| `delivery.display` | visible delivery promise text | Optional and ZIP/session dependent. |
|
||||||
|
| `isSponsored` | visible sponsored marker | Sponsored results may be included but must be labeled. |
|
||||||
|
|
||||||
|
## Detail Page Fields
|
||||||
|
|
||||||
|
Open only normalized product detail URLs under `/dp/<ASIN>` or `/gp/product/<ASIN>`. Extract visible fields:
|
||||||
|
|
||||||
|
| Output field | Detail-page source | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `title` | `#productTitle` or equivalent heading | Detail title can replace search title. |
|
||||||
|
| `price` | buy-box/current price selectors | Variant pages can omit price. |
|
||||||
|
| `delivery` | delivery message near buy box | Report as text, not guaranteed. |
|
||||||
|
| `availability` | availability block | Optional. |
|
||||||
|
| `seller` | seller/ships-from visible text | Optional. |
|
||||||
|
| `bullets` | feature bullets list | Trim empty and hidden items. |
|
||||||
|
| `specs` | product overview/details/technical tables | Preserve name/value pairs. |
|
||||||
|
| `starBreakdown` | visible customer-review histogram | Percent or count basis only. Do not crawl review pages. |
|
||||||
|
|
||||||
|
## Filter Semantics
|
||||||
|
|
||||||
|
- `over 200 reviews` means `reviewCount > 200`.
|
||||||
|
- `at least 200 reviews` means `reviewCount >= 200`.
|
||||||
|
- `more than 4.5 stars` means `rating > 4.5`.
|
||||||
|
- `4.5 stars or better` means `rating >= 4.5`.
|
||||||
|
- `less than $4 each` means visible unit price first, then high-confidence unit-count inference. Unknown unit prices do not pass strict unit-price filters.
|
||||||
|
- Missing fields must be represented as `null` or noted in `missingFields` / `extractionNotes`; never fabricate values.
|
||||||
|
|
||||||
|
## Official Alternatives
|
||||||
|
|
||||||
|
Amazon Business Product Search API and Product Advertising API are official API paths for structured product data when the operator has credentials. This skill uses bounded web automation because the current install request requires `web-automation` scraping.
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Compliance And Failure Modes
|
||||||
|
|
||||||
|
This reference is operational guidance, not legal advice. The operator is responsible for making sure a run complies with Amazon terms, robots directives, local law, and account obligations.
|
||||||
|
|
||||||
|
## Required Guardrails
|
||||||
|
|
||||||
|
- Fetch and evaluate `https://www.amazon.com/robots.txt` before live scraping planned Amazon paths.
|
||||||
|
- Stop if the effective rules disallow the planned search or detail paths.
|
||||||
|
- Do not automate sign-in, checkout, cart, wishlist, review submission, customer-review pages, reviewer profiles, or any disallowed path.
|
||||||
|
- Do not bypass CAPTCHA, bot checks, blocked pages, or access-denied pages.
|
||||||
|
- Do not print cookies, profile state, session storage, or account/location-specific browser data.
|
||||||
|
|
||||||
|
## Allowed Scope
|
||||||
|
|
||||||
|
Allowed behavior is bounded read-only product research over search result pages and normalized product detail pages:
|
||||||
|
|
||||||
|
- `/s?k=<query>` search results.
|
||||||
|
- `/dp/<ASIN>` product details.
|
||||||
|
- `/gp/product/<ASIN>` product details.
|
||||||
|
|
||||||
|
Review data is limited to visible summary ratings/counts and visible histogram rows on search/detail pages. Do not navigate to `/product-reviews`, `/review`, `/gp/customer-reviews`, or review AJAX endpoints.
|
||||||
|
|
||||||
|
## Failure Modes
|
||||||
|
|
||||||
|
Return a structured warning and do not claim success when any of these happen:
|
||||||
|
|
||||||
|
- CAPTCHA or bot-check page.
|
||||||
|
- Sign-in wall.
|
||||||
|
- HTTP 429 or 503 that remains after the bounded retry budget.
|
||||||
|
- Robots rules disallow a planned path.
|
||||||
|
- Product markup changes enough that required fields cannot be found.
|
||||||
|
- Amazon returns localized, personalized, or ZIP/session-dependent delivery text that cannot be verified.
|
||||||
|
|
||||||
|
## Output Rules
|
||||||
|
|
||||||
|
- Unknown fields stay unknown.
|
||||||
|
- Partial extraction is acceptable only when the response includes warnings and missing-field notes.
|
||||||
|
- Sponsored products can be returned by default but must be labeled.
|
||||||
|
- Counts above 30 require operator confirmation or batch splitting.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Web-Automation Prompts
|
||||||
|
|
||||||
|
Use these patterns when debugging or extending the `amazon-shopping` browser workflow. The TypeScript helper is the default interface; these prompts document the intended rendered-page behavior.
|
||||||
|
|
||||||
|
## Search Page
|
||||||
|
|
||||||
|
```text
|
||||||
|
Use the installed web-automation skill. Open https://www.amazon.com/s?k=<encoded query>. Wait for rendered search results. If a CAPTCHA, bot check, sign-in wall, or access denied page appears, stop and return a challenge status. Otherwise extract visible product cards with ASIN, title, product URL, displayed price, rating text, review count text, delivery text, sponsor marker, and image URL. Do not open cart, sign-in, wishlist, or review-listing pages.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Detail Page
|
||||||
|
|
||||||
|
```text
|
||||||
|
For each candidate product detail URL under /dp/<ASIN> or /gp/product/<ASIN>, open the page slowly one at a time. Extract title, canonical link, visible buy-box/current price, delivery summary, availability, seller when visible, feature bullets, product details/specification table rows, rating score, review count, and visible customer-review histogram percentages if present on the detail page. Do not navigate to /product-reviews or reviewer profile pages.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pagination
|
||||||
|
|
||||||
|
```text
|
||||||
|
Follow only the visible Amazon pagination control for the next search page, or construct page=<n> only after the current page exposes normal search results and no challenge/block. Stop when enough candidates have been collected, no next page exists, a challenge appears, or maxSearchPages is reached.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Robustness Notes
|
||||||
|
|
||||||
|
- Prefer Playwright locator/actionability behavior and bounded waits over fixed sleeps.
|
||||||
|
- Never follow sponsored redirect URLs, sign-in links, cart links, wishlist links, or review-page links.
|
||||||
|
- Return partial results with warnings when Amazon markup changes or fields are hidden.
|
||||||
Executable
+54
@@ -0,0 +1,54 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
import { readdir, readFile, stat } from "node:fs/promises";
|
||||||
|
import { join, relative } from "node:path";
|
||||||
|
|
||||||
|
const root = new URL("..", import.meta.url).pathname;
|
||||||
|
const scannedExtensions = new Set([".md", ".json", ".ts", ".js", ".mjs", ".sh"]);
|
||||||
|
const installSpecificPath = ["", "Users", "stefano"].join("/");
|
||||||
|
const forbidden = [
|
||||||
|
{
|
||||||
|
pattern: installSpecificPath,
|
||||||
|
message: "Source files must not hardcode this install path"
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
function extensionOf(path) {
|
||||||
|
const dot = path.lastIndexOf(".");
|
||||||
|
return dot === -1 ? "" : path.slice(dot);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function walk(dir) {
|
||||||
|
const entries = await readdir(dir);
|
||||||
|
const files = [];
|
||||||
|
for (const entry of entries) {
|
||||||
|
if (["node_modules", "dist", "coverage", "tmp", "out"].includes(entry)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const path = join(dir, entry);
|
||||||
|
const info = await stat(path);
|
||||||
|
if (info.isDirectory()) {
|
||||||
|
files.push(...await walk(path));
|
||||||
|
} else if (scannedExtensions.has(extensionOf(path)) || entry === "SKILL.md") {
|
||||||
|
files.push(path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return files;
|
||||||
|
}
|
||||||
|
|
||||||
|
const failures = [];
|
||||||
|
for (const file of await walk(root)) {
|
||||||
|
const text = await readFile(file, "utf8");
|
||||||
|
for (const rule of forbidden) {
|
||||||
|
if (text.includes(rule.pattern)) {
|
||||||
|
failures.push(`${relative(root, file)}: ${rule.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (failures.length > 0) {
|
||||||
|
for (const failure of failures) {
|
||||||
|
console.error(failure);
|
||||||
|
}
|
||||||
|
process.exitCode = 1;
|
||||||
|
}
|
||||||
+13
@@ -0,0 +1,13 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
SKILL_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||||
|
TSX="${SKILL_DIR}/node_modules/.bin/tsx"
|
||||||
|
|
||||||
|
if [ ! -x "$TSX" ]; then
|
||||||
|
echo "Missing local dependencies. Run: cd \"$SKILL_DIR\" && npm install" >&2
|
||||||
|
exit 127
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec "$TSX" "$SKILL_DIR/src/cli.ts" "$@"
|
||||||
Executable
+10
@@ -0,0 +1,10 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
SKILL_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||||
|
|
||||||
|
cd "$SKILL_DIR"
|
||||||
|
npm install
|
||||||
|
npm run lint
|
||||||
|
npm test
|
||||||
@@ -0,0 +1,228 @@
|
|||||||
|
import { execFile } from "node:child_process";
|
||||||
|
import { pathToFileURL } from "node:url";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { promisify } from "node:util";
|
||||||
|
|
||||||
|
import { extractDetailPage } from "./detail-page.js";
|
||||||
|
import { applyFiltersAndLimit } from "./filters.js";
|
||||||
|
import { createResponse } from "./report.js";
|
||||||
|
import { extractSearchPage } from "./search-page.js";
|
||||||
|
import type { ProductSearchResult, SearchProductsRequest, SearchProductsResponse } from "./types.js";
|
||||||
|
import { resolveWebAutomationRuntime } from "./web-automation-runtime.js";
|
||||||
|
|
||||||
|
const execFileAsync = promisify(execFile);
|
||||||
|
const AMAZON_ROOT = "https://www.amazon.com";
|
||||||
|
const DEFAULT_WAIT_MS = 4500;
|
||||||
|
|
||||||
|
export type HttpClassification = "ok" | "retryable" | "challenge";
|
||||||
|
|
||||||
|
interface BrowserDeps {
|
||||||
|
fetchText?: (url: string) => Promise<string>;
|
||||||
|
sleep?: (ms: number) => Promise<void>;
|
||||||
|
now?: () => Date;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function plannedAmazonPaths(asins: string[]): string[] {
|
||||||
|
return [
|
||||||
|
"/s",
|
||||||
|
...asins.flatMap((asin) => [`/dp/${asin}`, `/gp/product/${asin}`])
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function classifyHttpStatus(status: number | null | undefined): HttpClassification {
|
||||||
|
if (status === 429 || status === 503) return "retryable";
|
||||||
|
if (status === 401 || status === 403) return "challenge";
|
||||||
|
return "ok";
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isPathAllowedByRobots(robots: string, userAgent: string, path: string): boolean {
|
||||||
|
const groups: Array<{ agents: string[]; disallows: string[] }> = [];
|
||||||
|
let current: { agents: string[]; disallows: string[] } | undefined;
|
||||||
|
let hasDirectives = false;
|
||||||
|
|
||||||
|
const lines = robots.split(/\r?\n/);
|
||||||
|
|
||||||
|
for (const rawLine of lines) {
|
||||||
|
const line = rawLine.replace(/#.*/, "").trim();
|
||||||
|
if (!line) continue;
|
||||||
|
const [rawKey, ...rest] = line.split(":");
|
||||||
|
const key = rawKey.trim().toLowerCase();
|
||||||
|
const value = rest.join(":").trim();
|
||||||
|
|
||||||
|
if (key === "user-agent") {
|
||||||
|
if (!current || hasDirectives) {
|
||||||
|
current = { agents: [], disallows: [] };
|
||||||
|
groups.push(current);
|
||||||
|
hasDirectives = false;
|
||||||
|
}
|
||||||
|
current.agents.push(value.toLowerCase());
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (key === "disallow") {
|
||||||
|
hasDirectives = true;
|
||||||
|
if (current && value) {
|
||||||
|
current.disallows.push(value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const normalizedAgent = userAgent.toLowerCase();
|
||||||
|
const exactGroups = groups.filter((group) => group.agents.includes(normalizedAgent));
|
||||||
|
const matchedGroups = exactGroups.length > 0 ? exactGroups : groups.filter((group) => group.agents.includes("*"));
|
||||||
|
const disallows = matchedGroups.flatMap((group) => group.disallows);
|
||||||
|
|
||||||
|
return !disallows.some((rule) => path.startsWith(rule));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function defaultFetchText(url: string): Promise<string> {
|
||||||
|
const response = await fetch(url);
|
||||||
|
return response.text();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function checkRobots(paths: string[], deps: BrowserDeps): Promise<string[]> {
|
||||||
|
const warnings: string[] = [];
|
||||||
|
const robots = await (deps.fetchText ?? defaultFetchText)(`${AMAZON_ROOT}/robots.txt`);
|
||||||
|
for (const path of paths) {
|
||||||
|
if (!isPathAllowedByRobots(robots, "*", path)) {
|
||||||
|
warnings.push(`Amazon robots directives disallow planned path: ${path}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return warnings;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadCloakBrowser(runtimeDir: string): Promise<{
|
||||||
|
ensureBinary?: () => Promise<void>;
|
||||||
|
launchContext: (options: Record<string, unknown>) => Promise<any>;
|
||||||
|
}> {
|
||||||
|
const moduleUrl = pathToFileURL(join(runtimeDir, "node_modules", "cloakbrowser", "dist", "index.js")).toString();
|
||||||
|
return import(moduleUrl) as Promise<any>;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function checkRuntime(): Promise<string> {
|
||||||
|
const runtime = await resolveWebAutomationRuntime();
|
||||||
|
await execFileAsync(runtime.checkInstall.command, runtime.checkInstall.args, { cwd: runtime.checkInstall.cwd });
|
||||||
|
return runtime.scriptsDir;
|
||||||
|
}
|
||||||
|
|
||||||
|
function searchUrl(query: string, pageNumber: number): string {
|
||||||
|
const url = new URL("/s", AMAZON_ROOT);
|
||||||
|
url.searchParams.set("k", query);
|
||||||
|
if (pageNumber > 1) {
|
||||||
|
url.searchParams.set("page", String(pageNumber));
|
||||||
|
}
|
||||||
|
return url.toString();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function pageHtml(page: any, url: string, deps: BrowserDeps): Promise<{ html: string; status: number | null }> {
|
||||||
|
let lastStatus: number | null = null;
|
||||||
|
for (let attempt = 0; attempt < 3; attempt += 1) {
|
||||||
|
const response = await page.goto(url, { waitUntil: "domcontentloaded", timeout: 45000 });
|
||||||
|
await page.waitForTimeout?.(DEFAULT_WAIT_MS);
|
||||||
|
lastStatus = response?.status?.() ?? null;
|
||||||
|
if (classifyHttpStatus(lastStatus) !== "retryable") {
|
||||||
|
return {
|
||||||
|
html: await page.content(),
|
||||||
|
status: lastStatus
|
||||||
|
};
|
||||||
|
}
|
||||||
|
await (deps.sleep ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms))))((2 ** attempt) * 1000 + Math.floor(Math.random() * 500));
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
html: await page.content(),
|
||||||
|
status: lastStatus
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function enrichDetails(page: any, products: ProductSearchResult[], deps: BrowserDeps): Promise<ProductSearchResult[]> {
|
||||||
|
const enriched: ProductSearchResult[] = [];
|
||||||
|
for (const product of products) {
|
||||||
|
await (deps.sleep ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms))))(1500 + Math.floor(Math.random() * 1500));
|
||||||
|
const loaded = await pageHtml(page, product.url, deps);
|
||||||
|
const classification = classifyHttpStatus(loaded.status);
|
||||||
|
if (classification === "challenge") {
|
||||||
|
enriched.push({
|
||||||
|
...product,
|
||||||
|
extractionNotes: [...product.extractionNotes, "Detail page returned a challenge/block status."]
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
enriched.push(extractDetailPage(loaded.html, product));
|
||||||
|
}
|
||||||
|
return enriched;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function searchProducts(request: SearchProductsRequest, deps: BrowserDeps = {}): Promise<SearchProductsResponse> {
|
||||||
|
const warnings: string[] = [];
|
||||||
|
const robotsWarnings = await checkRobots(plannedAmazonPaths([]), deps);
|
||||||
|
if (robotsWarnings.length > 0) {
|
||||||
|
return createResponse({
|
||||||
|
query: request.query,
|
||||||
|
filters: request.filters,
|
||||||
|
limit: request.limit,
|
||||||
|
maxSearchPages: request.maxSearchPages,
|
||||||
|
results: [],
|
||||||
|
filteredOutCount: 0,
|
||||||
|
warnings: robotsWarnings,
|
||||||
|
now: deps.now
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const runtimeDir = await checkRuntime();
|
||||||
|
const cloak = await loadCloakBrowser(runtimeDir);
|
||||||
|
await cloak.ensureBinary?.();
|
||||||
|
const context = await cloak.launchContext({
|
||||||
|
headless: process.env.CLOAKBROWSER_HEADLESS !== "false",
|
||||||
|
locale: "en-US",
|
||||||
|
viewport: { width: 1440, height: 900 },
|
||||||
|
humanize: true
|
||||||
|
});
|
||||||
|
const page = await context.newPage();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const candidates: ProductSearchResult[] = [];
|
||||||
|
let nextUrl: string | undefined = searchUrl(request.query, 1);
|
||||||
|
for (let pageNumber = 1; pageNumber <= request.maxSearchPages && nextUrl; pageNumber += 1) {
|
||||||
|
const loaded = await pageHtml(page, nextUrl, deps);
|
||||||
|
const classification = classifyHttpStatus(loaded.status);
|
||||||
|
if (classification === "challenge" || classification === "retryable") {
|
||||||
|
warnings.push(`Amazon returned status ${loaded.status}; stopping without bypass.`);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
const extracted = extractSearchPage(loaded.html, nextUrl);
|
||||||
|
warnings.push(...extracted.warnings);
|
||||||
|
if (extracted.status === "challenge") {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
candidates.push(...extracted.products);
|
||||||
|
if (candidates.length >= request.limit * 3) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
nextUrl = extracted.nextPageUrl ?? (pageNumber + 1 <= request.maxSearchPages ? searchUrl(request.query, pageNumber + 1) : undefined);
|
||||||
|
}
|
||||||
|
|
||||||
|
let detailCandidates = candidates;
|
||||||
|
if (!request.skipDetails) {
|
||||||
|
const detailPaths = plannedAmazonPaths(candidates.map((candidate) => candidate.asin)).filter((path) => path !== "/s");
|
||||||
|
const detailRobotsWarnings = await checkRobots(detailPaths, deps);
|
||||||
|
if (detailRobotsWarnings.length > 0) {
|
||||||
|
warnings.push(...detailRobotsWarnings, "Detail enrichment skipped because robots directives disallow at least one planned detail path.");
|
||||||
|
} else {
|
||||||
|
detailCandidates = await enrichDetails(page, candidates.slice(0, request.limit * 3), deps);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const filtered = applyFiltersAndLimit(detailCandidates, request.filters, request.limit);
|
||||||
|
return createResponse({
|
||||||
|
query: request.query,
|
||||||
|
filters: request.filters,
|
||||||
|
limit: request.limit,
|
||||||
|
maxSearchPages: request.maxSearchPages,
|
||||||
|
results: filtered.results,
|
||||||
|
filteredOutCount: filtered.filteredOutCount,
|
||||||
|
warnings,
|
||||||
|
now: deps.now
|
||||||
|
});
|
||||||
|
} finally {
|
||||||
|
await context.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,191 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
import minimist from "minimist";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
|
||||||
|
import { searchProducts } from "./browser.js";
|
||||||
|
import { parseNaturalLanguageRequest } from "./query-parser.js";
|
||||||
|
import { createMarkdownReport } from "./report.js";
|
||||||
|
import type { ProductFilters, SearchProductsRequest, SearchProductsResponse } from "./types.js";
|
||||||
|
|
||||||
|
export interface CliDeps {
|
||||||
|
stdout: Pick<NodeJS.WriteStream, "write">;
|
||||||
|
stderr: Pick<NodeJS.WriteStream, "write">;
|
||||||
|
now?: () => Date;
|
||||||
|
searchProducts?: (request: SearchProductsRequest) => Promise<SearchProductsResponse>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function usage(): string {
|
||||||
|
return `amazon-shopping
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
scripts/search-products "<product request>" [options]
|
||||||
|
scripts/search-products --query "<product request>" [options]
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--json Print JSON output
|
||||||
|
--markdown Print markdown output
|
||||||
|
--limit N, --max N Maximum products to return (default: 15)
|
||||||
|
--allow-large-limit Permit limits above 30
|
||||||
|
--min-rating N Minimum rating score
|
||||||
|
--min-reviews N Minimum review count
|
||||||
|
--max-price N Maximum displayed product price
|
||||||
|
--max-unit-price N Maximum price per unit
|
||||||
|
--max-search-pages N Search result pages to scan, 1-5 (default: 2)
|
||||||
|
--skip-details Do not open product detail pages
|
||||||
|
--dry-run Parse and print the planned request without Amazon network access
|
||||||
|
--help Show this help
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parsePositiveInteger(value: unknown, name: string): number | undefined {
|
||||||
|
if (value === undefined) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const parsed = Number(value);
|
||||||
|
if (!Number.isInteger(parsed) || parsed < 1) {
|
||||||
|
throw new Error(`${name} must be an integer greater than 0`);
|
||||||
|
}
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseNumber(value: unknown, name: string): number | undefined {
|
||||||
|
if (value === undefined) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const parsed = Number(value);
|
||||||
|
if (!Number.isFinite(parsed)) {
|
||||||
|
throw new Error(`${name} must be a number`);
|
||||||
|
}
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildSearchUrl(query: string): string {
|
||||||
|
return `https://www.amazon.com/s?k=${encodeURIComponent(query)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseCliRequest(argv: string[]): SearchProductsRequest {
|
||||||
|
const args = minimist(argv, {
|
||||||
|
boolean: ["help", "json", "markdown", "allow-large-limit", "dry-run", "skip-details"],
|
||||||
|
string: [
|
||||||
|
"query",
|
||||||
|
"limit",
|
||||||
|
"max",
|
||||||
|
"min-rating",
|
||||||
|
"min-reviews",
|
||||||
|
"max-price",
|
||||||
|
"max-unit-price",
|
||||||
|
"max-search-pages"
|
||||||
|
],
|
||||||
|
alias: { h: "help", max: "limit" }
|
||||||
|
});
|
||||||
|
|
||||||
|
const rawQuery = String(args.query ?? args._.join(" ")).trim();
|
||||||
|
if (!rawQuery) {
|
||||||
|
throw new Error("A product query is required");
|
||||||
|
}
|
||||||
|
|
||||||
|
const natural = parseNaturalLanguageRequest(rawQuery);
|
||||||
|
const limit = parsePositiveInteger(args.limit, "limit") ?? natural.limit ?? 15;
|
||||||
|
if (limit > 30 && !args["allow-large-limit"]) {
|
||||||
|
throw new Error("Requested limits above 30 require --allow-large-limit or a batched run");
|
||||||
|
}
|
||||||
|
|
||||||
|
const maxSearchPages = parsePositiveInteger(args["max-search-pages"], "max-search-pages") ?? 2;
|
||||||
|
if (maxSearchPages > 5) {
|
||||||
|
throw new Error("max-search-pages must be 5 or less");
|
||||||
|
}
|
||||||
|
|
||||||
|
const filters: ProductFilters = { ...natural.filters };
|
||||||
|
const minRating = parseNumber(args["min-rating"], "min-rating");
|
||||||
|
const minReviews = parsePositiveInteger(args["min-reviews"], "min-reviews");
|
||||||
|
const maxPrice = parseNumber(args["max-price"], "max-price");
|
||||||
|
const maxUnitPrice = parseNumber(args["max-unit-price"], "max-unit-price");
|
||||||
|
if (minRating !== undefined) filters.minRating = minRating;
|
||||||
|
if (minReviews !== undefined) filters.minReviews = minReviews;
|
||||||
|
if (maxPrice !== undefined) filters.maxPrice = maxPrice;
|
||||||
|
if (maxUnitPrice !== undefined) filters.maxUnitPrice = maxUnitPrice;
|
||||||
|
|
||||||
|
const json = Boolean(args.json);
|
||||||
|
const markdown = Boolean(args.markdown);
|
||||||
|
|
||||||
|
return {
|
||||||
|
query: natural.query || rawQuery,
|
||||||
|
filters,
|
||||||
|
limit,
|
||||||
|
maxSearchPages,
|
||||||
|
skipDetails: Boolean(args["skip-details"]),
|
||||||
|
dryRun: Boolean(args["dry-run"]),
|
||||||
|
output: json && markdown ? "both" : markdown ? "markdown" : "json"
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function createDryRunResponse(request: SearchProductsRequest, now: () => Date): SearchProductsResponse {
|
||||||
|
return {
|
||||||
|
query: request.query,
|
||||||
|
filters: request.filters,
|
||||||
|
limit: request.limit,
|
||||||
|
maxSearchPages: request.maxSearchPages,
|
||||||
|
results: [],
|
||||||
|
filteredOutCount: 0,
|
||||||
|
warnings: [`Dry run only. Planned search URL: ${buildSearchUrl(request.query)}`],
|
||||||
|
source: {
|
||||||
|
site: "amazon.com",
|
||||||
|
scrapedAt: now().toISOString(),
|
||||||
|
automation: "web-automation/CloakBrowser"
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function defaultSearchProducts(request: SearchProductsRequest, deps: CliDeps): Promise<SearchProductsResponse> {
|
||||||
|
if (request.dryRun) {
|
||||||
|
return createDryRunResponse(request, deps.now ?? (() => new Date()));
|
||||||
|
}
|
||||||
|
return searchProducts(request, { now: deps.now });
|
||||||
|
}
|
||||||
|
|
||||||
|
function writeResponse(response: SearchProductsResponse, output: SearchProductsRequest["output"], deps: CliDeps): void {
|
||||||
|
if (output === "markdown") {
|
||||||
|
deps.stdout.write(createMarkdownReport(response));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (output === "both") {
|
||||||
|
deps.stdout.write(`${JSON.stringify(response, null, 2)}\n\n${createMarkdownReport(response)}`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
deps.stdout.write(`${JSON.stringify(response, null, 2)}\n`);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function runCli(
|
||||||
|
argv: string[],
|
||||||
|
deps: CliDeps = { stdout: process.stdout, stderr: process.stderr }
|
||||||
|
): Promise<number> {
|
||||||
|
const rawArgs = minimist(argv, { boolean: ["help"], alias: { h: "help" } });
|
||||||
|
if (rawArgs.help || argv.length === 0) {
|
||||||
|
deps.stdout.write(usage());
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const request = parseCliRequest(argv);
|
||||||
|
const response = deps.searchProducts
|
||||||
|
? await deps.searchProducts(request)
|
||||||
|
: await defaultSearchProducts(request, deps);
|
||||||
|
writeResponse(response, request.output, deps);
|
||||||
|
return 0;
|
||||||
|
} catch (error: unknown) {
|
||||||
|
const message = error instanceof Error ? error.message : String(error);
|
||||||
|
deps.stderr.write(`${message}\n`);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {
|
||||||
|
runCli(process.argv.slice(2)).then((code) => {
|
||||||
|
process.exitCode = code;
|
||||||
|
}).catch((error: unknown) => {
|
||||||
|
const message = error instanceof Error ? error.message : String(error);
|
||||||
|
process.stderr.write(`${message}\n`);
|
||||||
|
process.exitCode = 1;
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
import { HTMLElement, parse } from "node-html-parser";
|
||||||
|
|
||||||
|
import { parseMoney, parseRating, parseReviewCount, parseStarBreakdown } from "./parsers.js";
|
||||||
|
import type { DeliverySummary, ProductSearchResult, ProductSpec } from "./types.js";
|
||||||
|
|
||||||
|
function textOf(node: HTMLElement | null | undefined): string {
|
||||||
|
return cleanText(node?.textContent ?? "");
|
||||||
|
}
|
||||||
|
|
||||||
|
function attrOf(node: HTMLElement | null | undefined, name: string): string {
|
||||||
|
return cleanText(node?.getAttribute(name) ?? "");
|
||||||
|
}
|
||||||
|
|
||||||
|
function cleanText(text: string): string {
|
||||||
|
return text
|
||||||
|
.replace(/\s+/g, " ")
|
||||||
|
.replace(/\s*\{".*$/g, "")
|
||||||
|
.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
function isScriptLike(text: string): boolean {
|
||||||
|
return /\(function\s*\(|window\.|P\.when|ue\.count|tracking\(\)|logShoppableMetrics|buying options|add to cart/i.test(text);
|
||||||
|
}
|
||||||
|
|
||||||
|
function firstText(root: HTMLElement, selectors: string[]): string {
|
||||||
|
for (const selector of selectors) {
|
||||||
|
const text = textOf(root.querySelector(selector));
|
||||||
|
if (text) {
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
|
||||||
|
function extractBullets(root: HTMLElement): string[] {
|
||||||
|
const spanBullets = root.querySelectorAll("#feature-bullets li span")
|
||||||
|
.map((node) => textOf(node))
|
||||||
|
.filter((text) => text && !/make sure this fits/i.test(text));
|
||||||
|
if (spanBullets.length > 0) {
|
||||||
|
return spanBullets;
|
||||||
|
}
|
||||||
|
return root.querySelectorAll("#feature-bullets li")
|
||||||
|
.map((node) => textOf(node))
|
||||||
|
.filter((text) => text && !/make sure this fits/i.test(text));
|
||||||
|
}
|
||||||
|
|
||||||
|
function extractSpecs(root: HTMLElement): ProductSpec[] {
|
||||||
|
const specs: ProductSpec[] = [];
|
||||||
|
const seen = new Set<string>();
|
||||||
|
const excludedNames = new Set(["customer reviews"]);
|
||||||
|
for (const row of root.querySelectorAll("tr")) {
|
||||||
|
const cells = row.querySelectorAll("th,td").map((cell) => textOf(cell)).filter(Boolean);
|
||||||
|
if (cells.length >= 2) {
|
||||||
|
const name = cells[0];
|
||||||
|
const value = cells.slice(1).join(" ");
|
||||||
|
const key = name.toLowerCase();
|
||||||
|
if (seen.has(key) || excludedNames.has(key) || isScriptLike(name) || isScriptLike(value)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
seen.add(key);
|
||||||
|
specs.push({ name, value });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return specs;
|
||||||
|
}
|
||||||
|
|
||||||
|
function extractHistogramText(root: HTMLElement): string {
|
||||||
|
const rows = root.querySelectorAll("#histogramTable tr, [aria-label*='star'] tr");
|
||||||
|
const parts: string[] = [];
|
||||||
|
for (const row of rows) {
|
||||||
|
const cells = row.querySelectorAll("th,td").map((cell) => textOf(cell)).filter(Boolean);
|
||||||
|
if (cells.length >= 2) {
|
||||||
|
parts.push(`${cells[0]} ${cells[1]}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return parts.join(" ");
|
||||||
|
}
|
||||||
|
|
||||||
|
function deliveryFromText(text: string): DeliverySummary | undefined {
|
||||||
|
const display = text.replace(/\s+/g, " ").trim();
|
||||||
|
if (!display) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
display,
|
||||||
|
free: /\bfree\b/i.test(display),
|
||||||
|
prime: /\bprime\b/i.test(display)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function extractDetailPage(html: string, base: ProductSearchResult): ProductSearchResult {
|
||||||
|
const root = parse(html);
|
||||||
|
const title = firstText(root, ["#productTitle", "h1"]) || base.title;
|
||||||
|
const priceText = firstText(root, [
|
||||||
|
"#corePriceDisplay_desktop_feature_div .a-offscreen",
|
||||||
|
".a-price .a-offscreen",
|
||||||
|
".a-price"
|
||||||
|
]);
|
||||||
|
const deliveryText = firstText(root, [
|
||||||
|
"#mir-layout-DELIVERY_BLOCK-slot-PRIMARY_DELIVERY_MESSAGE_LARGE",
|
||||||
|
"#deliveryBlockMessage",
|
||||||
|
"[data-csa-c-delivery-price]"
|
||||||
|
]);
|
||||||
|
const availability = firstText(root, ["#availability", "#availabilityInsideBuyBox_feature_div"]);
|
||||||
|
const seller = firstText(root, ["#merchant-info", "#sellerProfileTriggerId"]);
|
||||||
|
const ratingText = attrOf(root.querySelector("#acrPopover"), "title") || textOf(root.querySelector("#acrPopover"));
|
||||||
|
const reviewText = textOf(root.querySelector("#acrCustomerReviewText"));
|
||||||
|
const histogram = parseStarBreakdown(extractHistogramText(root));
|
||||||
|
|
||||||
|
const product: ProductSearchResult = {
|
||||||
|
...base,
|
||||||
|
title,
|
||||||
|
price: parseMoney(priceText) ?? base.price,
|
||||||
|
rating: parseRating(ratingText) ?? base.rating,
|
||||||
|
reviewCount: parseReviewCount(reviewText) ?? base.reviewCount,
|
||||||
|
delivery: deliveryFromText(deliveryText) ?? base.delivery,
|
||||||
|
availability: availability || base.availability,
|
||||||
|
seller: seller || base.seller,
|
||||||
|
bullets: extractBullets(root),
|
||||||
|
specs: extractSpecs(root),
|
||||||
|
starBreakdown: histogram ?? base.starBreakdown,
|
||||||
|
missingFields: [...base.missingFields],
|
||||||
|
extractionNotes: [...base.extractionNotes]
|
||||||
|
};
|
||||||
|
|
||||||
|
for (const field of ["price", "delivery", "rating", "reviewCount", "starBreakdown"] as const) {
|
||||||
|
if (product[field] === undefined && !product.missingFields.includes(field)) {
|
||||||
|
product.missingFields.push(field);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return product;
|
||||||
|
}
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
import type { FilteredProducts, ProductFilters, ProductSearchResult } from "./types.js";
|
||||||
|
|
||||||
|
function passesMin(value: number | undefined, threshold: number, comparison: "gt" | "gte" | undefined): boolean {
|
||||||
|
if (value === undefined) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return comparison === "gt" ? value > threshold : value >= threshold;
|
||||||
|
}
|
||||||
|
|
||||||
|
function filterReasons(product: ProductSearchResult, filters: ProductFilters): string[] {
|
||||||
|
const reasons: string[] = [];
|
||||||
|
if (filters.minRating !== undefined && !passesMin(product.rating, filters.minRating, filters.ratingComparison)) {
|
||||||
|
reasons.push(product.rating === undefined ? "rating unknown" : `rating ${product.rating} below filter`);
|
||||||
|
}
|
||||||
|
if (filters.minReviews !== undefined && !passesMin(product.reviewCount, filters.minReviews, filters.reviewCountComparison)) {
|
||||||
|
reasons.push(product.reviewCount === undefined ? "review count unknown" : `review count ${product.reviewCount} below filter`);
|
||||||
|
}
|
||||||
|
if (filters.maxPrice !== undefined) {
|
||||||
|
if (!product.price) {
|
||||||
|
reasons.push("price unknown");
|
||||||
|
} else if (product.price.amount > filters.maxPrice) {
|
||||||
|
reasons.push(`price ${product.price.display} above filter`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (filters.maxUnitPrice !== undefined) {
|
||||||
|
if (!product.unitPrice) {
|
||||||
|
reasons.push("unit price unknown");
|
||||||
|
} else if (product.unitPrice.amount > filters.maxUnitPrice) {
|
||||||
|
reasons.push(`unit price ${product.unitPrice.display} above filter`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (filters.requirePrime && !product.delivery?.prime) {
|
||||||
|
reasons.push("Prime delivery not verified");
|
||||||
|
}
|
||||||
|
if (filters.requireFreeDelivery && !product.delivery?.free) {
|
||||||
|
reasons.push("free delivery not verified");
|
||||||
|
}
|
||||||
|
return reasons;
|
||||||
|
}
|
||||||
|
|
||||||
|
function rankProducts(a: ProductSearchResult, b: ProductSearchResult): number {
|
||||||
|
const ratingDiff = (b.rating ?? -1) - (a.rating ?? -1);
|
||||||
|
if (ratingDiff !== 0) return ratingDiff;
|
||||||
|
const reviewDiff = (b.reviewCount ?? -1) - (a.reviewCount ?? -1);
|
||||||
|
if (reviewDiff !== 0) return reviewDiff;
|
||||||
|
return (a.price?.amount ?? Number.POSITIVE_INFINITY) - (b.price?.amount ?? Number.POSITIVE_INFINITY);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function applyFiltersAndLimit(
|
||||||
|
products: ProductSearchResult[],
|
||||||
|
filters: ProductFilters,
|
||||||
|
limit: number
|
||||||
|
): FilteredProducts {
|
||||||
|
const filteredOutReasons: Record<string, string[]> = {};
|
||||||
|
const uniqueProducts = new Map<string, ProductSearchResult>();
|
||||||
|
for (const product of products) {
|
||||||
|
if (!uniqueProducts.has(product.asin)) {
|
||||||
|
uniqueProducts.set(product.asin, product);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const passing: ProductSearchResult[] = [];
|
||||||
|
|
||||||
|
for (const product of uniqueProducts.values()) {
|
||||||
|
const reasons = filterReasons(product, filters);
|
||||||
|
if (reasons.length > 0) {
|
||||||
|
filteredOutReasons[product.asin] = reasons;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
passing.push({
|
||||||
|
...product,
|
||||||
|
matchedFilters: [
|
||||||
|
...product.matchedFilters,
|
||||||
|
...(filters.minRating !== undefined ? [`rating ${filters.ratingComparison ?? "gte"} ${filters.minRating}`] : []),
|
||||||
|
...(filters.minReviews !== undefined ? [`reviews ${filters.reviewCountComparison ?? "gte"} ${filters.minReviews}`] : []),
|
||||||
|
...(filters.maxPrice !== undefined ? [`price <= ${filters.maxPrice}`] : []),
|
||||||
|
...(filters.maxUnitPrice !== undefined ? [`unit price <= ${filters.maxUnitPrice}`] : [])
|
||||||
|
]
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
results: passing.sort(rankProducts).slice(0, limit),
|
||||||
|
filteredOutCount: uniqueProducts.size - passing.length,
|
||||||
|
filteredOutReasons
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
import type { MoneyValue, StarBreakdown, UnitCountExtraction } from "./types.js";
|
||||||
|
|
||||||
|
export function parseMoney(text: string | undefined | null): MoneyValue | undefined {
|
||||||
|
if (!text) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const compact = text.replace(/\s+/g, " ").trim();
|
||||||
|
const match = compact.match(/\$\s*([0-9][0-9,]*(?:\.[0-9]{1,2})?)/);
|
||||||
|
if (!match) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const amount = Number(match[1].replace(/,/g, ""));
|
||||||
|
if (!Number.isFinite(amount)) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
amount,
|
||||||
|
currency: "USD",
|
||||||
|
display: compact
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseUnitPrice(text: string | undefined | null): MoneyValue | undefined {
|
||||||
|
if (!text) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const compact = text.replace(/\s+/g, " ").trim();
|
||||||
|
const unitMatch = compact.match(/(\$\s*[0-9][0-9,]*(?:\.[0-9]{1,2})?)(?:\s*\$\s*[0-9][0-9,]*(?:\.[0-9]{1,2})?)?\s*(?:\/|\bper\b\s*)\s*(?:count|unit|item|piece|pack|each)\b/i)
|
||||||
|
?? compact.match(/(\$\s*[0-9][0-9,]*(?:\.[0-9]{1,2})?)\s*(?:each)\b/i);
|
||||||
|
if (!unitMatch) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const display = unitMatch[0]
|
||||||
|
.replace(/\$\s*([0-9][0-9,]*(?:\.[0-9]{1,2})?)\s*\$\s*\1/i, "$$$1")
|
||||||
|
.replace(/\s+/g, "");
|
||||||
|
return parseMoney(display);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseRating(text: string | undefined | null): number | undefined {
|
||||||
|
if (!text) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const match = text.match(/([0-5](?:\.[0-9])?)\s*(?:out of\s*)?5\s*stars?/i)
|
||||||
|
?? text.match(/\brated\s+([0-5](?:\.[0-9])?)/i);
|
||||||
|
if (!match) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const rating = Number(match[1]);
|
||||||
|
return Number.isFinite(rating) ? rating : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseReviewCount(text: string | undefined | null): number | undefined {
|
||||||
|
if (!text) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const match = text.match(/([0-9][0-9,]*)\s*(?:ratings?|reviews?)/i);
|
||||||
|
if (!match) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const count = Number(match[1].replace(/,/g, ""));
|
||||||
|
return Number.isInteger(count) ? count : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseStarBreakdown(text: string | undefined | null): StarBreakdown | undefined {
|
||||||
|
if (!text) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const breakdown: Partial<Omit<StarBreakdown, "basis">> = {};
|
||||||
|
const words: Record<string, keyof Omit<StarBreakdown, "basis">> = {
|
||||||
|
"5": "five",
|
||||||
|
"4": "four",
|
||||||
|
"3": "three",
|
||||||
|
"2": "two",
|
||||||
|
"1": "one"
|
||||||
|
};
|
||||||
|
const percentMatches = [...text.matchAll(/([1-5])\s*star\s*([0-9]{1,3})\s*%/gi)];
|
||||||
|
if (percentMatches.length === 0) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
for (const match of percentMatches) {
|
||||||
|
const key = words[match[1]];
|
||||||
|
if (key) {
|
||||||
|
breakdown[key] = Number(match[2]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
...breakdown,
|
||||||
|
basis: "percent"
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function extractUnitCount(text: string | undefined | null): UnitCountExtraction | undefined {
|
||||||
|
if (!text) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const patterns = [
|
||||||
|
{ pattern: /(\d{1,4})\s*[- ]?(?:count|ct)\b/i, confidence: "high" as const },
|
||||||
|
{ pattern: /\bpack\s+of\s+(\d{1,4})\b/i, confidence: "high" as const },
|
||||||
|
{ pattern: /\b(\d{1,4})\s*[- ]?pack\b/i, confidence: "high" as const },
|
||||||
|
{ pattern: /\bset\s+of\s+(\d{1,4})\b/i, confidence: "medium" as const },
|
||||||
|
{ pattern: /\b(\d{1,4})\s+(?:bulbs?|cables?|pieces?|pcs)\b/i, confidence: "low" as const }
|
||||||
|
];
|
||||||
|
for (const { pattern, confidence } of patterns) {
|
||||||
|
const match = text.match(pattern);
|
||||||
|
if (!match) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const count = Number(match[1]);
|
||||||
|
if (Number.isInteger(count) && count > 0) {
|
||||||
|
return {
|
||||||
|
count,
|
||||||
|
confidence,
|
||||||
|
source: match[0]
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
import type { ParsedNaturalLanguageRequest, ProductFilters } from "./types.js";
|
||||||
|
|
||||||
|
function cleanQuery(text: string): string {
|
||||||
|
return text
|
||||||
|
.replace(/\b(?:that|and|with|have)\b/gi, " ")
|
||||||
|
.replace(/\s+/g, " ")
|
||||||
|
.replace(/\s+(and|or|a)$/i, "")
|
||||||
|
.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
function removeMatched(text: string, match: RegExpMatchArray | null): string {
|
||||||
|
if (!match) {
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
return text.replace(match[0], " ");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseNaturalLanguageRequest(input: string): ParsedNaturalLanguageRequest {
|
||||||
|
let remaining = input.trim();
|
||||||
|
const filters: ProductFilters = {
|
||||||
|
includeKeywords: [],
|
||||||
|
excludeKeywords: []
|
||||||
|
};
|
||||||
|
let limit: number | undefined;
|
||||||
|
|
||||||
|
const limitMatch = remaining.match(/\b(?:return|limit|top)\s+(\d{1,3})\b/i);
|
||||||
|
if (limitMatch) {
|
||||||
|
limit = Number(limitMatch[1]);
|
||||||
|
remaining = removeMatched(remaining, limitMatch);
|
||||||
|
}
|
||||||
|
|
||||||
|
const unitPriceMatch = remaining.match(/\b(?:cost\s+)?(?:less than|under|below)\s+\$([0-9]+(?:\.[0-9]{1,2})?)\s*(?:each|per\b|\/\s*(?:count|unit|item))\b/i);
|
||||||
|
if (unitPriceMatch) {
|
||||||
|
filters.maxUnitPrice = Number(unitPriceMatch[1]);
|
||||||
|
remaining = removeMatched(remaining, unitPriceMatch);
|
||||||
|
}
|
||||||
|
|
||||||
|
const maxPriceMatch = remaining.match(/\b(?:cost\s+)?(?:less than|under|below)\s+\$([0-9]+(?:\.[0-9]{1,2})?)\b/i);
|
||||||
|
if (maxPriceMatch) {
|
||||||
|
filters.maxPrice = Number(maxPriceMatch[1]);
|
||||||
|
remaining = removeMatched(remaining, maxPriceMatch);
|
||||||
|
}
|
||||||
|
|
||||||
|
const exclusiveReviews = remaining.match(/\b(?:over|more than|above)\s+([0-9][0-9,]*)\s*(?:reviews?|ratings?)\b/i);
|
||||||
|
const inclusiveReviews = remaining.match(/\b(?:at least|minimum|min\.?)\s+([0-9][0-9,]*)\s*(?:reviews?|ratings?)\b/i);
|
||||||
|
const reviewMatch = exclusiveReviews ?? inclusiveReviews;
|
||||||
|
if (reviewMatch) {
|
||||||
|
filters.minReviews = Number(reviewMatch[1].replace(/,/g, ""));
|
||||||
|
filters.reviewCountComparison = exclusiveReviews ? "gt" : "gte";
|
||||||
|
remaining = removeMatched(remaining, reviewMatch);
|
||||||
|
}
|
||||||
|
|
||||||
|
const exclusiveRating = remaining.match(/\b(?:a\s+)?(?:(?:review score|rating)\s+of\s+|rating\s+)?(?:more than|over|above|rated above)\s+([0-5](?:\.[0-9])?)\s*(?:stars?)?\b/i);
|
||||||
|
const inclusiveRating = remaining.match(/\b([0-5](?:\.[0-9])?)\s*stars?\s+or\s+better\b/i)
|
||||||
|
?? remaining.match(/\b(?:at least|minimum|min\.?)\s+([0-5](?:\.[0-9])?)\s*(?:stars?|rating)?\b/i);
|
||||||
|
const ratingMatch = exclusiveRating ?? inclusiveRating;
|
||||||
|
if (ratingMatch) {
|
||||||
|
filters.minRating = Number(ratingMatch[1]);
|
||||||
|
filters.ratingComparison = exclusiveRating ? "gt" : "gte";
|
||||||
|
remaining = removeMatched(remaining, ratingMatch);
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
query: cleanQuery(remaining),
|
||||||
|
filters,
|
||||||
|
limit
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
import type { ProductFilters, ProductSearchResult, SearchProductsResponse } from "./types.js";
|
||||||
|
|
||||||
|
export interface ResponseInput {
|
||||||
|
query: string;
|
||||||
|
filters: ProductFilters;
|
||||||
|
limit: number;
|
||||||
|
maxSearchPages: number;
|
||||||
|
results: ProductSearchResult[];
|
||||||
|
filteredOutCount: number;
|
||||||
|
warnings: string[];
|
||||||
|
now?: () => Date;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createResponse(input: ResponseInput): SearchProductsResponse {
|
||||||
|
return {
|
||||||
|
query: input.query,
|
||||||
|
filters: input.filters,
|
||||||
|
limit: input.limit,
|
||||||
|
maxSearchPages: input.maxSearchPages,
|
||||||
|
results: input.results,
|
||||||
|
filteredOutCount: input.filteredOutCount,
|
||||||
|
warnings: input.warnings,
|
||||||
|
source: {
|
||||||
|
site: "amazon.com",
|
||||||
|
scrapedAt: (input.now ?? (() => new Date()))().toISOString(),
|
||||||
|
automation: "web-automation/CloakBrowser"
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatFilters(filters: ProductFilters): string {
|
||||||
|
const parts = [
|
||||||
|
filters.minRating !== undefined ? `rating ${filters.ratingComparison ?? "gte"} ${filters.minRating}` : "",
|
||||||
|
filters.minReviews !== undefined ? `reviews ${filters.reviewCountComparison ?? "gte"} ${filters.minReviews}` : "",
|
||||||
|
filters.maxPrice !== undefined ? `price <= $${filters.maxPrice}` : "",
|
||||||
|
filters.maxUnitPrice !== undefined ? `unit price <= $${filters.maxUnitPrice}` : ""
|
||||||
|
].filter(Boolean);
|
||||||
|
return parts.length > 0 ? parts.join(", ") : "none";
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatProduct(product: ProductSearchResult, index: number): string {
|
||||||
|
const specs = product.specs.slice(0, 3).map((spec) => `${spec.name}: ${spec.value}`).join("; ");
|
||||||
|
const lines = [
|
||||||
|
`${index}. ${product.title}`,
|
||||||
|
` Link: ${product.url}`,
|
||||||
|
` Price: ${product.price?.display ?? "unknown"}${product.unitPrice ? ` (${product.unitPrice.display})` : ""}`,
|
||||||
|
` Rating: ${product.rating ?? "unknown"} stars; reviews: ${product.reviewCount ?? "unknown"}`,
|
||||||
|
` Delivery: ${product.delivery?.display ?? "unknown"}`,
|
||||||
|
specs ? ` Specs: ${specs}` : "",
|
||||||
|
product.bullets[0] ? ` Notes: ${product.bullets.slice(0, 2).join(" ")}` : "",
|
||||||
|
product.missingFields.length > 0 ? ` Missing: ${product.missingFields.join(", ")}` : "",
|
||||||
|
product.isSponsored ? " Sponsored: yes" : ""
|
||||||
|
].filter(Boolean);
|
||||||
|
return lines.join("\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createMarkdownReport(response: SearchProductsResponse): string {
|
||||||
|
const lines = [
|
||||||
|
`# Amazon Shopping Results`,
|
||||||
|
"",
|
||||||
|
`Query: ${response.query}`,
|
||||||
|
`Filters: ${formatFilters(response.filters)}`,
|
||||||
|
`Results returned: ${response.results.length} (filtered out: ${response.filteredOutCount})`,
|
||||||
|
response.warnings.length > 0 ? `Warnings: ${response.warnings.join("; ")}` : "",
|
||||||
|
"",
|
||||||
|
...response.results.map((product, index) => formatProduct(product, index + 1))
|
||||||
|
].filter((line) => line !== "");
|
||||||
|
return `${lines.join("\n")}\n`;
|
||||||
|
}
|
||||||
@@ -0,0 +1,144 @@
|
|||||||
|
import { HTMLElement, parse } from "node-html-parser";
|
||||||
|
|
||||||
|
import { parseMoney, parseRating, parseReviewCount, parseUnitPrice } from "./parsers.js";
|
||||||
|
import type { DeliverySummary, ProductSearchResult, SearchPageExtraction } from "./types.js";
|
||||||
|
|
||||||
|
function textOf(node: HTMLElement | null | undefined): string {
|
||||||
|
return node?.textContent.replace(/\s+/g, " ").trim() ?? "";
|
||||||
|
}
|
||||||
|
|
||||||
|
function attrOf(node: HTMLElement | null | undefined, name: string): string | undefined {
|
||||||
|
return node?.getAttribute(name) ?? undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function absoluteAmazonUrl(href: string | undefined, currentUrl = "https://www.amazon.com/"): string | undefined {
|
||||||
|
if (!href) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
if (href.startsWith("https://www.amazon.com")) {
|
||||||
|
return href;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const parsed = new URL(href, currentUrl);
|
||||||
|
if (parsed.hostname !== "www.amazon.com") {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
return parsed.toString();
|
||||||
|
} catch {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeProductUrl(asin: string, href: string | undefined, currentUrl: string): string {
|
||||||
|
const absolute = absoluteAmazonUrl(href, currentUrl);
|
||||||
|
if (!absolute) {
|
||||||
|
return `https://www.amazon.com/dp/${asin}`;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const url = new URL(absolute);
|
||||||
|
const match = url.pathname.match(/\/(?:dp|gp\/product)\/([A-Z0-9]{8,14})/i);
|
||||||
|
if (match) {
|
||||||
|
return `https://www.amazon.com/dp/${match[1].toUpperCase()}`;
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return `https://www.amazon.com/dp/${asin}`;
|
||||||
|
}
|
||||||
|
return `https://www.amazon.com/dp/${asin}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function detectChallenge(html: string): boolean {
|
||||||
|
return /robot check|enter the characters you see|captcha|automated access|access denied/i.test(html);
|
||||||
|
}
|
||||||
|
|
||||||
|
function deliveryFromText(text: string): DeliverySummary | undefined {
|
||||||
|
const compact = text.replace(/\s+/g, " ").trim();
|
||||||
|
const deliveryMatch = compact.match(/((?:FREE\s+)?delivery[^.]*?(?:Tomorrow|Today|Monday|Tuesday|Wednesday|Thursday|Friday|Saturday|Sunday)?)/i);
|
||||||
|
if (!deliveryMatch) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const display = deliveryMatch[1].trim();
|
||||||
|
return {
|
||||||
|
display,
|
||||||
|
free: /\bfree\b/i.test(display),
|
||||||
|
prime: /\bprime\b/i.test(compact)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function firstText(card: HTMLElement, selectors: string[]): string {
|
||||||
|
for (const selector of selectors) {
|
||||||
|
const value = textOf(card.querySelector(selector));
|
||||||
|
if (value) {
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
|
||||||
|
function firstUnitPriceText(card: HTMLElement): string {
|
||||||
|
for (const node of card.querySelectorAll(".a-color-secondary, .a-size-base, span")) {
|
||||||
|
const value = textOf(node);
|
||||||
|
if (parseUnitPrice(value)) {
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
|
||||||
|
export function extractSearchPage(html: string, currentUrl: string): SearchPageExtraction {
|
||||||
|
if (detectChallenge(html)) {
|
||||||
|
return {
|
||||||
|
status: "challenge",
|
||||||
|
products: [],
|
||||||
|
warnings: ["Amazon returned a challenge or blocked page; stopping without bypass."],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const root = parse(html);
|
||||||
|
const cards = root.querySelectorAll("[data-asin]")
|
||||||
|
.filter((card) => /^[A-Z0-9]{8,14}$/i.test(card.getAttribute("data-asin") ?? ""));
|
||||||
|
const products: ProductSearchResult[] = [];
|
||||||
|
|
||||||
|
for (const card of cards) {
|
||||||
|
const asin = (card.getAttribute("data-asin") ?? "").toUpperCase();
|
||||||
|
const link = card.querySelector("h2 a") ?? card.querySelector("a[href*='/dp/']") ?? card.querySelector("a[href*='/gp/product/']");
|
||||||
|
const title = textOf(link) || firstText(card, ["h2", "[data-cy='title-recipe']"]);
|
||||||
|
if (!title) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const priceText = firstText(card, [".a-price .a-offscreen", ".a-price"]);
|
||||||
|
const allText = textOf(card);
|
||||||
|
const unitPriceText = firstUnitPriceText(card);
|
||||||
|
const ariaText = card.querySelectorAll("[aria-label]")
|
||||||
|
.map((node) => attrOf(node, "aria-label") ?? "")
|
||||||
|
.join(" ");
|
||||||
|
const delivery = deliveryFromText(allText);
|
||||||
|
const product: ProductSearchResult = {
|
||||||
|
asin,
|
||||||
|
title,
|
||||||
|
url: normalizeProductUrl(asin, attrOf(link, "href"), currentUrl),
|
||||||
|
imageUrl: attrOf(card.querySelector("img"), "src"),
|
||||||
|
price: parseMoney(priceText),
|
||||||
|
unitPrice: parseUnitPrice(unitPriceText),
|
||||||
|
rating: parseRating(ariaText || allText),
|
||||||
|
reviewCount: parseReviewCount(ariaText || allText),
|
||||||
|
delivery,
|
||||||
|
specs: [],
|
||||||
|
bullets: [],
|
||||||
|
isSponsored: /\bsponsored\b/i.test(allText),
|
||||||
|
matchedFilters: [],
|
||||||
|
missingFields: [],
|
||||||
|
extractionNotes: []
|
||||||
|
};
|
||||||
|
products.push(product);
|
||||||
|
}
|
||||||
|
|
||||||
|
const nextHref = attrOf(root.querySelector(".s-pagination-next[href]"), "href");
|
||||||
|
const nextPageUrl = absoluteAmazonUrl(nextHref, currentUrl);
|
||||||
|
|
||||||
|
return {
|
||||||
|
status: "ok",
|
||||||
|
products,
|
||||||
|
warnings: [],
|
||||||
|
nextPageUrl: nextPageUrl ?? undefined
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
export interface SearchProductsRequest {
|
||||||
|
query: string;
|
||||||
|
filters: ProductFilters;
|
||||||
|
limit: number;
|
||||||
|
maxSearchPages: number;
|
||||||
|
skipDetails: boolean;
|
||||||
|
dryRun: boolean;
|
||||||
|
output: "json" | "markdown" | "both";
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProductFilters {
|
||||||
|
minRating?: number;
|
||||||
|
ratingComparison?: "gt" | "gte";
|
||||||
|
minReviews?: number;
|
||||||
|
reviewCountComparison?: "gt" | "gte";
|
||||||
|
maxPrice?: number;
|
||||||
|
maxUnitPrice?: number;
|
||||||
|
includeKeywords: string[];
|
||||||
|
excludeKeywords: string[];
|
||||||
|
requirePrime?: boolean;
|
||||||
|
requireFreeDelivery?: boolean;
|
||||||
|
deliveryBy?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProductSearchResult {
|
||||||
|
asin: string;
|
||||||
|
title: string;
|
||||||
|
url: string;
|
||||||
|
imageUrl?: string;
|
||||||
|
price?: MoneyValue;
|
||||||
|
unitPrice?: MoneyValue;
|
||||||
|
rating?: number;
|
||||||
|
reviewCount?: number;
|
||||||
|
starBreakdown?: StarBreakdown;
|
||||||
|
delivery?: DeliverySummary;
|
||||||
|
specs: ProductSpec[];
|
||||||
|
bullets: string[];
|
||||||
|
seller?: string;
|
||||||
|
isSponsored?: boolean;
|
||||||
|
availability?: string;
|
||||||
|
matchedFilters: string[];
|
||||||
|
missingFields: string[];
|
||||||
|
extractionNotes: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface MoneyValue {
|
||||||
|
amount: number;
|
||||||
|
currency: "USD";
|
||||||
|
display: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DeliverySummary {
|
||||||
|
display: string;
|
||||||
|
prime?: boolean;
|
||||||
|
free?: boolean;
|
||||||
|
fastestDate?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface StarBreakdown {
|
||||||
|
five?: number;
|
||||||
|
four?: number;
|
||||||
|
three?: number;
|
||||||
|
two?: number;
|
||||||
|
one?: number;
|
||||||
|
basis: "percent" | "count";
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProductSpec {
|
||||||
|
name: string;
|
||||||
|
value: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SearchProductsResponse {
|
||||||
|
query: string;
|
||||||
|
filters: ProductFilters;
|
||||||
|
limit: number;
|
||||||
|
maxSearchPages: number;
|
||||||
|
results: ProductSearchResult[];
|
||||||
|
filteredOutCount: number;
|
||||||
|
warnings: string[];
|
||||||
|
source: {
|
||||||
|
site: "amazon.com";
|
||||||
|
scrapedAt: string;
|
||||||
|
automation: "web-automation/CloakBrowser";
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ParsedNaturalLanguageRequest {
|
||||||
|
query: string;
|
||||||
|
filters: ProductFilters;
|
||||||
|
limit?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface UnitCountExtraction {
|
||||||
|
count: number;
|
||||||
|
confidence: "high" | "medium" | "low";
|
||||||
|
source: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SearchPageExtraction {
|
||||||
|
status: "ok" | "challenge";
|
||||||
|
products: ProductSearchResult[];
|
||||||
|
warnings: string[];
|
||||||
|
nextPageUrl?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FilteredProducts {
|
||||||
|
results: ProductSearchResult[];
|
||||||
|
filteredOutCount: number;
|
||||||
|
filteredOutReasons: Record<string, string[]>;
|
||||||
|
}
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
import { access } from "node:fs/promises";
|
||||||
|
import { constants } from "node:fs";
|
||||||
|
import { dirname, join, resolve } from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
|
||||||
|
export interface RuntimeResolverOptions {
|
||||||
|
env?: NodeJS.ProcessEnv;
|
||||||
|
homeDir?: string;
|
||||||
|
skillDir?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WebAutomationRuntime {
|
||||||
|
scriptsDir: string;
|
||||||
|
checkInstall: {
|
||||||
|
cwd: string;
|
||||||
|
command: string;
|
||||||
|
args: string[];
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function assertFile(path: string, label: string): Promise<void> {
|
||||||
|
try {
|
||||||
|
await access(path, constants.F_OK);
|
||||||
|
} catch {
|
||||||
|
throw new Error(`web-automation runtime is missing ${label}: ${path}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function assertExecutableOrFile(path: string, label: string): Promise<void> {
|
||||||
|
try {
|
||||||
|
await access(path, constants.X_OK);
|
||||||
|
} catch {
|
||||||
|
await assertFile(path, label);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function defaultSkillDir(): string {
|
||||||
|
return resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function resolveWebAutomationRuntime(options: RuntimeResolverOptions = {}): Promise<WebAutomationRuntime> {
|
||||||
|
const env = options.env ?? process.env;
|
||||||
|
const homeDir = options.homeDir ?? process.env.HOME ?? "";
|
||||||
|
const skillDir = options.skillDir ?? defaultSkillDir();
|
||||||
|
const candidates = [
|
||||||
|
env.AMAZON_SHOPPING_WEB_AUTOMATION_DIR,
|
||||||
|
homeDir ? join(homeDir, ".openclaw", "workspace", "skills", "web-automation", "scripts") : undefined,
|
||||||
|
resolve(skillDir, "..", "web-automation", "scripts")
|
||||||
|
].filter((candidate): candidate is string => Boolean(candidate));
|
||||||
|
|
||||||
|
const errors: string[] = [];
|
||||||
|
for (const scriptsDir of candidates) {
|
||||||
|
try {
|
||||||
|
await assertFile(join(scriptsDir, "check-install.js"), "check-install.js");
|
||||||
|
await assertFile(join(scriptsDir, "package.json"), "package.json");
|
||||||
|
await assertExecutableOrFile(join(scriptsDir, "node_modules", ".bin", "tsx"), "node_modules/.bin/tsx");
|
||||||
|
return {
|
||||||
|
scriptsDir,
|
||||||
|
checkInstall: {
|
||||||
|
cwd: scriptsDir,
|
||||||
|
command: "node",
|
||||||
|
args: ["check-install.js"]
|
||||||
|
}
|
||||||
|
};
|
||||||
|
} catch (error: unknown) {
|
||||||
|
errors.push(error instanceof Error ? error.message : String(error));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error(`Unable to locate usable web-automation runtime.\n${errors.join("\n")}`);
|
||||||
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import { classifyHttpStatus, isPathAllowedByRobots, plannedAmazonPaths } from "../src/browser.js";
|
||||||
|
|
||||||
|
describe("browser compliance helpers", () => {
|
||||||
|
it("plans only search and product-detail paths", () => {
|
||||||
|
assert.deepEqual(plannedAmazonPaths(["B0TEST0001"]), ["/s", "/dp/B0TEST0001", "/gp/product/B0TEST0001"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("honors robots disallow rules for planned paths", () => {
|
||||||
|
const robots = `
|
||||||
|
User-agent: *
|
||||||
|
Disallow: /cart
|
||||||
|
Disallow: /product-reviews
|
||||||
|
Disallow: /dp/private
|
||||||
|
`;
|
||||||
|
|
||||||
|
assert.equal(isPathAllowedByRobots(robots, "*", "/s"), true);
|
||||||
|
assert.equal(isPathAllowedByRobots(robots, "*", "/product-reviews/B0TEST0001"), false);
|
||||||
|
assert.equal(isPathAllowedByRobots(robots, "*", "/dp/private/B0TEST0001"), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not leak disallow rules from other user-agent groups", () => {
|
||||||
|
const robots = `
|
||||||
|
User-agent: specialbot
|
||||||
|
Disallow: /dp
|
||||||
|
|
||||||
|
User-agent: *
|
||||||
|
Disallow: /cart
|
||||||
|
`;
|
||||||
|
|
||||||
|
assert.equal(isPathAllowedByRobots(robots, "*", "/dp/B0TEST0001"), true);
|
||||||
|
assert.equal(isPathAllowedByRobots(robots, "specialbot", "/dp/B0TEST0001"), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("classifies retryable and challenge statuses", () => {
|
||||||
|
assert.equal(classifyHttpStatus(429), "retryable");
|
||||||
|
assert.equal(classifyHttpStatus(503), "retryable");
|
||||||
|
assert.equal(classifyHttpStatus(403), "challenge");
|
||||||
|
assert.equal(classifyHttpStatus(200), "ok");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import { buildSearchUrl, parseCliRequest, runCli } from "../src/cli.js";
|
||||||
|
|
||||||
|
function createOutput() {
|
||||||
|
let stdout = "";
|
||||||
|
let stderr = "";
|
||||||
|
return {
|
||||||
|
stdout: { write: (chunk: string) => { stdout += chunk; return true; } },
|
||||||
|
stderr: { write: (chunk: string) => { stderr += chunk; return true; } },
|
||||||
|
get stdoutText() { return stdout; },
|
||||||
|
get stderrText() { return stderr; }
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("amazon-shopping CLI", () => {
|
||||||
|
it("prints help", async () => {
|
||||||
|
const output = createOutput();
|
||||||
|
const code = await runCli(["--help"], output);
|
||||||
|
|
||||||
|
assert.equal(code, 0);
|
||||||
|
assert.match(output.stdoutText, /scripts\/search-products/);
|
||||||
|
assert.match(output.stdoutText, /--dry-run/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("defaults to 15 results and two search pages", () => {
|
||||||
|
const request = parseCliRequest(["usb c cable"]);
|
||||||
|
|
||||||
|
assert.equal(request.query, "usb c cable");
|
||||||
|
assert.equal(request.limit, 15);
|
||||||
|
assert.equal(request.maxSearchPages, 2);
|
||||||
|
assert.equal(request.output, "json");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("maps kebab-case CLI filters into the request contract", () => {
|
||||||
|
const request = parseCliRequest([
|
||||||
|
"--query",
|
||||||
|
"100w led bulbs",
|
||||||
|
"--min-rating",
|
||||||
|
"4.5",
|
||||||
|
"--min-reviews",
|
||||||
|
"200",
|
||||||
|
"--max-unit-price",
|
||||||
|
"4",
|
||||||
|
"--max-search-pages",
|
||||||
|
"3",
|
||||||
|
"--skip-details",
|
||||||
|
"--dry-run"
|
||||||
|
]);
|
||||||
|
|
||||||
|
assert.equal(request.query, "100w led bulbs");
|
||||||
|
assert.equal(request.filters.minRating, 4.5);
|
||||||
|
assert.equal(request.filters.minReviews, 200);
|
||||||
|
assert.equal(request.filters.maxUnitPrice, 4);
|
||||||
|
assert.equal(request.maxSearchPages, 3);
|
||||||
|
assert.equal(request.skipDetails, true);
|
||||||
|
assert.equal(request.dryRun, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("maps output modes", () => {
|
||||||
|
assert.equal(parseCliRequest(["usb c cable", "--json"]).output, "json");
|
||||||
|
assert.equal(parseCliRequest(["usb c cable", "--markdown"]).output, "markdown");
|
||||||
|
assert.equal(parseCliRequest(["usb c cable", "--json", "--markdown"]).output, "both");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts max as a natural agent alias for limit", () => {
|
||||||
|
const request = parseCliRequest(["100w led bulbs", "--max", "5"]);
|
||||||
|
|
||||||
|
assert.equal(request.limit, 5);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("normalizes natural-language filters for the target request", () => {
|
||||||
|
const request = parseCliRequest([
|
||||||
|
"100w led bulbs that cost less than $4 each and have over 200 reviews with a review score of more than 4.5 stars",
|
||||||
|
"--dry-run"
|
||||||
|
]);
|
||||||
|
|
||||||
|
assert.equal(request.query, "100w led bulbs");
|
||||||
|
assert.equal(request.filters.maxUnitPrice, 4);
|
||||||
|
assert.equal(request.filters.minReviews, 200);
|
||||||
|
assert.equal(request.filters.reviewCountComparison, "gt");
|
||||||
|
assert.equal(request.filters.minRating, 4.5);
|
||||||
|
assert.equal(request.filters.ratingComparison, "gt");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects limits below one", () => {
|
||||||
|
assert.throws(
|
||||||
|
() => parseCliRequest(["usb c cable", "--limit", "0"]),
|
||||||
|
/limit must be an integer greater than 0/
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects unsafe large limits unless explicitly allowed", () => {
|
||||||
|
assert.throws(
|
||||||
|
() => parseCliRequest(["usb c cable", "--limit", "31"]),
|
||||||
|
/require --allow-large-limit/
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects search page caps above five", () => {
|
||||||
|
assert.throws(
|
||||||
|
() => parseCliRequest(["usb c cable", "--max-search-pages", "6"]),
|
||||||
|
/max-search-pages must be 5 or less/
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("builds the Amazon search URL without live network access", () => {
|
||||||
|
assert.equal(
|
||||||
|
buildSearchUrl("100w led bulbs"),
|
||||||
|
"https://www.amazon.com/s?k=100w%20led%20bulbs"
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import { extractDetailPage } from "../src/detail-page.js";
|
||||||
|
|
||||||
|
const fixturePath = join(import.meta.dirname, "fixtures", "product-detail.html");
|
||||||
|
|
||||||
|
describe("extractDetailPage", () => {
|
||||||
|
it("extracts visible product detail fields from sanitized HTML", async () => {
|
||||||
|
const html = await readFile(fixturePath, "utf8");
|
||||||
|
const details = extractDetailPage(html, {
|
||||||
|
asin: "B0TESTLED1",
|
||||||
|
title: "Search title",
|
||||||
|
url: "https://www.amazon.com/dp/B0TESTLED1",
|
||||||
|
specs: [],
|
||||||
|
bullets: [],
|
||||||
|
matchedFilters: [],
|
||||||
|
missingFields: [],
|
||||||
|
extractionNotes: []
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(details.title, "Bright Daylight LED Bulbs 100W Equivalent, 50 Count");
|
||||||
|
assert.equal(details.price?.amount, 18.99);
|
||||||
|
assert.equal(details.delivery?.free, true);
|
||||||
|
assert.equal(details.availability, "In Stock");
|
||||||
|
assert.equal(details.seller, "Ships from Amazon.com");
|
||||||
|
assert.equal(details.bullets.length, 2);
|
||||||
|
assert.deepEqual(details.specs[0], { name: "Brand", value: "BrightCo" });
|
||||||
|
assert.equal(details.rating, 4.6);
|
||||||
|
assert.equal(details.reviewCount, 1234);
|
||||||
|
assert.equal(details.starBreakdown?.five, 72);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("records missing detail-only fields", () => {
|
||||||
|
const details = extractDetailPage("<html><body><h1 id=\"productTitle\">Sparse Product</h1></body></html>", {
|
||||||
|
asin: "B0SPARSE01",
|
||||||
|
title: "Sparse",
|
||||||
|
url: "https://www.amazon.com/dp/B0SPARSE01",
|
||||||
|
specs: [],
|
||||||
|
bullets: [],
|
||||||
|
matchedFilters: [],
|
||||||
|
missingFields: [],
|
||||||
|
extractionNotes: []
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(details.price, undefined);
|
||||||
|
assert.ok(details.missingFields.includes("price"));
|
||||||
|
assert.ok(details.missingFields.includes("starBreakdown"));
|
||||||
|
});
|
||||||
|
|
||||||
|
it("drops script-like spec rows and trims availability metadata", () => {
|
||||||
|
const details = extractDetailPage(`
|
||||||
|
<h1 id="productTitle">Messy Product</h1>
|
||||||
|
<div id="availability">In Stock {"merchantId":"secretish"}</div>
|
||||||
|
<table>
|
||||||
|
<tr><td>Special Feature</td><td>(function(P) { tracking(); }) Real feature text</td></tr>
|
||||||
|
<tr><td>A19 Add to Cart logShoppableMetrics("x", true)</td><td>Buying Options</td></tr>
|
||||||
|
<tr><td>Wattage</td><td>15 watts</td></tr>
|
||||||
|
<tr><td>Customer Reviews</td><td>4.7 out of 5 stars tracking payload</td></tr>
|
||||||
|
</table>
|
||||||
|
`, {
|
||||||
|
asin: "B0MESSY001",
|
||||||
|
title: "Messy",
|
||||||
|
url: "https://www.amazon.com/dp/B0MESSY001",
|
||||||
|
specs: [],
|
||||||
|
bullets: [],
|
||||||
|
matchedFilters: [],
|
||||||
|
missingFields: [],
|
||||||
|
extractionNotes: []
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(details.availability, "In Stock");
|
||||||
|
assert.deepEqual(details.specs, [{ name: "Wattage", value: "15 watts" }]);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import { applyFiltersAndLimit } from "../src/filters.js";
|
||||||
|
import type { ProductSearchResult } from "../src/types.js";
|
||||||
|
|
||||||
|
function product(overrides: Partial<ProductSearchResult>): ProductSearchResult {
|
||||||
|
return {
|
||||||
|
asin: "B0BASE0001",
|
||||||
|
title: "Base Product",
|
||||||
|
url: "https://www.amazon.com/dp/B0BASE0001",
|
||||||
|
specs: [],
|
||||||
|
bullets: [],
|
||||||
|
matchedFilters: [],
|
||||||
|
missingFields: [],
|
||||||
|
extractionNotes: [],
|
||||||
|
...overrides
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("applyFiltersAndLimit", () => {
|
||||||
|
it("applies strict rating, review, and unit-price filters", () => {
|
||||||
|
const result = applyFiltersAndLimit([
|
||||||
|
product({
|
||||||
|
asin: "B0PASS0001",
|
||||||
|
rating: 4.6,
|
||||||
|
reviewCount: 201,
|
||||||
|
unitPrice: { amount: 3.99, currency: "USD", display: "$3.99/Count" }
|
||||||
|
}),
|
||||||
|
product({
|
||||||
|
asin: "B0FAIL0001",
|
||||||
|
rating: 4.5,
|
||||||
|
reviewCount: 200,
|
||||||
|
unitPrice: { amount: 3.99, currency: "USD", display: "$3.99/Count" }
|
||||||
|
}),
|
||||||
|
product({
|
||||||
|
asin: "B0UNKNOWN1",
|
||||||
|
rating: 4.7,
|
||||||
|
reviewCount: 300
|
||||||
|
})
|
||||||
|
], {
|
||||||
|
includeKeywords: [],
|
||||||
|
excludeKeywords: [],
|
||||||
|
minRating: 4.5,
|
||||||
|
ratingComparison: "gt",
|
||||||
|
minReviews: 200,
|
||||||
|
reviewCountComparison: "gt",
|
||||||
|
maxUnitPrice: 4
|
||||||
|
}, 10);
|
||||||
|
|
||||||
|
assert.deepEqual(result.results.map((item) => item.asin), ["B0PASS0001"]);
|
||||||
|
assert.equal(result.filteredOutCount, 2);
|
||||||
|
assert.match(result.filteredOutReasons["B0UNKNOWN1"]?.join(" ") ?? "", /unit price unknown/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("sorts by rating, reviews, then price", () => {
|
||||||
|
const result = applyFiltersAndLimit([
|
||||||
|
product({ asin: "B0LOWPRICE", rating: 4.7, reviewCount: 1000, price: { amount: 15, currency: "USD", display: "$15.00" } }),
|
||||||
|
product({ asin: "B0HIGHRATE", rating: 4.9, reviewCount: 100, price: { amount: 40, currency: "USD", display: "$40.00" } }),
|
||||||
|
product({ asin: "B0MOREREV", rating: 4.7, reviewCount: 2000, price: { amount: 20, currency: "USD", display: "$20.00" } })
|
||||||
|
], { includeKeywords: [], excludeKeywords: [] }, 2);
|
||||||
|
|
||||||
|
assert.deepEqual(result.results.map((item) => item.asin), ["B0HIGHRATE", "B0MOREREV"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("deduplicates repeated ASINs before limiting", () => {
|
||||||
|
const result = applyFiltersAndLimit([
|
||||||
|
product({ asin: "B0DUP0001", rating: 4.8, reviewCount: 1000 }),
|
||||||
|
product({ asin: "B0DUP0001", rating: 4.8, reviewCount: 1000 }),
|
||||||
|
product({ asin: "B0UNIQUE1", rating: 4.7, reviewCount: 900 })
|
||||||
|
], { includeKeywords: [], excludeKeywords: [] }, 10);
|
||||||
|
|
||||||
|
assert.deepEqual(result.results.map((item) => item.asin), ["B0DUP0001", "B0UNIQUE1"]);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
# Fixtures
|
||||||
|
|
||||||
|
Fixtures in this directory are hand-crafted sanitized HTML snippets. They are not live Amazon snapshots and contain no cookies, account details, delivery location, scripts, tracking identifiers, or browser profile data.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
<!-- Hand-crafted sanitized fixture. Not a live Amazon snapshot. -->
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<h1 id="productTitle">Bright Daylight LED Bulbs 100W Equivalent, 50 Count</h1>
|
||||||
|
<span id="productTitle_feature_div"></span>
|
||||||
|
<span class="a-price"><span class="a-offscreen">$18.99</span></span>
|
||||||
|
<div id="mir-layout-DELIVERY_BLOCK-slot-PRIMARY_DELIVERY_MESSAGE_LARGE">FREE delivery Tomorrow</div>
|
||||||
|
<div id="availability">In Stock</div>
|
||||||
|
<div id="merchant-info">Ships from Amazon.com</div>
|
||||||
|
<div id="feature-bullets">
|
||||||
|
<ul>
|
||||||
|
<li><span>Energy efficient 100W equivalent bulbs.</span></li>
|
||||||
|
<li><span>Daylight color temperature for kitchens and garages.</span></li>
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
<table id="productOverview_feature_div">
|
||||||
|
<tr><td>Brand</td><td>BrightCo</td></tr>
|
||||||
|
<tr><td>Light Type</td><td>LED</td></tr>
|
||||||
|
</table>
|
||||||
|
<span id="acrPopover" title="4.6 out of 5 stars"></span>
|
||||||
|
<span id="acrCustomerReviewText">1,234 ratings</span>
|
||||||
|
<table id="histogramTable">
|
||||||
|
<tr><td>5 star</td><td>72%</td></tr>
|
||||||
|
<tr><td>4 star</td><td>15%</td></tr>
|
||||||
|
<tr><td>3 star</td><td>7%</td></tr>
|
||||||
|
<tr><td>2 star</td><td>3%</td></tr>
|
||||||
|
<tr><td>1 star</td><td>3%</td></tr>
|
||||||
|
</table>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
<!-- Hand-crafted sanitized fixture. Not a live Amazon snapshot. -->
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<div data-component-type="s-search-result" data-asin="B0TESTLED1">
|
||||||
|
<h2><a class="a-link-normal s-line-clamp-2" href="/Bright-Daylight-Equivalent/dp/B0TESTLED1/ref=sr_1_1">Bright Daylight 100W Equivalent LED Bulbs, 50 Count</a></h2>
|
||||||
|
<span class="a-price"><span class="a-offscreen">$18.99</span></span>
|
||||||
|
<span class="a-size-base a-color-secondary">$0.38/Count</span>
|
||||||
|
<span aria-label="4.6 out of 5 stars"></span>
|
||||||
|
<a aria-label="1,234 ratings"></a>
|
||||||
|
<div class="a-row a-size-base a-color-secondary">FREE delivery Tomorrow</div>
|
||||||
|
<img class="s-image" src="https://m.media-amazon.com/images/I/test-led.jpg" />
|
||||||
|
</div>
|
||||||
|
<div data-component-type="s-search-result" data-asin="B0TESTLED2">
|
||||||
|
<span>Sponsored</span>
|
||||||
|
<h2><a href="https://www.amazon.com/gp/product/B0TESTLED2">Value LED Bulbs Soft White, Pack of 24</a></h2>
|
||||||
|
<span class="a-price"><span class="a-offscreen">$21.99</span></span>
|
||||||
|
<span aria-label="4.3 out of 5 stars"></span>
|
||||||
|
<a aria-label="543 ratings"></a>
|
||||||
|
<div>Delivery Friday</div>
|
||||||
|
</div>
|
||||||
|
<a class="s-pagination-next" href="/s?k=led+bulbs&page=2">Next</a>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import {
|
||||||
|
extractUnitCount,
|
||||||
|
parseMoney,
|
||||||
|
parseRating,
|
||||||
|
parseReviewCount,
|
||||||
|
parseStarBreakdown,
|
||||||
|
parseUnitPrice
|
||||||
|
} from "../src/parsers.js";
|
||||||
|
|
||||||
|
describe("parsers", () => {
|
||||||
|
it("parses USD money", () => {
|
||||||
|
assert.deepEqual(parseMoney("$19.99"), { amount: 19.99, currency: "USD", display: "$19.99" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses rating text", () => {
|
||||||
|
assert.equal(parseRating("4.6 out of 5 stars"), 4.6);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses review count text", () => {
|
||||||
|
assert.equal(parseReviewCount("1,234 ratings"), 1234);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses visible star histogram percentages", () => {
|
||||||
|
assert.deepEqual(parseStarBreakdown("5 star 72% 4 star 15% 3 star 7% 2 star 3% 1 star 3%"), {
|
||||||
|
five: 72,
|
||||||
|
four: 15,
|
||||||
|
three: 7,
|
||||||
|
two: 3,
|
||||||
|
one: 3,
|
||||||
|
basis: "percent"
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("extracts high-confidence unit counts", () => {
|
||||||
|
assert.deepEqual(extractUnitCount("LED bulbs, 100 Count, daylight"), {
|
||||||
|
count: 100,
|
||||||
|
confidence: "high",
|
||||||
|
source: "100 Count"
|
||||||
|
});
|
||||||
|
assert.deepEqual(extractUnitCount("Pack of 6 USB-C cables"), {
|
||||||
|
count: 6,
|
||||||
|
confidence: "high",
|
||||||
|
source: "Pack of 6"
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("distinguishes lower-confidence unit count phrases", () => {
|
||||||
|
assert.deepEqual(extractUnitCount("Set of 8 replacement filters"), {
|
||||||
|
count: 8,
|
||||||
|
confidence: "medium",
|
||||||
|
source: "Set of 8"
|
||||||
|
});
|
||||||
|
assert.deepEqual(extractUnitCount("6 bulbs soft white"), {
|
||||||
|
count: 6,
|
||||||
|
confidence: "low",
|
||||||
|
source: "6 bulbs"
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses visible unit prices", () => {
|
||||||
|
assert.deepEqual(parseUnitPrice("$0.33/Count"), {
|
||||||
|
amount: 0.33,
|
||||||
|
currency: "USD",
|
||||||
|
display: "$0.33/Count"
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("prefers the unit price when product price appears first", () => {
|
||||||
|
assert.deepEqual(parseUnitPrice("$9.99 ($5.00$5.00/count)"), {
|
||||||
|
amount: 5,
|
||||||
|
currency: "USD",
|
||||||
|
display: "$5.00/count"
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses whole-dollar and one-decimal prices", () => {
|
||||||
|
assert.deepEqual(parseMoney("$20"), { amount: 20, currency: "USD", display: "$20" });
|
||||||
|
assert.deepEqual(parseMoney("$19.9"), { amount: 19.9, currency: "USD", display: "$19.9" });
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import { parseNaturalLanguageRequest } from "../src/query-parser.js";
|
||||||
|
|
||||||
|
describe("parseNaturalLanguageRequest", () => {
|
||||||
|
it("extracts the target LED bulb filters from natural language", () => {
|
||||||
|
const parsed = parseNaturalLanguageRequest(
|
||||||
|
"100w led bulbs that cost less than $4 each and have over 200 reviews with a review score of more than 4.5 stars"
|
||||||
|
);
|
||||||
|
|
||||||
|
assert.equal(parsed.query, "100w led bulbs");
|
||||||
|
assert.equal(parsed.filters.maxUnitPrice, 4);
|
||||||
|
assert.equal(parsed.filters.minReviews, 200);
|
||||||
|
assert.equal(parsed.filters.reviewCountComparison, "gt");
|
||||||
|
assert.equal(parsed.filters.minRating, 4.5);
|
||||||
|
assert.equal(parsed.filters.ratingComparison, "gt");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("distinguishes inclusive review and rating phrasing", () => {
|
||||||
|
const parsed = parseNaturalLanguageRequest("usb c charger at least 500 reviews and 4.3 stars or better");
|
||||||
|
|
||||||
|
assert.equal(parsed.query, "usb c charger");
|
||||||
|
assert.equal(parsed.filters.minReviews, 500);
|
||||||
|
assert.equal(parsed.filters.reviewCountComparison, "gte");
|
||||||
|
assert.equal(parsed.filters.minRating, 4.3);
|
||||||
|
assert.equal(parsed.filters.ratingComparison, "gte");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("cleans rating filter phrases from search query text", () => {
|
||||||
|
const parsed = parseNaturalLanguageRequest("usb c cable with over 1000 reviews and rating over 4 stars");
|
||||||
|
|
||||||
|
assert.equal(parsed.query, "usb c cable");
|
||||||
|
assert.equal(parsed.filters.minReviews, 1000);
|
||||||
|
assert.equal(parsed.filters.minRating, 4);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("extracts limit and max product price phrases", () => {
|
||||||
|
const parsed = parseNaturalLanguageRequest("return 5 wireless mouse under $30");
|
||||||
|
|
||||||
|
assert.equal(parsed.query, "wireless mouse");
|
||||||
|
assert.equal(parsed.limit, 5);
|
||||||
|
assert.equal(parsed.filters.maxPrice, 30);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import { createMarkdownReport, createResponse } from "../src/report.js";
|
||||||
|
|
||||||
|
describe("report", () => {
|
||||||
|
it("creates a structured JSON response", () => {
|
||||||
|
const response = createResponse({
|
||||||
|
query: "usb c cable",
|
||||||
|
filters: { includeKeywords: [], excludeKeywords: [], minReviews: 1000 },
|
||||||
|
limit: 1,
|
||||||
|
maxSearchPages: 2,
|
||||||
|
results: [],
|
||||||
|
filteredOutCount: 4,
|
||||||
|
warnings: ["partial extraction"],
|
||||||
|
now: () => new Date("2026-04-15T00:00:00.000Z")
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(response.source.site, "amazon.com");
|
||||||
|
assert.equal(response.filteredOutCount, 4);
|
||||||
|
assert.equal(response.source.scrapedAt, "2026-04-15T00:00:00.000Z");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("creates concise markdown with product details and warnings", () => {
|
||||||
|
const markdown = createMarkdownReport(createResponse({
|
||||||
|
query: "usb c cable",
|
||||||
|
filters: { includeKeywords: [], excludeKeywords: [] },
|
||||||
|
limit: 1,
|
||||||
|
maxSearchPages: 2,
|
||||||
|
filteredOutCount: 0,
|
||||||
|
warnings: ["price missing for one item"],
|
||||||
|
now: () => new Date("2026-04-15T00:00:00.000Z"),
|
||||||
|
results: [{
|
||||||
|
asin: "B0TEST0001",
|
||||||
|
title: "USB-C Cable",
|
||||||
|
url: "https://www.amazon.com/dp/B0TEST0001",
|
||||||
|
price: { amount: 9.99, currency: "USD", display: "$9.99" },
|
||||||
|
rating: 4.7,
|
||||||
|
reviewCount: 1234,
|
||||||
|
delivery: { display: "FREE delivery Tomorrow", free: true },
|
||||||
|
specs: [{ name: "Length", value: "6 ft" }],
|
||||||
|
bullets: ["Braided cable"],
|
||||||
|
matchedFilters: [],
|
||||||
|
missingFields: ["starBreakdown"],
|
||||||
|
extractionNotes: []
|
||||||
|
}]
|
||||||
|
}));
|
||||||
|
|
||||||
|
assert.match(markdown, /USB-C Cable/);
|
||||||
|
assert.match(markdown, /\$9\.99/);
|
||||||
|
assert.match(markdown, /4\.7 stars/);
|
||||||
|
assert.match(markdown, /price missing/);
|
||||||
|
assert.match(markdown, /https:\/\/www\.amazon\.com\/dp\/B0TEST0001/);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import { extractSearchPage } from "../src/search-page.js";
|
||||||
|
|
||||||
|
const fixturePath = join(import.meta.dirname, "fixtures", "search-results.html");
|
||||||
|
|
||||||
|
describe("extractSearchPage", () => {
|
||||||
|
it("extracts normalized product candidates from sanitized search HTML", async () => {
|
||||||
|
const html = await readFile(fixturePath, "utf8");
|
||||||
|
const extracted = extractSearchPage(html, "https://www.amazon.com/s?k=led+bulbs");
|
||||||
|
|
||||||
|
assert.equal(extracted.status, "ok");
|
||||||
|
assert.equal(extracted.products.length, 2);
|
||||||
|
assert.equal(extracted.products[0]?.asin, "B0TESTLED1");
|
||||||
|
assert.equal(extracted.products[0]?.url, "https://www.amazon.com/dp/B0TESTLED1");
|
||||||
|
assert.equal(extracted.products[0]?.price?.amount, 18.99);
|
||||||
|
assert.equal(extracted.products[0]?.unitPrice?.amount, 0.38);
|
||||||
|
assert.equal(extracted.products[0]?.rating, 4.6);
|
||||||
|
assert.equal(extracted.products[0]?.reviewCount, 1234);
|
||||||
|
assert.equal(extracted.products[0]?.delivery?.free, true);
|
||||||
|
assert.equal(extracted.products[0]?.isSponsored, false);
|
||||||
|
assert.equal(extracted.products[1]?.isSponsored, true);
|
||||||
|
assert.equal(extracted.nextPageUrl, "https://www.amazon.com/s?k=led+bulbs&page=2");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("detects Amazon challenge pages", () => {
|
||||||
|
const extracted = extractSearchPage("<html><title>Robot Check</title><body>Enter the characters you see below</body></html>", "https://www.amazon.com/s?k=x");
|
||||||
|
|
||||||
|
assert.equal(extracted.status, "challenge");
|
||||||
|
assert.match(extracted.warnings[0] ?? "", /challenge/i);
|
||||||
|
assert.equal(extracted.products.length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns ok with no products for empty or cardless pages", () => {
|
||||||
|
const extracted = extractSearchPage("<html><body>No results</body></html>", "https://www.amazon.com/s?k=x");
|
||||||
|
|
||||||
|
assert.equal(extracted.status, "ok");
|
||||||
|
assert.deepEqual(extracted.products, []);
|
||||||
|
assert.equal(extracted.nextPageUrl, undefined);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("skips malformed ASINs and cards without titles", () => {
|
||||||
|
const extracted = extractSearchPage(`
|
||||||
|
<div data-asin="bad"><h2><a href="/dp/bad">Bad ASIN</a></h2></div>
|
||||||
|
<div data-asin="B0VALID1234"></div>
|
||||||
|
`, "https://www.amazon.com/s?k=x");
|
||||||
|
|
||||||
|
assert.equal(extracted.status, "ok");
|
||||||
|
assert.equal(extracted.products.length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps candidates with missing price and records missing price later", () => {
|
||||||
|
const extracted = extractSearchPage(`
|
||||||
|
<div data-asin="B0NOPRICE1">
|
||||||
|
<h2><a href="/dp/B0NOPRICE1">No Price Product</a></h2>
|
||||||
|
</div>
|
||||||
|
`, "https://www.amazon.com/s?k=x");
|
||||||
|
|
||||||
|
assert.equal(extracted.products.length, 1);
|
||||||
|
assert.equal(extracted.products[0]?.price, undefined);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { mkdtemp, mkdir, writeFile } from "node:fs/promises";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { describe, it } from "node:test";
|
||||||
|
|
||||||
|
import { resolveWebAutomationRuntime } from "../src/web-automation-runtime.js";
|
||||||
|
|
||||||
|
async function createRuntime() {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), "amazon-shopping-runtime-"));
|
||||||
|
await writeFile(join(dir, "check-install.js"), "console.log('ok');\n");
|
||||||
|
await writeFile(join(dir, "package.json"), "{\"type\":\"module\"}\n");
|
||||||
|
await mkdir(join(dir, "node_modules", ".bin"), { recursive: true });
|
||||||
|
await writeFile(join(dir, "node_modules", ".bin", "tsx"), "#!/usr/bin/env node\n");
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("resolveWebAutomationRuntime", () => {
|
||||||
|
it("uses AMAZON_SHOPPING_WEB_AUTOMATION_DIR first", async () => {
|
||||||
|
const runtimeDir = await createRuntime();
|
||||||
|
const resolved = await resolveWebAutomationRuntime({
|
||||||
|
env: { AMAZON_SHOPPING_WEB_AUTOMATION_DIR: runtimeDir },
|
||||||
|
homeDir: "/missing-home",
|
||||||
|
skillDir: "/missing-skill"
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(resolved.scriptsDir, runtimeDir);
|
||||||
|
assert.deepEqual(resolved.checkInstall, {
|
||||||
|
cwd: runtimeDir,
|
||||||
|
command: "node",
|
||||||
|
args: ["check-install.js"]
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns a clear error when required files are missing", async () => {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), "amazon-shopping-runtime-missing-"));
|
||||||
|
await assert.rejects(
|
||||||
|
() => resolveWebAutomationRuntime({
|
||||||
|
env: { AMAZON_SHOPPING_WEB_AUTOMATION_DIR: dir },
|
||||||
|
homeDir: "/missing-home",
|
||||||
|
skillDir: "/missing-skill"
|
||||||
|
}),
|
||||||
|
/check-install.js/
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022",
|
||||||
|
"module": "NodeNext",
|
||||||
|
"moduleResolution": "NodeNext",
|
||||||
|
"strict": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"types": ["node"]
|
||||||
|
},
|
||||||
|
"include": ["src/**/*.ts", "tests/**/*.ts"]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user