Skill: web-scraper
When to Use
Use this skill when the user asks to:
- Scrape or fetch content from a URL
- Extract text from a webpage
- Read an article from a link
- Get the content of a website
- Download and convert a webpage to text/markdown
- Summarize a web page
Input Parameters
| Parameter | Required | Description | Example |
|---|---|---|---|
url | Yes | URL to scrape | https://example.com/article |
format | No | Output format: markdown (default), text, html | markdown |
output | No | Save output to file path | article.md |
Procedure
-
Get the URL from the user's request
-
Run the bundled script:
# Fetch and extract as markdown (default) python3 skills/web-scraper/scripts/scrape.py "https://example.com/article" # Extract as plain text python3 skills/web-scraper/scripts/scrape.py "https://example.com" --format text # Save to file python3 skills/web-scraper/scripts/scrape.py "https://example.com" --output article.md -
The script auto-installs
trafilaturaandrequestsif needed -
Present the extracted content to the user
Bundled Scripts
| Script | Type | Description |
|---|---|---|
scripts/scrape.py | Python | Fetch URL and extract clean content |
Script Usage
# Extract article content as markdown
python3 scripts/scrape.py "https://example.com/blog-post"
# Extract as plain text
python3 scripts/scrape.py "https://example.com" --format text
# Keep raw HTML
python3 scripts/scrape.py "https://example.com" --format html
# Save to file
python3 scripts/scrape.py "https://example.com" --output page.md
# Include metadata (title, author, date)
python3 scripts/scrape.py "https://example.com/article" --metadata
Example
scrape this URL: https://example.com/article
extract the text from this webpage
fetch the content of this link and save it
read this article: https://blog.example.com/post-1
get the content from this website