Browser Content Capture

Capture web content that traditional scrapers cannot access using agent-browser CLI.

Overview

This skill enables content extraction from sources that require browser-level access:

• JavaScript-rendered SPAs (React, Vue, Angular apps)

• Login-protected documentation (private wikis, gated content)

• Dynamic content (infinite scroll, lazy loading, client-side routing)

• Multi-page site crawls (documentation trees, tutorial series)

When to Use

Use when:

• WebFetch returns empty or partial content

• Page requires JavaScript execution to render

• Content is behind authentication

• Need to navigate multi-page structures

• Extracting from client-side routed apps

Do NOT use when:

• Static HTML pages (use WebFetch

• faster)

• Public API endpoints (use direct HTTP calls)

• Simple RSS/Atom feeds

Quick Start

Basic Capture Pattern

1. Navigate to URL

agent-browser open https://docs.example.com

2. Wait for content to render

agent-browser wait --load networkidle

3. Get interactive snapshot

agent-browser snapshot -i

4. Extract text content

agent-browser get text body

5. Take screenshot

agent-browser screenshot /tmp/capture.png

6. Close when done

agent-browser close

agent-browser Commands Reference

Command Purpose When to Use

open <url>

Go to URL First step of any capture

snapshot -i

Get interactive element tree Understanding page structure

eval "<script>"

Run custom JS Extract specific content

click @e#

Click elements Navigate menus, pagination

fill @e# "value"

Fill inputs Authentication flows

wait @e#

Wait for element Dynamic content loading

screenshot <path>

Capture image Visual verification

console

Read JS console Debug extraction issues

network requests

Monitor XHR/fetch Find API endpoints

Quick reference: See references/agent-browser-commands.md or run agent-browser --help

Capture Patterns

Pattern 1: SPA Content Extraction

For React/Vue/Angular apps where content renders client-side:

Navigate and wait for hydration

agent-browser open https://react-docs.example.com agent-browser wait --load networkidle

Get snapshot to identify content element

agent-browser snapshot -i

Extract after framework mounts (use ref from snapshot)

agent-browser get text @e5 # Main content area

Or use eval for custom extraction

agent-browser eval "document.querySelector('article').innerText"

Details: See references/spa-extraction.md

Pattern 2: Authentication Flow

For login-protected content:

Navigate to login

agent-browser open https://docs.example.com/login agent-browser snapshot -i

Fill credentials (refs from snapshot)

agent-browser fill @e1 "user@example.com" # Email field agent-browser fill @e2 "password123" # Password field

Click submit and wait for redirect

agent-browser click @e3 agent-browser wait --url "**/dashboard"

Save authenticated state for reuse

agent-browser state save /tmp/auth-state.json

Now navigate to protected content

agent-browser open https://docs.example.com/private-docs

Details: See references/auth-handling.md

Pattern 3: Multi-Page Crawl

For documentation with navigation trees:

Get all page links from sidebar

agent-browser open https://docs.example.com agent-browser snapshot -i

Extract links via eval

LINKS=$(agent-browser eval "JSON.stringify(Array.from(document.querySelectorAll('nav a')).map(a => a.href))")

Iterate and capture each page

for link in $(echo "$LINKS" | jq -r '.[]'); do agent-browser open "$link" agent-browser wait --load networkidle agent-browser get text body > "/tmp/content-$(basename $link).txt" done

Details: See references/multi-page-crawl.md

Session Management

Save and Reuse Authentication

Login once and save state

agent-browser open https://app.example.com/login agent-browser snapshot -i agent-browser fill @e1 "$USERNAME" agent-browser fill @e2 "$PASSWORD" agent-browser click @e3 agent-browser wait --url "**/dashboard" agent-browser state save /tmp/app-auth.json

Later: restore state

agent-browser state load /tmp/app-auth.json agent-browser open https://app.example.com/protected-content

Parallel Sessions

Run isolated sessions for different tasks

agent-browser --session scrape1 open https://site1.com agent-browser --session scrape2 open https://site2.com

Extract from each

agent-browser --session scrape1 get text body > site1.txt agent-browser --session scrape2 get text body > site2.txt

Fallback Strategy

Use this decision tree for content capture:

User requests content from URL │ ▼ ┌─────────────┐ │ Try WebFetch│ ← Fast, no browser needed └─────────────┘ │ Content OK? ──Yes──► Done │ No (empty/partial) │ ▼ ┌──────────────────┐ │ Use agent-browser│ └──────────────────┘ │ ├─ Known SPA (react, vue, angular) ──► wait --load networkidle ├─ Requires login ──► Authentication flow with state save └─ Dynamic content ──► wait @element or wait --text

Best Practices

1. Minimize Browser Usage

• Always try WebFetch first (10x faster, no browser overhead)

• Cache extracted content to avoid re-scraping

• Use get text @e# to extract only needed content

2. Handle Dynamic Content

• Always use wait after navigation

• Use wait --load networkidle for heavy SPAs

• Use wait --text "Expected" for specific content

3. Respect Rate Limits

• Add delays between page navigations

• Don't crawl faster than a human would browse

• Honor robots.txt and terms of service

4. Clean Extracted Content

• Use targeted refs from snapshot to extract main content

• Use eval to remove noise elements before extraction

• Convert to clean markdown for downstream processing

Troubleshooting

Issue Solution

Empty content Add wait --load networkidle after navigation

Partial render Use wait --text "Expected content"

Login required Use authentication flow with state save/load

CAPTCHA blocking Manual intervention required

Content in iframe Use frame @e# then extract

Related Skills

• browser-automation

• agent-browser CLI quick start and integration

• webapp-testing

• Playwright test automation patterns

• streaming-api-patterns

• Handle SSE progress updates

Version: 2.0.0 (January ) Browser Tool: agent-browser CLI (replaces Playwright MCP)

Capability Details

spa-extraction

Keywords: react, vue, angular, spa, javascript, client-side, hydration, ssr Solves:

• WebFetch returns empty content

• Page requires JavaScript to render

• React/Vue app content extraction

auth-handling

Keywords: login, authentication, session, cookie, protected, private, gated Solves:

• Content behind login wall

• Need to authenticate first

• Private documentation access

multi-page-crawl

Keywords: crawl, sitemap, navigation, multiple pages, documentation, tutorial series Solves:

• Capture entire documentation site

• Extract multiple pages

• Follow navigation links

agent-browser-commands

Keywords: agent-browser, open, snapshot, click, fill, eval, get text Solves:

• Which command to use

• Browser automation reference

• agent-browser CLI guide