HotTrender Basic Crawler
Core Rule
For four-region daily hotspot trends or vertical/custom-keyword hotspot discovery, use the bundled crawler runtime first. Do not reimplement platform crawling until the existing providers and scripts are checked.
Before changing code, answer these questions:
- Is there already a script, API, provider, doc, or test covering this need?
- Can the user goal be satisfied by running or configuring that capability?
- If not, what exact gap remains, and where is the smallest extension point?
Only edit code after that evaluation.
Repository Layout
This skill bundles a sanitized basic crawler runtime under assets/hottrender-runtime/. It does not bundle DingTalk, OSS, ActionCard, lp-ads, worker queues, databases, logs, LLM, or any secrets.
First resolve the runtime path from environment variables, the current workspace, or the bundled runtime:
HOTTRENDER_APP_DIR # directory containing scripts/fetch_daily_trends.py
If HOTTRENDER_APP_DIR is missing, install the bundled runtime:
python assets/install_hottrender_runtime.py --target ./HotTrenderRuntime
export HOTTRENDER_APP_DIR="$PWD/HotTrenderRuntime"
If variables are missing but a local checkout may exist, discover it safely:
find "$PWD" "$HOME" -maxdepth 5 -path '*/scripts/fetch_daily_trends.py' 2>/dev/null
Fast Path
Use these references only when needed:
- Setup requirements and portability constraints: setup.md
- Daily trend and custom keyword commands: commands.md
- Existing provider capabilities and extension points: capabilities.md
- How to decide whether to configure, run, or modify code: extension-policy.md
Operating Workflow
- For "四地区热点", "每日热点", "daily trends", or "jp/us/tw/kr", start from
scripts/fetch_daily_trends.py. - For "垂类热点", "关键词热点", "自定义关键词", or "custom keyword", start from
scripts/fetch_keyword_hotspots.py. - For "抓取是否有效", "平台数据不对", or "为什么没结果", inspect
configs/providers.yaml, run offline mode first, then real mode. - For code changes, keep the runtime basic. Do not add DingTalk, OSS, lp-ads, database, worker, or LLM features back into this skill.
Guardrails
- Never print API keys, cookies, tokens, msToken, proxy credentials, or other secrets.
- Do not fabricate live platform data. Offline/sample mode must be called out as offline/sample.
- Do not introduce push, publishing, database, queue, or workspace features into this basic crawler.
- If the user has no HotTrender checkout, use the bundled runtime installer before proposing code rewrites.
- Keep changes scoped: provider logic in
src/providers, orchestration insrc/crawler.py, CLI entrypoints inscripts/.
Verification
Prefer focused verification:
cd "$HOTTRENDER_APP_DIR"
python -m pytest tests/test_basic_crawler.py -q
python scripts/fetch_daily_trends.py --config configs/providers.yaml --output out/daily_trends.md