Skills Guide

Taoguba Crawler

VerifiedCaution

Skill to run Taoguba (tgb.cn) web crawlers for scraping forum articles and generating HTML outputs.

Download and use
Sby Skills Guide Bot
Data & AIBeginner
7/28/2026
Claude Code
#web-scraping#python#taoguba#data-extraction#html-export

Recommended for

Backend developerClaude CodeData scientistData & analysis

Automated summary

This skill runs the project's two Python crawlers for Taoguba (tgb.cn), pulling BBS forum articles and homepage recommendations and exporting them as JSON plus HTML with base64-embedded images.

Strengths

  • Two ready-made crawlers: HTML scraping for the BBS board and a JSON API for the homepage
  • Always writes a dated JSON file plus a self-contained HTML rendering with base64-encoded images
  • 0.5–1s delay between requests to reduce blocking, with COOKIE-based auth loaded from .env
  • Clear troubleshooting section covering the most common failures (invalid cookie, image errors, timeouts)

Limitations

  • Requires a .env file with a valid COOKIE that is not supplied or explained in depth
  • CSS selectors (a.overhide.mw300, #first, .comment-data) are hard-coded and will break if the site changes
  • No pagination control beyond defaults and no output format other than JSON/HTML
When to use it

Use it when you want to run the existing Taoguba crawlers to archive forum or homepage articles with minimal setup.

When not to use it

Avoid it if you need a configurable general-purpose scraper, or if you do not have a valid Taoguba cookie.

Security analysis

Caution
Quality score78/100

The skill only orchestrates Python crawlers that fetch pages from tgb.cn using a stored cookie, apply rate limiting, and write local JSON/HTML output. No obfuscation, no curl|sh, no sending secrets to unknown endpoints, and no destructive file operations are described, so it is not risky. It is rated caution because it uses Bash plus network access and handles an authenticated session cookie, which warrants awareness but is normal for a scraper.

Findings
  • •Reads a site session COOKIE from a .env file and sends it to tgb.cn for authenticated scraping (legitimate use, but credential handling is sensitive)
  • •Performs outbound network requests and downloads arbitrary remote images, then base64-embeds them into generated HTML
  • •Web scraping may conflict with the target site's terms of service; no robots/ToS check mentioned
  • •Scripts are executed via Bash but no destructive commands, piping to shell, or exfiltration to third parties are described
How this analysis is made

Examples

Run both crawlers
Crawl taoguba with both crawlers and tell me where the JSON and HTML files landed.
BBS only
Run the BBS crawler for tgb.cn/bbs/1/1 and summarize how many articles were extracted.
Debug empty output
The taoguba crawler returned no articles — check my .env COOKIE setup and tell me what's wrong.

name: taoguba-crawler description: This skill should be used when the user asks to "crawl taoguba", "crawl tgb", "scrape taoguba articles", "run the crawler", "crawl bbs", "crawl home page", "generate article HTML", or needs to run the Taoguba (tgb.cn) web crawlers. version: 0.1.0 allowed-tools: Bash, Read

Taoguba Crawler

This skill runs the Taoguba (tgb.cn) article crawlers located in the project root.

Prerequisites

  • Python 3 with requests, beautifulsoup4, python-dotenv installed
  • A .env file in the project root containing COOKIE and optionally USER_AGENT

Available Crawlers

1. BBS Crawler (crawler_bbs.py)

Crawl the forum board at tgb.cn/bbs/1/1 using HTML scraping.

python crawler_bbs.py
  • Extracts article list by parsing a.overhide.mw300 elements
  • Gets each article's main post and author replies
  • Downloads images and embeds them as base64 in HTML
  • Outputs: output/bbs_YYYY-MM-DD.json and output/bbs_YYYY-MM-DD_HHMMSS.html

2. Home Crawler (crawler_home.py)

Crawl the homepage recommendations via JSON API (/newIndex/getZh).

python crawler_home.py
  • Fetches articles from the JSON API (default 2 pages)
  • Same content extraction and HTML generation as BBS crawler
  • Outputs: output/home_YYYY-MM-DD.json and output/home_YYYY-MM-DD_HHMMSS.html

Common Workflow

To run both crawlers:

python crawler_bbs.py && python crawler_home.py

Key Implementation Details

  • Authentication: Both scripts read COOKIE from .env via python-dotenv
  • Rate limiting: 0.5-1s delay between requests to avoid being blocked
  • Image handling: Images are downloaded and embedded as base64 in the HTML output
  • Article content: Extracts main post (#first) and author replies (.comment-data with author badge)
  • Output directory: All results saved to output/ folder

Scripts

The crawler scripts are bundled in scripts/:

  • scripts/crawler_bbs.py - BBS forum crawler (HTML scraping)
  • scripts/crawler_home.py - Homepage crawler (JSON API)

To run the bundled scripts directly:

python scripts/crawler_bbs.py
python scripts/crawler_home.py

Troubleshooting

  • If no articles are returned, check that .env contains a valid COOKIE value
  • If image downloads fail, the HTML will show error messages inline
  • Network timeouts default to 10-15 seconds per request

From the file to a first result

The download contains exactly the content displayed on this page. This copy may differ from the latest version in the source repository.

View the source and supporting files

This download contains SKILL.md only. Any scripts, references, templates and dependencies are not included. Get the required files from the source, preserving their directory structure, before continuing.

1. Download the file

Download SKILL.md

From the root of a test project, save the file as .skill-downloads/taoguba-crawler-ms47vohh/SKILL.md, or use the command below. This folder is for inspecting the skill before installation.

Command for macOS/Linux or WSL: requires a POSIX shell and curl. On Windows without WSL, use the download button and create the folder manually.

mkdir -p '.skill-downloads/taoguba-crawler-ms47vohh' && curl --fail --show-error --location 'https://skills-guides.com/api/skills/taoguba-crawler-ms47vohh/download' --output '.skill-downloads/taoguba-crawler-ms47vohh/SKILL.md.part' && mv '.skill-downloads/taoguba-crawler-ms47vohh/SKILL.md.part' '.skill-downloads/taoguba-crawler-ms47vohh/SKILL.md'

2. Check prerequisites

Read SKILL.md before using it. If it references scripts or external services, inspect them and prepare the required access in your test environment.

Tools declared by the author (check availability in your agent): Bash, Read

Original excerpt: Prerequisites
  • Python 3 with requests, beautifulsoup4, python-dotenv installed
  • A .env file in the project root containing COOKIE and optionally USER_AGENT

The platforms on this page are catalog indications. The paths below follow tool documentation; they do not prove this skill has been run in those tools. No minimum tool version is confirmed for this listing.

3. Load the skill

After inspection, copy the complete folder into your project using one of the commands below, then open a new tool session. The command stops if a folder with the same name already exists: compare the files before replacing it.

Claude Code
test -f '.skill-downloads/taoguba-crawler-ms47vohh/SKILL.md' && test -s '.skill-downloads/taoguba-crawler-ms47vohh/SKILL.md' && mkdir -p '.claude/skills' && test ! -e '.claude/skills/taoguba-crawler' && cp -R '.skill-downloads/taoguba-crawler-ms47vohh' '.claude/skills/taoguba-crawler'

OpenSkills

OpenSkills must already be installed. The local install/read procedure is checked with OpenSkills 1.5.0, which requires Node.js ≥ 20.6.0. Catalog slugs are not resolved automatically.

Official OpenSkills installation
openskills install './.skill-downloads/taoguba-crawler-ms47vohh'

openskills read 'taoguba-crawler-ms47vohh'

Reading should display the installed file content and its base directory. It does not run the skill or install its dependencies.

For an agent that reads AGENTS.md, then sync the index. This command changes AGENTS.md: review the diff before using it.

openskills sync

Manual method or another tool: attach SKILL.md to a conversation, or open it in an agent that can read files. Include any required supporting resources. No universal folder is recognized by every tool.

4. Try a first task

Use a short, synthetic input: an API endpoint, a small document or an interface brief. In a new conversation, attach the file or give the agent its path, then adapt this request:

Read .skill-downloads/taoguba-crawler-ms47vohh/SKILL.md and apply its instructions to this example: [describe a test input]. Identify any missing files, dependencies or access. Produce the result requested by the skill without inventing data or running unreviewed scripts.

Expected result: output matching the skill instructions, or a precise list of missing prerequisites. A read confirmation alone does not validate a first use. If an external resource is unavailable, that path remains untested.

Related skills

Create Interactive Maps with kepler.gl

Data & AI

Create interactive map visualizations and export standalone HTML using the keplergl Python package. Use to plot geospatial data from DataFrames, GeoJSON, or CSV.

Claude CodeCursorCodexintermediateSafe 11,995
Skills Guide Bot

Agent-Driven Parameter Optimization

Data & AI

Guide for optimizing configurable systems with a measurable objective, using bounded trials and evidence-based recommendations.

Claude CodeCursorCodexintermediateSafe 45,609
Skills Guide Bot

ICP Visitor Outreach

Data & AI

Identify signed-up users matching the ideal customer profile but stalled, and draft personalized outreach to re-engage them.

Claude CodeintermediateSafe 6,843
Skills Guide Bot