Crawl4AI Web Crawler

Converts web pages to clean, LLM-friendly markdown using crawl4ai.

Requirements

• uv must be installed (https://docs.astral.sh/uv/)
• The skill manages its own isolated Python environment via uv sync

Setup

Before first use, run the setup script:

skills/crawl4ai/scripts/setup.sh

This will:

1. Check for uv installation
2. Run uv sync to create .venv/ and install dependencies
3. Run crawl4ai-setup to install playwright browsers
4. Run crawl4ai-doctor to verify installation

Usage

To crawl a URL and get markdown (from the skill directory):

cd skills/crawl4ai
uv run python scripts/crawl.py "https://example.com"

Options:

• --output FILE - Save markdown to file instead of stdout
• --include-links - Include hyperlinks in markdown output
• --include-images - Include image references
• --no-headless - Show browser window (default: headless)
• --timeout SECONDS - Page load timeout (default: 30)

Examples

Basic crawl

cd skills/crawl4ai
uv run python scripts/crawl.py "https://docs.python.org/3/"

Save to file

uv run python scripts/crawl.py "https://example.com" --output content.md

With all options

uv run python scripts/crawl.py "https://example.com" \
  --include-links \
  --include-images \
  --timeout 60 \
  --output result.md

Troubleshooting

If crawling fails:

1. Run uv run crawl4ai-doctor to diagnose
2. Ensure playwright browsers are installed: uv run python -m playwright install chromium
3. Check the URL is accessible and not blocked by robots.txt