| name | context-scrapers |
| description | Web scraping spiders (Scrapy), parsers, and source management. |
| tags | backend, data, scrapers |
Scrapers Context
Overview
Web scraping infrastructure using Scrapy. Handles data acquisition from city councils (Legistar) and municipal codes (Municode).
Active Files
Spiders
backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_meetings.py- Legistar meeting scraperbackend/affordabot_scraper/affordabot_scraper/spiders/sanjose_municode.py- Municode scraper
Configuration
backend/affordabot_scraper/affordabot_scraper/pipelines.py- Item pipelines (DB storage)backend/affordabot_scraper/affordabot_scraper/settings.py- Scrapy settings
Verification Scripts
scripts/verify_raw_scrapes.py- Check raw_scrapes tablescripts/verify_municode_discovery.py- Verify discovery logic
Usage
Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion.