Agents
Manage and monitor data pipeline agents
Pipeline Flow
Card Agent
→Set Agent
→External Map Agent
→Price Agent (legacy)
→Aggregation Agent
Card Agent
Fetches all YGO cards from YGOProDeck API
Never run
IdleWeekly
Set Agent
Populates sets and prints from TCGCSV
Never run
IdleWeekly
Deck Agent
Scrapes FormLibrary decklists into decks + deck_cards
Never run
IdleWeekly
Aggregation Agent
Price retention (raw→daily→weekly→monthly) + is_latest maintenance
Never run
IdleNightly
Listings Agent
Scrapes ALL TCGPlayer listings → listings table + active prices. Distributed fleet.
Last run: 2h ago
IdleDaily
History Backfill Agent
3-year TCGPlayer weekly price history → market prices. Distributed fleet.
Never run
IdleOn demand
Image Agent
Downloads card images from YGOProDeck CDN
Never run
IdleManual
External Map Agent
Maps cards/prints to TCGPlayer product IDs
Never run
IdleManual
Card Format Agent
Tags cards with formats they appear in
Never run
IdleManual
Price Agent (legacy)
Old cheapest-only scrape — superseded by Listings Agent
Never run
IdleReplaced
Seller Agent (legacy)
Seller directory — now built automatically by Listings Agent
Never run
IdleReplaced