Monitor and manage the automated background web crawler for your agents.
💡 How the Auto-Scraper works
The nightly cron job uses each agent's Auto-Crawler Prompt to search Google and queue new URLs as pending recommendations.
When new URLs are found, an email notification is sent to the agent creator so they can log back in and approve or dismiss them here.
→ Configure Google API keys and email settings in Global Config.
→ Set each agent's crawler prompt in Agent Studio (Update Agent tab).
🤖 Agent Crawler Status
Loading agents...
⏳ Pending Recommendations
URLs discovered by the nightly crawler across all your agents. Approve to scrape into the knowledge base, or dismiss.