Skip to main content

​
Website Onboarding

Website onboarding bootstraps an agent from content you already have. Point it at your site and it generates rules, attributes, and knowledge base entries — so a new agent starts with your FAQs, product details, and tone rather than an empty configuration.

​
Crawling modes

Crawl entire website

Discovers and crawls pages starting from your homepage. Respects robots.txt and crawl limits. Default maximum 20 pages, configurable up to 100.Best for complete website analysis.

Process specific pages

Processes only the URLs you provide — no crawling or discovery.Best for targeted extraction from pages you already know matter.

​
What it generates

Choose any combination:

​
AI enhancement

​
Running it

  1. Select your agent — choose which agent receives the generated content.
  2. Choose a crawling mode — full crawl or specific pages.
  3. Enter URLs — your homepage for a full crawl, or each page individually.
  4. Configure — maximum pages, what to generate, and whether to enable AI analysis.
  5. Process — start and wait. Duration depends on page count, page complexity, and AI settings.
Always review generated content before activating rules. Onboarding produces a starting point from whatever your site happens to say — it can pick up outdated policies or marketing copy that shouldn’t become an automated response.

​
Getting good results

  • Start small. Run 10–20 pages first and check the output before scaling up.
  • Target high-value pages in specific-pages mode: FAQs, product catalogs, About and Contact, terms of service, and pricing.
  • Enable AI analysis when you want better FAQ extraction, product information parsing, and business context.

​
Troubleshooting

​
How it works

  • Respects robots.txt.
  • Falls back to alternative scrapers for JavaScript-heavy sites.
  • Processes concurrently with rate limiting.
  • Handles a range of content types and page structures.