> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stateset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Website Onboarding

> Generate rules, attributes, and knowledge base entries automatically from your website content.

# Website Onboarding

Website onboarding bootstraps an agent from content you already have. Point it at your site and
it generates rules, attributes, and knowledge base entries — so a new agent starts with your
FAQs, product details, and tone rather than an empty configuration.

## Crawling modes

<CardGroup cols={2}>
  <Card title="Crawl entire website" icon="spider">
    Discovers and crawls pages starting from your homepage. Respects `robots.txt` and crawl
    limits. Default maximum 20 pages, configurable up to 100.

    Best for complete website analysis.
  </Card>

  <Card title="Process specific pages" icon="list-check">
    Processes only the URLs you provide — no crawling or discovery.

    Best for targeted extraction from pages you already know matter.
  </Card>
</CardGroup>

## What it generates

Choose any combination:

| Output             | Purpose                                       |
| ------------------ | --------------------------------------------- |
| **Rules**          | Automated response rules for common inquiries |
| **Attributes**     | Agent behavior and knowledge settings         |
| **Knowledge Base** | Training data for AI responses                |

## AI enhancement

| Mode         | Behavior                                                                                                     |
| ------------ | ------------------------------------------------------------------------------------------------------------ |
| **Standard** | Basic content extraction and analysis                                                                        |
| **Enhanced** | Deeper analysis of FAQs and common questions, product information, business details, and communication style |

## Running it

1. **Select your agent** — choose which agent receives the generated content.
2. **Choose a crawling mode** — full crawl or specific pages.
3. **Enter URLs** — your homepage for a full crawl, or each page individually.
4. **Configure** — maximum pages, what to generate, and whether to enable AI analysis.
5. **Process** — start and wait. Duration depends on page count, page complexity, and AI
   settings.

<Warning>
  **Always review generated content before activating rules.** Onboarding produces a starting
  point from whatever your site happens to say — it can pick up outdated policies or marketing
  copy that shouldn't become an automated response.
</Warning>

## Getting good results

* **Start small.** Run 10–20 pages first and check the output before scaling up.
* **Target high-value pages** in specific-pages mode: FAQs, product catalogs, About and Contact,
  terms of service, and pricing.
* **Enable AI analysis** when you want better FAQ extraction, product information parsing, and
  business context.

## Troubleshooting

| Problem                  | Try                                                                                                                                |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- |
| **No content extracted** | The site may block scrapers. Switch to specific pages, and verify the URLs are publicly reachable.                                 |
| **Timeout errors**       | Reduce the page count, switch to specific-pages mode, or check the site's response time.                                           |
| **Poor quality results** | Enable AI analysis, target high-value pages, and check the pages have real content structure rather than being JS-rendered shells. |

## How it works

* Respects `robots.txt`.
* Falls back to alternative scrapers for JavaScript-heavy sites.
* Processes concurrently with rate limiting.
* Handles a range of content types and page structures.

## Related

* [ResponseCX Platform](/stateset-response/responsecx-platform)
* [Workflow Studio](/stateset-response/responsecx-workflow-studio)
* [Public API — knowledge endpoints](/stateset-response/response-public-api)