--- title: "Import a Website as a Resource" url: "https://help.outdoo.ai/en/article/import-a-website-as-a-resource" language: "en" last_updated: "2026-11-10T16:29:38.621Z" --- # Import a Website as a Resource Import Website turns a public website, such as your product docs or help center, into a resource in **Manage > Resources**. Outdoo AI crawls the pages and keeps them in the library, so you can use your site as a data source for your agents without downloading and uploading files. ## Add a website - Go to **Manage > Resources** and click **Add Resources**. - Click **Import Website**. This opens a full page, not a window. - Under **Website**, enter the **Website URL**. This is required. - Under **Resource Details**, enter a **Name**, for example "Competitor battlecard". This is required. When you enter a valid URL and the name is still empty, Outdoo AI fills it in with the site's name, and you can change it. - Optionally enter a **Sitemap** URL, for example `https://example.com/sitemap.xml`. - Optionally set up **Page Filters**, **Extra URLs** and **Custom Headers** (described below). - Click **Import Website** at the top of the page. Click **Cancel** to leave without importing. Up to 5,000 pages are crawled per website. ### Sitemap validation The sitemap must be a full URL on a real domain. A bare word such as "sitemap" is rejected. When the sitemap is not valid, the field turns red and shows "Enter a valid sitemap URL, e.g., https://example.com/sitemap.xml". **Import Website** (or **Save Changes** on an existing website resource) stays disabled until you correct or clear it. ## Control what is crawled You can set these options when you add the website, and change them later from the website resource's details page. ### Page Filters Use filters to skip pages you do not want to import. Each rule has three parts: the field (**Page Path**), a condition, and a value such as `/en/pricing/`. The conditions are **Contains**, **Doesn't Contain**, **Starts With**, **Ends With** and **Equals**. Pages that match a rule are excluded. You can add up to 120 rules. Once pages have been crawled, you can also switch individual pages or whole folders on or off from the page list. ### Extra URLs Add pages the crawler did not find. Upload a CSV file with one URL per line. URLs you have already added are listed, and you can remove them. ### Custom Headers Send custom HTTP headers with every crawl request, for example an `Authorization` header with a value such as `Bearer abc123`. Enter the header name and value, and click **Add Header** to include more than one. ## Review and re-crawl Click **Edit** in the three-dot menu on a website resource, or click the website's name, to open its details page. The left menu has two sections, **General** and **Info**. Click **Save Changes** to apply any edits, or **Cancel** to leave. - **General** has **Resource Details** and **Website**, where you change the name, website URL and sitemap. It also has **Page Filters**, **Extra URLs** and **Custom Headers**, which work as described above. - **Info** shows the crawl status and the pages imported from the site. In **Info**, **Crawl Status** is one of **Queued**, **Crawling...**, **Indexing...** or **Completed**. When the crawl is complete, it also shows how long ago the last crawl ran. **Imported Pages** shows how many pages are in use. A progress bar shows how many pages were imported, skipped or failed. Below that, the page list lets you: - Search for a page with **Search pages...**. - Turn individual pages on or off. Turn a page off to exclude it from your agents' knowledge. - Click **Re-crawl this page** on a single page to fetch only that page again. Crawling a large site can take a while. The same status (**Queued**, **Crawling...**, **Indexing...** or **Completed**) also appears in the **Status** column of the library list. To fetch the whole site again after its content changes, click **Re-crawl** in the three-dot menu on the resource's row in the library, or click **Re-crawl Website** in the **Info** section. ## Use a website resource Website resources appear in the resource library alongside files, so you can attach them in the same places. See [Managing Resources](https://help.outdoo.ai/en/article/managing-resources) for where resources can be used. Hope this article was helpful. For any queries, please write to support@outdoo.ai ## Related articles - [Managing Resources](https://help.outdoo.ai/en/article/managing-resources) - [Create a Roleplay Agent from a File or Resource](https://help.outdoo.ai/en/article/create-a-roleplay-agent-from-file-or-resource) - [Intro to Roleplay Agents](https://help.outdoo.ai/en/article/intro-to-roleplay-agents) - [Add a Text Snippet](https://help.outdoo.ai/en/article/add-a-text-snippet)