Import a Website as a Resource
Import Website turns a public website, such as your product docs or help center, into a resource in Manage > Resources. Outdoo AI crawls the pages and keeps them in the library, so you can use your site as a data source for your agents without downloading and uploading files.
Add a website
- Go to Manage > Resources and click Add Resources.
- Click Import Website. This opens a full page, not a window.
- Under Website, enter the Website URL. This is required.
- Under Resource Details, enter a Name, for example "Competitor battlecard". This is required. When you enter a valid URL and the name is still empty, Outdoo AI fills it in with the site's name, and you can change it.
- Optionally enter a Sitemap URL, for example
https://example.com/sitemap.xml.
- Optionally set up Page Filters, Extra URLs and Custom Headers (described below).
- Click Import Website at the top of the page. Click Cancel to leave without importing.
Up to 5,000 pages are crawled per website.
Sitemap validation
The sitemap must be a full URL on a real domain. A bare word such as "sitemap" is rejected. When the sitemap is not valid, the field turns red and shows "Enter a valid sitemap URL, e.g., https://example.com/sitemap.xml". Import Website (or Save Changes on an existing website resource) stays disabled until you correct or clear it.
Control what is crawled
You can set these options when you add the website, and change them later from the website resource's details page.
Page Filters
Use filters to skip pages you do not want to import. Each rule has three parts: the field (Page Path), a condition, and a value such as /en/pricing/. The conditions are Contains, Doesn't Contain, Starts With, Ends With and Equals. Pages that match a rule are excluded. You can add up to 120 rules.
Once pages have been crawled, you can also switch individual pages or whole folders on or off from the page list.
Extra URLs
Add pages the crawler did not find. Upload a CSV file with one URL per line. URLs you have already added are listed, and you can remove them.
Custom Headers
Send custom HTTP headers with every crawl request, for example an Authorization header with a value such as Bearer abc123. Enter the header name and value, and click Add Header to include more than one.
Review and re-crawl
Click Edit in the three-dot menu on a website resource, or click the website's name, to open its details page. The left menu has two sections, General and Info. Click Save Changes to apply any edits, or Cancel to leave.
- General has Resource Details and Website, where you change the name, website URL and sitemap. It also has Page Filters, Extra URLs and Custom Headers, which work as described above.
- Info shows the crawl status and the pages imported from the site.
In Info, Crawl Status is one of Queued, Crawling..., Indexing... or Completed. When the crawl is complete, it also shows how long ago the last crawl ran. Imported Pages shows how many pages are in use. A progress bar shows how many pages were imported, skipped or failed.
Below that, the page list lets you:
- Search for a page with Search pages....
- Turn individual pages on or off. Turn a page off to exclude it from your agents' knowledge.
- Click Re-crawl this page on a single page to fetch only that page again.
Crawling a large site can take a while. The same status (Queued, Crawling..., Indexing... or Completed) also appears in the Status column of the library list.
To fetch the whole site again after its content changes, click Re-crawl in the three-dot menu on the resource's row in the library, or click Re-crawl Website in the Info section.
Use a website resource
Website resources appear in the resource library alongside files, so you can attach them in the same places. See Managing Resources for where resources can be used.
Hope this article was helpful. For any queries, please write to support@outdoo.ai