Create a Webcrawler external content connector

task brazil Updated Oct 20, 2026 7 min read create-ext-cont-connector-webcrawler.md
Raw file servicenow.com

Create an external content connector to retrieve searchable content from pages and subdomains in a public web source system. Select from a list of predefined web sources or specify your own web source.

Before you begin

Role required: sn_ext_conn.xcc_admin

About this task

Follow this procedure to create a Webcrawler connector to retrieve searchable content and metadata from URLs found in a predefined or custom web source.

By default, you can configure up to three Webcrawler connectors for custom web sources. If you need to retrieve items from more than three custom web sources, you can create a Customer Service and Support case at https://support.servicenow.com/now to request a limit increase for the Webcrawler connector.

Procedure

  1. Navigate to All > External Content Connectors > External Content Admin Home.

  2. If prompted to switch scope, select Switch scope and select the application scope specified in the prompt.

  3. In the Connectors section, select New.

  4. On the Choose source page, select the Webcrawler tile, then select Next.

  5. On the Connection settings page, specify the web source that you want to retrieve searchable content from.

    • To retrieve searchable content from a web source in the list of predefined web sources, select the Predefined web source tile, then select your desired web source from the list.
    • To retrieve searchable content from a public web source not included in the list, select the Custom web source tile, then fill in the following fields.

Field

Description

Name

Unique name for this Webcrawler external content connector.

URL

Start point URL for your selected web source. As an example, you might enter https://example.com/mysite.

Max crawling depth

The maximum crawl depth for the connector, expressed in link hops from the start point URL.The start point URL page has depth 0. Pages linked directly from the start point URL have depth 1, pages linked from those depth-1 pages have depth 2, and so on. The connector follows links up to and including the configured maximum depth. A value of 0 means the connector only crawls the start point URL page.

Type: Integer

Supported values: 0 to 5000

Default value: 10

Note: This setting only applies when the Use sitemap option isn't selected. When the Use sitemap option is selected, the connector ignores this setting.

Use sitemap

Option to retrieve content and links from URLs found in the source system's sitemaps that include the specified start point URL.If you select this option, the connector reads matching URLs from all sitemaps referenced in the source system's robots.txt file and all sitemaps located in common sitemap locations.

As an example, suppose you select the Use sitemap option and then specify https://example.com/mysite as the start point URL for a content crawl. When you run the content crawl, the Webcrawler connector retrieves content and links from sitemap URLs that include https://example.com/mysite. In this case, the connector retrieves content and links from sitemap URLs https://example.com/mysite/a and https://example.com/mysite/b but ignores sitemap URLs https://example.com/othersite/c and https://example.com/yoursite/d because they don't include the start point URL.

Note: When this option is selected, the connector ignores the Max crawling depth setting.
6. Select the **The administrator acknowledges and agrees…** legal disclaimer option.

  1. Save and validate your connection settings by selecting Validate Connection.

    Note: If validation of your connection settings fails, the system shows an error message. Verify your connection settings to ensure they're correct. If permissions required by the connector are missing or incorrectly configured in the source system, a warning message appears showing the permissions that must be corrected. Provide the information from this message to your source system administrator.

  2. On the Crawl settings page, modify any default crawl settings that you want to override for this connector, then select Next.

    If you want to skip this step for now, select Skip instead of Next. You can modify the crawl settings for this connector from the External Content Admin Home page. For details on this procedure and the available crawl settings, see Configure crawl settings for an Atlassian Confluence Cloud external content connector.

  3. On the Create crawl page, create a content crawl for this connector by selecting a crawl scope (if supported) and any desired options, then select Next.

    If you want to skip this step for now, select Skip instead of Next. You can create and run crawls for this connector from the External Content Admin Home page. For details on creating content crawls, see Create a content crawl for an external content connector.

  4. On the Connect search profile page, use the Connect to search profile field and Add button to add any search profiles that you want to connect this external content connector's default search source to, then select Save.

    If you want to skip this step for now, select Skip instead of Next. You can connect search sources for this connector to search profiles from the External Content Admin Home page. For details on connecting an external content connector to search profiles, see Connect an external content connector to a search profile.

    When you link a connector search source to a search profile in this step, the system automatically publishes the search profile to make the new link take effect.

Result

Your new external content connector appears in the Connectors list on the External Content Admin Home page.

What to do next

To retrieve searchable content with your new connector, you must configure and run content crawls for it. You can modify your new connector's crawl settings and create crawls for it from the External Content Admin Home page even if you skipped these steps during connector creation.

To make content crawled by your new connector searchable in portals and search applications, you need to link one of its search sources to the search profile used by each portal or search application. You can use the connector's default search source or create your own custom search sources.

  • Search sources for predefined web sources

    The Webcrawler external content connector includes a dedicated search source for each predefined web source. Each dedicated search source includes content from the specified predefined web source that's stored in the connector's indexed source. Use these search sources to access searchable content and metadata retrieved from predefined web sources. As an example, searchable content and metadata retrieved from the Google Drive predefined web source is accessible in a search source named Google Drive.

  • Default search source

    By default, the system creates a search source that includes all searchable content from your Webcrawler connectors.

  • Custom search sources

    You can create your own search sources with filters to specify which content from the connector's indexed source is searchable. To view the connector's indexed source, navigate to All > AI Search > AI Search Index > Indexed Sources. For information about creating search sources, see Search sources in AI Search.

You can link connector search sources to search profiles from the External Content Admin Home page. For details on this procedure, see Connect an external content connector to a search profile.

Parent Topic:Webcrawler external content connector

Keywords ServiceNow OttoAI Agentsgenerative AIagentic AI
Search documentation
Search document titles and file paths on the brazil branch