Indexed source retention policies and filter conditions
To limit the set of records indexed from source tables, you can configure retention policies and filter conditions for your indexed sources. AI Search also uses these settings to automatically purge stale records from the index, optimizing search performance.
Indexed source retention policies
Indexing large source tables, such as the Task [task] table and tables that extend it, can add significant numbers of records to the AI Search index. To limit the set of source table records indexed based on the time since they were last updated, select a retention policy for your indexed source.
AI Search only indexes source records updated within the time period defined for the retention policy. For example, if you select a two-year retention policy, AI Search excludes source records that were last updated more than two years ago.
When the time since a source record's last update exceeds the limit from the indexed source's retention policy, AI Search marks the corresponding indexed record as stale.
Note: Retention policies are required for indexed sources that index records from the Task [task] table or tables that extend it. They are optional for other indexed sources.
Indexed source filter conditions
To limit the set of records indexed from a source table, define filter conditions for your indexed source. AI Search only indexes records that match all defined filter conditions.
Adding filter conditions can reduce the number of records indexed from a source table and reduce indexing frequency. As an example, if you exclude frequently updated open records based on their status, AI Search doesn't index data from those records. Excluding these records reduces your index size and the compute resources needed to index changes from the source table.
When a source record no longer satisfies the indexed source's filter conditions, AI Search marks the corresponding indexed record as stale.
Note: Adding filter conditions doesn't reduce the number of AI Search indexing events generated for the indexed source. The system generates indexing events for every change to an indexed source table, unless the change is to a column that has a no_text_index field setting defined in the indexed source. The filter condition can reduce the number of records indexed for each event, however.
Indexed source filter conditions only operate on source record fields that are indexed. Fields in the following table aren't indexed by default.
| Field | Description |
|---|---|
| Tags | The Tags field on source records isn't indexed by default. To use tags in an indexed source filter condition, you must first configure indexing for tags in the indexed source. For details on this procedure, see Enable indexing of tags for an AI Search indexed source. |
| Fields on referenced records | Fields on referenced records aren't indexed by default. To use a referenced record field in an indexed source filter condition, you must first configure indexing for the field in the indexed source. For details on this procedure, see Enable indexing of referenced table fields for an AI Search indexed source. |
Purging stale records
AI Search automatically purges stale records from the index daily. Users with the admin role can manually purge stale records. For details on this procedure, see Purge stale records from the AI Search index.
- Purge stale records from the AI Search index
Execute a scheduled job to delete stale records from the AI Search index.
Parent Topic:Indexed sources in AI Search
--- title: Indexed source retention policies and filter conditions description: To limit the set of records indexed from source tables, you can configure retention policies and filter conditions for your indexed sources. AI Search also uses these settings to automatically purge stale records from the index, optimizing search performance. locale: en-US canonical_url: https://www.servicenow.com/docs/r/australia/platform-administration/ai-search/retention-policies-conditions-ais.html release: australia product: AI Search classification: ai-search topic_type: concept last_updated: "2026-09-30" reading_time_minutes: 3 breadcrumb: [Indexed sources, Configure, AI Search, Search administration, Configure core features, Administer the ServiceNow AI Platform] --- # Indexed source retention policies and filter conditions To limit the set of records indexed from source tables, you can configure retention policies and filter conditions for your indexed sources. AI Search also uses these settings to automatically purge stale records from the index, optimizing search performance. ## Indexed source retention policies Indexing large source tables, such as the Task \[task\] table and tables that extend it, can add significant numbers of records to the AI Search index. To limit the set of source table records indexed based on the time since they were last updated, select a retention policy for your indexed source. AI Search only indexes source records updated within the time period defined for the retention policy. For example, if you select a two-year retention policy, AI Search excludes source records that were last updated more than two years ago. When the time since a source record's last update exceeds the limit from the indexed source's retention policy, AI Search marks the corresponding indexed record as stale. **Note:** Retention policies are required for indexed sources that index records from the Task \[task\] table or tables that extend it. They are optional for other indexed sources. ## Indexed source filter conditions To limit the set of records indexed from a source table, define filter conditions for your indexed source. AI Search only indexes records that match all defined filter conditions. Adding filter conditions can reduce the number of records indexed from a source table and reduce indexing frequency. As an example, if you exclude frequently updated open records based on their status, AI Search doesn't index data from those records. Excluding these records reduces your index size and the compute resources needed to index changes from the source table. When a source record no longer satisfies the indexed source's filter conditions, AI Search marks the corresponding indexed record as stale. **Note:** Adding filter conditions doesn't reduce the number of AI Search indexing events generated for the indexed source. The system generates indexing events for every change to an indexed source table, unless the change is to a column that has a `no_text_index` field setting defined in the indexed source. The filter condition can reduce the number of records indexed for each event, however. Indexed source filter conditions only operate on source record fields that are indexed. Fields in the following table aren't indexed by default. |Field|Description| |-----|-----------| |Tags|The Tags field on source records isn't indexed by default. To use tags in an indexed source filter condition, you must first configure indexing for tags in the indexed source. For details on this procedure, see [Enable indexing of tags for an AI Search indexed source](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/enable-tag-indexing-ais.md).| |Fields on referenced records|Fields on referenced records aren't indexed by default. To use a referenced record field in an indexed source filter condition, you must first configure indexing for the field in the indexed source. For details on this procedure, see [Enable indexing of referenced table fields for an AI Search indexed source](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/enable-ref-table-field-index-ais.md).| ## Purging stale records AI Search automatically purges stale records from the index daily. Users with the admin role can manually purge stale records. For details on this procedure, see [Purge stale records from the AI Search index](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/purge-stale-records-ais-index.md). - **[Purge stale records from the AI Search index](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/purge-stale-records-ais-index.md)** Execute a scheduled job to delete stale records from the AI Search index. **Parent Topic:**[Indexed sources in AI Search](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/indexed-sources-ais.md)
| title | Indexed source retention policies and filter conditions |
| description | To limit the set of records indexed from source tables, you can configure retention policies and filter conditions for your indexed sources. AI Search also uses these settings to automatically purge stale records from the index, optimizing search performance. |
| locale | en-US |
| canonical_url | https://www.servicenow.com/docs/r/australia/platform-administration/ai-search/retention-policies-conditions-ais.html |
| release | australia |
| product | AI Search |
| classification | ai-search |
| topic_type | concept |
| last_updated | 2026-09-30 |
| reading_time_minutes | 3 |
| breadcrumb | Indexed sourcesConfigureAI SearchSearch administrationConfigure core featuresAdminister the ServiceNow AI Platform |