new

Site Crawler Controlled Crawls

✅

Site Crawler Update!

Controlled Crawls

Users can now

require

exclude

key words or URL strings to be crawled & captured, making it easier to filter out irrelevant content and only capture what is needed.

In the Advanced Crawl Options section of the Site Crawler (both pv10 & portal), you’ll see two new fields:

Require (
One Keyword or URL String): Use this field to specify a single keyword or URL pattern that must be present in every URL collected. For example, if you’re only interested in pages with the path
"/collections/womens-all",
enter it here. Our crawler will ensure that only URLs containing this keyword are included.
Exclude
(Multiple Keywords or URL Strings): Use this field to list any keywords or URL strings you want to exclude from your crawl results. Enter multiple keywords by separating them with line breaks. For instance, if you don’t need URLs with
"/news"
or
"/product/mens",
add these terms here to filter them out.

👉🏻 Here's a 90 second demo walking through this enhancement.