Site Audit Project List Options

Each project in your list includes these options:

• Project Name: Click the project name to open its Site Audit page.
• Pencil icon (Edit): Rename the project.
• Trash icon (Delete): Permanently remove the project and all of its crawls.
• Gear icon (Settings): View the project's starting URL, depth, speed, and exclusions.
• Lightning bolt (Alerts): View the Site Audit alert configuration for the project, if enabled.

How to Set Up a New Site Audit Project

Click New Site Audit to open a popup where you can set up a new project and crawl, or run a new crawl within an existing project. The popup is organized into five tabs, described below.

Basic Settings Tab

This tab covers the essential information needed to start a new Site Audit.

• Project Type: Choose Existing Project to reuse a previously set-up project (including its custom settings), or New Project to start fresh with no inherited settings.
• Project Name: If you selected an existing project, its name displays automatically. If you're creating a new project, enter a name here.
• Language: The language used to tokenize and store your crawl data, enabling efficient broad-match searching on the title, meta description, H1, and H2. Defaults to English.
• Description: An optional field for any notes about the crawl project.

Choose What to Crawl

Select one of these sources:

• Starting URL: Enter the protocol (http/https) and the URL where the crawl should begin.
• Sitemap(s): Enter the protocol and the URL where the sitemap is located.
• RSS: Enter the protocol and the URL where the RSS feed is located.
• Upload CSV: Upload a CSV with your target URLs listed in a single column.

Starting URL tip: Subdomains are allowed here when Broad Match is selected under Domain Settings > Ranking Configuration, so you can start a crawl from any subdomain as long as the root domain matches. A validation message confirms the URL's current status code — an error usually means a slow response (over 25 seconds) or a blocked user agent.

Multi-subdomain sitemaps: If you're crawling several subdomains of one site, you can filter each subdomain in Site Audit and generate a separate sitemap for each. If all subdomain sitemaps are hosted under a single domain, edit the sitemap index to point to each individual sitemap.

Crawler Type

Choose how the crawler processes each page:

• Standard Crawl: Crawls the page source without rendering, similar to most crawlers. Use this for maximum compatibility.
• JavaScript Crawl: Renders every page exactly as a browser would, so you can catch issues Google's own JavaScript crawler might run into. This crawl type is slower, since the crawler waits for JavaScript to finish rendering on each page.
• Block Resources: Enter URL patterns (one per line) for resources you want blocked from rendering during the JavaScript crawl. Learn more

JavaScript crawl warning: By enabling a JavaScript crawl, you confirm that:

• You're authorized to run a crawl on the target website.
• You're responsible for any issues the crawl causes.
• The crawl will trigger all JavaScript and load all resources as a browser would.
• If the site uses any resources or JavaScript billed by number of loads/displays/triggers, you're solely responsible for those costs.

If you don't agree, do not enable the JavaScript crawl option. Starting a crawl with JavaScript enabled means you accept these terms.

Crawl Speed & Depth

• Crawl Speed: The number of concurrent requests sent to the site, equal to pages crawled per second. Total crawl time depends on how the site handles requests, page size, download time, and the number of URLs. Speeds above 8 pages per second trigger a cluster crawl (multiple pages crawled at the same time, using multiple crawlers).
  • Advanced: Optionally limit the number of pages crawled per day.
• Crawl Depth: Choose how far the crawl should go:
  • Custom: Crawl a set number of link levels away from the starting URL.
  • Full Site Crawl: Crawl every URL found for the domain (this can take significant time, depending on configuration).
  • Crawl only pages uploaded/found: Crawl only the specific URLs provided.

Advanced Settings Tab

Use this tab to configure commonly used advanced crawl options.

• Obey Robots.txt: On by default. Select No to bypass the site's robots.txt settings.
• Store Blocked Links: Off by default. Select Yes to have the crawler store links blocked by robots.txt.
• Enable Cookies: Have the crawler track cookies sent by web servers and return them on later requests. Use this for sites that redirect based on persisted cookies.
• Select Region: Optionally crawl from a location closer to where the site is hosted.

User Agent

Set a custom user agent, or leave blank to default to ClarityBot. Some domains may require you to add this bot to an allow list so it isn't blocked.

Link Parameter Handling

• Enter URL parameter(s) to remove: Enter parameters to remove automatically, separated by commas, or enter * to remove all URL parameters before crawling. This helps avoid duplicate content served under multiple URLs.
• Internal Links Analysis: Enable this to crawl the internal links found on each page.
• HREFLang Crawl: Hreflang tags tell search engines which language or regional version of a page to show. By default, all hreflang tags found while crawling appear in the Hreflang Audit tab of Site Health. Enable this option to also crawl rel="alternate" hreflang URLs (these can be crawled if Validate is also enabled).
• Canonical Crawl: A canonical marks the preferred version of a page when duplicate or similar content exists at multiple URLs. By default, all canonicals found while crawling appear in the Canonical Audit tab of Site Health. Enable this option to crawl all canonical URLs.

Crawling Rules Tab

Use this tab to customize which pages are crawled and how query parameters are handled.

Domain Crawling Rules

Enter one string match pattern per line.

• Allow domain(s): List the domains and subdomains you want crawled. By default, the crawler only follows the subdomain of your starting URL (or, for CSV/sitemap crawls, the subdomain of the first URL found).
• Deny domain(s): List domains and subdomains to deny. Links discovered for denied domains won't be crawled further.

To crawl all subdomains, enter the root domain in the Allow field — for example, entering xyz.com will crawl www.xyz.com, support.xyz.com, blog.xyz.com, and so on. A subdomain must either be discovered on an already-crawled page or added as an additional starting URL.

Multiple subdomains, one sitemap host: If you're crawling several subdomains of one site, you can filter each subdomain in Site Audit and generate a separate sitemap for each. If all subdomain sitemaps are hosted under a single domain, edit the sitemap index to point to each individual sitemap.

Link Crawling Rules

• Follow nofollow links: Off by default. Select Yes to crawl links marked rel="nofollow".
• URL pattern(s) to allow: Enter a regex (a pattern-matching expression) — matching URLs are included and their links followed. Disallow patterns take precedence over allow patterns.
• URL pattern(s) to disallow: Enter a regex pattern; matching URLs are excluded from crawling. These take precedence over allow patterns.
• URLs to crawl but not index: Matching URLs are crawled (and their links discovered), but their content isn't indexed.
• URLs to index but not crawl: Matching URLs are indexed, but their links aren't followed or added to the crawl list.

Link Discovery

Enter one string match pattern per line to restrict where links are found and crawled from.

• Restrict to Xpath: An XPath is a way to point to a specific location in a page's code. Specify one or more XPaths defining the regions to scan for links — only content within those regions is scanned.
• Restrict to CSS: Specify one or more CSS selectors defining the regions to scan for links. Works the same way as Restrict to Xpath.

Custom Tab

Use this tab to configure additional content or custom searches for the crawler to capture and store.

    Content Extraction

Capture additional content beyond the standard HTML elements (Title, Meta Description, H1, H2). Add more elements using the Content Extraction button. If a custom element appears multiple times on a page, only the first instance is retrieved. 

• Selector Type	Description	Example Target	Example Input
  XPATH	Targets unique XML/HTML path structures.	Product Price	//span[@class="price"]/text()
  CSS	Targets elements using standard CSS syntax.	Article Author	div.author-info > span
  DIV_ID	Targets a specific element by its unique ID attribute.	Main Call-to-Action	cta-button-primary
  DIV_CLASS	Targets an element by its assigned CSS class.	Rating Summary	product-rating-score

  Content Match

  Filters crawled pages and displays results under the Custom Search Tab in Site Health based on text or pattern matching.

  Match Type	Description	Example Scenario & Input	Output
  Contains	Returns pages matching the string and counts total occurrences per page.	Target: Find pages missing Google Tag Manager container. Input: GTM-XXXXXXX	Flags pages containing the string and provides the occurrence count per page.
  Does Not Contain	Returns pages that completely lack the specified string.	Target: Identify pages missing copyright or compliance text. Input: © 2026 Company Name	Lists all pages where this string is missing.
  Regex	Evaluates pages against a Regular Expression pattern and counts matches.	Target: Detect unmasked US phone numbers on pages. Input: \b\d{3}[-.]?\d{3}[-.]?\d{4}\b	Lists matching pages and the count of matched phone number instances.

Start Audit Tab

Choose when your audit should start.

• Frequency: Run a one-time crawl, or set up Weekly, Bi-Weekly, or Monthly recurring crawls. Recurring crawls let you choose how many months to schedule them for.
• Launch Crawl:
  1. Start Now — the crawl queues up and begins shortly after you select Start Site Audit.
  2. Start Later — schedule the crawl for a future date and time under Start Date.
• Schedule Interval: Set the specific hours and days of the week the crawler should run — useful for restricting crawl activity to off-peak hours or low server load. This setting takes precedence over Launch Crawl.

Setting Up Crawl Alerts

Crawl alerts email you and your team when something happens during a crawl. You can be notified when:

• A crawl starts, completes, or hits an error
• The crawl detects an increase in selected issues
• There's a change to any additional custom content you've designated

To set up alerts:

1. Toggle Enable Alerts.
2. Under Crawl Launch, choose to receive an email when the crawl launches and completes.
3. Under Increase in Issues, click the pencil icon to open the side panel and select the specific issues you want to be alerted about.
4. Under Changes to Additional Content/Custom Content, select this option if you've already entered custom content to track (it's only available if you have).
5. Under Send Crawl Alerts To, enter the email addresses of any teammates who should also receive alerts.

When any condition you've set occurs during a crawl, everyone on the alert list receives an email describing the change. Crawl initiation, completion, and error emails are sent only to the user who set up the crawl.

Update Crawl Config: Select this option to apply your current settings to all future scheduled crawls within the project.

Managing Your Crawls

• Archiving old crawls: Crawl data is available for up to 12 months, after which it's archived. Summary data for archived crawls remains visible in Site Audit Projects and Site Audit Reports. To retrieve archived crawl data, contact support@seoclarity.net.
• Pausing crawls: You can pause a crawl temporarily. If it's paused for more than 7 days, it's automatically stopped.

How to Cancel Scheduled Crawls

1. Go to your Site Audit Projects page.
2. Select the Crawl Log tab.
3. Select the scheduled crawl(s) you want to cancel.
4. Click the Action dropdown and select Remove Crawls.

Note: Existing Site Audit projects with a recurring frequency can't be edited. Instead, cancel the future scheduled crawls and set up a new Site Audit project with your updated settings. This keeps your existing project intact while letting you apply the changes you need.

Troubleshooting: Why Is My Crawl Running Slowly?

If your crawl is taking longer than expected, check these common causes:

• Crawl Speed: A low crawl speed setting can significantly slow things down. Try increasing it in your Site Audit project settings.
• Page Size and Complexity: Large or complex pages — especially those with heavy multimedia or complex code — take longer to crawl.
• Crawl Depth: A deep crawl depth takes longer to complete. Consider narrowing the depth to focus on your most important pages or sections first.
• Crawl Schedule: Crawls scheduled during peak hours or high server load run slower. Try rescheduling to off-peak hours or times of low server activity.

If you've tried these steps and your crawl is still unusually slow, contact support@seoclarity.net for further assistance.