How to Disable Crawling for Specific Pages in Page Clarity
You may not always want Page Clarity to crawl all your managed pages. In seoClarity you have the ability to disable crawling for specific pages in your managed pages set. Pages with crawling disabled still count as managed pages in the platform.
Once disabled, the following crawl activities will cease for the specified page or pages:
- Daily crawling for content changes
- Retrieval of Page Speed
- Retrieval of backlinks
The following will not be affected:
- Page tags. Page tags can have pages with crawling either enabled or disabled.
- Ability to filter in other parts of the platform by the URL or page tag the page is associated with.
How to Disable Crawling in Page Clarity
Pages can be disabled in two ways:
- Manually select the pages in Page Clarity and set them to enable or disable crawling (details below).
- Automatic disabling: if we encounter an error when crawling the page consistently for seven days, we will automatically disable the crawling for the page
Whether or not a page was disabled manually or automatically, you have the option of re-enabling crawling at any time. In the case of automatic disabling, we will again crawl the page for seven days and if still returning an error consistently, disable the crawling for it again.
How To Manually Disable Crawling
1) Check the boxes in front of any pages for which you want to change the crawl status. Click the Actions drop down menu and select either "Disable Crawler" or "Enable Crawler." The action will be applied to the checked pages.
2) A new icon (spider with a line through it) will be displayed before each URL that has crawling disabled.
3) This icon also shows up on the Page Clarity Detail page, along with a message alerting you to the disabled status, and facilitating a quick undoing of that status if desired.
Related Articles
Page Clarity Managed Pages
Page Clarity Overview This overview provides an easy understanding of the Page Clarity tool. It was designed to enable you to monitor and manage your Target URLs at any level you choose. This guide will walk you through the process of adding URLs, ...Site Audit Report
Site Audit Report Overview This overview will help you understand exactly what Site Audit Reports displays, which is a summary of the most recently completed crawls. It contains a summarized view of site health scores of crawls run within a project, ...Page Tags
Page Tags Overview This walkthrough will take you through the process of adding, editing, and deleting Page Tags in the seoClarity platform. This is a high-level overview of the Page Tags basics which will get you started in the tool. After going ...Page Optimizer (enabled feature)
Page Optimizer Overview Page Optimizer enables the dynamic replacement of content in any web page. It can be used to replace or update content in the HTML of the page for a given location. A video introduction to using Page Optimizer and ...Page Crawler settings
Overview Page Crawler settings allow you to specify custom content to be tracked for your managed pages in Page Clarity. To set up email alerts for Page Crawler settings see Setting Up Email Alerts in Page Clarity. Managed elements can utilize the ...