How do I configure Cloudflare to allow the Claritybot user agent to crawl my site?
Please note that the following settings are part of the Cloudflare platform and could be changed by them without notice.
- In https://dash.cloudflare.com,/ select the domain name that will be the crawl target
- Navigate to the “Firewall” tab and click “Firewall Rules”
- Click "Create Firewall rule"
- Input: a Rule name, for example “Allow Claritybot”
Field: User Agent
Operator: equals
Value: {select a user agent string from below}
- Choose an action: Allow
Compare your page to the diagram below. When you are happy, click "Deploy".
Related Articles
Why did my crawl return an error?
Crawl Error Video Malformed or Incorrect URL Check that the starting URL is correct and is accessible (200 response) to the user agent setup for the crawl. User Agent We recommend adding our crawler as an allowed crawler based on the user agent name, ...
Allowing The seoClarity Crawler To Crawl Your Site
Overview The seoClarity crawler can only crawl your site if you allow it to. With the volume of bad bots increasing day by day, most sites are enhancing their security to block unknown bots from accessing their site. In that respect, it is important ...
Site Audit Projects
Site Audit Projects Overview The Site Audit Projects List gives you a high level view of the different crawls that have been setup for the domain. Watch the video below: "How to Create a Clarity Audit Project" Background & Requirements Some sites ...
Site Audit Settings
Site Audit Settings Overview Site Audits provides a variety of reports and analysis based on a crawl, that can impact the health of a site. Site Audit Settings allow for the customization of those Site Audit reports by enabling the ability to ...
Site Audit Details
Site Audit Details Overview Site Audit Details provides a variety of reports and analysis based on a crawl, that can impact the health of a site. Site Audit Details Use Cases Identify and fix potential user experience issues. Learn more Identify ...