How do I configure Cloudflare to allow the Claritybot user agent to crawl my site?
Please note that the following settings are part of the Cloudflare platform and could be changed by them without notice.
- In https://dash.cloudflare.com,/ select the domain name that will be the crawl target
- Navigate to the “Firewall” tab and click “Firewall Rules”
- Click "Create Firewall rule"
- Input: a Rule name, for example “Allow Claritybot”
Field: User Agent
Operator: equals
Value: {select a user agent string from below}
- Choose an action: Allow
Compare your page to the diagram below. When you are happy, click "Deploy".