Why did my crawl return an error?

Why did my crawl return an error?

Crawl Error Video



Malformed or Incorrect URL




Check that the starting URL is correct and is accessible (200 response) to the user agent setup for the crawl.



User Agent

We recommend adding our crawler as an allowed crawler based on the user agent name, ClarityBot or the full string: "Mozilla/5.0 (compatible; ClarityBot/9.0; +https://www.seoclarity.net/bot.html)". Learn more
  1. Some systems, like Cloudflare, will need to be configured to allow the Claritybot user agent to crawl your site. Learn more

Sitemap Crawl

A common mistake when setting up a Sitemap crawl is entering a sitemap in the URL text box but forget to update the type of crawl to sitemap. 


CSV Crawl

A common mistake setting up CSV crawls is selecting the option full site crawl when the intention is to crawl only URLs in a CSV.


Robots.txt

Obey Robots.txt: If no is selected, the crawl bypasses the settings in the robots.txt file of the site to be crawled. By default, the crawler obeys the robots protocol.


To check the settings use https://www.xxxxxxx.com/robots.txt in a browser window, entering the domain that is being crawled in the URL. Learn more