Why did my crawl return an error?

Why did my crawl return an error?

Crawl Error Video



Malformed or Incorrect URL




Check that the starting URL is correct and is accessible (200 response) to the user agent setup for the crawl.



User Agent

We recommend adding our crawler as an allowed crawler based on the user agent name, ClarityBot or the full string: "Mozilla/5.0 (compatible; ClarityBot/9.0; +https://www.seoclarity.net/bot.html)". Learn more
  1. Some systems, like Cloudflare, will need to be configured to allow the Claritybot user agent to crawl your site. Learn more

Sitemap Crawl

A common mistake when setting up a Sitemap crawl is entering a sitemap in the URL text box but forget to update the type of crawl to sitemap. 


CSV Crawl

A common mistake setting up CSV crawls is selecting the option full site crawl when the intention is to crawl only URLs in a CSV.


Robots.txt

Obey Robots.txt: If no is selected, the crawl bypasses the settings in the robots.txt file of the site to be crawled. By default, the crawler obeys the robots protocol.


To check the settings use https://www.xxxxxxx.com/robots.txt in a browser window, entering the domain that is being crawled in the URL. Learn more


    • Related Articles

    • Allowing The seoClarity Crawler To Crawl Your Site

      Overview The seoClarity crawler can only crawl your site if you allow it to. With the volume of bad bots increasing day by day, most sites are enhancing their security to block unknown bots from accessing their site. In that respect, it is important ...

    • Site Audit Projects

      Site Audit Projects Overview The Site Audit Projects List gives you a high level view of the different crawls that have been setup for the domain. Watch the video below: "How to Create a Clarity Audit Project" Background & Requirements for Site Audit ...

    • How do I configure Cloudflare to allow the Claritybot user agent to crawl my site?

      Please note that the following settings are part of the Cloudflare platform and could be changed by them without notice. In https://dash.cloudflare.com,/ select the domain name that will be the crawl target Navigate to the “Firewall” tab and click ...

    • URL Parameter settings

      URL Parameter Settings Overview Customize how ClarityBot crawls your site. The URL Parameters Setting gives users the ability to instruct the seoClarity crawler bot with customizable crawl settings for your website, as well as setup pattern matching ...

    • Site Audit Details

      Site Audit Details Overview Site Audit Details provides a variety of reports and analysis based on a crawl, that can impact the health of a site. Site Audit Details Use Cases Identify and fix potential user experience issues. Learn more Identify ...