What are meta robots or robots.txt ?

What are meta robots or robots.txt ?

You can use an HTML <META> tag to tell robots not to index the content of a page, or follow it for links. Robots (aka crawlers or spiders) can ignore your Meta tag as it is only a guideline and not all Robots will respect it. The Meta Name attribute must be "ROBOTS". Valid values for the attribute are: "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW". Multiple values are allowed as long as they are comma separated, they should not be conflicting such as "FOLLOW", "NOFOLLOW". If there is no robots <META> tag, the default is "INDEX,FOLLOW", so it's not absolutely necessary to implement. This tag can be applied to individual pages or entire sitemaps.


The NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (this could even come from another other site) and so can still land on your page. This should not be confused with the link attribute rel="nofollow".


    Examples of the most common HTML snippets for robots.txt:


        <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

        <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">

        <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

        <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">


    • Related Articles

    • New Domain Provisioning: How To Request Profile Creation In seoClarity (Modular Apps)

      Overview To have a new domain profile provisioned within the seoClarity platform, the attached form needs to be filled out. Once filled out, the form will automatically create a Support Ticket and put you in touch with our Support team. Notice: ...

    • New Domain Provisioning: How To Request Profile Creation In seoClarity

      Overview To have a new domain profile provisioned within the seoClarity platform, the attached form needs to be filled out. Once filled out, the form will automatically create a Support Ticket and put you in touch with our Support team. Notice: ...

    • Glossary of Terms

      For more information on Platform Features, review seoClarity’s menu navigation. seoClarity Product features: Actions, Data, Filters, Metrics Alert: Email notification on changes; user-customizable. Annotations: Denote changes, updates or anything ...

    • Site Audit Projects

      Site Audit Projects Overview The Site Audit Projects List gives you a high level view of the different crawls that have been setup for the domain. Watch the video below: "How to Create a Clarity Audit Project" Background & Requirements Some sites ...

    • SEO Tech Issues: Robots.txt blocking pages from being crawled (Video)

      If you have important pages that are not being indexed by Google, one culprit could be your robots.txt file. Find out why: