You can use an HTML <META> tag to tell robots not to index the content of a page, or follow it for links. Robots (aka crawlers or spiders) can ignore your Meta tag as it is only a guideline and not all Robots will respect it. The Meta Name attribute must be "ROBOTS". Valid values for the attribute are: "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW". Multiple values are allowed as long as they are comma separated, they should not be conflicting such as "FOLLOW", "NOFOLLOW". If there is no robots <META> tag, the default is "INDEX,FOLLOW", so it's not absolutely necessary to implement. This tag can be applied to individual pages or entire sitemaps.

The NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (this could even come from another other site) and so can still land on your page. This should not be confused with the link attribute rel="nofollow".

    Examples of the most common HTML snippets for robots.txt:

        <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

        <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">

        <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

        <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">