| -link"> | | | | This tag will allow your page to be indexed in the |
| The "robots" meta tag, when used properly, | | | | search engines, but any links on that page will not be |
| will tell the search engine spiders whether or not to | | | | followed. |
| index and follow a particular page. Some examples of | | | | 4. <meta name="robots" |
| usage are as follows: | | | | content="noindex,nofollow"> |
| <meta name="robots" | | | | When using this tag, the search engine spiders will not |
| content="index,follow"> | | | | include this page in their directory and will not follow |
| <meta name="robots" | | | | any links on the page either. |
| content="noindex,follow"> | | | | |
| <meta name="robots" | | | | Where does the "robots" tag belong? |
| content="index,nofollow"> | | | | The "robots" meta tag should be used |
| <meta name="robots" | | | | within the <head> and </head> tags of your |
| content="noindex,nofollow"> | | | | page. These tags are located at the top of the html |
| Let us first examine what these terms mean before | | | | coding. It will look something like this: |
| we explain the usage for each one: | | | | <html> |
| "index"- This directive tells the search | | | | <head> |
| engine robots (or spiders) that it is okay to index the | | | | <title>Title of your page goes here</title> |
| page. Another words, you are allowing the search | | | | <meta name="keywords" |
| engine to include your page within their search | | | | content="word1,word2,word3,word4"> |
| directory. | | | | <meta name="description" |
| "noindex"- Using this tag, you are letting the | | | | content="A brief description of the content ofthis |
| robots know that this page should not be indexed. | | | | page."> |
| Simply put, this page will not appear in their search | | | | <meta name="robots" |
| directory. | | | | content="index,follow"> |
| "follow"- When you use this tag, you are | | | | </head> |
| telling the search engines that you want their robot to | | | | <body> |
| follow any links that are found on that page. | | | | Your webpage information here. |
| "nofollow"- The opposite of the above | | | | </body> |
| definition, this directive will tell the robots not to follow | | | | </html> |
| any links on your page. | | | | More Robots Tags |
| Putting it all together: | | | | Google automatically archives a page as it crawls it. |
| With the robots tags explained, let's examine the | | | | This is called a "cached" version of the |
| usage for each one. | | | | page. Visitors can retrieve the archived version of the |
| 1. <meta name="robots" | | | | page by clicking on the "cached" link within |
| content="index,follow"> | | | | Google's search results. If you do not want your |
| This tag will be used when you want the search | | | | content to be archived, you can use the following tag: |
| engine spiders to index the page and follow the links to | | | | <meta name="robots" |
| other pages. Most search engines use this setting as a | | | | content="noarchive"> |
| "default" setting. It is possible that you may | | | | *This will only prevent your page from being |
| not even need to use this tag if you want the search | | | | "cached". If you do not want your page to |
| engines to follow and index the page. However, an | | | | be indexed at all, you will still need to include the |
| article at Search Engine World | | | | "noindex" tag. |
| (searchengineworld.com/metatag/robots.htm) suggests | | | | Another alternative to the above tag is the tag that |
| that Inktomi does not use this as their default setting. | | | | specifically addresses Google only. If you want other |
| Instead, they use the "index, nofollow" tag. | | | | search engine robots to archive your site, but you |
| Better safe than sorry! | | | | would like to prevent Google from doing so, then you |
| There has been much debate over whether or not it is | | | | can use the following tag: |
| necessary to use this tag. If there is even a slight | | | | <meta name="googlebot" |
| possibility that some search engines do not use this as | | | | content="noarchive"> |
| the default setting, then it would only make sense to | | | | The Misuse of Robots Tags |
| include this tag if you want your page included in their | | | | Something that has been popping up on websites |
| search directory AND your links to be followed. Do the | | | | everywhere is the Google indexing tag. This is a silly |
| research and decide for yourself. | | | | little tag that is not necessary. Some people think this |
| 2. <meta name="robots" | | | | tag helps Google to spider your site, but this simply isn't |
| content="noindex,follow"> | | | | true. The tag looks like this: <meta |
| This tag can be used to tell the search engines that | | | | name="googlebot" |
| you do not want the page included in their directory, | | | | content="index,follow">. |
| but you DO want them to follow the links that lead to | | | | Some website owners believe that by specifying |
| other pages. A good example of its usage would be | | | | "googlebot" that their site has the |
| your disclaimer or privacy policy pages. You may not | | | | advantage of being spidered faster and listed by |
| want these pages to show up in the search engines if | | | | Google. According to Google's web crawler |
| they are only important to your actual visitors. | | | | information at you only need to use the noindex, |
| However, if the links on these pages point to other | | | | nofollow, or noarchive tags when you don't want |
| pages that you want the search engines to find, then | | | | Google to cache, index, or follow that page. Google's |
| you would still want the spiders to "follow" | | | | default setting is to index and follow the links on the |
| those links. | | | | page, so this "so called" googlebot index |
| 3. <meta name="robots" | | | | follow tag is completely unnecessary. |
| content="index,nofollow"> | | | | |