| In my last article about the file I had
| |
| | Here are some examples of a file
|
| spelled it wrong. It should have been
| |
| | User-agent: *
|
| instead of . The article should read like
| |
| | Disallow:
|
| this:
| |
| | The above would let all spiders index
|
| When optimizing your web site most
| |
| | all content.
|
| webmasters don't consider using the s is
| |
| | Here another example
|
| a very important file for your site. It
| |
| | User-agent: *
|
| let the spiders and crawlers know what
| |
| | Disallow: /cgi-bin/
|
| they can and can not index. This is
| |
| | The above would block all spiders from
|
| helpful in keeping them out of folders
| |
| | indexing the cgi-bin directory.
|
| that you do not want index like the admin
| |
| | User-agent: googlebot
|
| or stats folder.
| |
| | Disallow:
|
| Here is a list of variables that you can
| |
| | User-agent: *
|
| include in a file and there meaning:
| |
| | Disallow:
|
| 1) User-agent: In this field you can
| |
| | Disallow: /cgi-bin/
|
| specify a specific robot to describe
| |
| | Disallow: /admin/
|
| access policy for or a "*" for all robots
| |
| | Disallow: /stats/
|
| more explained in example.
| |
| | In the above example googlebot can index
|
| 2) Disallow: In the field you specify
| |
| | everything while all other spiders can
|
| the files and folders not to include in
| |
| | not index , cgi-bin, admin, and stats
|
| the crawl.
| |
| | directory. Notice that you can block
|
| 3) The # is to represent comments
| |
| | single files like .
|
