| In my last article about the file I had spelled it wrong. It | | | | Disallow: |
| should have been instead of . The article should read | | | | The above would let all spiders index all content. |
| like this: | | | | Here another example |
| When optimizing your web site most webmasters | | | | User-agent: * |
| don't consider using the s is a very important file for | | | | Disallow: /cgi-bin/ |
| your site. It let the spiders and crawlers know what | | | | The above would block all spiders from indexing the |
| they can and can not index. This is helpful in keeping | | | | cgi-bin directory. |
| them out of folders that you do not want index like the | | | | User-agent: googlebot |
| admin or stats folder. | | | | Disallow: |
| Here is a list of variables that you can include in a file | | | | User-agent: * |
| and there meaning: | | | | Disallow: |
| 1) User-agent: In this field you can specify a specific | | | | Disallow: /cgi-bin/ |
| robot to describe access policy for or a "*" for all | | | | Disallow: /admin/ |
| robots more explained in example. | | | | Disallow: /stats/ |
| 2) Disallow: In the field you specify the files and folders | | | | In the above example googlebot can index everything |
| not to include in the crawl. | | | | while all other spiders can not index , cgi-bin, admin, and |
| 3) The # is to represent comments | | | | stats directory. Notice that you can block single files |
| Here are some examples of a file | | | | like . |
| User-agent: * | | | | |