| When optimizing your web site most webmasters | | | | The above would let all spiders index all content. |
| don't consider using the robot.txt file. This is a very | | | | Here another |
| important file for your site. It let the spiders and | | | | User-agent: * |
| crawlers know what they can and can not index. This | | | | Disallow: /cgi-bin/ |
| is helpful in keeping them out of folders that you do not | | | | The above would block all spiders from indexing the |
| want index like the admin or stats folder or content | | | | cgi-bin directory. |
| that they can not index. | | | | User-agent: googlebot |
| Here is a list of variables that you can include in a | | | | Disallow: |
| robot.txt file and there meaning: | | | | User-agent: * |
| 1)User-agent: In this field you can specify a specific | | | | Disallow: /admin.php |
| robot to describe access policy for or a "*" for all | | | | Disallow: /cgi-bin/ |
| robots more explained in example. | | | | Disallow: /admin/ |
| 2)Disallow: In the field you specify the files and folders | | | | Disallow: /stats/ |
| not to include in the crawl. | | | | In the above example googlebot can index everything |
| 3)# the number sign represents comments | | | | while all other spiders can not index admin.php, cgi-bin, |
| Here are some examples of a robot.txt file for | | | | admin, and stats directory. Notice that you can block |
| User-agent: * | | | | single files like admin.php. |
| Disallow: | | | | |