| What is the Robot Text File? | | | | User-agent: * |
| The robot text file is used to disallow specific or all | | | | The * acts as a wildcard and disallows all spiders. You |
| search engine spider's access to folders or pages that | | | | may want to use this to stop search engines listing |
| you don't want indexed. | | | | unfinished pages. |
| Why would you want to do this? | | | | To disallow an entire directory use |
| You may have created a personnel page for | | | | Disallow:/mydirectory/ |
| company employees that you don't want listed. Some | | | | To disallow an individual file use |
| webmasters use it to exclude their guest book pages | | | | Disallow:/file.htm |
| so to avoid people spamming. There are many | | | | You have to use a separate line for each disallow. |
| different reasons to use the robots text file. | | | | You cannot you for example use |
| How do I use it? | | | | Disallow:/file1.htm,file2.html |
| You need to upload it to the root of your web site or it | | | | You should use |
| will not work - if you don't have access to the root | | | | Use-agent/* |
| then you will need to use a Meta tag to disallow | | | | Disallow:/file1.htm |
| access. You need to include both the user agent and | | | | Disallow:/file2.htm |
| a file or folder to disallow. | | | | For a list of spider names visit |
| What does it look like? | | | | Make sure you use the right syntax if you don't it will |
| It's really nothing more than a "Notepad" type .txt file | | | | not work. You can check you syntax here |
| named "robots.txt" | | | | For help on creating robot text files there is a program |
| The basic syntax is | | | | call robogen. |
| User-agent: spiders name here | | | | There is a free version and an advanced version, |
| Disallow:/ filename here | | | | which costs $12. |
| If you use | | | | |