| by: Alan murray | | | | If you use |
| | | | |
| What is the Robot Text File? | | | | User-agent: * |
| | | | |
| The robot text file is used to disallow | | | | The * acts as a wildcard and disallows all |
| specific or all search engine spider's access | | | | spiders. You may want to use this to stop |
| to folders or pages that you don't want | | | | search engines listing unfinished pages. |
| indexed. | | | | |
| | | | To disallow an entire directory use |
| Why would you want to do this? | | | | |
| | | | Disallow:/mydirectory/ |
| You may have created a personnel page for | | | | |
| company employees that you don't want listed. | | | | To disallow an individual file use |
| Some webmasters use it to exclude their guest | | | | |
| book pages so to avoid people spamming. There | | | | Disallow:/file.htm |
| are many different reasons to use the robots | | | | |
| text file. | | | | You have to use a separate line for each |
| | | | disallow. You cannot you for example use |
| How do I use it? | | | | |
| | | | Disallow:/file1.htm,file2.html |
| You need to upload it to the root of your web | | | | |
| site or it will not work - if you don't have | | | | You should use |
| access to the root then you will need to use | | | | |
| a Meta tag to disallow access. You need to | | | | Use-agent/* |
| include both the user agent and a file or | | | | |
| folder to disallow. | | | | Disallow:/file1.htm |
| | | | |
| What does it look like? | | | | Disallow:/file2.htm |
| | | | |
| It's really nothing more than a "Notepad" | | | | For a list of spider names visit |
| type .txt file named "robots.txt" | | | | |
| | | | Make sure you use the right syntax if you |
| The basic syntax is | | | | don't it will not work. You can check you |
| | | | syntax here |
| User-agent: spiders name here | | | | |
| | | | For help on creating robot text files there |
| Disallow:/ filename here | | | | is a program call robogen. |
| | | | |