| by: Alan murray
| |
| | Disallow:/ filename here
|
| What is the Robot Text File?
| |
| | If you use
|
| The robot text file is used to disallow
| |
| | User-agent: *
|
| specific or all search engine spider's
| |
| | The * acts as a wildcard and disallows
|
| access to folders or pages that you don't
| |
| | all spiders. You may want to use this to
|
| want indexed.
| |
| | stop search engines listing unfinished
|
| Why would you want to do this?
| |
| | pages.
|
| You may have created a personnel page for
| |
| | To disallow an entire directory use
|
| company employees that you don't want
| |
| | Disallow:/mydirectory/
|
| listed. Some webmasters use it to exclude
| |
| | To disallow an individual file use
|
| their guest book pages so to avoid people
| |
| | Disallow:/file.htm
|
| spamming. There are many different
| |
| | You have to use a separate line for each
|
| reasons to use the robots text file.
| |
| | disallow. You cannot you for example use
|
| How do I use it?
| |
| | Disallow:/file1.htm,file2.html
|
| You need to upload it to the root of your
| |
| | You should use
|
| web site or it will not work - if you
| |
| | Use-agent/*
|
| don't have access to the root then you
| |
| | Disallow:/file1.htm
|
| will need to use a Meta tag to disallow
| |
| | Disallow:/file2.htm
|
| access. You need to include both the user
| |
| | For a list of spider names visit
|
| agent and a file or folder to disallow.
| |
| | Make sure you use the right syntax if you
|
| What does it look like?
| |
| | don't it will not work. You can check you
|
| It's really nothing more than a "Notepad"
| |
| | syntax here
|
| type .txt file named "robots.txt"
| |
| | For help on creating robot text files
|
| The basic syntax is
| |
| | there is a program call robogen.
|
| User-agent: spiders name here
| |
| |
|
