| If you have a website you really need to have a | | | | Disallow: |
| robots.txt file. It gives search engine spiders specific | | | | The first line tells all spiders to listen up because the |
| commands and it is easy to use and easy to maintain. | | | | following command is for you. The second line means |
| Here is an easy guide to a robots.txt file in five | | | | do not index any of the following pages. And it is here |
| minutes. | | | | you put the url of any pages you don't want spidered. |
| There are times when you don't want a search engine | | | | So if you wanted the spiders to skip your private page |
| to index a page or a folder on your website. Maybe | | | | it looks like this: |
| you have some information you just don't want to | | | | Disallow:/privatepage.htm |
| have show up in google. This may include your | | | | If you want the spiders to skip a whole folder you put |
| statistics page, a page of notes, or a dynamic page. | | | | the url of that folder with a slash like this: |
| And, importantly, if you use google adsense and the | | | | Disallow:/privatefolder/ |
| search tool that displays search results on your | | | | Simply place this text file in the root folder of your |
| website google mandates you exclude this page from | | | | website and you are done. In the future you can add |
| search engines. Which means they mandate you | | | | and remove commands easily. |
| having a robots.txt file. | | | | The robots.txt file is a very easy file to write and |
| A robots.txt file is a simple document named robots.txt | | | | maintain and it is a very powerful tool that will help you |
| and saved in the root folder of your website. Search | | | | interact successfully with search engines. This disallow |
| engines see this and follow any commands it contains. | | | | command is the simplest and most used command but |
| Create a simple text document using any word | | | | there are also many other commands you can use |
| processor program like notepad and put these two | | | | and if you have a website it is well worth your time to |
| lines it: | | | | have a robots.txt file and even to research it a bit |
| User-agent: * | | | | further. |