| Generally in case of small websites webmasters are | | | | more control over the robots and this indicates that |
| under false assumption that they do not require to | | | | you can issue indexing instructions to various search |
| create a robot.txt file. But it is required. First of all let us | | | | engines. |
| define the robot.txt file. Even prior to it we need to | | | | Robots.txt invites the search engines. Some of the |
| define what a web robot is. A web robot is also called | | | | good bots can also step away from your website in |
| spider or crawler which should not be confused with | | | | case you have not created robot.txt in the top level of |
| the normal web browser as web browser is not a | | | | your website. Some time there is requirement to |
| robot. | | | | exclude some pages from search engines. These are |
| The main use of robots.txt file for webmasters in SEO | | | | those pages that are still under construction and those |
| services is to give instructions to the robot what they | | | | directories that you do not want to get them indexed. |
| crawl and what should not be crawled. This can give | | | | You may also want to exclude those search engines |
| you some control over the robots. This give you little | | | | whose main aim is to collect email addresses. |