| What is website indexability, you ask? Basically it | | | | the robots.txt file is to tell friendly SEO crawlers which |
| refers to how easily and efficiently Internet search | | | | pages they should ignore while indexing the site. This is |
| engines are able to read the pages on a website and | | | | helpful in the case of an infinite domain space. An |
| determine their ranking. One option for doing this is to | | | | example of an infinite domain namespace might be |
| add a robots.txt file. By adding the file to the root of | | | | one where users upload files into an online document |
| the particular website, it is instructing an SEO robot | | | | repository. These documents are considered to be |
| which pages not to index. This ensures that the most | | | | media and not content so the webmaster should add |
| relevant pages get indexed. | | | | a line to the robots.txt file to disallow access to the |
| Not all SEO robots will read a robots.txt file. Most | | | | root URL for this document repository. The more |
| malware crawlers will unfortunately ignore it as their | | | | results returned from a site with the same URL will |
| purpose is malicious. They do not care if they are only | | | | often degrade the result ranking of a page. |
| allowed access to certain pages. The main purpose of | | | | |