| Fundamentally, we like the content of our websites to | | | | important to put the file on the proper directory for |
| be indexed immediately so that traffic will be driven | | | | engines look first in the main directory like Once the |
| and search engine ranking will be improved. But in | | | | user agent or the search engine was not able to find |
| some situations, a file or online tool is used to hide the | | | | the file, they will think that all the web pages should be |
| pages and personal files we have in our website. | | | | indexed. |
| The most useful and popular way of hiding the files | | | | How does the Robots.txt look like? |
| from search engines is using the Robots.txt file. | | | | Robots.txt has a really plain structure which is only |
| Actually robots meta tag is good too but some | | | | composed of user agent and disallowed directories |
| engines cannot read meta tags. If Robots.txt is used, all | | | | and files. The term used for search engine is "user |
| search engines will be notified. | | | | agent" and "disallow" for the lists of files and directories |
| Robots.txt is also referred as Robot Exclusion | | | | you wanted to exclude from indexing. If you want to |
| Standard or Robot Exclusion Standard. It was | | | | insert comments, put # sign at the start of the line. |
| developed in 1994 and it was only popular on Lycos | | | | Robots.txt is one of the many tools for your website. |
| and AltaVista. It is a text file as what the extensions | | | | Having a personal website spells fun! You can benefit |
| says and not an HTML tool. Robots.txt can help you | | | | from the advancements in the technology world. If you |
| from preventing the search machines from crawling | | | | got one, don't forget to deliver your audience quality |
| search engines, but like any other tools, Robots.txt can | | | | content to drive site traffic. Contact an organic SEO |
| be unreliable at times so don't rely your top secret | | | | company for clean and proper SEO techniques if you |
| information to a text file. | | | | want to optimize your website for the target niche. |
| Put the Robots.txt file in the main directory. It is | | | | |