| The results web users get when they type in words | | | | content. That information is then entered into its large |
| or phrases that are associated with your blog or | | | | database to be processed into the search ranks you |
| website is one of the best and cheapest ways to gain | | | | see when you type in the terms you're searching for |
| traffic (if you aren't worried about gaining traffic to | | | | (i.e.: "how do I bake the best chocolate chip cookies"). |
| your site you can stop reading now). New users to | | | | The "User-agent" tells what specific search engine bot |
| Wordpress and websites in general may not be | | | | gets the instructions (typically you leave it as * to |
| familiar with many of the terms and tools used by the | | | | signify all search engines). |
| professionals to create and maintain a website, | | | | In this example, we tell all search engines to follow all |
| especially when it comes to SEO and search engine | | | | the directories it can find and index them: |
| ranking.* SEO, or search engine optimization can seem | | | | User-agent: * |
| like voodoo magic to the uninitiated users. Since no one | | | | Disallow: |
| except Google (or any search engine) knows exactly | | | | Here, we tell all search engines to not index anything |
| how the searches are ranked and computed, the best | | | | (useful if you want your site to be as private as |
| we can do is some trial and error to see what works | | | | possible and do not care about gaining traffic): |
| best. There is no one formula that always works. A | | | | User-agent: * |
| practical approach for the uninitiated webmaster is | | | | Disallow: / |
| what is needed. A properly configured robots.txt file will | | | | This is an example of how a typical Wordpress blog |
| help, and is easy to configure once you know what to | | | | should be configured (depending on what plugins and |
| look for. | | | | content you have there can be variations, but the |
| Wordpress by default has a virtual robots.txt file that's | | | | concept remains the same): |
| created automatically. This file, however, is not setup | | | | User-agent: * |
| properly (as far as SEO is concerned). Ordinarily you | | | | Disallow: /wp-admin |
| would create your own robots.txt file and place it in the | | | | Disallow: /wp-includes |
| root of your site files that make up your website in the | | | | Disallow: /wp-content |
| web server. But, since Wordpress does this | | | | Disallow: /feed/ |
| automatically with a virtual file, we must disable that | | | | Disallow: /tag/ |
| feature first. | | | | Disallow: /author/ |
| Download the robots.txt plugin (just do a google | | | | Disallow: /comments/ |
| search for kb-robotstxt for wordpress) and activate it. | | | | Disallow: /archive/ |
| In the control panel for the plugin, we can add or | | | | The main purpose of this configuration is to protect |
| remove the options and lines of code that make up | | | | any admin or core Wordpress files and folders from |
| the file. It's these lines that will help to make our site | | | | accidentally being indexed (this can be a security issue) |
| more search engine friendly. | | | | and to help reduce the chances of the search engines |
| The two things you must concern yourself with in the | | | | regarding duplicate content (which can hurt your page |
| code (see examples below) are pretty | | | | rank and search results). For example, you may have |
| straightforward: | | | | enough posts on your site to have one or many |
| 1) What parts of my site do I want the search engines | | | | archive pages for users to go back and see the |
| to crawl | | | | previous old posts. The search engines may see the |
| 2) What parts of my site do I not want the search | | | | original post and then the post from the archive page |
| engines to crawl | | | | as duplicate content. In the long run, this can hurt your |
| By crawling your site with what's called a "bot", the | | | | search results (Google sees this as a sort of cheat |
| search engines comb over your site finding links and | | | | and penalizes you for it). |