| Understanding the way a search engine crawler | | | | content. It is important to build your site in a way so the |
| indexes pages and the specific algorithm elements | | | | crawlers understand the most important to least |
| involved in their programming is key to determining | | | | important information. |
| which optimization techniques to use. The algorithms | | | | Some ways to control crawlers: |
| use a combination of page content and structure, | | | | - .htaccess - Use to set up allowed Server Side |
| loading time, and analysis of inbound links to determine | | | | Includes and 301 Redirects. |
| the page rank for keywords and phrases. To achieve | | | | - sitemap.xml - Set up and verify with Google |
| the best results with search engines all aspects of the | | | | Webmaster Tools. Its fast, simple, and easy to |
| algorithm are needed. | | | | schedule crawler visits and page importance. These |
| There are many methods for controlling the way | | | | tools also offer a way to analyze how the crawler |
| search engines will crawl and index your pages. I like to | | | | sees a website. |
| think of search engine crawlers reading a web page | | | | - robot.txt - Set up to allow and disallow access to |
| like a human would read a book. First to find out what | | | | certain pages. |
| the book is about, a person would view the page title | | | | - Set up easy to follow or nofollow links - Search |
| and summary (page description) of the book. Next a | | | | engine robots use links to crawl the world wide web. |
| person would examine the chapters (page topics and | | | | Both inbound and outbound links should be properly |
| titles) and next the headlines and then the paragraph | | | | structured. |