| ternet surfers still hold on to the mistaken belief that | | | | by the search engine spider. Submissions can be |
| actual people visit each and every website and then | | | | optional though because most spiders will be able to |
| input it for inclusion in the search engine's database. | | | | find the content in a web page if other websites link to |
| Imagine, if these were true! With billions of websites | | | | the page. This is the reason why it is a good idea to |
| available on the internet and with a majority of these | | | | build reciprocal links with other website. By enhancing |
| sites offering fresh content it will take thousands of | | | | the link popularity of your website and getting links |
| people to achieve the tasks made by search engine | | | | from other sites that have the same topic as your |
| spiders and robots - and even then they won't be as | | | | website. |
| efficient or as thorough. | | | | When the search engine spider robot visits the |
| Search engine spiders and robots are pieces of code | | | | website, it checks if there is an existing robots.txt file. |
| or software that have only one aim - seek content on | | | | The file tells the robot which areas of the site are off |
| the internet and within each and every individual web | | | | limits to its probe - like certain directories that have no |
| page out there. These tools have a very important | | | | use for search engines. All search engine bots look for |
| role in how effectively search engines operate. | | | | this text file so it is a good idea to put one even if it is |
| Search engine spiders and robots visit websites and | | | | blank. |
| get the necessary information that it needs to | | | | The robots list and store all of the links found on a |
| determine the nature and content of the website and | | | | page and they follow each link to its destination |
| then adds the data to the search engine's index. | | | | website or page. |
| Search engine spiders and robots follow links from one | | | | The robots then submit all of this information to the |
| website to another so that it can consistently and | | | | search engine, which in turn compiles the data received |
| infinitely gather the necessary information. The ultimate | | | | from all the bots and builds the search engine |
| goal of search engine spiders and robots is to compile | | | | database. This part of the process already has the |
| a comprehensive and valuable database that can | | | | intervention of search engine engineers who write the |
| deliver the most relevant results to the search queries | | | | algorithms employed in evaluating and scoring the |
| of visitors. | | | | information that the search engine bots compiled. The |
| But how exactly do search engine spiders and robots | | | | moment all of the information is added to the search |
| work? | | | | engine database this information is already made |
| The whole process begins when a web page is sent | | | | available to search engine visitors who are making |
| to a search engine for submission. The submitted URL | | | | search queries in the search engine. |
| is added to the queue of websites that will be visited | | | | |