| ext"> | | | | robots.txt to block any inquiries on speeches by the |
| The Robots.txt protocol, also called the | | | | Vice President, a photo essay of the First Lady, and |
| ârobots exclusion standardâ is | | | | profiles of the 911 victims. |
| designed to lock out web spiders from accessing part | | | | How does the protocol work? It lists the files that |
| of a website. It is a security or privacy measure, the | | | | shouldnât be scanned, and places it in the |
| equivalent of hanging a âKeep Outâ | | | | top-level directory of the website. The robots.txt |
| sign on your door. | | | | protocol was created by consensus in June 1994 by |
| This protocol is used by web site administrators when | | | | members of the robots mailing list |
| there are sections or files that they would rather not | | | | (robots-request@nexor.co.uk). There is no official |
| be accessed by the rest of the world. This could | | | | standards body or RFC for the protocol, so |
| include employee lists, or files that they are circulating | | | | itâs difficult to legislate or mandate that the |
| internally. For example, the White House website uses | | | | protocol be followed. |