I learned today, via Search Tools, that Microsoft, Yahoo! and Google have agreed to specific extensions to the robots.txt file protocol. All of their search engines will now honor additional directives. More info from Yahoo! and Google.

What is robots.txt some of you may be asking? It is a simple text file you can place on your web site to tell search engine spiders what parts of your site they should index and which they should ignore. It has been around for a long time and these are the first additions to the standard in at least a decade.