Robots.txt Protocol Enhanced by Big Search Engine Companies

I learned today, via Search Tools, that Microsoft, Yahoo! and Google have agreed to specific extensions to the robots.txt file protocol. All of their search engines will now honor additional directives. More info from Yahoo! and Google.

What is robots.txt some of you may be asking? It is a simple text file you can place on your web site to tell search engine spiders what parts of your site they should index and which they should ignore. It has been around for a long time and these are the first additions to the standard in at least a decade.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s