Google provides Robots.txt file generator for web publishers

Google provides Robots.txt file generator for web publishers

Search engine giant Google has made available a new service through its Webmaster Tools.

The web publishers would now be able to use this service to generate robots.txt file for their web sites.

The company said that the tool has been designed to give the publishers an easy and interactive way to build a robots.txt file.

They are also providing a robots.txt analysis tool to check the effect of the resulted robots.txt file.

The company highlighted some points about this file in a blog post:

Not every search engine will support every extension to robots.txt files
Robots.txt is simply a request

Checkout: Speaking the language of robots