How to setup your Robots file
Login to the CMS > Go to Robots TXT in the node tree.
Enter the relevant information in the field 'Crawl Instructions' provided. If not set already, assign the Sitemap XML Location using the picker provided.
If you are using the Robots TXT node to control your robots.txt file you will need to add the following rewrite rule to your web.config file:
<rule name="Robots TXT" stopProcessing="true">
<match url="robots.txt" />
<action type="Rewrite" url="robots-txt" />
IIS URL Rewrite module must be installed on your webserver to use this feature.