How to setup your Robots file

Step 1

Login to the CMS > Go to Robots TXT in the node tree.

Step 2

Enter the relevant information in the field 'Crawl Instructions' provided. If not set already, assign the Sitemap XML Location using the picker provided.

Step 3

If you are using the Robots TXT node to control your robots.txt file you will need to add the following rewrite rule to your web.config file:

<rewrite>
 <rules>
  <rule name="Robots TXT" stopProcessing="true">
  <match url="robots.txt" />
   <action type="Rewrite" url="robots-txt" />
  </rule>
 </rules>
</rewrite>

IIS URL Rewrite module must be installed on your webserver to use this feature.