seo toolkit - request is disallowed by a robots txt rule

Question

I am trying to run the SEO toolkit IIS extension on an application I have running but I keep getting the following error:

The request is disallowed by a Robots.txt rule

Now I have edited the robots.txt file in both the application and the root website so they both have the following rules:

User-agent: *
Allow: /

But this makes no difference and the toolkit still won't run.

I have even tried deleting both robots.txt files and that still doesn't make any difference.

Answer 1

actually robot.txt dont allow google to crawl on pages thats forbidden by admin so google ignores that pages
and such pages are basically not ranked
Check to make sure the DNS record is pointing to the correct server
If you're searching for the file, account for case sensitivity - robots.txt vs Robots.txt
Verify that the Toolkit is actually attempting to visit the site. Check the IIS logs for the presence of the "iisbot" user-agent.