OverTheFenceBot
OverTheFenceBot reads public websites for Over the Fence, a service that shows a business how its website compares with its rivals’.
It reads a site’s robots.txt first, then its homepage and sitemap when robots.txt allows. The rest of the site’s public pages are read by our crawling partner, DataForSEO, two seconds apart. We only read pages anyone can see without signing in.
Keeping it off your site
Add these lines to your robots.txt:
User-agent: OverTheFenceBot Disallow: /
We hand the lines you write for OverTheFenceBot to our partner’s crawler with every read, and it follows your rules for every crawler too.
Questions
Write to hello@overthefence.io.