For website owners
AdnaveBot, our website crawler
If you found AdnaveBot in your server logs, this page explains what it was doing and how to control it.
What is AdnaveBot?
AdnaveBot is the crawler behind Adnave's website audits. It reads public web pages to check them for search, technical and speed problems, and reports what it finds to whoever asked for the audit.
Why did it visit my website?
Either someone ran an audit of your site, or your site is one of the world's most popular, which we audit about once a month for our public reports. A visit reads the page entered, up to 30 more pages, and checks up to 60 links.
How do I recognise it?
It identifies itself in the User-Agent header as: Mozilla/5.0 (compatible; AdnaveBot/1.0; +https://adnave.com/bot)
Does it follow robots.txt?
Yes. It follows rules written for AdnaveBot by name, and otherwise the general rules for all crawlers. It honours a Crawl-delay of up to three seconds, and at most 40 audits of one site are allowed per day across all our users.
How do I block it or slow it down?
Add this to your robots.txt to block it entirely:
User-agent: AdnaveBot
Disallow: /
To slow it down instead, add a line such as: Crawl-delay: 3
Does it store my content?
No page content is kept. An audit stores findings, such as a missing title, and a list of the words the pages emphasise.