Loading the figures…
Does your site pass?
Run the same audit on your own website, free.
AI assistants allowed in: how the world's top websites do
Of 551 of the world's most popular websites that we audited, 436 pass this check and 115 fall short.
79%
pass
436 of 551 sites
21%
fall short
115 of 551 sites
30 of 51
by how often it fails
1 is the most-failed check · area: AI search
What this check means
AI assistants use separate crawlers for training and for search. Blocking a search crawler in robots.txt stops that assistant fetching pages to cite; blocking a training crawler keeps pages out of model training only.
Guide: AI search readiness: how to be found by AI assistants
Popular sites that fall short (115)
facebook.comfbcdn.nettwitter.comlinkedin.comamazon.commail.rudzen.ruwhatsapp.netx.comdoubleclick.netpinterest.commyfritz.nettiktok.comwhatsapp.comyahoo.comwa.mespotify.comamazonvideo.comvimeo.combaidu.commsn.comt-online.dechatgpt.comyandex.ruregistrar-servers.comskybridge.clicknytimes.comsnapchat.comgravatar.comgoogle-analytics.comflickr.comcnn.comsoundcloud.comtheguardian.combbc.combbc.co.ukt.coimdb.comlaunchpad.netnature.comspringer.comalibaba.comwiley.comweibo.comamazon.co.ukespn.comnaver.comsmartadserver.comchatgpt.siteamazon.dego.comglobo.comaol.comnetcraze.ioyahoo.co.jpgoodreads.comcnbc.combusinessinsider.comscribd.comamazon.co.jpstatista.comamzn.toslideshare.netwired.comthreads.comsohu.comdailymail.comuol.com.brunesco.orgtechcrunch.comnintendo.comamazon.framazon.cahstgr.netamazon.inprimevideo.comspiegel.dekick.comcbsnews.comgosuslugi.rulemonde.frbandcamp.comfb.comglobalsign.comamazon.espinimg.comdailymail.co.uksharethrough.comamazon.ithihonorcloud.comcnet.combild.dewattpad.comnikkei.comwelt.dewp.plted.comairbnb.comimgsmail.rutheverge.comdaum.netamazon.com.brnbcnews.commercadolivre.com.brhbr.orgfigma.comlatimes.comamazonalexa.comrepubblica.itpbs.orgcorriere.ithuffpost.comamazon.com.aumercadolibre.com.arn-tv.de
Popular sites that pass (436)
google.comcloudflare.comamazonaws.comyoutube.commicrosoft.comapple.comazure.comtriplinkintl.comgooglevideo.comoffice.comlive.comgithub.comlsrelayaccess.comcloud.microsoftwikipedia.orgfastly.netnetflix.combing.comgooglesyndication.comyoutu.bewindows.neticloud.comzoom.usadobe.comwordpress.orggandi.netgoogledomains.comskype.comdigicert.comopera.comb-cdn.netroblox.comntp.orgworkers.devblogspot.comqq.commozilla.orgnginx.orgf5.comwordpress.comt.mesentry.iolencr.orgnginx.comyandex.netui.comeuropa.eubit.lysamsung.comadobe.ioipify.orggithub.iodropbox.comandroid.comwindows.comcloudflare-dns.comduckdns.orgapache.orgdiscord.ggarchive.orgvk.runic.rutelekom.netcrashlytics.comvk.comtelegram.orgsamsungcloud.comubuntu.commeraki.comshopify.compages.devastrumu.ioexample.comcreativecommons.orgforter.comdiscord.comscorecardresearch.comw3.orgreg.ruquickconnect.tokaspersky.compki.googbrave.comunity3d.comdoi.orghcaptcha.comafternic.comstripe.comoracle.compublicnode.comroku.comwikimedia.orgmts.ruweather.comamplitude.comcriteo.comtwitch.tvmit.edunist.govcharter.netapplovin.comcisco.comadtrafficquality.googlesourceforge.netdnsmadeeasy.comsalesforce.comcpanel.nethubspot.comlinktr.eegoogleblog.comwho.intalidns.comdatadoghq.comcentinel-analytica.comslack.comhp.comlaunchdarkly.comtelegram.mebooking.comwww.gov.uksteampowered.comtinyurl.comgamepass.comonelink.meip-api.comgrammarly.comcloudflareinsights.commiui.comcasalemedia.comanthropic.comzendesk.comdebian.orgvungle.comuserapi.comharvard.edurubiconproject.comnflxvideo.netphp.netunisys.compaloaltonetworks.comok.ruwithgoogle.comea.combilibili.comokta.combranch.ionvidia.comgnu.orgtradingview.comgmail.com
And 286 more. The full list is in the data file.
How this was measured
The figure comes from Adnave's own audits of 551 popular websites, each read as a search engine first receives it and re-checked about once a month. A site “falls short” when the result is anything other than a pass. That is not always a mistake: a very large site may skip something on purpose. Sites that refuse automated visits are not in the sample, so the figure describes the popular sites that could be read, not the whole web. Full method and limits
Put this figure on your own page
The picture is drawn fresh from our data, so it stays up to date wherever you place it. Free to use; the code links back to this page as the source.
<a href="https://adnave.com/research/stats/ai-crawlers"><img src="https://adnave.com/embed/ai-crawlers.svg" alt="Share of the world's top websites that pass the check “AI assistants allowed in”. Live figure by Adnave." width="480" height="200" loading="lazy"></a>
Use this figure
Free to quote, chart and republish with a link to this page. Check the page before quoting: the figure moves as sites are re-audited.
Suggested citation: Adnave, “AI assistants allowed in: how the world's top websites do”, https://adnave.com/research/stats/ai-crawlers
Download the data (CSV)