AI crawler policy tracker
Every day we re-read the rules of the world's most popular websites and record which ones start, or stop, blocking AI crawlers. This page is the log.
Loading the log…
How this is measured
We read each site's robots.txt file, the public file where a site states which crawlers may read it. A crawler counts as blocked when it is disallowed from the whole site. A change is published only after a second reading, hours later, gives the same answer, so a site that is briefly unreachable does not appear as a change of policy. If a rules file disappears from a site that had blocks, we treat it as a fault and wait.
Limits: this shows what a site asks of crawlers, not what its firewall does, and not whether a crawler obeys. Sites that refuse our own crawler are not watched. Tracking is recent, so the log starts short. Full method