What does robots.txt do?
It is a text file at the root of your site that tells crawlers which sections they may visit. It is advice about crawling, not a lock: a blocked page can still be listed if other sites link to it, just without a description.
The most damaging mistake is a leftover “Disallow: /” from when the site was being built, which shuts every crawler out of everything.
What is a sitemap and do I need one?
A sitemap is a list of the pages you want found, usually at /sitemap.xml. Search engines can discover pages by following links, but a sitemap makes sure new and deep pages are found quickly. Most website platforms generate one automatically. Name it in robots.txt and submit it in Google Search Console.
What is noindex?
A tag or header that tells search engines not to list a page at all. It is the right choice for carts, login pages and thank-you pages. It is a serious problem when it lingers on a page you want customers to find, and it is easy to miss because the page looks perfectly normal to visitors.
What is a canonical tag?
It names the preferred address for a page. The same page is often reachable at several addresses, for example with tracking codes on the end. Without a canonical tag, search engines may split the page's standing between those copies.
Why is my page not on Google?
Check these in order.
- Is it blocked by robots.txt or marked noindex?
- Is it in the sitemap, and linked from at least one other page?
- Does it answer with a normal page, not an error or a redirect?
- Is it new? Indexing can take days or weeks.
- Is it too thin or too similar to another page to be worth listing?