Type a query, get results in under a second. Between those two moments sits a system Google has spent 25 years building, and most SEO confusion starts with guessing at how it works. So I’ll skip the guessing. Every claim in this guide is checked against Google’s own documentation, and I’ll say so when something is my reading rather than their statement.
Google describes 3 stages: crawling (finding and downloading pages), indexing (analysing and storing them) and serving (matching them to queries). Each stage can fail quietly. A page with no links pointing at it may never be crawled. A crawled page can be judged too thin to store. An indexed page can rank on page 9 forever. When you diagnose a visibility problem, your first job is working out which stage it died at.
Stage 1: crawling
Googlebot works from an enormous, constantly updated list of known URLs. New URLs join that list mainly through links from pages Google already knows, plus XML sitemaps and manual submission. It fetches pages from that list on a schedule Google sets algorithmically, per site, balancing how much it wants your content against how much load your server can take.
Fetching the HTML is half the job. Google then renders the page in a recent version of Chrome and runs the JavaScript it finds, which is how content that only exists after scripts run still gets seen. Rendering usually follows soon after the fetch, though it can lag, and anything critical that only appears via JavaScript carries a bit more risk. I cover the mechanics in the crawling and indexing guide.
Two things worth stating because people pay money based on the opposite belief. Crawl frequency reflects demand and capacity, so getting crawled often is a description of your site’s freshness and popularity rather than a reward you can chase. And Google’s docs state plainly that it doesn’t accept payment to crawl a site more often or rank it higher. Ads buy ads. That’s all they buy.
Stage 2: indexing
After rendering, Google analyses what the page is: the text, the images, the video, the language it’s written in, where it’s aimed, whether it’s usable. It also groups duplicates and near-duplicates together and picks one version, the canonical, to represent the group in results.
Then comes the sentence that explains half the anguish in SEO forums, quoted from Google’s documentation: “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Google crawls far more than it keeps. Pages that add nothing beyond what’s already indexed are the usual casualties, and ‘Crawled, currently not indexed’ in Search Console is where they show up. That status gets its own section in the companion guide because it’s the one that generates the most panic and needs the least of it.
Stage 3: serving and ranking
When someone searches, Google pulls candidate pages from the index and orders them using hundreds of signals, weighted by context like location, language and device. This stage is where all the named machinery lives: the core ranking systems, the spam systems, the freshness systems, the reviews system. I walk through each one in my guide to Google’s ranking systems, and the update history tracks how they’ve changed over time.
The part worth internalising: ranking is query-dependent. Your page holds no fixed rank. It gets scored against a specific query, from a specific place, on a specific day. That’s why rank tracking is a sample, and why two people rarely see quite the same results.
Where AI Overviews fit
AI Overviews sit on top of this pipeline rather than replacing it. The summary is generated from search results, so a page has to be crawled, indexed and ranked before it can be cited in one. The pipeline still decides who’s in the room; the Overview decides who gets quoted. What that does to clicks is a moving story, and I keep the data in my AI Overviews guide rather than here, because it changes faster than the fundamentals do.
Six common claims, checked
- “Submitting a sitemap gets you indexed.” Sitemaps help discovery. Indexing stays unguaranteed regardless of how the URL was found. Docs are explicit on this.
- “Google crawls my site daily, so it must like me.” Crawl rate tracks how often your content changes and how much demand exists for it. It says little about ranking.
- “Buying Google Ads helps organic rankings.” The documentation rules this out in plain words. In 20 years I’ve never seen credible data showing otherwise.
- “Meta keywords matter.” Google has ignored the meta keywords tag since 2009 and said so publicly.
- “Fresh content always wins.” Freshness systems apply to queries that deserve fresh results. For evergreen queries, the better page wins whether it’s 3 weeks or 3 years old.
- “Everything gets indexed eventually.” The docs say the opposite, and index selectivity has visibly tightened in recent years.
What this means for your site
- Make every page you care about reachable by links from your own site. Discovery starts at home; internal linking is the lever you fully control.
- Keep critical content and links in the server-rendered HTML where you can.
- Give Google a reason to keep the page: something on it that isn’t already in the index in a better form.
- When a page underperforms, find the failing stage before touching anything. Search Console will tell you whether the problem is crawling, indexing or ranking, and the fixes for each barely overlap.
Sources
How Google Search works and the in-depth version, both Google Search Central. Claims about payment, rendering and indexing guarantees come from those pages. Opinions about what deserves your attention are mine.