Ever wondered how Google decides which version of a webpage to show in search results when multiple versions exist? The process, often called “canonicalization,” is much more complex than simply choosing the page with the rel=”canonical” tag. In a recent episode of Google’s Search Off the Record podcast, Google Search engineers Martin, John, and special guest Allan Scott shed light on this intricate system.
Canonicalization: A Two-Step Process
Allan explained that canonicalization is often misunderstood as a single magic box. In reality, it’s a two-step process:
- Clustering: Grouping pages that are considered duplicates or near-duplicates.
- Canonical Selection: Choosing the best representative (canonical) URL from each cluster.
Often, what seem like canonicalization problems are actually clustering issues. If two pages shouldn’t be grouped together in the first place, then the “wrong” canonical being chosen is a moot point.
Beyond rel=”canonical”: The 40 Signals While rel=”canonical” is a strong signal for both clustering and canonical selection, it’s just one of around 40 signals Google considers.
These signals range from the obvious (HTTP vs. HTTPS) to the more nuanced (sitemaps, PageRank, and even the now-deceased “Redirect to Shorter” signal). The complexity arises because website owners often provide conflicting signals (e.g., a 301 redirect to one URL and a rel=”canonical” to another). These conflicting signals make it difficult for Google’s systems to determine the correct canonical URL.
The Localization Iceberg
Localization adds another layer of complexity. Google differentiates between boilerplate translations (like social media feeds) and full translations. Full translations should not be clustered together, while boilerplate translations should be. This helps consolidate signals and avoid unnecessary crawling. However, cases like price differences on otherwise identical product pages present a unique challenge.
Hreflang tags play a crucial role in localization, helping Google serve the correct language variant to users. Google is working on improving its use of hreflang, including verifying site accuracy to increase trust and expand reach. The x-default hreflang tag is also a signal for canonical selection, acting as a fallback when a specific locale can’t be determined.
The Black Hole of Error Pages
A particularly interesting problem is the “marauding black hole” of error pages. Identical error pages (often serving a 200 OK status code) can cluster together, preventing them from being recrawled even after the original issue is fixed. This can be especially problematic for transient errors or temporary product unavailability.
How to Avoid the Black Hole:
Serve correct HTTP status codes: Use 404, 403, or 503 for errors. Only 200s go into the black hole.
Provide clear error messages: If you can’t change the HTTP code (e.g., in single-page applications), use JavaScript to redirect to a dedicated error page or provide a clearly discernable error message.
Be cautious with noindex: Only use noindex on error pages that are permanently gone. For temporary errors, avoid noindex as it signals immediate removal from the index.
Key Takeaways for Website Owners:
Consistency is key: Ensure all signals (redirects, rel=”canonical”, hreflang, sitemaps) point to the desired canonical URL.
Use HTTP status codes correctly: This is the most crucial step in avoiding error page clustering.
Understand the complexities of localization: Use hreflang effectively and differentiate between boilerplate and full translations.
Be mindful of error pages: Implement proper error handling to prevent them from clustering and hindering recrawling.
By understanding the nuances of canonicalization and avoiding common pitfalls, website owners can ensure their content is properly indexed and served to the right audience. Keep an eye out for updated documentation on canonicalization signals, as mentioned in the podcast.
Also Read: