Crawl Depth: 10-Point Guide for SEOs

Did you know that every article on The New York Times is no more than five clicks from the homepage?

I mean every article—going all the way back to 1851. 

That’s what good crawl depth looks like. 

The results? Happy bots, faster indexing, and an all-around better user experience. 

Key Takeaways

• Crawl depth measures a page’s distance from the homepage in clicks (a site’s homepage has a crawl depth of zero). 
• The biggest contributors to optimal crawl depth are a well-organized site structure and consistent internal linking. 
• Creating an XML sitemap is also crucial and will immediately improve the crawl depth of all your pages. 
• Over the long term, automation tools monitor your site’s crawl health and identify issues like broken links. 

What Is Crawl Depth?

Crawl depth is the minimum number of clicks it takes to reach a page starting from the homepage. 

A site’s homepage has a crawl depth of zero. An “About Us” page linked in a site’s header or footer has a crawl depth of one. A little-used product page on an ecommerce page might have a crawl depth upwards of five. 

Crawl depth is important because it affects indexing. It makes it easier for bots to navigate your site. This in turn maximizes “crawl budget,” the number of pages search engines like Google can index in one visit due to server resources. 

1. Build a Clear Site Architecture

Your site architecture is the way you organize your pages. A well-linked, hierarchical, and relatively “flat” architecture is best for crawlability. 

Bigger sites will have more complex architectures, but it’s generally best to keep pages within five clicks of the homepage. 

Here’s how to build a site architecture optimized for crawl depth:

• Organize informational content in clusters of topics and subtopics. 
• Use category pages on ecommerce sites. 
• Avoid orphan pages (pages without any links). 
• Use dropdowns in your header navigation bar to link to your main pages.
• Use clear URL structures to reflect categories and topics, like neilpatel.com/training/growth-hacking-unlocked/untold-laws-of-growth.

If you’re not sure what your existing site structure looks like, tools like PowerMapper create easy-to-digest visualizations. 

2. Create HTML and XML Sitemaps

Maintaining up-to-date HTML and XML sitemaps is one of the easiest ways to optimize crawl depth.

HTML sitemaps are designed for human browsers. XML sitemaps, on the other hand, are written in XML schema, a special language that tells bots about your pages. 

Google encourages both XML and HTML sitemaps. While search engines rely primarily on XML sitemaps for indexing, an HTML sitemap also provides useful information about your site hierarchy. 

The New York Times’ HTML sitemap is simple but effective: 

Here’s an example of one of my XML sitemaps on NeilPatel.com: 

If you run a big site with more than 50,000 pages, you’ll need to create a sitemap index page that links to multiple sitemaps. 

While you can create XML sitemaps manually, there are lots of tools that do the job with virtually no input. Yoast SEO is the one I use on NeilPatel.com. 

3. Maintain Consistent Internal Linking Practices

Even if you have the perfect site structure, internal linking further reduces the crawl depth of pages. It also makes the best possible use of crawl budget. 

This is especially the case for large websites. Let’s take Wikipedia—which has one of the best internal link structures on the web—as an example. 

If only the navbar were used, certain pages might have a crawl depth in the double digits. However, due to extensive internal cross-linking, search engines can find even the most niche content quickly. 

Here are the top best practices for internal linking:

• Use keyword-rich, descriptive anchor text. 
• Avoid using the same anchor text for two different links on the same page. 
• Prioritize pages with deep crawl depth and few existing links. 
• Keep relevance in mind when linking. 
• Link to subpages and parent pages. 
• Regularly update old pages with links to new relevant content. 

It’s also helpful to use a link suggestion tool when you’re creating new pages. Internal link automation gets a bad rap. But when used in conjunction with human judgment, it can be very helpful.

It’s simply impossible for one person to pick evenly from a pool of thousands or even tens of thousands of pages. 

4. Think in Terms of “Information Scent”

Optimizing crawl depth is as much about improving user experience as it is about speeding up indexing. 

That’s why it’s important to balance technical SEO considerations with the needs of your users. This approach will have knock-on SEO effects. Fortunately, sites that work well for Google also tend to work well for real human beings. 

“Information scent” is a helpful tool for optimizing UX while reducing crawl depth. According to Nielsen Norman Group, it’s an “estimate of how relevant the page will be, if visited.”

A link with a stronger information scent is easier for users to evaluate and, therefore, more likely to be clicked. 

There are three components that determine information scent: 

• The “link label” or anchor text
• The surrounding content
• The broader context of the link

For example, a visitor looking for furniture deals can quickly evaluate whether or not clicking on “Shop Now” on the central image on the Walmart page below will satisfy their needs. It has a clear link label, useful surrounding content, and sits in a broader context of product promotions. 

5. Don’t Over-Optimize Larger Sites

Sometimes, achieving a universal crawl depth of below five isn’t desirable. In fact, it’s possible it could make your site less user-friendly.

In these cases, the best thing to do is focus on optimizing your site structure and sitemap. A complete sitemap will maximize the number of pages indexed per session. And a solid site structure will help avoid a messy user experience. 

Microsoft, for example, has a gargantuan site. But it makes use of a hierarchical structure with clear navigation labels and plenty of internal links. 

Microsoft also provides a comprehensive HTML sitemap and provides XML sitemaps for its multiple subdomains. 

6. Link to Regularly Updated Pages From the Homepage

If you have pages that change frequently, it’s important they’re re-indexed as soon as possible so they rank for appropriate keywords. Reduce their crawl depth by linking to them from the homepage.

For example, Target’s “New Arrivals” page reflects its spring range:

Etsy includes its Mother’s Day page in the navbar for the relevant month:

7. Follow Pagination Best Practices

Pagination is when you split content over multiple pages. For example, on blog index pages or ecommerce category pages.

The Semrush blog uses pagination: 

The issue with pagination is that it creates pages with high crawl depth. Ecommerce sites can often have hundreds, and sometimes thousands, of pages for a product category. 

Follow these pagination best practices to prevent indexing issues:

• Give every page its own URL, such as with a ?page=n parameter.
• Include links to the subsequent and previous pages on all pages, along with the first page in the pagination sequence.
• Give every page its own canonical URL to prevent content duplication problems.
• Prevent Google from indexing URLs with filters (like a product category page filtered by size) by using the noindex tag.
• Remove SEO elements from all but the first two pages to discourage these pages from appearing in search results. 

Keep in mind that search engine crawlers can discover new content by following paginated pages, so don’t stop these pages from being indexed. 

8. Manage Your “URL Inventory”

Your URL inventory is the list of URLs you want search engines to crawl. 

By regularly pruning your inventory, you free up bandwidth for search engines to crawl other areas of your site. This means that even pages with a high crawl depth are more likely to be indexed quickly. 

Google gives the following tips for increasing crawlability: 

• Remove duplicate content where possible.
• Use the robots.txt file for pages that don’t require high-priority indexing. 
• Use 404 or 410 status codes for permanently deleted pages. 
• Remove soft 404 errors (redirects to irrelevant content).
• Make sure your XML sitemaps are up to date. 
• Try to remove long 301 redirect chains.

All of these technical adjustments will also improve site usability, which will likely give a small SEO boost. 

9. Use Tools to Fix Broken Links

Broken links increase crawl depth for obvious reasons.

If a page exists but internal links to it aren’t working correctly, there’s every likelihood that the crawl pathway will lengthen. In the worst-case scenario, it might become an orphan page.  

A tool like Ubersuggest runs regular audits and identifies broken links, which you can then fix. 

10. Check Your Index Coverage Regularly

While tools can automate the process of identifying issues like broken links, manual reviews have a part to play, too. There’s no substitute for checking your index coverage in Google Search Console. 

Specifically, the “Why pages aren’t indexed” section of the report alerts you to any issues with indexing that may be related to crawl depth. 

If new and updated pages are quickly indexed, that’s a good sign you’re on the right track with crawl depth. 

Bonus Tip: Increase Your Site Speed

If your site renders quickly, you will use less of your crawl budget. Search engine bots will access and read more pages on a single crawl. 

Use PageSpeed Insights to test and troubleshoot the speed of your pages. Aim for a page speed below two seconds. 

Crawl Depth Is One Part of the SEO Puzzle

Crawl depth is a small but important part of good SEO. 

The speed with which Google indexes your pages and the ease with which users can navigate them have a direct impact on your rankings, traffic, and conversions. 

What’s more, maintaining a shallow crawl depth (where appropriate) is a straightforward task. If you’re dealing with a poorly organized, large site, there will be a little more upfront work. 

But after that, it’s all about regular monitoring and following best practices.