How to Use Screaming Frog SEO Spider for Website Audits

How do you keep an eye on your technical SEO?

Technical SEO is essential if you want to get ahead in the search engine results. However, it can be tricky to monitor and implement, especially if you don’t have technical or web development experience.

A comprehensive technical SEO tool can help you understand how your website is performing and provide you with quick wins when optimizing your pages.

One of my favorite auditing tools is Screaming Frog SEO Spider. It’s brilliant for checking your site for technical faults and finding quick wins to improve your search engine rankings!

In this post, I’m going to introduce you to Screaming Frog and show you all the ways you can use it to improve the user experience of your website. I’ll also talk you through the Screaming Frog pricing structure and what alternatives there are in the marketplace.

Let’s begin!

What Is Screaming Frog?

Screaming Frog is a UK-based technical SEO consulting agency. They developed a tool designed for SEO auditing in 2010 called “Screaming Frog SEO Spider,” typically referred to as “Screaming Frog.”

You can use Screaming Frog to crawl the individual URLs of a specific website.

The Screaming Frog SEO spider works exactly how spiders for search engines like Google and Bing do. It analyzes the HTML code of the starting page, identifying elements like headings, images, meta descriptions, page titles, and structured data.

It then follows internal links within the HTML code to identify other pages, reviewing them in the same way.

Screaming Frog presents this data in its detailed interface. This provides SEO specialists with a wealth of valuable information they can use to fix issues that might otherwise affect a website’s search engine rankings. For example, broken links, page loading speed, and duplicate content.

Installing and Getting Started

In a world of cloud-based applications, Screaming Frog stands out in that it is desktop-based. This means you have to install it onto your computer to use it.

There are benefits to this. You don’t have to log into a cloud-based account each time, and it’s more secure as your data is stored on your own computer.

Screaming Frog is available for Windows, MacOS, Ubuntu, and Fedora. The step-by-step process I’ve listed below is for Windows, but it may be slightly different depending on your operating system.

System Requirements

Screaming Frog recommends that you have 16GB of RAM to crawl extremely large websites, but for smaller sites, 8GB of RAM is fine.

A 64-bit operating system is required, but if you’re using a 32-bit machine, Screaming Frog can provide an alternative version of the software.

Step 1: Download Screaming Frog

Here’s how to install Screaming Frog on your computer.

To get started, visit the Screaming Frog SEO Spider landing page and click the blue download button. Screaming Frog will suggest the right version of the tool for your operating system, but you can choose an alternative if you prefer.

Step 2: Install Screaming Frog

Wait for Screaming Frog to download; it’s a large file, so depending on the speed of your computer, it might take a while. Go to the location where your downloads are saved and click the .exe file.

If your computer asks if you’re happy for Screaming Frog to make changes to your device, click Yes.

Follow the installation instructions and click Install when you’re ready.

Click Finish to complete the Installation, and Screaming Frog will be ready to launch!

If you’ve left the Launch Screaming Frog SEO Spider box checked, it will load immediately, otherwise find the shortcut on your desktop and click it to get started.

Step 3 (Optional): Change Your Storage Settings

If you’re using the paid-for version and own a solid-state drive, Screaming Frog recommends changing your storage settings. This means your crawl data is saved directly to your drive rather than in memory, giving you more flexibility.

To do this, access Screaming Frog, go to File > Settings > Storage Mode and choose Database Storage Mode.

Conduct a Website Audit

The thing I love most about Screaming Frog for SEO is that you can jump in right away and get started. Here’s how to conduct your first website audit.

Input Your URL

Open Screaming Frog: at the top of the tool, you’ll see a gray box that says Enter URL to Spider. Type your website URL here.

If you click the down arrow, you can view all the URLs you’ve previously audited.

The default option is All Subdomains, which crawls your entire site. You can change this to crawl one specific URL or just the subdomain/subfolder you’ve entered.

When you’re ready to go, click the green Start Button.

Start the Crawl

Screaming Frog will automatically start to spider your chosen site in real-time, meaning URLs appear as soon as Screaming Frog crawls them. This can be great for getting a rough idea of the state of your technical SEO, even if the scan is still running.

If you’re on the free version of Screaming Frog, you may see a dialog box if you hit the limit of 500 URLs. We’ll talk about the differences between the free and paid-for versions of Screaming Frog later in this post.

The amount of time an audit takes depends on how many URLs you want to review and how fast your computer is. A crawl of a large website with thousands of pages typically takes a few hours.

Screaming Frog shows a progress bar so you can see how far you are through the audit, and you can press pause to temporarily stop the scan at any time.

Read Your Report

When the progress bar hits 100 percent, your data is ready to read, and your reports are ready to access.

Go to Reports and choose the report you want to download. This provides you with lots of valuable information you can use to optimize your site.

There are also lots of different categories of data available to view in Screaming Frog, which can feel overwhelming if you’re new to it.

My advice is to click on everything—you can’t “break” your website in Screaming Frog, so just go for it! I’ll talk you through some specific reports you can access in Screaming Frog SEO Spider soon.

Each tab lets you search for specific information or apply a filter. For example, the Internal URL report enables you to filter by specific file type.

You can also view a list of results or access a tree table view, which lets you see the hierarchy of the URLs on your site.

Each tab also summarizes the essential findings and provides a data chart, which is great for reports or presentations.

Finally, you can export your data as a CSV, Excel Spreadsheet, or Google Sheet. Click Export on the report you want to save and complete the details.

If you’re using the paid version of Screaming Frog, you can save your crawl, meaning you can reopen it and revisit it later. Just go to File > Save or File > Save As.

Use Screaming Frog to Identify and Resolve SEO Issues

We’ve looked at how to use Screaming Frog to audit your site; now let’s get into what you should do with all that data. In this section, I’ll show you how to use Screaming Frog to analyze your site, improve your SEO, and even carry out keyword research.

Fix Technical SEO Issues

Screaming Frog can help you fix a wide range of technical SEO problems on your website. Once you’ve run a crawl, it’s just a case of knowing where to find the right data.

Here are some ideas to get you started.

Find Broken Links

Broken links don’t just frustrate your visitors but can negatively affect your SEO. Here’s how to locate them all using Screaming Frog SEO.

Run a crawl and go to the Response Codes tab. Next, filter for Client Error (4XX) results. This will bring up a list of all 404 and 403 errors on your site.

Click on an error and click on the Inlinks tab at the bottom of Screaming Frog. This will tell you the page where the broken link was found, meaning you can go in and remove or update it.

You can also export a broken link report for your site by going to Bulk Export > Response Codes > Client Error (4XX) Inlinks.

Review Page Titles and Meta Descriptions

You can use Screaming Frog to review your page titles and meta descriptions, meaning you can ensure a unique, relevant, and detailed search engine presence.

Go to the Page Titles or Meta Descriptions tabs, where you can see the relevant information for each URL on your site.

You can filter to see missing, duplicate, too long, or too short titles and export this information into a spreadsheet.

Identify Missing Alt Text

Alt text is essential for accessibility. You can check the images on your website to see if they have suitable alt text by checking out the Images tab.Images tab.

Select a file and access the Inlinks tab at the bottom of Screaming Frog to see where the image is and what the alt text is

If you want to find images with missing alt text, choose the Missing Alt Text filter at the top of the report.

Analyze Backlinks

One of Screaming Frog’s strengths is that you can use it to review your internal and external links.

Both types of links are essential for strong performance in search engines. Linking to external websites boosts your credibility and provides additional value to visitors.

Solid internal linking:

• Keeps visitors on your site for longer
• Makes it easier for search engines to index your entire site
• Passes “link juice,” helping search engines determine which pages to prioritize in the results

You can use Screaming Frog to get data about all your internal and external backlinks.

Start by going to the Internal Links report and filter it to only show HTML files (i.e. pages). Click on the URL (or URLs) of your choice.

Next, scroll down to the tabs at the bottom of the page. Choose Inlinks to see the other pages on your site that link to that URL and Outlinks to see what internal links feature on that URL. You can also see which anchor text is in use and whether the link is follow or nofollow.

When you have the information you need, you can export it into a spreadsheet for further review.

You can repeat the process using the External tab and Inlinks report to see which external links you promote on your pages.

Conduct Keyword Research and Analysis

Screaming Frog isn’t just useful for technical SEO. You can use it as part of your keyword research strategy too.

Competitive Analysis

You can use Screaming Frog to check out your competitors’ websites as well as your own.

Run a crawl of their site, then go to the Page Titles, Meta Descriptions, H1, and H2 tabs to see what words and phrases they use.

Custom Extraction

Another strategy is to use the custom extraction feature in Screaming Frog to scrape web pages for additional information. For example, Screaming Frog doesn’t collect data about H3 tags by default, but you can use custom extraction to do this.

As an aside, you can use the custom extraction feature to identify a wide range of data about your site. You can use it to find which cookies are in use, which social media sharing buttons are on each page, and if your documents contain any broken links.

Duplicate Content

You can use Screaming Frog to find exact and nearly exact content on your site.

While duplicate content won’t tank your SEO rankings, it can confuse the search engines. Go to the Content tab and filter for Exact Duplicates and Near Duplicates.

Once you’ve identified duplicate content, you can review these pages and add new keywords to offer unique content to your visitors.

Advanced Features to Explore

Screaming Frog is a powerful tool that can tell you a lot about your website. Consider these additional features to enhance your experience even further.

• You can schedule audits, either as a one-off or at selected intervals. Just go to File > Scheduling, and your crawl will run in the background.
• You can integrate tools into Screaming Frog to provide additional functionality. For example, you can use PageSpeed Insights to get information about page load speed and performance, or Google Analytics to add information about page visits or bounce rates to your selected data. Go to Configuration > API Access to do this.
• You can crawl JavaScript elements on your website, for example, AJAX content. This means you can see which pages depend on JavaScript and if any elements are blocked. Go to the JavaScript tab to access this information.
• You can compare two sites in Screaming Frog—this is ideal if you’ve made technical changes to your website and want to see the improvements. Go to File > Crawls and choose the reports you want to review.
• You can generate an XML sitemap and robots.txt file to upload to your website. Go to Sitemaps > XML Sitemap or Configuration > robots.txt and follow the instructionsConfiguration > robots.txt and follow the instructions
• You can create cool visualizations that let you easily see the relationship between the pages and files on your site. Go to the Visualizations menu, choose your visualization, and tweak it to your requirements

For example, the most recent update allows paid users to filter Google Analytics tracking parameters, ideal if you need to change or remove the tracking code attached to your URLs.

Screaming Frog Pricing

The Screaming Frog SEO platform is free to use. You can crawl up to 500 URLs for free and have access to functionality and reporting including:

• Broken links and redirects
• Page titles and meta descriptions
• Meta robots and directives
• Hreflang attributes
• Exact duplicate pages
• XML sitemap generation
• Visualizations

It’s important to remember that any link on your website counts as a URL, not just pages. This means JavaScript files, CSS files, and images all contribute towards your 500 URL limit.

However, by paying for a license, you can unlock additional functionality and reporting including:

• Unlimited crawls (dependent on allocated memory and storage)
• Scheduled reporting
• JavaScript rendering
• AMP crawling and validation
• Integration with tools including PageSpeed Insights, Google Search Console, and Google Analytics
• Structured data crawling and validation
• Near duplicate pages
• Robots.txt generation
• Free technical support

Each license costs $259 a year.

These licenses are per person rather than device, so you can pay once and use Screaming Frog on different devices. You can also pay for additional licenses, with a price break if you buy more than five at a time.

To apply a license to your version of Screaming Frog, buy online, where you get a username and license key. Go to Screaming Frog and select License > Enter License Key from the menu. Enter your details when requested, restart Screaming Frog, and you’re good to go!

Screaming Frog Alternatives

Screaming Frog is one of the best tools for carrying out a technical SEO audit, but I appreciate that it’s not for everyone.

The learning curve can be steep if you’re new to technical SEO, and if you have a slow computer, it can take time to crawl large websites.

These are some of my favorite Screaming Frog alternatives if you want to explore other options:

• Sitebulb. While Sitebulb is more expensive than Screaming Frog, it is both desktop and cloud-based. It also offers a “Hints” system which suggests how you can improve the technical SEO of your website in plain English.
• Visual SEO. Of all the tools on this list, Visual SEO is the most similar to Screaming Frog in terms of looks and pricing. The critical difference is that Visual SEO is easier to use while Screaming Frog offers more advanced functionality.
• Spotibo. An online SEO crawler, Spotibo lets you analyze 500 URLs for free, providing jargon-free suggestions for improving your SEO. It’s a lot more basic than Screaming Frog but ideal if you want to carry out a quick spot check.
• Scrutiny. Scrutiny is a desktop-based crawler for Mac, which works similarly to Screaming Frog. It also offers web accessibility testing.
• Oncrawl. Oncrawl comes with an easy-to-understand dashboard that shows you technical SEO issues at a glance.
• Lumar. Lumar is a fast and comprehensive platform that offers suggestions about how you can improve your technical SEO and accessibility, as well as benchmark against your industry. The downside of Lumar is that it’s for enterprise companies only, with no free trial or pricing plans for smaller businesses.
• Botify. Another enterprise platform, Botify is more technical than Lumar and targeted at SEO professionals. Its Intelligence feature prioritizes which technical SEO actions you should undertake first.
• Xenu’s Link Sleuth. While Link Sleuth only identifies broken links, it’s simple to use, free, and very thorough—it even checks to see if any email addresses are still active.

FAQs

Conclusion

Technical SEO can feel like a lot, especially if you’re a marketer who doesn’t have web development experience.

However, get it right, and you’re one step closer to improving your search engine presence.

Google has gone on record to say that page speed and mobile responsiveness contribute to SEO, so the more you can optimize your site, the better!

Screaming Frog is a fantastic tool to have in your MarTech stack. It’s fast, free, and provides valuable insights you can use to make your website better.

I’d definitely recommend downloading it and giving it a try. If you need a little extra support understanding Screaming Frog and how to use SEO to your advantage, my technical SEO agency is here to help.

Are you a fan of using Screaming Frog for SEO? Which technical SEO platform is your favorite?