Beyond Broken Links – Some Alternative Uses For Screaming Frog

Screaming Frog LogoFirstly, let me apologise in case some elements of today’s post are stating the blinking obvious. I’m sure a lot of you will be using these techniques on a daily basis, but I’m hoping that I’ll be covering a couple of things that might be new to some people, or taking a different approach that you might not have thought of before.

Secondly, sorry if you were expecting the second part of my Local Domains Strategy post. That’ll be coming soon.

Site crawlers are an essential part of any SEO arsenal. You need to be able to pinpoint broken links, 302 redirects and other undesirable elements on your site and these are the quickest, easiest way to find them. The difference between most of them and the Screaming Frog SEO Spider is how much else it can do. The Froggy can find pretty much anything on your site or in your backlink profile if you set it up right. Today, I’m going to talk through a few alternative ways you can use it.

Check for Incorrectly Configured Canonical Tags

Duplicate Values ExcelThis one’s a setting that’s available right out of the box and, with a bit of Excel know-how, it can help you identify badly implemented canonicals on even the biggest site without needing a lot of patience. After running your crawl, go to the Meta and Canonical tab, set the filter to Canonical and hit Export. Now open it up in Excel, highlight the URL and the Canonical columns, go to Conditional Formatting and in the Highlight Cells dialogue, select Duplicate Data. Of course, you want it to be duplicated, so set it to Unique. Now you’ll have highlights for any URL’s that don’t match the canonical tag. Wow, that was easy, and it’s really handy too, especially if you’ve got a big site with a canonical issue somewhere.

Finding Duplicated or Over-Limit Meta Titles or Descriptions

Again, another easy one that a lot of people miss. Once your crawl’s done, go to the Page Titles or Descriptions tab, export it to Excel and do the “Highlight Duplicates” thing from the above point. From here, you can either do separate exports or – my preferred approach – do some further highlighting in Excel. Back to the Conditional Formatting section and you can highlight the ones that are above and below the optimal length for either titles or descriptions.

If you’d like more information about the optimal meta title and description lengths for different search engines, you can find it in my unnecessarily long post from a couple of months back.

Checking PPC Display URLs

We’ve got a lot of PPC bods at work. They’re very good at what they do, and they work with some amazing brands, several of whom I’m lucky enough to work on too. However, when you work on such large sites and accounts, putting together tons of display URL’s, you need a quick way to check for errors. That’s where Screaming Frog can help out.

Before the campaigns go live, get the team to send you their display URL’s as a txt or csv file. Run them as a list and be sure to go in to the Configuration section and uncheck Check External Links. Any redirects or 404’s, highlight to the team and that’s a nice, clean campaign for them.

Checking Tracking Codes

Screaming Frog Custom FilterThis one’s been covered a lot since the tool came out, but it’s mostly focused on Google Analytics. Not that Google Analytics isn’t essential, but you can also use Screaming Frog to check for any other tracking platform that puts a unique identification code into the page source, including Doubleclick, Infinity Call Tracking, Clicky and Omniture.

Go into your site’s source, find your tracking code, copy the unique identification ID and then go to Custom inside Screaming Frog. From here, check one filter for Includes and one for Does Not Include and paste your ID code into both boxes. Now when you’ve run your crawl, go to the Custom tab and export Filter 1 and Filter 2. If pages that should have the code in are coming up in filter 2, have a word with the dev team.

Broken Link Prospecting

This one’s my new favourite. You need a couple of other toys to do it properly, but they’re well worth the investment.

Broken links are one of the main “in’s” for link builders, but it’s one of those techniques that can be tough to scale. Fortunately though, thanks to Screaming Frog, there’s a solution.

Firstly, you’re going to need a link prospector. I use quite a few, including the Citation Labs Link Prospector, but whatever you use, as long as you can get a big list of prospects into a spreadsheet, you should be good to go.

Run your prospecting report using your platform of choice, but be sure to be as specific as possible. You’re trying to scale your broken link building, but you don’t want such a huge list of pages to test that your crawl’s running for a month – if you think about how many links are on the average webpage, you’ll see why. All the prospectors I mentioned above let you drill down your exports a fair way, so if you’ve done that, we should be good to go.

Most prospectors let you export your opportunities into csv. Do that. Now paste the paths into a new csv or text file – these are the pages we want to test.

Set Screaming Frog to the List mode, but rather than the usual list crawl, go into Configuration, check the External Links and – this is the important part – set the level to one. It’ll then crawl outbound links and, hopefully, give you a few opportunities to hit for links on very relevant pages. I’ve used this one quite a bit recently and had some good success, but only if the prospector is set properly. There’s a great post here offering some excellent settings for the Citation Labs Link Prospector, which is well worth a read.

And We’re Done

There’s so much that you can do with this tool that I haven’t even scratched the surface. I’d love to hear some other ways that you guys are using it, so leave me a comment if there are any I’ve missed. Happy Frogging.

Stay Up To Date

Get my latest posts delivered to your inbox for free. No spam, I promise.
Or Subscribe By RSS


  1. Screaming Frog seems like an incredibly useful tool.  It does look like a cut above the rest of the SEO spiders for the amount of options and tools that it provides.

    •  @richescorner It really is, and I haven't even started to cover everything it can do. There might be a follow up post in the future. Thanks for the comment.

  2. Hey, Ben! You forgot to mention that Screaming Frog also builds xml sitemaps. I use a combination of Screaming Frog and Webmeup for SEO. To me these tools compliment each other. There is a chunk of work which is most conveniently performed in Webmeup, but then there are tasks that only Screaming Frog can help me with.

    Anyway, it's always nice to read about other things one can do with the tool, I might been to do this one day!



    • Hi Steve,

      Thanks for the comment. At the time of writing this post, I don't think Screaming Frog actually did XML sitemaps or URL rewrites, but it's cool to see the way that the tool's evolved over the last year. In fact, this post's probably due an update or followup soon.

      Thanks again,


  3. Even though I use another tool for on-page optimization tasks, I have to say Screaming Frog is a good one. Mine is WebSite Auditor, which has a similar set of features regarding structure (broken links, page code, response errors), content (keywords in titles, H1-H6 tags, anchor texts) and other on-site elements (robot.txt, sitemap). Other than that, I really like reports in WebSite Auditor: all on-page factors are conveniently grouped and illustrated with widgets and graphs in the report templates. So, I can't even imagine how some SEOs can do without a good on-page optimization tool, like Screaming Frog or WebSite Auditor..