Screaming Frog

Image - Screaming Frog
Image – Screaming Frog

The Screaming Frog SEO Spider crawl website URLs and collects key elements to audit onsite technical SEO. It optimizes a website by analyzing on-site elements, such as page titles, meta descriptions, URL structure, response codes, images, etc. In the Directives tab, find information about the meta robots tag, canonical links, and rel=next/prev annotations. The SEO spider tool helps improve the performance of websites in search result pages. The tool finds broken links, generates XML sitemaps, solves duplicate content issues, crawls JavaScript websites and a lot more.

Read more about Search Engine Optimization by clicking on SEO Fundamentals.

Screaming Frog Interface

fig 1.0 - Screaming Frog Interface
fig 1.0 – Screaming Frog Interface

The Screaming Frog Interface [fig 1.0] is made up of 1) Screaming Frog SEO Spider Menu 2) Crawl Bar 3) Tab Bar with Menu at top 4) Side Bar Scroll and menu at top 5) Main Window with Slider 6) Detailed Window with Menu at bottom 7) Graph

Crawl a Website

First, in the Filter drop-down select ‘HTML’ [refer fig 1.0]. Next, in the screaming frog SEO spider menu bar, go to ‘Mode’ and from the drop-down select ‘Spider’. Enter the URL in the crawl bar located at the top of the screen and click ‘Start’. All the crawled URLs get displayed in the Main Window. Click on any of the URL in the Main Window and go the Menu located at the bottom of the Detailed Window and click on ‘Inlinks’. All the details pertaining to that URL gets displayed in the Detailed Window. Find out which pages show a 404 error that needs to be addressed and these details can be exported in a CSV file.

To crawl a list of URLs go to Mode and select List. From the Upload drop-down located in the crawl bar, upload a list of URLs from a file or enter manually by pasting in the displayed box.

fig 2.0 - Side Bar API
fig 2.0 – Side Bar API

Side Bar Tab

The Side Bar Scroll gives you a summary of URLs crawled. Details of all the SEO elements are available by scrolling down in the sidebar. Connect [fig2.0] to Google Analytics, Google Search Console, Majestic, Ahrefs & Mozscape for external link audits available only in the paid version.

fig 2.1 - Side Bar Response Times
fig 2.1 – Side Bar Response Times

Tabs: Overview> Site Structure> Response Times> API.

Under response time you can find how fast your pages are loading. The response times shown in the [fig2.1] is 0.25 seconds which is as per the standard norms.

Screaming Frog Tab Bar

Tab Bar: Internal> External> Protocol> Response Codes> URI> Page Titles> Meta Description> Meta Keywords> H1> H2> Images> Canonicals> Pagination> Directives> Hreflang> AJAX> AMP> Sitemaps> Custom> Analytics> Search Console> Link Metrics.

To be concise some of the main tabs, in the above is considered as per below.

fig 3.0 - Screaming Frog Internal
fig 3.0 – Screaming Frog Internal

Internal

Internal [fig 3.0] refers to all that is on-page. Filter by All> HTML> JavaScript> CSS> Images> PDF>Flash > Others.

Scroll along the bar and check to optimize column wise. Tabs: Address> Content> Status Code> Status [HTTP response code]> Indexability [URLs are indexable or not]> Indexability Status> Title 1> Title 1 Length> Title 1 Pixel Width [can be upto 50-60 characters or 512 pixels]> Meta Description 1> Meta Description 1 Length [under 158 characters]> Meta Description 1 Pixel Width> Meta Keyword 1> Meta Keywords 1 Length> H1-1> H1-1 length> H2-1> H2-1 length> Meta Robots 1> Meta Refresh 1> Canonical Link Element 1 (tells search engine which URL is the master copy]> rel=”next”1> rel=”prev”1> Size> Word Count [at least 350 words]> Text Ratio> Crawl Depth [upto 3 is acceptable but level 4 and 5 are considered bad]> Link Score> Inlinks [check each page lowest number/ highest number]> Unique Inlinks> % of Total> Outlinks [check each page lowest number/ highest number]> Unique Outlinks> Unique External Outlinks> Hash> Response time [0.25 second]> Last Modified> Redirect URI [301, 302 etc]> redirect Type> URL Encoded Address.

fig 3.1 - Screaming Frog: External
fig 3.1 – Screaming Frog: External

External

The external tab [fig 3.1] gives information about external URIs.

Tabs: Address> Content> Status Code> Crawl Depth> Inlinks>

fig 3.2 - Screaming Frog: Protocol display in side bar
fig 3.2 – Screaming Frog: Protocol display in side bar

Screaming Frog Protocol

The protocol tab [fig 3.2] gives data HTTP Vs HTTPS from internal and external URIs.

Tabs: Address> Content> Status Code> Status> Indexability> Indexability Status> Canonical Link Element1> Meta Robots1> X-Robots-Tag 1.

fig 3.3 - Screaming Frog: Response codes display in side bar
fig 3.3 – Screaming Frog: Response codes display in side bar

Screaming Frog Response Codes

The response code tab [fig 3.3] gives response information from internal and external URIs. To find redirects go to bulk exports and select response codes and click on redirection (3xx) in-links. The redirection_(3xx)_inlinks.csv file saved on your computer shows a table where the redirects are on that need to be changed to the actual redirects.

Tabs: Address> Status Code> Status> Indexability> Indexability Status> Inlinks> Response Times> Redirect URI> Redirect Type.

fig 3.4 - Screaming Frog-URI
fig 3.4 – Screaming Frog-URI

URI

The URI tab [fig 3.4] includes data related to the URLs requested which is displayed in the main box.

Tabs: Address> Content> Status Code> Status> Indexability> Indexability Status> Hash> Length> Canonical Link element 1> URL Encoded Address.

Page Titles

Go to page title tab. Export the report and get information about length and character/length and pixels of the title. Should the title be long, it will get truncated in Google search results. Check your titles to spot similarities so that you can merge such similar pages into one.

Tabs: Address> Occurrences> Title 1 Length> Title 1 Pixel Length> Title 1 Pixel Width> Indexability> Indexability Staus.

Meta Descriptions

Keep meta description between 120 to 158 characters. Google’s meta description length is up to 920 pixels.

Tabs: Address> Occurrences> Meta Description 1> Meta Description 1 Length> Meta Description 1 Pixel> Indexability> Indexability Status.

Meta Keywords

Ignore the Meta Keywords tag as Google doesn’t use the “keywords” meta tag in web search ranking.

H1

Each of your pages should have a single H1 heading. Check if anything is marked zero.

Tabs: Address> Occurrences> H1-1> H1-1 Length> Indexability> Indexability Status.

H2

Every page is not required to have an H2 heading. Check if any page that is not having an H2.

Tabs: Address> Occurrences> H2-1> H2-2> H2-2 length> Indexability> Indexability Status.

fig 3.5-Screaming Frog: H1, H2, Images
fig 3.5 – Screaming Frog: H1, H2, Images

Images

The image tab includes data pertaining to any images crawled. The Image has to be under 100 KB for it to load faster on web pages. Check for missing or long text alt tags.

Tabs: Address> Content> Size> IMG Inlinks> Indexability> Indexability Status.

Link Metrics

Tabs: Address> Status Code> Title 1.

Connect Screaming Frog with Ahrefs, Majestic etc. These Link Profilers scan websites and databases to determine the authority of websites for these links from where they come. Backlinks are the Off-page SEO that is outside the scope of normal SEO Audits.

fig 3.6 - Screaming Frog>Detailed Window>Lower Tabs > SERP Snippet” class=”wp-image-1710″/><figcaption>fig 3.6 – Screaming Frog>Detailed Window>Lower Tabs > SERP Snippet</figcaption></figure>



<h2><span class=Detailed Window – Lower Tab

Detailed Window Box displays results for any single URL that is clicked in the Main Window Box for any selection of lower tab menu: URL Info > Inlinks> Outlinks> Image Info> Resources> SERP Snippet> Rendered Page> View Source.

The SERP Snippet tab [fig 3.6] shows results for the URL: https://sapcanvas.com/yoast-seo/ providing information on the truncated ‘text’ for the Title and Description (if any) in Search Results. This can be rectified.

fig 4.0 - Screaming Frog Menu.
fig 4.0 – Screaming Frog Menu

Screaming Frog Menu

Configuration

The default settings [fig 4.0] are good enough.

fig 4.1 - Mode
fig 4.1 – Mode

Mode

The limit for List mode is up to 500 URLs for the free version.

fig 4.2 - Bulk Exports
fig 4.2 – Bulk Exports

Bulk Exports

The Bulk Exports tab allows saving of any of the elements from the drop-down list [fig 4.2] as a .csv file to your computer. For eg to obtain a list of 404 go the menu – bulk exports and select response code and click on client error (4xx) and it will save this file for you.

fig 4.3 - Screaming Frog Reports
fig 4.3 – Screaming Frog Reports

Screaming Frog Reports

In the same way, all the reports refer [fig 4.3] can be saved in a .csv file on your computer. For example, under reports select the redirect & canonical chains and the redirect_and_canonical_chains.csv file saved will show the link to a redirect and that redirect is redirecting to another page.

fig 4.4 - Screaming Frog Sitemaps
fig 4.4 – Screaming Frog Sitemaps

Screaming Frog Sitemaps

In the menu select site map and in the window that pops up leave the setting at default and click next. This will then allow you to save the sitemap.xml file wherever you want. Check this sitemap file errors for broken links, redirects, nasty URLs etc. In a similar manner, you can save the images_sitemap.xml file and check errors for the missing alt text and image file sizes.

fig 4.5 - Visualizations
fig 4.5 – Visualizations

Visualizations

Graphical representation for each drop down elements in the visualization tab [fig 4.5] gets displayed.

fig 4.6 - Crawl Analysis
fig 4.6 – Crawl Analysis

Crawl Analysis

Crawl Analysis configuration is only available for the paid version.

Other Options to Screaming Frog

Visual SEO Studio

Wildshark SEO

BeamUsUp

The Screaming frog SEO spider tool helps quickly audit your website by finding- and fixing of redirects 301, 302, duplicate content, broken links, pages with thin content etc; checking- XML site maps, server response times etc; optimizing- page titles, internal links, SERP snippets etc; so as to address these issue.

Digital Marketing Solutions

Thank you for subscribing.

Something went wrong.