Website Auditor SEO Spider: Powerful Website Crawler for Fast Audit

Enter your e-mail to start the download. We will also send the download links to your email.

Site auditing is no longer a challenge.

Lightning-fast site audit
All resources checked: crawl JavaScript, CSS, images, video, etc.
Crawling like Google, Bing, or Yandex
Custom extraction of any piece of content
Scheduling SEO audits and reporting to clients

The Website Auditor SEO Spider tool includes everything one can ask of a crawling software: scraping dynamically generated content and password-protected sites, identifying redirect chains, broken links, and technical errors; auditing onsite SEO and linking factors, searching for any piece of content sitewide, etc. But more importantly, it enables you to see any site comprehensively, just like search engines do — regardless of what it's built on, how big it is, or which SEO factors you are looking to measure.

Fast and comprehensive SEO audit.

The WebSite Auditor SEO Spider tool digs through your site just like search engine bots do, leaving no tag unturned, and helps you spot problems in all types of your site's resources. It scrapes not just HTML, but also CSS, JavaScript, Flash, images, videos, PDFs, and other resources, internal and external.

Crawl websites with all their subdomains.

If your site includes subdomains and directories, the WebSite Auditor will let you conveniently crawl them along with your main domain, so you can audit and analyze all your pages and resources — and your site structure as a whole — in a single project.

Advanced crawler settings for in-depth site audit

Where to find it: if you are creating a new project, check the Show expert options box at Step 1, go to Advanced Options, and check the Crawl subdomains box. For an existing project, go to Preferences > Crawler Settings to do the same, and click Rebuild Project.

Crawl password-protected sites.

The tool allows crawling and auditing websites that require authentication and optimize password-protected pages.

Where to find it: when you are creating a new project, check the Show expert options box at Step 1 and pick a bot. For an existing project, go to Preferences > Crawler Settings to do the same, and click Rebuild Project.

Crawl JavaScript websites and dynamically generated content.

The WebSite Auditor SEO Spider can execute JavaScript and crawl dynamically gеnerated content to render web pages fully, including copy and links.

Where to find it: if you are creating a project, check the Show expert options box at Step 1, go to Advanced Options, and turn on Execute JavaScript. For an existing project, go to Preferences > Crawler Settings to do the same, and hit Rebuild Project.

Identify redirect chains.

Scan URLs to analyze if there are any redirect chains on the site. You'll see a list of pages the redirects originate from, along with the full redirect path — so you can quickly see what needs to be fixed and where.

Where to find it: Under Site Structure > Site Audit, in the Redirects section.

Audit technical SEO factors to spot slow pages.

The WebSite Auditor SEO Spider identifies if pages are mobile-friendly, or probably too big and may negatively affect SEO score and user experience. Page size is calculated comprehensively by adding the size of its HTML code to that of every resource used on the page, including images, videos, and JavaScript. The tool checks if any of your pages could use some speed optimization. You'll find a list of such URLs right in the Site Audit dashboard along with each page's size in megabytes.

Where to find it: Under Site Structure > Site Audit, in the Encoding and technical factors section.

Check for broken links and server errors.

Use the SEO Spider tool to find broken links (page 404) and server errors in no time. Also, take care of pages with excessive external links to avoid search engine sanctions.

Where to find it: Under Site Structure > Site Audit, in the Indexing and Crawlability, and Links section.

Fast audit of all site's images.

The WebSite Auditor SEO Spider will see if any of them are broken or have empty alt text tags, and show you a list of pages the problematic images are found on.

Where to find it: Under Site Structure > Site Audit, in the Images section.

Improve onsite SEO.

Find page titles that are too long, duplicate URLs, empty or duplicate tags, and meta descriptions. Fix duplicate content in page titles.

Where to find it: Under Site Structure > Site Audit, in the On-page section.

Find pages buried too deep on your site.

The Click Depth metric allows identifying pages on your site that are hard to find for users and search engines. Click Depth shows the number of clicks it takes to get to a given page from the home page. Ideally, all pages should be accessible at 4 clicks max to boost crawlability and avoid losing too much PageRank.

Where to find it: For new projects, a Click Depth column will be automatically added to the All pages workspace (in the Pages dashboard). For existing projects, you'll need to add it manually by right-clicking the header of any column.

See how your pages and resources interlink.

Links are crucial for a site's SEO score. Wondering how well-linked-to a page is? Examine all internal and external links to and from the page in one click.

Where to find it: Under Site Structure > All Resources and Site Structure > Pages.

Visualize site architecture.

The interactive visualization tool allows building your website structure and pages' interlinking. Insightful diagrams will show the strongest pages, as well as those that need more connections.

Review microdata and Open Graph markup.

See which of your pages don't have structured data in place, the markup that's used by Google for Knowledge Graph listings, rich snippets, and local ranks. You can also check each page's Open Graph titles, descriptions, and images — this type of metadata is used in snippets when your pages are shared on social media.

Where to find it: In Site Structure > Pages, under the Structured Data Markup, OG Title, OG Image, OG Description columns.

Conduct a localization audit.

The Website Auditor SEO Spider tool will analyze your localized pages and report issues with Hreflang elements. To avoid duplicate content issues, use Hreflang elements or Sitemap where you tell Google and other search engines about what language version is better to load for users who speak other languages.

Generate XML Sitemaps and robot instructions.

Generate XML Sitemaps to make your URL structure better accessible to bots. Create robots.txt files with meta robots directives (like 'noindex', 'dofollow', 'nofollow') to instruct bots how to scan URLs on your site. The tool allows saving XML Sitemaps and meta robots to your computer or upload straight to the site via FTP.

Generate sitemaps and robots.txt with website tools for webmasters

Use Custom Search to get any piece of content on your site.

WebSite Auditor's Custom Search option lets you find all instances of any piece of content across a website — be it an HTML tag, a script, a plugin, or keywords in headings and texts. And if you're a designer or web developer, you're sure to love the CSS Selector option to search for specific elements with CSS syntax.

Find any resource on the website with a custom search

Where to find it: Under Site Structure > Pages > Custom Search button on the top menu bar. Check out our 10 use cases of Custom Search.

Discover hidden issues with the All Resources submodule.

The WebSite Auditor SEO Spider parses every HTML tag and header field of your pages to uncover issues other crawlers wouldn't find. These include everything from common HTML issues like empty "href" attributes for links or "src" attributes for images to the numerous <link> tags (often created automatically by the CMS), iframes, Google Analytics scripts, and social media sharing plugins.

Making sure you've got all the right scripts and elements on the right pages only takes a moment. Spotted a page that's too heavy or slow? View all resources used on it to find the culprit.

Review all resources on pages, internal and external

Where to find it: In Site Structure > All Resources, by clicking on a resource and inspecting the Found in column.

Ready to test? If you don't have the Website Auditor installed yet, download it for free and create a new project for your site — it only takes a few clicks.

Lots of in-depth options for the tech-savvy.

Here are some tweaks and use cases advanced SEOs will love.

Customize everything.

The webSite Auditor's SEO Spider allows customizing your crawl configuration in plenty of ways:

Select a bot or user agent to crawl as;
Adjust crawl speed;
Use filters to only crawl pages with (or without) specific symbols in URLs;
Treat pages with certain URL parameters as different pages (or as the same page) and specify parameters to use or ignore.

The latter can be especially useful if you are auditing a CMS-powered site, which often creates numerous versions of the same page with different parameters, thus confusing search engines, causing duplication issues, and wasting your crawl budget.

Where to find it: Under Preferences > Crawler Settings. For your customizations to be applied, click Rebuild Project.

See your site like Google robot.

Or Bing's, or Yandex', or any other bots. With the WebSite Auditor SEO Spider, you can choose to crawl any website following its robots.txt instructions, either for all spiders or any specific bot of your choice.

Choose various crawler bots to check how different search engines see your site

Where to find it: If you are creating a new project, check the Show expert options box at Step 1 and pick a bot. For an existing project, go to Preferences > Crawler Settings to do the same, and click Rebuild Project.

Crawl any site in a matter of minutes.

WebSite Auditor's SEO Spider ensures powerful, thorough, and fast analysis. Even if you're crawling large websites for e-commerce, the audit will now only take a few minutes; and of course, there's a separate setting so you can increase (or reduce) the crawling speed to your liking.

Where to find it: Under Preferences > Crawler Settings in the Speed tab.

Audit redirects and canonicals.

You can see the destination URLs of every redirect along with canonical URLs and meta refresh URLs in your WebSite Auditor workspace, which lets you quickly make sure each redirect and canonical is set up correctly and points to the right page, and find & fix doorway pages by removing the meta refresh.

Where to find it: Under Site Structure > Pages, by right-clicking the header of any column and adding the Redirect URL, Canonical URL, and Meta Refresh URL columns to your workspace.

Search for orphan pages.

This option searches Google index and Sitemap for pages that aren't linked to from other pages of your site but do exist. They could be old pages you forgot to link to, pages missed in a site migration or redesign, or even a sign that your site has been hacked. You'll find these pages in your project by the Orphan Pages tag. If they are important pages that aren't linked to by mistake, start linking to them internally, to encourage search engines to crawl them more frequently.

Where to find it: check the Show expert options box at Step 1 and select Look for orphan pages. For an existing project, go to Preferences > Crawler Settings to do that, and click Rebuild Project.

Mind your response headers.

The WebSite Auditor SEO Spider parses not only the HTML of your pages but also their HTTP response headers, which can be especially useful for professional SEO consultants. Here are a few examples:

The 'Server' field can be important if you're looking to implement server-side redirects — different server types call for distinct kinds of redirection. E.g., an Apache server might require .htaccess edits, while an IIS server means you'd need to use the URL Rewrite Module.
'Link' is the header field used to indicate a certain type of relationship between the requested resource and other resources. Like the HTML <link> tag, this header field can be used to specify canonicals, hreflang, shortlinks, and many other link types that matter for SEO.
The 'X-Robots-Tag' header field contains robots directives, including 'noindex' rule. If you've found a page that's not indexed and can't figure out why, this field can be the answer.

Where to find it: Under Site Structure > All Resources, by clicking on any resource and switching to Response header in the bottom part of the screen.

Schedule tasks, save crawls and report to clients.

With its rich custom extraction possibilities, the Website Auditor SEO Spider is a great SEO auditing tool for professionals. Schedule your SEO audits, save crawls as spreadsheets or CSV files, and report issues to clients with several clicks.

Stop overlooking site issues that are hurting your rankings.

A thorough site crawl is the technical foundation of SEO success. Whether you're a small business owner and only run site audits once a month, or a technical SEO auditing dozens of large websites for clients, the Website Auditor SEO Spider tool will be an uncontested must-have.

This SEO software works on any operating system, be it Windows, MacOS, or Ubuntu, optimizes sites in any language and for any country's search engines, and is available in different interface languages.

Download the WebSite Auditor tools for free with the link below to crawl up to 500 of a site's resources, or get a WebSite Auditor license for unlimited crawling.

Happy auditing!