Crawling Slower Sites

At times, some pages in your project may return status codes like '403 Forbidden', '400 Bad Request' or '500 Internal Server Error'. The pages may be listed under 'Resources with 4xx or 5xx status codes' sections while the URLs are valid and load fine in a browser.

This may happen with huge or slower websites hosted on servers with high security or crawling budget restrictions. To decrease the load and get valid results, you can adjust the crawling settings in WebSite Auditor to limit the number and the speed of the requests sent to a site.

Firstly, limit the number of tasks WebSite Auditor will run simultaneously. Go to Preferences > Misc Global Settings and set 1-2 tasks in the Number of Tasks field.

This way, the load on the server will decrease so your IP shouldn't get compromised, and the pages should start returning actual status codes.

Next, once you create or rebuild a project, enable Expert Options, and click Next.

In the Speed section, limit the number of requests to the website to prevent heavy load on the server. We recommend setting up the limit to 1 request per 10 seconds if the server tends to block the requests.

These settings can be accessed or modified any time under Preferences > Crawler Settings before rebuilding an existing project.