With the rise in usage of Javascript frameworks, such as React, Vue, Angular etc in Websites of today, crawling requires all the code and resource urls on the page to be processed and rendered.
Having resource urls that are not accessible or returning the incorrect response could mean that crawlers would not be able to render the page as intended which may impact how users view the page and cause inaccurate indexing by search engine crawlers.
This makes it important to audit these resource urls to confirm that all resources are returning the correct response. An additional use case this could be used for is to ensure that resources pertaining to paid apis, google ads and analytics are blocked while crawling.
Our latest update to the capability of our Javascript crawler and the performance tab of our Site Health 3.0 UI allows you to capture, view and download the resource urls details for all Javascript crawls. These details include the count of resource urls found per page along with the resource url, the status, whether it was cached, the request type and the resource type.
This information is captured for all Javascript crawls and is visible in the Performance Tab. Simply click on the number of resources loaded per url to view the details.