Crawling

In this pointless crawl budget experiment, I blocked two resources which returned a 404 and 502 Googlebot spent a lot of time crawling each day.

I did a test in April 2019 to see if rel=“next” and “prev” tags were used in crawling to discover new content.