Tales of Robots.txt: Uncrawled URLs in Google Search
In this case study, I identified that Google was reporting on URLs blocked in the robots.txt file in the “Valid with warnings” report and how I resolved the problem very slowly.
How Blocking 4xx and 5xx Rendering Resources can Reveal Googlebot’s True Server Response
In this pointless crawl budget experiment, I blocked two resources which returned a 404 and 502 Googlebot spent a lot of time crawling each day.
How to simulate Googlebot using Chrome
I created a way to replicate Googlebot to debug crawling issues called the Chromebot technique.
Googlebot doesn’t use rel=next/prev in crawling
I did a test in April 2019 to see if rel=“next” and “prev” tags were used in crawling to discover new content.
