Tales of Robots.txt: Uncrawled URLs in Google Search
In this case study, I identified that Google was reporting on URLs blocked in the robots.txt file in the “Valid with warnings” report and how I resolved the problem very slowly.
In this case study, I identified that Google was reporting on URLs blocked in the robots.txt file in the “Valid with warnings” report and how I resolved the problem very slowly.
In this pointless crawl budget experiment, I blocked two resources which returned a 404 and 502 Googlebot spent a lot of time crawling each day.
I created a way to replicate Googlebot to debug crawling issues called the Chromebot technique.
I did a test in April 2019 to see if rel=“next” and “prev” tags were used in crawling to discover new content.