A key problem is monitoring the crawling and indexing of important pages on any website.
For the last few months, I’ve teamed up with a senior developer to play around with the URL Inspection API. Our goal was to see if we could create an indexing monitoring tool by storing the daily fetch requests from the URL Inspection API.
We did it. This blog post is a few notes and lessons learned from the data.
In this case study, I identified that Google was reporting on URLs blocked in the robots.txt file in the “Valid with warnings” report and how I resolved the problem very slowly.
In this case study, I found one of my own blog posts had not been indexed by Google. So, naturally being a technical SEO specialist I investigated it.
In this pointless crawl budget experiment, I blocked two resources that returned a 404 and 502 Googlebot spent a lot of time crawling each day.
In this guide, I’m going to explain how to make your own Googlebot simulator in Google Chrome to debug complex technical SEO issues.
In 2018 I ran a quick test to find out if an HTML canonical tag and the rel=”canonical” HTTP header are analysed at the same speed in Google’s index.
