To crawl or not to crawl?
https://www.seroundtable.com/google-force-crawl-links-28347.html
For anyone trying to build a presence on the internet, whether it be a social-media influencer, a startup or a budding artist, ranking high up in search results is undoubtedly essential. To the extent that several specialists like to closely follow each new development or detail in the search industry. One such constantly evolving aspect that is of importance is the manner in which search giants such as Google crawl the web.
Given the size of the web, obviously Google can’t crawl and re-crawl every single page on the web at short intervals of time. Rather, Google has it’s own way of determining which pages to re-crawl and when to do it. However, the way Google does this not exactly clear to the general public. in 2018, Google updated their “Search Console” a tool that the company describes allows “webmasters to check indexing status and optimize visibility of their websites”. The update allowed users to find out when Google’s algorithms last crawled a specific URL. This development was of high importance because before although one could get a general sense of if their page placed poorly on certain queries we could not know whether this was because of a factor such as the content of the page or simply the fact that the page hadn’t been crawled since it was updated/created. This topic is important because say an old webpage, which was published several years ago, suddenly becomes active and starts to grow in number of subpages and links/tabs then it still may not be crawled for a considerable amount of time because Google’s algorithms continue to think the page is dead. Now, though Google does allow you to submit sitemaps or specific URLs and have those pages crawled again, not everyone uses or even knows of this provision. However, a recent Twitter post by John Mueller, a top Google executive, indicates that even force crawling a webpage wouldn’t be of much use. Mueller explains that even if you were to force crawl a page and find new links on it, those links will not be ranked very high as the fact that they weren’t being crawled naturally means that they aren’t of much value anyway. No doubt Mueller is extremely confident in Google’s crawling algorithms.
Further readings:
https://www.searchenginejournal.com/find-out-when-google-last-crawled-a-specific-url/259195/#close
https://support.google.com/webmasters/answer/6065812