I recently had the opportunity to speak at the Pubcon Conference in Las Vegas, US, and spoke on a subject which has intrigued and fascinated me now for a number of years.
The subject of web crawling efficiency.
It occurred to me that there are a number of myths and misunderstandings around the subject of ‘crawl budget’ and I often see blog posts and tweets on social media which seem to counter some of the findings I have discovered over the years. Whilst I am aware that none of us know ‘all of the secrets’ to crawl budget and web crawling, there are some which I have found the answers to via asking questions and which appear to be well supported by research papers in academia and also various Search Engine organisation patents. My assumptions are therefore drawn largely from a tying together of numerous pieces of information rather than based upon individual items in isolation.
Whilst in Las Vegas, I also got to meet some people ‘in real life’ in the industry whom I’ve communicated with over the past couple of years via social media. That was really special and I enjoyed their offline company as much as I have enjoyed their online company, if not more.
The deck from my Pubcon talk has now been published on Slideshare if anyone would like to take a look and review the slides.