SEO of zoekmachineoptimalisatie heeft zoveel veranderende onderdelen dat het soms lijkt alsof we, zodra we klaar zijn met het optimaliseren van het ene deel, terug moeten naar het andere deel waar we vorige week ook al aan hadden gewerkt.
Je kunt continu bezig zijn met het optimaliseren van de drie SEO pijlers: techniek, content en off-page. Maar de belangrijkste onderdelen zijn misschien wel de indexeerbaarheid en het crawlbudget, want die bepalen of jouw website wordt getoond in de zoekresultaten.
In deze blog bespreken we hoe zoekmachines werken, de betekentis van de termen crawlen, indexeren en ranken, het belang van crawlbudget en tot slot geven we 9 tips om de indexatie te stimuleren.
Table of Contents
How do search engines work?
It is important to know how search engines work. Search engines have crawlers that go through all web pages looking for information. This information it is stored in a database.
When someone performs a search, the search engine will show results coming from the aforementioned database in no time. But how does the search engine know what is the best result?
Crawlen, indexeren & ranken
This has to do with crawling, indexing and ranking. Web page crawling involves collecting and storing information, such as content and images, in the database or library. Just like in a real library, the information obtained here is archived and organized, for example, by topic and search intent.
Then the database is going to rank this indexed information in the search results based on a query.
When you perform a search, the search engine generates a list of results that may answer the query.
Indexability & crawl budget
Sounds like waiting for the search engine crawlers to pass by the web page to crawl, index and rank everything. Unfortunately, it's not that easy. Important factors are indexability and crawl budget.
The crawl budget determines the number of pages the search engine can crawl daily. If your crawl budget is not optimal, it ensures that not all your pages are indexed and therefore do not appear (quickly) in search results. We discuss 9 tips to boost indexability.
Benieuwd wat Google zelf te zeggen heeft over Crawl Budget? Bekijk dan onderstaande video!
1. Track crawl status with Google Search Console
Errors in crawl status can indicate a deeper problem on your website. It is important to check the crawl status every month. This way, you can identify possible errors that affect the overall performance of your website. It is literally the first step of SEO, because without it, all other efforts are worth nothing.
To examine some of these errors, you can use the URL inspection tool to see how Google views your website. In the example to the right, there are no problems.
Common crawl errors & solutions
If your website has the misfortune of a crawl error, it may require a simple fix or point to a much larger technical problem.
If a page is not fetching and displaying correctly, it may indicate a deeper DNS error that needs to be fixed by your DNS provider.
Fixing a server error requires investigating a specific error. The most common errors are: timeouts, connection failed or refused or no response. Usually a server error is temporary, although if you have a persistent problem you should contact the hosting provider directly.
Robots.txt errors can be more problematic. If the robots.txt file returns a 200 or 404 error, it means search engines are having trouble retrieving this file. You can either submit a robots.txt or avoid the protocol altogether. In the latter case, you can choose to manually set pages to no-index so that those pages are not crawled and indexed.
Fixing these errors quickly will ensure that all your pages are crawled and indexed the next time the search engine crawler visits your website.
2. Create mobile-friendly web pages
With the mobile-first index, the mobile version is considered the primary version of your website. So with the mobile-first index, Google looks at the mobile version first and also gives it more weight in the ranking.
There are many technical tweaks that can instantly make your website more mobile-friendly, including:
Responsive webdesign
Setting viewpoint metatag for browsers and screens
Optimizing and compressing images for faster load times
Reduce the size of UI elements on the page
Make sure you test your website on a mobile platform and run a test through Google PageSpeed Insights.
Page speed is an important ranking factor and can affect how quickly search engines can crawl and index your website.
How exactly does Google mobile first indexing work? Martin Splitt explains!
3. Update content regularly
Search engines will crawl your website more regularly if you regularly publish new content or update old content. If you produce content on a regular basis you signal to search engines that your website is constantly evolving and therefore needs to be crawled more often to reach an intended audience.
Also, the latest SEO trends show that updating old content will become increasingly important. It can even get you more SEO traffic!
4. Submit a sitemap
One of the best tips for good indexation remains submitting a sitemap to Google Search Console.
You can create an XML version using a sitemap generator or create one manually in Google Search Console.
In the video below, Daniel Waisberg explains what a sitemap is, whether you need one and how to submit it to Google search console.
5. Optimize the internal link structure
Establishing a consistent internal link structure is crucial to ensuring that your website is not only well indexed, but also well organized. A good structure ensures that search engine crawlers can crawl and index all pages.
Creating categories under which related pages can appear can further help search engines correctly understand and display page content based on searches.
Neil Patel explains what you should and shouldn't do when getting started with internal links:
6. Deep link to poorly accessible pages
If a page on your website cannot be crawled or takes a long time to be indexed, you can boost indexation by obtaining a link on an external domain.
This is a particularly useful strategy to promote new content on your website and get it indexed faster. Moreover, backlinks contribute to ranking for certain keywords.
7. Reduce resources on the page and increase load times
Forcing search engines to crawl large and non-optimized images quickly consumes your crawl budget and prevents the rest of your website's pages from being indexed (regularly).
Search engines also have trouble crawling certain backend elements of your website. For example, Google has had trouble crawling JavaScript in the past. Other sources, such as Flash and CSS, can also perform poorly on mobile devices and eat up the crawl budget. In a sense, it's a lose-lose scenario where page speed and crawl budget are sacrificed at the expense of intrusive elements on the page.
Make sure you optimize your Web page for speed, especially via mobile, by reducing resources on the page, such as images and videos. You can also enable caching and compression to help spiders crawl your website faster.
Why is the speed of your website important? Martin Splitt (Developer Advocate, Google) and Eric Enge (General Manager of Digital, Perficient) elaborate on this...
8. Pages with noindex tags
A noindex tag is a meta tag that tells search engines not to include certain pages in their index.
Over time, it may make sense to add a noindex tag to pages on your website, for example to pages that are duplicate or pages that are not important enough to be included in the index. In the latter case, you can think of a thank you page that follows an order placement.
By excluding pages from crawling and indexing in this way, you ensure that the crawl budget goes to pages that are important and thus boost indexation.
9. Removing duplicate content
Having huge amounts of duplicate content can significantly slow down your crawl speed and eat up your crawl budget. You can solve these problems by preventing these pages from being indexed (noindex tag) or by placing a canonical tag on the page you want to index.
Along the same lines, it is worth optimizing the meta tags, such as title and description, of each individual page so that you prevent search engines from considering similar pages as duplicate content during their crawl.
Want to know exactly how Google deals with duplicate content? Matt Cutts explains how.
Conclusion
The problems with crawling and indexing are largely related to your own knowledge of SEO and web design. If you are tinkering on the backend all the time, you may have identified these problems early and thus prevented them from starting to affect your rankings.
If you're not sure, run a quick scan in Google Search Console to see if your pages are being reached by Google's crawlers. The results can be really instructive!
This blog was made possible in part by Jeroen from Promoboer.
