When doing search engine optimization on your website, it is very important to understand exactly how a search engine spider sees your website so that you can have the best chance of a full crawl (we will talk about crawling in a minute) and so the search engine spiders understand what your site is about and rank it accordingly in the SERPs.
What Are Search Engine Spiders?
So first, let’s talk about spiders… also known as crawlers. These are programs that the search engines have to comb through a web page to determine what it is about.
How Do Search Engines See A Website?
The way that a search engine sees each page of your website is very different that what you or I see. For example, when I go to Nintendo.com, this is what I see:
However, when a search engine spider comes to Nintendo.com, this is what they see:
Two very different things, right? That is why it is incredibly important to have a structurally sound website that has clean code and that is also full of rich content that optimized for a set of keywords.
What Happens After A Crawl?
After a search engine spider crawls a web page, they then need to determine what the page is about. They do this by reading the text and trying to determine the intent of it. That is why it is crucial to include your keywords in your title tags, headings, and content. To demonstrate how a spider does this, this is a simple tag cloud of a web page about paintball guns.
From this, a search engine spider would determine that the page is about paintball guns, paintball fields, playing paintball, shopping for paintball guns, etc.
After a spider determines what a page is about, they will then decide how to rank you in their results pages. This is based on two factors, relevancy (keyword intent) and authority. As we have talked about and will talk about in other posts, authority comes from links to the web page and the sharing of the content socially.
I hope that this has helped paint a pretty good picture of how search engines see websites as a whole and individual web pages. If you have any questions or would like to add to the conversation, please do so in the comments section below.