How Search Engines Rank

What happens when you type a query into Google? We tend to take this for granted, but as marketers, we can improve our content’s organic performance by reflecting on what goes on behind the scene.

Search engines continue to improve their ability to understand content, whether on a web page or in the form of a search phrase. Every year Google launches thousands of improvements to Search. Ranking well in the SERP is no longer just a matter of keyword stuffing.

Marketers often learn search engine optimization by jumping into such tactics, in an attempt to exploit a loophole in a perceived ranking factor. It’s an approach which is prone to inconsistent results.

Let’s (re)view the basics with an eye to understanding how this knowledge helps improve SEO ranking. What follows is a conceptual framework for how Google search could operate.

1. Understand the query
2. Factor in context and settings
3. Generate an initial list of candidates
4. Apply neural-based ranking models
5. Assemble the SERP

The reason we’re using a theoretical framework is simple. We don’t know exactly how Google works and we don’t need to get bogged down in technicalities. Keeping it simple allows us to focus on improving content optimization for organic traffic.

1. Understand the Query With Google BERT and MUM

Understanding how search engines work starts first with understanding the query. The better a search engine can comprehend a request, the better it can provide relevant results. Google’s BERT and MUM act as two stellar examples of artificial intelligence in search engine ranking.

What’s BERT?

Google first introduced bidirectional encoder representations from transformers (BERT) in 2019. BERT is focused on natural language processing. More specifically, BERT improves Google’s understanding of the relation between words. A few years ago, Google Search wouldn’t consider prepositions like “to” and “for” in a search, believing them irrelevant. This issue often led to incorrect results, forcing a Searcher to complete a few more search queries before getting their answer. Now, Google understands every word in a query, thus improving the relevance of the results.

What’s MUM?

Google MUM, or Multitask Unified Model, takes its AI to a whole new level. First revealed in Spring 2021, MUM is both multilingual and multimodal.

It works across 75 different languages to provide the most accurate results for a complex search query. For instance, if a Searcher is completing a complex query about the weather in Italy, MUM will analyze both English-speaking and non-English-speaking results to give the most insightful results for the relevant keywords.

MUM will also work simultaneously across different formats in the future. Instead of just searching for an answer using text, you can upload a picture to use it as a question.

MUM tries to solve queries where the answer isn’t that simple. Here’s the example Google used in their announcement of MUM:

“I’ve hiked Mt. Adams and now want to hike Mt. Fuji in the fall, what should I do differently to prepare?

Since MUM can surface insights based on its deep knowledge of the world, it could highlight that while both mountains are roughly the same elevation, fall is the rainy season on Mt. Fuji so you might need a waterproof jacket. MUM could also surface helpful subtopics for deeper exploration — like the top-rated gear or best training exercises — with pointers to helpful articles, videos and images from across the web.”

If you want to dig deeper into some of the concepts relating to MUM, read the research paper Rethinking Search: Making Domain Experts out of Dilettantes.

Keep in mind that MUM is a term Google has applied to a group of technologies, and it’s very new – so it will be interesting to see how this evolves.

2. Factor in Context and Settings

Once a query is understood a search engine needs to factor in context and settings. Like with all other aspects of its search algorithm, Google has developed these factors over time, with its newest addition being the About This Result panel. This results page technology was launched in summer 2021 to help users understand the “why” behind the search engine results they were getting.

You can access the About This Result panel by clicking on the three dots next to a search result. Once here, you’ll get all the insight on how context and settings affect a Google ranking.

Context

The context of a Google search result is split into four categories:

• Keywords: Google ensures the search results shown are relevant and useful to the keywords a searcher uses.
• Related keywords: Google takes keyword phrases a step further by searching for words related to what a user searched. For example, if they looked up “recipe,” Google could show web pages that also use words like “cook” or “bake.”
• Links: web pages may link to a source that uses words related to a search. These links can provide searchers with even more useful information.
• Local information: lastly, Google compiles information like the language a user prefers as well as their location to provide localized results. Localization is especially helpful in finding nearby businesses.

Settings

Settings are incorporated to show personalized and local results.  

Personalization and settings

Google wants to personalize the search engine experience as much as possible. To do this, the search engine site incorporates a user’s previous search history, stored data (like cookies), search settings, and location. You see an example of personalization whenever you click on the search bar and see a list of your recent previous searches.

Location and settings

The location setting impacts search results by ranking local results above others. For example, if a user looks up “plumber,” they’ll first see results for plumbers near them.

Localization doesn’t only work on a small scale like a particular city, though. On a larger local scale, users can get results that affect their territory or even their entire country. For instance, if a user looks up “hurricane,” they can see results about a hurricane currently affecting their country.

3. Generate an Initial List of Candidates

After considering context and settings, it’s time to create a list of pages relevant to the query. If you’ve ever wondered how Google can come up with millions of search results for one keyword so quickly, it’s all due to how it retrieves, analyzes, and displays webpages. On a technical scale, Google likely uses a combination of retrieval over an inverted index and semantic retrieval. Using this formula ensures that Google maintains both speed and comprehension throughout the process.

Inverted indexing

Inverted indexing matches terms from a search with relevant documents found on the internet. In doing so, it cuts down on the time search engines have to spend searching for relevant content. Search engines may also consider term frequency, position, and structure on a document with inverted indexing. Essentially, without this type of web indexing, search engines would have to examine web pages one by one to see if they contain the query terms.

Inverted indexing has been used for many years and was often seen as the go-to of data retrieval. The one problem with this type of indexing is that it can’t analyze the meaning behind search terms or their semantics. Used by itself, inverted indexing can lead to inaccurate results.

Semantic retrieval

And that’s why semantic retrieval matters. While AI can’t think exactly as humans can, semantic search can do a lot more for a user than keyword retrieval can. This information retrieval method uses context, user intent, and the relation between keywords to understand the reasoning behind a user’s search. In doing so, semantic retrieval provides improved results.

It’s also beneficial when a user wants to look a query up but forgets the words associated with their search. For instance, if they want to look up a singer and they forget the name, they can instead describe the characteristics of the person, and Google can do the work for them. With inverted indexing alone, this would prove almost impossible.

4. Apply Neural-Based Ranking Models

With so many relevant search results, how do search engines choose the order in which they appear? The answer is through neural-based ranking models that use machine learning. And one of the most common neural models that search engines use is Learning to Rank (LTR).

What’s Learning to Rank?

LTR consists of machine learning techniques that create ranking models for information retrieval systems. They monitor a Searchers interactions with the SERP using each click as feedback to create a better ordered list of results. Over time, the models train themselves to go over a list of possible result options they collect and assign a relevancy rating to each one. The higher the rating, the higher the ranking. By following this method, LTR models show a more optimal order of results for the user.

How does it work exactly?

There are a few strategies LTR models can use to generate their results, including Pointwise, Pairwise, and Listwise. The difference between these three strategies is the number of documents that each strategy considers. Pointwise examines one document, Pairwise examines two, and Listwise looks at an entire list of documents.

Pointwise and Pairwise use one of two strategies to sort through documents: classification or regression. These machine learning techniques examine a document’s class and function value. On the other hand, Listwise goes through multiple documents to pick a ranking order. Out of all three, Listwise tends to be the most complex.

Learning to Rank algorithm options

These days, there are several Learning to Rank algorithm options developed by Microsoft, including RankNet, LambdaRank, and LambdaMART. They all use pairwise ranking, although each approaches the process in its own way, giving you plenty of options.

5. Assemble the SERP

The last element to know for how search engines rank is the search engine results page or SERP. SERPs have evolved greatly in the past few years and will continue to do so. As a result, there are many factors that make up a Google SERP, with the most common being:

• Snippets: these are the boxes of information that appear with a simple search intent, quickly giving users results. Snippets can show definitions of words, instructional steps, or lists.
• Knowledge panel: this Google search feature appears on the right-hand side of a SERP and shows core facts and figures related to a search query. Knowledge panels may also show images if relevant to the keywords.
• “People also ask:” For informational queries, Google will list a “People also ask” box that poses related questions that in which the user might be interested.
• Organic results: these are the non-ad results of a SERP and are considered the most relevant options by Google after crawling and indexing them.
• Paid results: these appear at the top of a SERP and are tagged with the word “ad.” Businesses pay to have their web pages ranked first using a pay-per-click model. Users tend to prefer organic results over advertisements, which is why many website owners choose to develop their SEO strategy.
• Visual results: visuals include image or video suggestions related to the keywords. They can help bring context to a user’s search.
• “Related searches:” At the very bottom of the SERP, the Google algorithm lists search opportunities related to a user’s original search. This feature is like “People also ask,” which can lead to similar results listed.
• Top Stories: if a user’s search query has appeared in recent news stories, Google shows three links to relevant news articles.

Other kinds of searches may yield additional features. For example, a local search will usually show three suggestions for local businesses, along with info like hours of operation and address. Another common situation is when a user searches for anything with travel. Google offers a feature where users can input their dates of travel to look for flights or hotels.

A user’s search query greatly affects the layout of the SERP. The format that you see after typing in a search can tell you a lot about the types of information that individuals are interested in seeing more of when they look up keywords, which is invaluable.

The composition of the SERP can also change based on the Searcher’s interaction. Using the search term “Aretha Franklin” creates the following SERP.

Using the search term “famous american soul singers female” and then clicking on Aretha Franklin in the carousel creates a distinctly different SERP, based on your previous search.

The Takeaway

Search engines like Google use a system, not just one algorithm, to produce a SERP that’s relevant to the query. It’s complicated. Still, improving your understanding of search will enhance your decision on what content you create, how you create it, and ultimately how it performs in search.

But writing quality content is no longer enough. Neither is just getting high-quality links. To succeed in the SERP, you need an awareness of how each piece of content fits together and how it helps a visitor’s journey, from when they first enter a search term into Google.