Keyword Research: the Fundamental Practice of Good SEO

Last updated:

The importance of keyword research in digital marketing

Keyword research should be one of the first tasks you perform for any digital marketing campaign. The keywords you choose will determine who sees your website, in what context and where it will rank, so the words and phrases you identify as targets for each page should never be far from your mind when producing web copy, writing a new blog post or optimising pages within your site.

But how do you go about researching the best words and phrases to use? How can you tell whether your key words are too specific or too generic, and what can you do to find the best phrases for your website? In this guide we’ll cover the entire A-Z of keyword research, and put you on the road to SEO success.

Research relevant key words and key phrases

First you have to consider relevance to your audience. This may sound obvious, but there is no point in optimising your site for a search term or phrase that is not relevant to the products or services that you offer; your goal should be to prioritise quality traffic to your site, rather than sheer quantity of visitors. I’d sooner have a single visitor who’s actually looking for the digital marketing services I offer than 100 visitors who are looking for something else.

That one visitor will find what they’re looking for, be engaged with my content and hopefully be very close to making a purchase decision, whereas the 100 will be looking for something that I don’t provide, making a quick getaway from my site, skewing my web analytics and increasing my bounce rate (a negative ranking factor in Google’s eyes).

With that basic premise in mind you need to consider what your potential customers will be searching for. Try and put yourself in their shoes; avoid technical jargon and try to focus on the ‘longer tail’ phrases that are most descriptive of your product or service.

For example, people no longer search for “Cornwall holiday”; instead, they search for exactly what they’re looking for – “luxury holiday cottage to rent in Cornwall”, or “cheap hotel in Bude”. And don’t just take my word for it, according to Moz (an authority in the Search Engine Optimisation world) over 70% of searches now lie in what’s called the ‘long tail’ of search.

Gaining Inspiration From Your Own Site

Searching for keyword ideasAssuming that you aren’t launching a brand new business, your company’s own website can provide you with a wealth of information about the best search terms to target.

Firstly, look at how you’ve divided the content on your site, and the navigation elements you use; all are likely to be rich in relevant key words. On our site, for example, we list “Content Marketing and Outreach” under our “What We Do” menu, and within this page we have the subheadings “Offsite SEO or Link Building”, “Outreach”, “100% Unique Editorial” and “Content Marketing”.

This gives us a good starting point – Offsite SEO, Link Building and Content Marketing are all good candidates for developing keyphrases. My first action would be to copy these primary keywords and list them in an Excel spreadsheet, grouping related keywords so we can start to build up a map of the different areas we’ll want to cover.

At this point I also tend to separate out my brand related terms – we can safely exclude phrases like “Bespoke Digital Bristol” from traditional keyphrase research because it is unlikely we will be competing with anyone else for these terms on Google.

I’ll usually also build a list of the primary URLs that I am trying to optimise, be they the home page, product pages, blog URLs or an entire sitemap. This way you can begin to match potential search terms to specific URLs where there is an obvious fit – the phrase “copywriting”, for example, should be a target for our ‘Content Creation’ page rather than for our ‘onsite optimisation’ pages.

A key point to remember is to send as clear a signal as possible to the search engines about the content of each page. If you use the same set of keyphrases across multiple pages, the effect of each one is diluted; even though your website may contain many references to “holidays in Cornwall”, it needs to be clear to the search engines exactly which page they ought to send visitors to. Because if they can’t pick out a page which clearly supplies the answer, they’re less likely to return your actual Cornish holidays page as a result. That’s why we’re dividing up our keyphrases; we want to use different groups of keywords for different pages, so that we’re sending search engines a clear signal of what each page is about.

How to group keywords & phrases into silos

If your site has a search function, you may be able to use it to see what words and phrases active users have been looking for within the content of your site. Not all sites are able to track the use of their search bars, but if you do have access to this data it can provide some valuable insight into your user’s intent.

Gathering Data From Google Analytics

Now that we have some ideas of the keywords we’d like to use, we need to know how people have been finding our site already. This should tell us how effective our keywords have been until now, and may also be a useful source of information on what people are looking for when they click-through to our site. Google Analytics and Google’s Search Console (formerly Webmaster Tools) are great tools for helping us to understand user behaviour, so we’ll need to know how to use them.

Google used to provide clear data on which keywords were bringing us traffic. However, in the past five years the keywords from many searches have been filtered out and now only display “not provided”. Despite this, Google Analytics can still show a broad set of search terms that people have used to find your site via organic search.

To view this data, go to Acquisition, All Traffic, Source/Medium and add in a secondary dimension for “keyword”. Remember to set your date range as wide as possible to get as comprehensive an overview as you can.

Google Analytics - Secondary Dimensions

Next, set an advanced filter to only show you “organic traffic” (not just Google Organic, don’t forget Bing and the other search engines).

Google Analytics - Advanced Filtering

And then filter out the “not provided” data (np -) to make things easier to look at. You may need to filter out “(not set)” in the same way as not provided and there will inevitably be a few other rogue terms or URLs within your list, however these are easily deleted within Excel.

Export this list to Excel, sort the search queries alphabetically and group them as before.

You can achieve a similar result to the above by filtering using the “primary dimension” fields provided by Google. Select “Organic” as your primary medium, add a secondary dimension of “keyword” and then filter out the (np-) and (not set) data.

Google Analytics - Primary Dimensions

I guess it’s a matter of preference to which route you prefer as both give the same result, the difference being that the first method shows you which search engine delivered the organic traffic whereas the latter groups by search query.

I recommend you run both reports and pick a format that you prefer. I use this report a lot, so I’ve set up a custom report to display this data at the click of a button. Since keyword research is a constantly-evolving process it may be worth your while to create your own custom reports; check out Google’s guide to custom reporting to get this set up.

What signals can you get from Search Console?

Another very useful but slightly limited feature is provided by Google’s Search Console. Search Analytics (found under the menu heading for ‘Search Traffic’) details the search queries that your site has appeared for within Google results pages over the past 90 days. You can set this date range to suit you but the maximum is 90 days and data from the most recent two days is unavailable.

This feature is a step above the data provided in Analytics as it not only shows you the search terms that have resulted in a click, but also the search terms where you may have only received an impression (an impression is where your URL has been displayed as a search result but not necessarily been clicked on).

Search Console - Search Analytics

This data can be filtered by device, country and destination URL, so you can see what terms are being used to find specific pages rather than having to look at data for your entire site.

Another helpful feature is the “position” metric which will show you the average ranking position for each term over the selected date range. This is a great indicator of how well you’re performing in the SERPs (Search Engine Results Pages), and is a key metric for onsite optimisation performance tracking.

As this is another Google tool, it’s possible to link your Analytics and Search Console accounts, and view the two together. Once you’ve done this, you can find ‘Search Analytics’ data under the Acquisition, Search Engine Optimisation and Queries tabs. Again, export this data, then sort and group it in your spreadsheet.

By now you should have a substantial set of keywords and phrases and there should be some clear themes emerging. I always have a random bunch of stragglers that are difficult to place in any one group, but for now you can ignore these (they will be useful in the future, though, so don’t delete them!).

What can you get from Google search?

Take each key phrase from your Excel sheet and search for it in Google. This can take some time to begin with, but you can also speed through it once you’re familiar with the process. As you type each query into Google’s search bar, take note of the autocomplete. What other search terms does Google tell you are related to this query? Add any relevant ones to your list.

Google's search suggestions relating to 'keyword research'

More of these can also be seen at the footer of each SERP:

Google - related searches

Note how within this feature Google highlights in bold the variations of your search term, rather than the search term itself, the opposite to how they highlight the SERPs.

This is a useful feature and helps us to begin a new list of search query variants that you can add to a primary or longer tail search term, to make it longer still.

For example, if my search term were “Holiday cottage in Cornwall”, variations could be:

Using prepositions & suppositions to vary your search terms

Or, in the “keyword research” results above, my list of variations might include; tool/tools, free, SEO, how to do, tutorial, software and best. Becoming familiar with the many different functions of Google is useful when working on SEO, and I’ve already created a guide to my favourite tips and tricks for getting the most out of Google.

What can you glean from your competitors?

Now it’s time to look at the organic results themselves. What sites are ranking within the SERPs for your chosen terms, and what variations of it are they optimised for?

Competitor Research

Much of this can be seen from the SERPs. By reading the page title and meta description of each listing you will identify a wealth of variations and you’ll quickly see patterns developing for the more popular terms.

Needless to say, you will need to make sure that the sites being returned within the SERPs are in the same marketplace as your own and that the search term is relevant to your business. Hopefully you will recognise many of the business names and already be familiar with their offering; if not, have a more detailed look at the site to ensure that they are indeed operating in your space.

By visiting the sites appearing in the first page of the SERPs for each term you analyse, you can assess them in a manner similar to how you assessed your own. Look at their content, look at their page titles, navigation elements, business sectors, service pages and blog titles etc. This should give you a new set of ideas, and confirm the ones you already have; we can expect to find keyphrases like “content marketing” on our competitor’s pages, but if they’re using some relevant variations of this, like “influencer outreach” or “content amplification”, then we might want to incorporate these into our own lists.

An old school technique for competitor keyphrase research is to inspect the source code of a site and look for the <meta name =”keywords”> list. Not many sites use this feature anymore, as it is now seen as a spammy signal by many search engine algorithms, however some sites still do, so it’s always worth a quick look just in case your opposition has generously provided you with a list of their target search terms.

Researching your competitors’ search terms can be very time consuming, especially if you have a list of a few hundred potentials, but the more thorough you are now the better the picture you will build and the more effective your keyword research will be.

What about sponsored listings and paid search results?

Don’t forget to pay attention to the Sponsored Listings or AdWords results. Other than their presence suggesting that it’s a competitive, high converting keyword, AdWords results will often contain well-optimised titles that are written to include closely related variations to your primary search term. Again this can be a rich source of keywords for your list and competitive sites to mine for further ideas.

What free tools can you use to help you out?

A lot of what we’ve been doing has focused on manually building up lists, but there are some great free tools to help you develop a list of keyphrases.

Google’s AdWords Keyword Planner is a good place to start and Chapter 2 of Backlinko’s Definitive Guide to Keyword Research is another great resource (a brilliant read if you’ve got a few spare hours). Chapter 5 of Moz’s Beginner’s Guide to SEO is also fantastic.

  • Google Trends

    Not that Google is the only source of information, but Google Trends (formerly Google’s Insight for Search) is a fantastic tool to show how the popularity of your chosen keyword is developing or diminishing over time. The graph displays the relative popularity of the search term over time versus its most popular point – you can’t tell how many people are searching for this term, only whether it’s growing more or less popular. This can still provide valuable information, though:

    From here we can see that searches for “content marketing” have been consistently growing over the past five years. Google also provides another set of related queries, which are displayed below the graph, providing further keywords for consideration.

  • Answer The Public

    Answer The Public is a great tool for helping to develop ideas for long tail search terms. Simply provide a short keyword and it will return results for searches which have been made using the same phrase. Because it shows us what questions are being asked about our chosen keyphrase, it can be a goldmine of useful information. Making the most of this tool takes a little practice, but my guide to using Answer The Public is a great place to start.

  • Übersuggest

    Übersuggest is a great tool for taking your search term and suggesting a range of longer tail variations of it. You’ll quickly see that it delivers back a lot of results, variations beginning with every letter in the alphabet, but within this mass of suggestions there will be some hidden gems.

There are needless to say thousands of paid for tools that offer varying levels of key word research, with many offering free trials along with additional insight into potential search volumes and competition, plus there are many other free tools that I cannot feasibly cover in this article.

Every digital marketing practitioner will have their own preference, but in my opinion by going to the search engines, your own site and your competitor sites, you should get more than enough data to conduct a very thorough set of research. Make sure you don’t over-rely on a single tool, though, as this can drastically skew your results. Using a combination of different sources is a valuable way to cross-check your data, so draw information from more than one location.

Matching keyphrases to the right pages

After all this research you should have a sizeable spreadsheet of keywords and phrases which are relevant to your business niche. These should be grouped into core themes of primary search terms and variant words that can be used as a preposition or supposition to increase the length of your key phrase and brand-focused terms.

Next, you need to divide these between the sectors and service areas of your site. Which are the most important phrases for your business? These should be targeted towards the home page of your site. Geographical ones, perhaps, will target your Contact page or pages specific to your regional offices, phrases relating to “blue welly boots” need to target the blue welly boots page of your site, etc.

blue wellington boot

A key point to remember is not to confuse the search engines. If you target the same set of phrases towards multiple pages within your site, it is sometimes difficult for the bots to make a decision on which page they see as most authoritative, and subsequently they are likely to reduce the authority of all the pages, and return in the SERPs a page from another site that is more obviously relevant for that term. The goal of all onsite SEO is to make it as easy as possible for the search engines to see what each page in your site is relevant for.

As such, each page on your site should have a set of search terms that you think are most applicable to it and most appropriate for the content on that page. Make sure that users who search for “red welly boots” aren’t being taken to a page full of sandals; you’re making it harder for them to find what they want, and they’re more likely to leave your site (and go to your competitors instead!).

There is no set number of phrases that a page can be seen as relevant for, but it’s a good idea to limit the number of primary phrases for each page to fewer than five. You can have as many variations of a primary phrase as you like, but it’s tricky to focus a page and its content for too many variables.

Sticking with the wellington boot example, “red wellies”, “spotty wellies” and “frog-eyed wellies” could easily all be targeted to the same page, since they’re all related. If, however, you then started talking about walking boots, riding boots and football boots, you would be watering down the page’s focus, and though you’ll gain some ranking authority for these other keyphrases you’ll reduce the power of the welly-specific keywords.

This isn’t a strict rule, but under normal circumstances I’d be inclined to have a page about wellington boots, another about football boots and another about walking boots, etc.. Better SEO results are achieved by focusing on a set of related keywords rather than over-diversifying.

Keyword Cannibalisation

This somewhat lurid description refers to in-site competition for keywords between different pages. An important part of key word usage is the construction of internal links, and the anchor text which is the visible part of the link incorporating a key phrase; the hyperlink text is hopefully related to the content of the article it links to, and so from this signal Google takes a hint that a link which reads “social media marketing” is likely to point to a page which discusses and is relevant to that specific topic.

So, if you want to show Google that you’re a social media marketer, you ought to publish as many pages about the topic as possible and link them all together, right? Well, Google is only looking to return the most relevant result it can find, and if they’re all interlinked with the same anchor text then it’s hard for Google to decide which one best matches the query. In this scenario, instead of having a dozen links targeting a single page about social media, there are several pages with three or four links each. This makes it hard for Google to understand which page to display, which could lead it to display the wrong page (or a competitor’s page).

The solution is to select the page which you want Google to rank, and point as many other pages as possible towards it using internal links with relevant anchor text. This sends the signal that this is the page which deals with “social media marketing”, giving Google the information it needs to make the best decision.

What to do with leftover keywords?

During this process you may find a set of words that you can’t find a home for. My footwear site, for example, may not have a dedicated page about riding boots and yet my research may have shown it to be a popular term. Because I do sell riding boots, this research would potentially lead me to create a dedicated page for them and incorporate the keywords which my research has suggested – this could be a valuable addition to my online store, which would have been overlooked had I not conducted keyword research.

Additionally, you may land up with terms related to your primary product pages that are important but not ones that you want to use on your main page – ones that are a little more specialised, or less valuable. These are perfect for deeper level content and blog posts so don’t delete them; save them for use within future content related to this primary page, or find a URL within your existing site that already has relevance and incorporate them within it..

You can read our guide to creating hierarchical webs of content and how to use blog posts and articles within your site to develop the relevance of multiple sets of long tail key phrases. We won’t go into that now as this guide is already more than long enough, but creating a framework for your content is an important part of developing good keyphrases and boosting your site’s SEO.

Finally, look at the list of variants that you have compiled. Have you used versions of them in combination with all of your primary terms for each of the pages in your site? If not then fill in the gaps to expand your list. You can’t have too many variations of a primary search term when it comes to writing content. For example, the more eagle-eyed of you may have noticed that I am varying between the use of “keyword and “key word” within this article, ensuring that this page is optimised for both closely-related search terms.

How many people are searching for these terms anyway?

Many people fixate on highly searched-for keywords, and if a search term doesn’t have thousands of potential searches a month then they’re not interested. I adopt a different approach. If I sell “red-eyed tree frog flip flops” but only 10 people a month are searching for this term, I’m not going to ignore it in favour of a more generic and highly searched-for term, I’m going to do everything I can to ensure that the 10 potential customers a month get to see my website at the top of their search results.

People who are looking for exactly the product I’m selling are much more likely to buy, and these are the people I want to show my site to. It’s for this reason that I leave until last the researching of potential search volumes that each term might expect per month.

Our full guide to using Google’s AdWords Keyword Planner covers this in more detail, but here’s a brief rundown on what to do:

Create a free account, select “get search volume data and trends”, copy and paste your search term groups into the ‘enter keywords’ box and click ‘get search volumes’.

You can target your results to UK only, drill into a specific city or other geographical regions or view all global traffic as well as implementing a range of other filters (not relevant to this exercise so I won’t go into them here). Personally I tend to use the UK only results as many of my clients’ businesses are UK focused, but I also really like the localised search insights that are available.

Next, you can select to see the average monthly search volumes for the past 12 months (the default setting) or you can specify a date range to suit you within the past 12 months and then compare this to the same period last year or another custom time slot. This is a particularly useful feature for observing how search trends are changing over time. Is your keyword getting more popular or less?

Export this data back into Excel and match the average monthly search volumes with your respective search term groups. Annoyingly these results never come out in the order in which you imported them, however you can easily use a VLOOKUP function in Excel to match the two back together again without having to reorder your entire list.

Using Your Keywords

Finally, we arrive at the end of your key phrase research. You now have a list of relevant target phrases, broken down and allocated across each of your web pages, and you have details of the average number of searches each term is likely to receive, based upon historical search trends. All that is left to do is highlight which are the primary terms for each page. I base this on the numbers of searches that they receive, and which are the less important secondary and tertiary terms and variations.

Actually putting your key words and phrases into use is a fairly straightforward matter, and if you’re used to writing your own content it should come naturally. Not all parts of a document are treated equally, though, and you need to know which parts are the most important so you can place your primary keywords accordingly. Read our guide to on-site SEO to discover more.

As a footnote, although I’ve focused predominantly on Google searches and Google’s offering of tools, Bing offer very similar tools and features as do many of the other search engines. The results will also be very similar, but if you are going to be really thorough you should conduct the same exercise using as many of the other search engine and key word suggestion tools available to you. The more you research you conduct into your potential customers and their search activities, the greater your understanding of their intent will be and ultimately the better placed you will be to make sure that your offering meets their needs.

Image Credits: “Postcards and magnifying glass” by Anna (Flickr: records) [CC BY 2.0], via Wikimedia Commons And “Spotter” by Thomas [CC BY 2.0], via Flickr