Measuring user behaviour: A comparison of products and practical testing

12 min read

Roosi-Ann Tõlgu

BY Roosi-Ann Tõlgu Nov, 17, 2020

Collecting data about your visitors and optimizing your webpage or service is very important in this day and age. But since there are so many tools, which ones should you choose?

Collecting data about your visitors and optimizing your webpage or service is very important in this day and age. But since there are so many tools, which ones should you choose? We compared 12 different tools and tested several of them. This post outlines the main differences between them and also talks a little bit about our own practical trials with some of them. For testing, we had a static website (our company website - https://concise.ee/) and a webapp (Keegliveeb - https://liiga.evml.ee/), both of which were built with React. Since every tool is a little bit different and suited for a different purpose, we have sorted this post into general categories that each includes a few different tools. So if you need…

General data about your visitors

The first step to getting any kind of idea of who, where and when is visiting your website is adding a general-purpose analytics tool that tells you what pages are visited, how long people are staying there and where they are coming from. From that information, you can extract whether or not you are getting to your target audience and if people are getting to the pages you want them to see. It also helps you to see where you should put more marketing efforts (which attribution channels).

Google analytics

Being the most popular user analytics tool out there with 36% of market share, Google Analytics boasts many powerful features including: reporting (audience, advertising, acquisition, behaviour, conversion, real-time reporting, user flows), analysis/visualization, and integration with Google Ads. In the non-paid plan, 50 custom dimension metrics and 5 custom variables are available. Its biggest strengths are:

  • It’s free
  • You can get a lot of data about who your visitors are and where they come from (traffic attribution)
  • Many integrations with other tools
  • It’s usable for both marketing people and developers
  • It’s relatively easy to set up

But it does come with caveats:

  • The UI may take some getting used to
  • You don’t have any governance of your data - Google owns it and uses it as it sees fit
  • There is a limit of 10 million hits per month, which may seem like a lot, but it can grow quite quickly depending on what you’re tracking
  • When you have more than 200,000 sessions per day, you only get reports once per day
  • Sampling of data can skew the statistics
  • You need to have cookie consent from visitors before collecting any data to be GDPR compliant which depending on your field can severely affect how much data you see

We tested Google Analytics for both Keegliveeb and our website. The main things we noticed were:

  • Since we couldn’t collect any data without cookie consent, the numbers GA showed us were relatively low compared to other products (avg. 10 users/day compared to ~20 from Mixpanel) and also a little bit skewed towards the people who clicked “accept” - i.e the percentage of mobile users was a lot higher than other tools showed us
  • Adding the tracking code was very easy, but the UI did seem a little confusing and complicated at first (especially the account/property/view system)
  • Creating charts and viewing data is relatively intuitive and easy
  • It also proved to be very useful for tracking one of our goals - how long are people spending on which pages

Google analytics 360

Vanilla GA is good for most companies for a long time, but if you are dealing with enterprise-level traffic, you might need some more capabilities. That’s when 360 comes into play.

GA 360 allows you to track more than 20 billion hits per month, have 200 custom dimensions and 50 custom variables. In addition, GA 360:

  • Has integrations with many more tools (e.g. SalesForce, Big Query, Doubleclick)
  • Data freshness is guaranteed to be less than 4 hours
  • The amount of data you can collect is essentially limitless
  • You get your own account manager and support
  • And, of course, many more features including unsampled reports, custom tables/reports and ad cost reports

The bad part is:

  • With enterprise-level traffic, comes the enterprise-level pricing. GA 360 pricing starts at $150,000 per year
  • It also may be hard to use if you don’t know what you’re dealing with

Adobe Analytics

Another popular, very thorough tool is Adobe Analytics. It is, like GA 360, best suited for large enterprises and it reflects that in the price - you have to ask for an offer suited for your particular needs, but you can be sure to have it in the tens of thousands of dollars per year.

Open Web Analytics

Contrary to ready-made tools like Google Analytics or Adobe Analytics, Open Web analytics is a self-hosted solution, which has most of the important stuff - Heatmaps, detailed usage of your site (what pages were visited and when), infinite custom site actions, search term tracking, return visitors, conversion goal tracking, goal funnels, campaign tracking and e-commerce tracking. The best parts about it are:

  • It’s free and open-source, so no limits for how much data you collect
  • It’s self-hosted, so you have complete governance of your data

Some drawbacks for these kinds of projects are:

  • The setup is harder than just adding a tracking code to your website
  • The user interface is a bit clunkier
  • It has fewer features than paid tools

Matomo

Matomo, formerly known as Piwik, is another great universal tool for web analytics. Benefits:

  • Includes many useful features, like conversion optimization (heatmaps, recordings, funnels, A/B testing, form analysis, goals), visitor profiles, SEO and integrations with WordPress, WooCommerce, Moodle and others
  • Offers a self-hosted version
  • A great alternative to Google Analytics
  • Data stored in the EU or country of your choice

Downsides:

  • It’s not free like Google Analytics - pricing starts from €19 per month.

We tested Matomo with our company website, mainly to compare it to Google Analytics. We noticed that:

  • There is a library for React for easier setup and also a Gatsby plugin (which we used)
  • It was, subjectively, a little bit easier to use at first than GA
  • A really useful feature was seeing individual sessions - the visitor’s country, browser, operating system, which pages were visited and how much time was spent on them

Heatmaps, clickmaps and scrollmaps

Heatmaps are maps of your website and highlights where users have clicked the most. Movemaps and scrollmaps follow the same principle - where users move and how far they scroll. This information is good for providing easy, visual feedback - i.e. which menu items are clicked, which parts of the page are viewed the most and if your CTAs are visible enough.

Crazyegg

Crazyegg is one of two most popular heatmapping tools. The features include basic heatmaps of your website, scrollmaps, and user recordings (which record some percentage of user sessions) and also some basic A/B testing functionality. It has a few pricing tiers from $24 to $249 a month, with the main differences being in the number of page views (from 75,000/month to 500,000/month) and recordings (from 500/month to 5,000/month) you can collect. The benefits of it include:

  • The simplicity of use - since it is a more focused product, there is not much to learn about using it
  • Easy install - just add a tracking code to your website and you’re done
  • Good mobile support
  • No cookie consent needed - the data is anonymous enough that it can be collected without the user’s permission

The downside is that:

  • It doesn’t have many features that other tools have, e.g traffic reporting and user profiles

We tested Crazyegg for Keegliveeb and found:

  • Adding the tracking code was easy as always and it also has a React library
  • If you are creating a heatmap, it creates 3 different ones for mobile, tablet and desktop (which might be what you need but uses 3x as many of your limited heatmaps)
  • Taking a snapshot is limited to URLs which is not very good for single-page applications

Hotjar

Another popular heatmapping tool is Hotjar, which is a direct competitor of Crazyegg. They share most of the same features (heatmaps and recordings), but Hotjar has another focus - customer feedback. This includes form analysis, customer feedback polls and surveys. Hotjar has two pricing tiers, personal and business, ranging from free to $2,000+ per month with the main differences being in the number of page views and recordings. At a higher price point, you can also remove Hotjar branding from customer surveys. Hotjar has many benefits:

  • Simple to use, as with Crazyegg, but the UI is even a little bit more polished
  • It includes conversion funnels and customer feedback functionality
  • It’s GDPR compliant, so no cookie consent needed
  • There is a free tier

But also some drawbacks:

  • There might be some problems integrating with JS
  • Not many integrations

We used Hotjar on our company website for a couple of different purposes:

  • Seeing where people are clicking - for that purpose, Hotjar worked very well. Generating a heatmap is easy, there are multiple options (i.e. target a single page, similar pages, javascript triggers) depending on what you want, and also better support for SPAs. There exists a React library, but it doesn’t include much, only a method for setting the tracking code.
  • A/B testing cookie banners - we made 2 versions, one with “accept” and “decline” buttons and another with close and “agree” buttons. Setting the test up felt a little bit hacky (Hotjar doesn’t have built-in A/B testing support, so dividing the traffic between two banners was done by choosing a random one to serve directly in the code. After that, we added javascript triggers to the banners and set up heatmaps with the triggers), but it worked and we even got some interesting results. Most people weren’t clicking on the buttons at all, but more people clicked on “accept” when there was also an option to decline (as opposed to just a close button).
  • What actually proved very useful was session recordings - if you don’t have much traffic, then it gives you quite a good picture
  • The main problem with using the free plan is that you have to keep an eye on your snapshots (for heatmaps and recordings) - they fill up every once in a while and then you have to make a new one. Also if the number of those is limited, you must delete old ones as well. This can get a little bit annoying if you have to do it too often.

Software product usage analysis

If you are developing or marketing a software product and would like some deeper insight about what your users are doing, then the following products are for you. These products are action-based (as opposed to session-based), so if you want to track anything, you have to add it to the code. But this allows you to create very detailed funnels and flows, which helps you to see exactly what users are doing in your app, how they are using it and where they are dropping off.

Mixpanel

Mixpanel is one of the biggest players in this field. It is action-based, not session-based (like GA), which means that setting up funnels and conversions is easier and allows for more interpretation. Mixpanel pricing ranges from free to $90 or more per month, depending on your needs. Benefits:

  • Suits well for mobile apps
  • Integrations (Github, iOS, Ionic, JS, Mailchimp, Survey Monkey and more)
  • Easy funnel setup
  • GDPR compliance
  • Integrated marketing platform, including SMS and e-mail functionality

Drawbacks:

  • Doesn’t really measure traffic attribution as well as Google analytics
  • Setup can be a little bit harder since you have to add action logging to places you want directly

We tested Mixpanel for our website and found:

  • The React library was thorough and included the functionality we needed
  • Adding the event loggers was a little bit tedious, but worth it since it was possible to compose very detailed graphs and funnels based exactly on the events we wanted
  • Initially, we wanted to conduct cookie banner A/B testing with Mixpanel, but it seemed that this feature was more suitable for mobile apps

Amplitude

A direct competitor to Mixpanel is Amplitude. They share a lot of the same features, but the main difference is that Amplitude doesn’t offer many marketing possibilities and isn’t as focused on mobile apps. The free plan does get you a little bit farther than Mixpanel’s.

SEO

SEO (search engine optimization) is another very important factor if you want to be visible to the world and improve your ranking on search engines. You may have problems with your site (i.e. broken links) that are affecting your ranking or have gaps in backlinks (sites that link to your site) without even knowing. This is true with keywords as well. SEO tools mostly have a few common features:

  • Site auditing - see how SEO-friendly your pages are and identify common problems (e.g. multiple title tags)
  • Backlink analysis - see which sites link to your site
  • Keyword exploration and generation - generate keywords that would improve your SEO rankings

Ahrefs

The first of the SEO tools in comparison is Ahrefs. It may be a little less well-known than the others, but it is nonetheless a strong competitor. Ahrefs offers four different pricing tiers, starting from $99 and ending at $999 a month, with differences being mostly in the amount and frequency of reports.

SEMrush

SEMrush has the basic features of an SEO tool. When we tested SEMrush for concise.ee, we found that:

  • The UI was mostly clean and simple
  • The site audit suggestions were easy to follow and explained exactly why and how to fix the problems that were found
  • Unfortunately, only Google results and rankings were accounted for (so no Bing, Yandex, etc)
  • There was only one user seat per organization and additional users cost $200 per user
  • The free trial was short (only 7 days)

Moz pro

Moz pro is another competitor to Ahrefs and SEMrush. We also tested this for concise.ee, and found that:

  • The site audit results were quite similar to SEMrush’s, and the suggestions also easy to follow
  • The UI was a little bit less polished, but still usable
  • Results from other search engines were also taken into account
  • There was also only one user per account, but any additional ones cost less ($50)
  • The free trial was a little longer (30 days)

Google Search Console

Google Search Console is of course Google-specific, but it does help you to identify key issues with your website, fix indexing and change the data people see about your organization on Google.

In conclusion, there are many different tools you could use to analyze your web traffic. What we found the most suitable for us were:

  • Mixpanel - mainly because it is very customizable and the funnel functionality helps us to easily identify where people are dropping off
  • Hotjar - because it provides quick, easy and visual feedback of where people are clicking. The recordings also proved to be very useful.
  • Google Search Console - For SEO, we decided it was both cheaper and more suitable for us to use since it was free and offered us enough functionality (other SEO tools were a bit of an overkill for us).