Measuring Meaningful Engagement
As alternatives in banking emerge, banks may find it difficult to prove their value to customers. Why should the customer dedicate time to their bank when they could just as well give it to Snapchat or Apple via their banking apps? We at Meniga believe the key to maintaining an edge over alternative banking services is to build the banking experience on meaningful engagement by building positive habits and catering to long-term relationships with customers. In our experience, many banks are willing to explore this path but, when the time comes, lack the tools to measure their success.
In this piece, we discuss how Meniga measures user engagement, how the measurement compliments the idea of meaningful engagement and how banks can easily implement the same measurement.
The Engagement Index
A while back we decided to start researching how banks and other organisations measured success in general and whether we could find something to compare ourselves to. We weren’t very surprised when we found that most organisations still use classical measurements; Daily or Weekly Active Users, Retention and App Installs. But looking at App Installs to measure engagement is like looking at fertility rates to predict a country’s results in the PISA survey. We came to the conclusion those indices could only vaguely imply usage while they ignored a person’s actions and intentions and could therefore not be a measure for engagement.
We felt an engagement measurement should
- Capture how the product is used,
- Capture temporal indicators of the product usage,
- Be standardised, and
- Be measurable on the person level.
Some time ago we found a paper by Eric T. Peterson & Joseph Carrabis, Measuring the Immeasurable: Visitor Engagement, which describes an index that meets all the requirements above. Even though the paper was written in 2008 and focuses on measuring engagement with blog and sales sites rather than mobile apps, this was a great inspiration for what we wanted to achieve.
The idea is simple: There are many qualities that indicate engagement. The more qualities a person displays strongly, the more engaged they are. With little modifications, We implemented some of the measurements described in the paper and refer to it as our Engagement Index.
The qualities Meniga believes indicate meaningful engagement with the app are:
- Loyalty: How often each person used the app in the last few days.
- Recency: How long it’s been since each person used the app last time.
- Consistency: How long each person has used the app on a regular basis.
- Interaction: How often each person actively interacts with high-engagement modules of the app.
- Click Count: How often each person dives deep into the app.
- Duration: How often each person spends long periods of time within the app.
Each of the above qualities are measured by an index on the scale of 0 to 10, described below. Therefore, for a given period, a person can display some of the qualities in a faint manner (index below 2.5), some in a more strong way (index between 2.5 and 7.5) and some very strongly (index above 7.5). The Engagement Index, the average of those sub-indexes, is then used as an indicator of the person’s overall engagement.
As a general rule, we will want the Engagement Index to increase over time. That is why we have designed it in such a way that there is room for improvement. Furthermore, since the Engagement Index is calculated down to the person level, we can compare the index between user segments, A/B groups or use it to cluster people during deeper analysis.
Let’s look at how we at Meniga have used it to gain insights into our user base and then how each of the six qualities is measured.
Challenging Yourself Increases Your Engagement
One of the things we have used the Engagement Index for is to compare engagement between user segments. Take for example a part of our challenges analysis. Meniga’s challenges are a way for users to challenge their spending patterns in order to save here and there and use the savings elsewhere. We segmented people based on their challenge activity and above, we see the Engagement Index between the segments. In the first segment are the people that have at no point accepted a challenge and in the second segment are the people that have accepted at least one challenge.
Using those segments, we see that people that accept challenges are significantly more engaged with the app than people that don’t. They are more invested to engage with Meniga and, thereby, their finances.
This is but an example of how the Engagement Index enables us to compare different groups of people. But how is the Engagement Index calculated?
The Sub-Indices
As mentioned above, each of the qualities of an engaged person is measured by a separate index, each on the scale of 0 to 10. The first three indices are similar in the way that they rely only on when the person uses the app, and not on what they do. However, the later three indices rely only on how people behave when using the app. The Engagement Index is then calculated as the average of those sub-indices.
First of the indices is the Loyalty index which is calculated from the number of sessions per week and is given by the equation
So, a person that only uses the app once a week has Li = 0, a person that uses it twice has Li = 5, and a person that has used the app seven times in a week has Li = 6.7. As stated above, we feel that a person that is active at least twice a week is likely to understand the value of frequently checking the status of their accounts, be actively challenging their financial habits or using CLO on a regular basis. This person is therefore considerably more engaged with the product than people that have only been active once in the last seven days.
Similarly to the Loyalty index, the Recency index measures usage frequency. The Recency index is a function of the time since last time the person used the app and is given by the equation
By this formula, a person that used the app yesterday and again today has Ri = 10, but a person that used the app five days ago has Ri = 2. The Recency index measures a slightly different type of frequency than the Loyalty index as one can have high Loyalty but low Recency for a given day and vice versa.
Those two indexes work very well to measure the activity of people that use the app at least weekly. Those measurements indicate a type of habit. However, not all Meniga users do that. We have a large group of people that only use Meniga for reviewing the year and budgeting the next, for calculating their spending in certain categories each month or for some other reason only need Meniga every now and then, but periodically. Those people have developed a long-term relationship with Meniga. This is why we introduce the Consistency index. The Consistency index is measured by how many preceding months, quarters or years in a row people have been active. To calculate this, we need historical data as far back as possible. The index is calculated thus
where time units can be months, quarters or years. A person that has used the app twice a month for the last three months therefore has COi = 6.7, the same as a person that has used the app only once a year for three consecutive years. This index is starts at 0 for all new users and is likely to become higher as the person ages, if the person uses the app every month, quarter or year.
We have looked at how we measure the times at which people are active. We will now continue to the indices that measure how the people are active. First of those is the Interaction index. The measurement of the Interaction index is based on the assumption that in order to perform some actions, the person needs to be in a relatively engaged mindset and therefore doing those actions is a token of engagement or intention. Following is the equation.
where an interactive session is any session where the person performs an action considered interactive. The definition of interactive actions varies greatly between apps and scenarios have to be considered specifically for each. For the Meniga app, we have identified over twenty actions to be interactive, ranging from recategorising transactions to starting a spending challenge or activating an offer. Many of the interactive actions are a part of our efforts to build positive habits, but they are not actions like opening the app, scrolling through or other trivial actions. For Twitter, examples of interactive actions might be tweeting, retweeting and searching for an account. Using those examples we see that person that opened Meniga four times in the last week and in one session they accepted some challenges and in two of the other session they recategorised transactions has Ii = 7.5 while a Twitter user that opened Twitter five times in the last week and only in one of the sessions they did anything else than scroll through their feed to kill time has Ii = 2.
In the next two indexes, the Click Count index and the Duration index, we use a similar approach as the Interaction index. We count the sessions where a certain criteria is met and divide it by the total number of sessions. For the Click Count index the criteria is met when the person does a certain amount of clicks or taps, and for the Duration index the criteria is for a sessions length to be of certain length. Those are based on the idea that a person that often clicks deep into an app or stays long is likely to be there for a reason and thereby be an engaged person. The equations for the Click Count index and the Duration index are
and
Those three indexes, the Interaction, Click Count and Duration indexes, measure similar types of engagement but it is easy to imagine a scenario where a person only scores high on one or two of them and not on the others. That is why we feel none of the indexes is sufficient alone but each contributes their own flavour to the whole.
Joining the Dots
Now that we have calculated all the indices for each person, combining them into the Engagement index is a simple question of averages. Since each index is on the scale of 0 to 10, the Engagement index will be on the scale of 0 to 10 as well, no matter if we use weighted averages or simple averages. This is up to each organisation to choose, along with the thresholds for the Click Count index and Duration index and the definition of interactive actions.
With all of this intact we have created a new indicator for our analysis arsenal. We should be well equipped to take on the challenge of meaningful engagement and measure the impact.
