Beyond Clicks: Discovering a Better Metric than CTR
A recent research paper written by the Personalization Science team at Yahoo Labs presented an effective yet simple, metric to measure and optimize user engagement. This newly-discovered metric is said to rival popular existing metrics, such as item-level click-through rate. It takes into account “the amount of time that users spend on content items, or dwell time”, instead of noisy click-based signals, which do not always effectively portray post-click user engagement. For instance, users might click on an item by accident or because of “link bait”, and as a consequence, may go back to the previous page or may not scroll down the page. The paper presents evidence supporting the claim that utilizing dwell time in a personalized recommender system can effectively capture and optimize user satisfaction.
However, relying on dwell time to measure user engagement can introduce a number of challenges.The following are discussed in the article:
- Users have less dwell time per article on mobile or tablet devices than on desktops.
- Users spend less time on slideshows than on articles.
- Users dwell more on longer articles, up to 1,000 words. Beyond that limit, there is very little correlation to article length.
- Users dwell the most on articles in the topics of politics or science and the least on food or entertainment.
These are just a few ways in which data provided by dwell time metrics can be skewed.
Different users exhibit different “content consumption behaviors”, despite reading the same content on the same device. To account for these differences, researchers at Yahoo Labs have introduced the concept of normalized user dwell time. When they chose to normalize dwell time instead of clicks, they managed to improve overall performance. And interestingly enough, they discovered that optimizing dwell time not only achieved better user engagement metrics but also improved click-through rate.
Sources:
http://yahoolabs.tumblr.com/post/99405569711/science-powering-product-and-personalization-going
http://marketingland.com/yahoo-scientists-measure-dwell-time-alternative-click-rate-102978