A primer on the current sorting algorithm
- The primary sort sums the number of stars, and divides by the number of reviews. Hence, a skill with three 4-star reviews ( (4+4+4) / 3 ), and a skill with one 3-star and one 5-star ( (3 + 5) / 2 ) review will both have an average rating of 4. This is calculated to at least two decimal places, possibly more. The skills with the highest scores show up first in the list.
- If two or more skills have an identical score, the secondary sort is calculated on total number of reviews. The more the better. So in the example above, the skill with three 4-star reviews would show before the other skill in the list, since its three reviews beat the two reviews of the other skill.
- Finally, if two skills are tied in both of the previous two categories, then the user viewing the skills will see the tied skills randomly sorted relative to each other each time they load the companion app. (As an aside, there's a caching bug in the Alexa browser site that can make certain skills in this situation disappear completely)
So, what's the problem?
Less Batman analogies, more algorithm, please
To achieve this, I propose the following algorithm to calculate the canonical value of a skill based on its reviews (where a higher score is better):
- m is the mean possible score (currently, 3 stars)
- r is the rating value of an individual skill review
- modifier(x,y) is a an arbitrary function that allows us to weight each review's contribution to the score in a non-linear fashion
To start out, lets assume that we do want things to scale perfectly linearly, so we just define modifier(x,y) to be:
- The Wayne Investigation - 36
- Magic 8-Ball - 19
- TIE: Utterly Body Quiz / CheerLights - 2
The Modifier Function
So what modifier should we use?
The question of what we "should" do really depends on which aspect we find more important. If it were up to me, I'd probably start with a linear implementation, see how things turned out, and then tune it accordingly.
What happens to unreviewed skills?
What are the downsides
- No matter how well you tune a system, people will figure out how to game it. One weird benefit of the current system is that it would theoretically be easy for any casual observer to ruin the schemes of someone cheating their average score.
- It doesn't account in any way for the variability among reviewers. One interesting things we've actually seen multiple times when researching posts about reviews is that some reviewers just will not give out full scores. The textual components of their reviews will be chock-full of effusive praise, and somehow they still deign only to grant 4-stars. This is a topic we can (and likely will) talk about at length.
- Skills are coming out fast enough now that even fixing sorting-by-review isn't going to solve all of the discoverability issues going forward. Some sort of faceted classification or search/sort/filter combination is going to be needed eventually.