How reviews are sorted
In sorting by Avg. Customer Review, the following rules are applied by Amazon:
- Sort by average review score, with highest average coming first
- Among all skills with a given average score, sort by number of reviews, with most reviews coming first.
- Among all skills with a given average score, and a given number of reviews, randomize all entries (note: this may not be an explicit randomization, they may just be randomized to begin with, and no additional sorting is done).
Lets look at how this actually works with current skills. As of February 6, 2016, if you open up the list of skills and sort by Avg. Customer Review, this is what you will see at the top of your list:
Does Any of This Even Matter?
Whereas the values for InsultiBot are fairly similar (the pattern and numbers of week two are pretty close to the first week), CompliBot completely diverges starting on the 26th, with a continued upward trajectory that is not matched by InsultiBot. It immediately becomes the higher-utilized of the two by both metrics, and easily holds on to that mantle.
Ergo...
- Quash astroturfing.
- Give us an avenue to work with those who give us negative reviews.
- Fix the ranking algorithm. Right now, anything other than 5-stars is a "bad" review. Alexa is certainly not the first to face this issue...
Let us know if you've run into any situations like this, or have similar supporting data. You can drop a comment below, or hit the contact page to email us directly.
2: It's also worth noting that, during this time period, we made no changes to the services on our side, nor did we do any advertising nor retrieve any official press. The only variable that changed during this period was the single 5-star review granted to CompliBot. We clip this data at the 29th, because at that point we got our first InsultiBot review, got an additional CompliBot review, and did our first content patch.