Self-Reviews in the Amazon Alexa Skill Store

One of the of the first signs that your software ecosystem is starting to take off is that people begin trying to game the system.  Well, folks, Alexa has crossed that threshold.

---

---

tldr; People are reviewing their own products (and in some cases organizing others to give them reviews) to make them appear better than they actually are.  I call out several examples in this story.  I also argue why self-reviewing is decidedly not cool.

Community Changing

For many months after it launched, the community surrounding the Alexa Skills Kit was rather pure and egalitarian.  The only people working on it were tinkerers, just looking to build something neat.  There was a strong sense of community among developers, with everyone perfectly happy to help everyone else.

Many of these aspects are still prevalent, but naturally as the install base of Alexa compatible devices began to grow, and the number of people jumping into the development game expanded, we've started to see a more aggressive and competitive undertone in the skill store.  In the right situation this can be great for a platform.  A modicum of competition can help the ecosystem pump out better skills for the users.

Unfortunately, along with the upsides of a marketplace comes the shady underbelly - the grey area where people are willing to do anything and everything not explicitly forbidden in order to make their product "win".  This could show up in all sorts of ways in the "Skill Store" - invocation squatting, naming a skill to show up at the front of the list, etc - but what I want to talk about today is astroturfing.

What is "Astroturfing"?

So, for those not in the know, astroturfing is a fairly recent term that describes the idea of a campaign that is meant to look grassroots (hence, AstroTurf), while actually being organized by an interested party.  Wikipedia gives a good overview of all the ways this might be done, but for the purposes of this article we're using the term specifically to describe a case where the creators of the skill review that same skill as if they were random users who discovered it (rather than the creator).

A Precedent Set

When reviews first became available on the skill store, it was as a slow trickle, with the majority of reviews actually coming from groups like LME and TsaTsaTzu that had already been doing full writeups of new skills on their blogs.  Most skills lived with 1 or no reviews for the first month or so.  It was surprising, then, when one day there was a brand new skill (I'm not revealing the name, for reasons that will become clear later) that had 3 reviews immediately upon launching, all of them 5-star rated.  Especially suspicious was that the skill itself was a business-oriented skill targeting an extremely niche product - not something a random user off the streets could install and use.

A post went up on the development forums about the topic, and it didn't take long for us to discover (thanks, LinkedIn!) that in fact all three reviewers were employees of the company that had just released the skill.  To further exacerbate things, all three reviews were completely unhelpful, saying nothing more than "Great work!".

The response was angry and immediate, with other members in the community filing 1-star reviews to balance the astroturfed reviews.  It sparked a great discussion on the Alexa forums about proper review etiquitte, and at least among a subset of developers there was a consensus that this sort of behavior was inappropriate.  

This specific instance did end up with a happy ending, however, as the proprietor of the skill ended up becoming involved in the conversation, resulting in a mutual agreement to remove the AstroTurf as well as the counter-reviews.  For a brief moment, the Alexa community was granted a reprieve from being spoiled by the internet, but it was clear to everyone that the illusory wall surrounding the platform couldn't hold out much longer.

Wherein I call out those who have crossed the line...

Flash forward to today, and skills are being released at an ever quickening pace. It's getting harder and harder to keep track of what's available on the skill store, and as such it's probably a lot easier for someone to lay down a bed of astroturf while passing it off as real.  That said, I did a bit of browsing, and here are a few examples:

Adnan Riaz

This one is fairly straightforward.  It's a skill with one review, 5-stars, and that review is by the same person who made the skill.

Bryant Schuck

Alright, this one is a bit weaker of a connection since we only have the first name of the reviewer here, but it seems fairly clear that this is the same person (5-star review immediately upon release of the skill, matching first name, no details in the review).

Rick Wargo

Now, this one is sort of interesting.  Rick Wargo has several skills released on the skill store, and he doesn't self-review every time.  In this specific case he's actually responding to two other reviewers who had trouble with his utterances, and therefore gave him a low score.  While I sympathize with his plight - there's currently no mechanism to respond to a poor review - I don't agree with the tactic of "balancing" reviews with your own 5-star rating.

Spriton Systems

Alright, now we're really getting into it.  Here we have a skill called "The Bible" by Spriton Systems. It seems lucky that Mr. Neal Sorensen has stumbled upon it, since he's super impressed with the random scripture feature.  I'd certainly hope he's happy with it, since he built it...

I am not sure what - if anything - the Bible says about the act of lying implicitly or through omission (maybe I should use Spriton's skill...), but I can't help but feel that there's a bit of irony here that a person is trying to cheat the system in order to promote a scripture meant to help you live a more Godly life.

Sterian Associates

Sometimes you really have to do some digging.  Sterian Associates was one of those cases where I had a hunch, and was able to confirm it in a roundabout way.

Alright, so Sterian has two skills on the store currently, both of them with perfect 5-star ratings.  That alone doesn't really mean anything.  Looking into the ratings, we see:

Above, we see the lone review for DogeCoin.

To the right, we have two of the three total reviews for MotoQuote.

What's notable here, is that all three of these reviews are by the same reviewer. In my research for this blog post I didn't recall seeing reviews of any other skills by this person.  Obviously I haven't read and memorized every review on the skill store, but this one happened to catch my eye.  That piqued my curiosity enough to look around a bit more.  The next hop was to the LinkedIn site for Sterian Associates:

Alright, so this doesn't actually prove anything by itself.  It shows that Sterian Associates has a single employee, and his name is Brian Schuster.  Unlike with Spriton, where the reviewer used their actual name to review, we have no way yet of saying that this lone employee of the skill creator was also the reviewer. Luckily, the name they used to review -"HuskyPup840" - is a fairly unique name, so I was able to Google it and only find a few results on the web.  As it turns out, reviews of skills in the skill store show up alongside reviews of other Amazon products...

Neat.  A few things become immediately apparent here, aside from the two reviews that prove that this is the same HuskyPup840.  First, the date of this user's previous review is October 2013, which is well before any Alexa enabled devices were released.  This confirms the notion above, that the user has only ever reviewed two skills.  Further, both skill reviews happened on the exact same day, January 24.  It's extremely improbable that a user would - on the same day - randomly stumble upon the only two skills (of 180 on the marketplace) from the same developer, and review those skills with 5-star ratings.  But there are even more clues here to follow.  On the left, we see the public lists for this user.  One of them is about Natural Language Processing - certainly a topic an Alexa Developer might be interested.  Plus, there's a list called "Brian's Books".  You'll recall that the lone employee of Sterian Associates was "Brian Schuster".  When we click through to that list, we get unassailable proof:

This is in fact a self-review, presumably from an account shared between a husband-and-wife.

Amazon... wait... what?

I got a laugh out of this one, because it's just so blatant.  Yes, Kevin@Amazon, I certainly hope that in your official capacity as an Amazon employee you'd approve of the skill your team produced.

And so many more...

As I was browsing through the skills, I stumbled on at least another half dozen that I'm personally convinced are astroturfed, but which lacked the sort of connective pieces to make a strong case.  As a result, I'm not going to call them out by name, but know that this issue isn't constrained to just these six examples.

So why does this even matter?

Alright, so obviously astroturfing is a thing that is happening.  People are reviewing their own skills to pump up the ratings, but why should anyone care? Isn't this just the way the internet works?  And won't other reviews just drown them out?  Well, there are a few things to consider:

• The Alexa ecosystem, and especially the skill store, are still in their infancy.  One of the strangest phenomena so far has been the spottiness of review volume - some skills are approaching the 100 review mark, whereas others sit unreviewed for long periods of time.  There doesn't yet seem to be a critical mass to drown out biased reviews.
• Along those lines, the skill store's sorting mechanisms are rudimentary. While Amazon did finally add a "best rated" sorting option within the last week, that algorithm looks only at star rating, and doesn't consider number of reviews.  As a result, of the skills listed in this article, all of them regularly show up on the first page of sorted-by-rating results, despite having little-to-no reviews from actual users of their systems.  This is especially bad in that it hides skills that are actually really good.  Something like TsaTsaTzu's Starlanes, which is fairly universally understood to be a top tier skill, is now buried several pages deep.
• This ecosystem is still fairly untainted - I don't think we've reached the point, yet, where we have to concede that the rabid anonymous masses of the internet are eventually going to ruin it.  Something that is standard behavior on other platforms doesn't have to become standard behavior here.

The end result of all this is that the review system is not able to provide meaningful information to new users looking to try something new on one of their Alexa Enabled devices.  No good comes of this from anyone's perspective - the users don't find the best skills, developers aren't rewarded for building the best skills, and Amazon has to deal with the negativity surrounding a system that regularly recommends mediocre or bad skills.

At this early stage, we are faced with something similar to the prisoner's dilemma. If we, as developers for this platform, could all agree to take the high road here, it would be better for everyone.  Barring that agreement, however, we are compelled to make a bad choice just to maintain parity with the other skills on the store.  At this point I have no solutions, just the quandary.