So, usually we use this blog to do things that are essentially self-promoting. "Look at all the sweet things we build", "Here's an idea to make Alexa better", "We discovered a hidden feature", etc. This is not one of those posts. This one is where we air some dirty laundry, which in turn exposes some issues with the platform as a whole...
tl;dr: One of our live (but unpromoted) skills was broken for three months without ever getting a bad review or being pulled from the store, which means it's likely nobody even noticed. This leads us to a couple key takeaways.
So, the story starts about 6 months ago. At the time, we had released two skills - CompliBot and InsultiBot, both of which were doing pretty well for themselves. We happened to be doing some testing of our live code, when all of a sudden things broke. After a few hours of investigation, we figured out what had happened - Alexa had been silently updated and its request payload had changed in a way that our service could not support. Eventually we hacked in a fix, and Amazon also rolled back the change so that it wouldn't effect others.
There was some drama about notifying devs before updates, etc, but the post-mortem takeaway was that our skills needed to be less strict about how they deserialized JSON payloads. Cool, whatever.
Flash forward a few months to May, and we had a third live skill (DiceBot) and a fourth one we were working on. One day while working on our (still to be released) fourth skill, we noticed it stopped working randomly. Another few hours of troubleshooting and we realized it was the same thing - Amazon had rolled the change back out to production (again, without notification). CompliBot and InsultiBot were working fine with the previous fix, and our in-development skill wasn't live yet, but DiceBot WAS live, and was not able to respond to any requests from Alexa. This was a problem...
There was some drama about notifying devs before updates, etc, but the post-mortem takeaway was that our skills needed to be less strict about how they deserialized JSON payloads. Cool, whatever.
Flash forward a few months to May, and we had a third live skill (DiceBot) and a fourth one we were working on. One day while working on our (still to be released) fourth skill, we noticed it stopped working randomly. Another few hours of troubleshooting and we realized it was the same thing - Amazon had rolled the change back out to production (again, without notification). CompliBot and InsultiBot were working fine with the previous fix, and our in-development skill wasn't live yet, but DiceBot WAS live, and was not able to respond to any requests from Alexa. This was a problem...
Mea Culpa
Now, this is the part where we admit that we can be a little bit lazy. As I said, we knew in May that things were broken, and we had the fix in CompliBot and InsultiBot (we just hadn't merged it up to the parent project we use. Whoops...), so it would've been an easy fix.
At the same time, though, both David and Eric had significant life-events happening at the time, and we hadn't done any promotion of DiceBot yet, so it was easy to let it start to slide. And the thing about a slippery slope is that it's hard to stop slipping.
So, a couple days went by and we were both suuuuper-busy and didn't get the fix in. Then days turned into weeks, and those weeks turned into months, until finally mid-August rolled around we realized we probably needed to handle it.
The fix went in, builds went up to production, and things were back to running properly. All-in-all, we took about 3 months of consecutive downtime on the skill. That triggered some questions, though, and this is what makes the whole episode interesting to us - If our skill has been utterly broken and unusable for 75% of its lifetime, why has nobody noticed?
At the same time, though, both David and Eric had significant life-events happening at the time, and we hadn't done any promotion of DiceBot yet, so it was easy to let it start to slide. And the thing about a slippery slope is that it's hard to stop slipping.
So, a couple days went by and we were both suuuuper-busy and didn't get the fix in. Then days turned into weeks, and those weeks turned into months, until finally mid-August rolled around we realized we probably needed to handle it.
The fix went in, builds went up to production, and things were back to running properly. All-in-all, we took about 3 months of consecutive downtime on the skill. That triggered some questions, though, and this is what makes the whole episode interesting to us - If our skill has been utterly broken and unusable for 75% of its lifetime, why has nobody noticed?
Ergo...
There are two fairly massive implications here that will definitely be interesting to developers, and maybe even to consumers:
- Discoverability is still broken. We've discussed this ad nauseum, and it's a fairly well trodden and clear problem among the development community, but this serves as a pretty clear experimental use case. The fact that we were not piling up negative reviews means that people were clearly not discovering our skill. The way the current system is built, having no reviews means you will never be seen by the users who could give you reviews. It's a catch-22, and it's what leads to shady practices like self-reviewing.
- Recertification is not happening. Amazon has been adamant in their assertion that skills will be revisited by the cert team on a regular basis to make sure that they are still in compliance with the rules. The idea here is to make sure people are not using the bait-n-switch approach - building their skill to match the spec and then changing it to do whatever they want after going live. Had there been even the most minimal recertification test pass, our skill would've failed miserably and been pulled from the store. The takeaway here is that our anecdote implies there are no repercussions for what your skill does after getting through cert. The ramifications of this are especially meaningful given the recent release of the long-form audio feature, and its capacity for IP infringement.
We (both our team, and the entire dev community) have been screaming about the issue of discoverability for close to a year now. The Alexa team is clearly aware of the problem, and they've been taking steps (the new Alexa GUI and voice activation are a start, and the developer marketing team does a great job of highlighting some top tier skills), but the fact remains that their developer outreach has greatly outpaced the platform's capacity to self-regulate.
The result for normal developers like us is a crapshoot where luck and/or deception seem to be the only two ways to get ahead, and where a single imperfect review can bury you forever. And the most frustrating part of all? There are a lot of really simple solutions that the dev community has come up with that could be implemented immediately, but cry as we might, we can't seem to get any traction.
The result for normal developers like us is a crapshoot where luck and/or deception seem to be the only two ways to get ahead, and where a single imperfect review can bury you forever. And the most frustrating part of all? There are a lot of really simple solutions that the dev community has come up with that could be implemented immediately, but cry as we might, we can't seem to get any traction.
RSS Feed
