3PO-LABS: ALEXA, ECHO AND VOICE INTERFACE
  • Blog
  • Bots
  • CharacterGenerator
  • Giants and Halflings
  • The Pirate's Map
  • Responder
  • Neverwinter City Guide
  • About
  • Contact

3PO-Labs: Alexa, Echo and Voice Interface

On the topic of certification...

1/10/2016

2 Comments

 
So, yesterday I put out what was essentially a plea for help on the Alexa front.  Our skills were rejected by the certification team again, despite making exactly the changes that they told us to make the first time.  I was hoping the conversation with some of the other developers would spur Amazon to take action...
Flash to this afternoon, and I started getting messages telling me my rant had been featured in a blog post that went to the top of HackerNews (see: https://news.ycombinator.com/item?id=10876409).  Surprising, but exactly what we needed!  Amazon's certification process is broken, and if the constant griping from their developers wasn't enough to get the ball moving, maybe a bit of internet shaming would be.
In any case, the HN thread has spurred a good conversation about the value of certification in an almost entirely mutable ecosystem - especially a brand new one where best practices and UX are not well defined or understood. Those insights cover most of the salient points, and there's no need for me to retread them...
What I'd rather talk about, then, is the broader subtext here - that Amazon has spent a ton of time building this oppressive certification infrastructure which is completely unsustainable even for people who are willing to play by their rules. 
Take for example the testing facet:  As part of certification, we are expected to have done functional testing of our skills.  

Ignore for the moment that this is a nigh impossible task to do in a repeatable way, since they provide no test automation hooks - the real kicker here is that they are constantly shifting the goalposts in this regard.  In the same way that we are capable of modifying our services after certification, the Alexa team is capable of modifying their voice model in a way that breaks production services.  Now, you may be saying to yourself "Sure, Amazon could do that, but they wouldn't, because they are a rational business who wants to maintain a high level of quality of their products", and you would be absolutely wrong.

As part of our initial rejection, the certification team gave us some suggestions they thought might help the hit-rate of our voice interface (it's worth noting that this sort of exchange is exactly the type of helpful thing the certification team could be spending its time on, instead of policing an unnecessary spec).  We went to work trying out these changes and undergoing a rigorous manual test pass after each change, comparing with various baselines.  After one particularly late evening of testing, we came back the next morning to see that things were no longer working, and we couldn't understand why.  Nothing on our end had changed at all (as an aside, this is exactly why baselines are valuable), but all of a sudden our successful invocations had begun failing.  We tried getting input from Amazon employees on the forums, to no avail.

The running theory here is that Amazon decided to change the enumeration they use to describe US first names.  Originally it was built as a list of the top N (maybe 1000?) names in the US?  Over that night, they seem to have changed it to include every single name registered in the US Census.  We know this because we were able to look at our new found failures and see that they were tied to phrases coming in that matched words we had never seen before.  What we noticed was that the pronoun "me" was all of a sudden being replaced by obscure names.  

There had always been the occasional false positive hit for the name "May" instead of the word "me", but all of a sudden we were seeing a massive spike in the failure rate, with words like "mi" and "mei" also showing up where they hadn't before.  The real smoking gun, though, was when we started matching "Mayme".  If this word looks obscure to you, that's because it is.  It is a female name that peaked in the 1880s, and has been almost non-existent among new births in the last 100 years.  It exists in no form other than as a proper noun, and we had seen exactly zero instances of it before this mystery event.  After the event, it became a fixture of our newly flakey interface.
Now, this is clearly anecdotal, and it seems obvious that Amazon has to have the ability to update their service to improve it.  But the corollary is that in granting them this capability we have also granted them carte blanche to decimate our interface, and we would've had no idea that it was happening if it weren't for the fact that we were slogging through a daily manual test process.  If we caught them breaking us, we have to assume that they've broken other skills as well and nobody was the wiser.

This is simply not a sustainable model.  To bring things full circle, I think the takeaway is that this is not a two-way street between the Alexa team and the development community.  They've designed a paradigm wherein we, as developers, are not able to make decisions about the best way to implement our own code, while at the same time they've undertaken a weirdly silent and proprietary approach that doesn't allow us to account for the changes they force on us.  

It feels very much as if they don't actually want us there - as if they feel they are doing us a service by opening up for development, but that it's a service they absolutely disdain.  

It's not hard to see the poison seeping into their new Alexa ecosystem day by day, and that's a real shame.
-Eric
P.S.: I want to call out both the blog post by Lawrence Krubner (http://www.smashcompany.com/business/amazon-has-no-idea-how-to-run-an-app-store) and the various attempts Jo Jaquinta has made (http://ocean-of-storms.com/tsatsatzu/explaining-amazons-indifference/) to try to get traction on the process.  Thanks for continuing to push on this.
2 Comments
Jo Jaquinta link
1/10/2016 06:51:47 pm

One point to be made is that those conducting certification are not subject matter experts. When you count the heads and add up the numbers, you can work out this is a side job for them. My theory is that the lowest on the totem pole get assigned certification so they can build their knowledge of the platform. Give the high turnover rate in Amazon, it means we're often dealing with someone with less Alexa experience than the people doing the submission. >_<

Reply
Thomas Worthington
1/11/2016 06:00:17 am

I sort of think this is no big deal. Amazon are really founded on the principle of treating content creators as dirt; it's why I'm boycotted them for a decade. They have acted as if they are doing writers a service by selling their books forever and it's never done them any financial harm as far as I can see; this won't be any different.

Reply



Leave a Reply.

    Author

    We're 3PO-Labs.  We build things for fun and profit.  Right now we're super bullish on the rise of voice interfaces, and we hope to get you onboard.



    Archives

    May 2020
    March 2020
    November 2019
    October 2019
    May 2019
    October 2018
    August 2018
    February 2018
    November 2017
    September 2017
    July 2017
    June 2017
    May 2017
    April 2017
    February 2017
    January 2017
    December 2016
    October 2016
    September 2016
    August 2016
    June 2016
    May 2016
    April 2016
    March 2016
    February 2016
    January 2016
    December 2015

    RSS Feed

    Categories

    All
    ACCELERATOR
    ALEXA COMPANION APPS
    BOTS
    BUSINESS
    CERTIFICATION
    CHEATERS
    DEEPDIVE
    EASTER EGG
    ECHO
    FEATURE REQUESTS
    MONETIZATION
    RECAP
    RESPONDER
    TESTING
    TOOLS
    VUXcellence
    WALKTHROUGH

Proudly powered by Weebly
  • Blog
  • Bots
  • CharacterGenerator
  • Giants and Halflings
  • The Pirate's Map
  • Responder
  • Neverwinter City Guide
  • About
  • Contact