3PO-LABS: ALEXA, ECHO AND VOICE INTERFACE
  • Blog
  • Bots
  • CharacterGenerator
  • Giants and Halflings
  • The Pirate's Map
  • Responder
  • Neverwinter City Guide
  • About
  • Contact

3PO-Labs: Alexa, Echo and Voice Interface

The Certification Saga: Problem Solving

1/18/2016

1 Comment

 
As a result of all of the huffing and puffing described in the previous article, as well as our repeated certification failures, we were invited to do a conference call with a few people on Amazon's side.  There were a few takeaways that I'd like to share.
tl;dr: They know the process needs work, their team understands the product space well, maybe a little too well for their own good, and the guidelines may be more "imperfect" than "arbitrary".

I want to preface this by saying that the intent of this article is not to describe technical solutions.  We'll keep talking about those on the the forums, and do writeups here on the blog if anything is novel or interesting.  This post is to talk about what we learned (both implicitly and explicitly) about the Alexa team, process, and state of the system.

First, I wanted to start by saying that Amazon understands that the certification and community engagement processes still need work.  They made very sure that we understood that they are trying to streamline the process and make it less of a headache for us.  A cynical view of this might be that they were simply trying to pacify us with platitudes, especially after the furor with the HackerNews post, but I legitimately got the feeling that this was a real concern for them and they wanted to make things better.  Failed process benefits nobody.

Second, if the people on the call are a representative cross-section, then the people who work on Alexa are all intelligent and well-versed.  Now, this statement may seem harsh or overly bombastic in its implication that anyone would have assumed otherwise.  The reason I even bring it up is that there was a growing sentiment among the development community that maybe the people in a position to handle certification were sort of "at the bottom rung of the ladder", and that resulted in a strict adherence to guidelines, rather than a more fluid consideration of of each skill on its merits within the context provided.  We talked to an engineer from the Speech and Language Understanding team, a Developer Advocate, a member of the Certification Team, and a more "businessy" guy whose role wasn't exactly clear (maybe program management?), and to a person they were all able to speak intelligently and at length about each of the topics we raised.  Which brings me to point number three...

It may not be a lack of nuance, but rather too much nuance that causes problems in certain very specific cases.  At one point during the call, we were discussing a problem that we had solved in a way they weren't initially happy with, and as we started to explain our reasoning, it turned out they were already familiar with the arguments in favor of our approach.  What they weren't expecting was to hear that argument made for a skill of the type we had built - generally the aspect in question applied to a different category.  It seems that, because of prior use cases, our skill got mentally classified into a bucket that didn't fit with the approach we were taking.  

The problem here is that the context around "why" we did what we were doing is lost when simple classifications are used, and as developers we had no reason to believe classification of this sort would even be happening, and as such had no way to preempt the problem.  Once we were able to explain to them that we understood the risks and rewards of our approach, and that our skill actually fell into a grey area between classifications, it was much easier to come to a middle ground that satisfied everyone.

Finally, it was good to hear that there is generally a reason behind most of their guidelines.  I won't go so far as to say that the reasons are all "good" or "valid", but they at least feel a little bit less arbitrary.  There are a lot of cases where the rules make sense within a fairly narrow definition of how the system is to be used, but where they become less meaningful when people start experimenting with the system's boundaries.  I was very happy to hear both the Developer Advocate and Business Guy on the call mention that they are in the process of reviewing the guidelines to understand which of them need clarification, which of them need to be standardized, and which of them can hypothetically have gray areas that require a modicum of leniency.

I don't expect this to be the last time we butt heads with the certification team, but after today I'm hopeful that next time we'll butt heads a little bit earlier, and at less of a break-neck pace.
1 Comment
Jo Jaquinta link
1/19/2016 07:19:47 am

Color me skeptical. I've been on the other side of these phone calls a lot. So I'm seeing things between the lines that may, or may not, be there.

>They know the process needs work
They have said this all along. It doesn't mean they are going to do anything about it. Or what priority they think doing something about it is. Or that the bits they think need work are the bits we think need work.

Fundamentally, they've been changing it, so it's a no-brainer that they think it needs work. However, the changes have not been good from our perspective. So I think there is still a disconnect between what we think needs changing and what they think needs changing.

>their team understands the product space well
I think they think they do. But I don't think the product space they think they know is the product space I see. Their focus is on stateless skills serving home automation. And that is the focus of their process. There are many more possibilities than that.

>the guidelines may be more "imperfect" than "arbitrary"
The problem is not with the guidelines, but the application of them. And that, I can definitively prove, is arbitrary.

Reply



Leave a Reply.

    Author

    We're 3PO-Labs.  We build things for fun and profit.  Right now we're super bullish on the rise of voice interfaces, and we hope to get you onboard.



    Archives

    May 2020
    March 2020
    November 2019
    October 2019
    May 2019
    October 2018
    August 2018
    February 2018
    November 2017
    September 2017
    July 2017
    June 2017
    May 2017
    April 2017
    February 2017
    January 2017
    December 2016
    October 2016
    September 2016
    August 2016
    June 2016
    May 2016
    April 2016
    March 2016
    February 2016
    January 2016
    December 2015

    RSS Feed

    Categories

    All
    ACCELERATOR
    ALEXA COMPANION APPS
    BOTS
    BUSINESS
    CERTIFICATION
    CHEATERS
    DEEPDIVE
    EASTER EGG
    ECHO
    FEATURE REQUESTS
    MONETIZATION
    RECAP
    RESPONDER
    TESTING
    TOOLS
    VUXcellence
    WALKTHROUGH

Proudly powered by Weebly
  • Blog
  • Bots
  • CharacterGenerator
  • Giants and Halflings
  • The Pirate's Map
  • Responder
  • Neverwinter City Guide
  • About
  • Contact