As a result of all of the huffing and puffing described in the previous article, as well as our repeated certification failures, we were invited to do a conference call with a few people on Amazon's side. There were a few takeaways that I'd like to share.
tl;dr: They know the process needs work, their team understands the product space well, maybe a little too well for their own good, and the guidelines may be more "imperfect" than "arbitrary".
I want to preface this by saying that the intent of this article is not to describe technical solutions. We'll keep talking about those on the the forums, and do writeups here on the blog if anything is novel or interesting. This post is to talk about what we learned (both implicitly and explicitly) about the Alexa team, process, and state of the system.
First, I wanted to start by saying that Amazon understands that the certification and community engagement processes still need work. They made very sure that we understood that they are trying to streamline the process and make it less of a headache for us. A cynical view of this might be that they were simply trying to pacify us with platitudes, especially after the furor with the HackerNews post, but I legitimately got the feeling that this was a real concern for them and they wanted to make things better. Failed process benefits nobody.
Second, if the people on the call are a representative cross-section, then the people who work on Alexa are all intelligent and well-versed. Now, this statement may seem harsh or overly bombastic in its implication that anyone would have assumed otherwise. The reason I even bring it up is that there was a growing sentiment among the development community that maybe the people in a position to handle certification were sort of "at the bottom rung of the ladder", and that resulted in a strict adherence to guidelines, rather than a more fluid consideration of of each skill on its merits within the context provided. We talked to an engineer from the Speech and Language Understanding team, a Developer Advocate, a member of the Certification Team, and a more "businessy" guy whose role wasn't exactly clear (maybe program management?), and to a person they were all able to speak intelligently and at length about each of the topics we raised. Which brings me to point number three...
It may not be a lack of nuance, but rather too much nuance that causes problems in certain very specific cases. At one point during the call, we were discussing a problem that we had solved in a way they weren't initially happy with, and as we started to explain our reasoning, it turned out they were already familiar with the arguments in favor of our approach. What they weren't expecting was to hear that argument made for a skill of the type we had built - generally the aspect in question applied to a different category. It seems that, because of prior use cases, our skill got mentally classified into a bucket that didn't fit with the approach we were taking.
The problem here is that the context around "why" we did what we were doing is lost when simple classifications are used, and as developers we had no reason to believe classification of this sort would even be happening, and as such had no way to preempt the problem. Once we were able to explain to them that we understood the risks and rewards of our approach, and that our skill actually fell into a grey area between classifications, it was much easier to come to a middle ground that satisfied everyone.
Finally, it was good to hear that there is generally a reason behind most of their guidelines. I won't go so far as to say that the reasons are all "good" or "valid", but they at least feel a little bit less arbitrary. There are a lot of cases where the rules make sense within a fairly narrow definition of how the system is to be used, but where they become less meaningful when people start experimenting with the system's boundaries. I was very happy to hear both the Developer Advocate and Business Guy on the call mention that they are in the process of reviewing the guidelines to understand which of them need clarification, which of them need to be standardized, and which of them can hypothetically have gray areas that require a modicum of leniency.
I don't expect this to be the last time we butt heads with the certification team, but after today I'm hopeful that next time we'll butt heads a little bit earlier, and at less of a break-neck pace.
First, I wanted to start by saying that Amazon understands that the certification and community engagement processes still need work. They made very sure that we understood that they are trying to streamline the process and make it less of a headache for us. A cynical view of this might be that they were simply trying to pacify us with platitudes, especially after the furor with the HackerNews post, but I legitimately got the feeling that this was a real concern for them and they wanted to make things better. Failed process benefits nobody.
Second, if the people on the call are a representative cross-section, then the people who work on Alexa are all intelligent and well-versed. Now, this statement may seem harsh or overly bombastic in its implication that anyone would have assumed otherwise. The reason I even bring it up is that there was a growing sentiment among the development community that maybe the people in a position to handle certification were sort of "at the bottom rung of the ladder", and that resulted in a strict adherence to guidelines, rather than a more fluid consideration of of each skill on its merits within the context provided. We talked to an engineer from the Speech and Language Understanding team, a Developer Advocate, a member of the Certification Team, and a more "businessy" guy whose role wasn't exactly clear (maybe program management?), and to a person they were all able to speak intelligently and at length about each of the topics we raised. Which brings me to point number three...
It may not be a lack of nuance, but rather too much nuance that causes problems in certain very specific cases. At one point during the call, we were discussing a problem that we had solved in a way they weren't initially happy with, and as we started to explain our reasoning, it turned out they were already familiar with the arguments in favor of our approach. What they weren't expecting was to hear that argument made for a skill of the type we had built - generally the aspect in question applied to a different category. It seems that, because of prior use cases, our skill got mentally classified into a bucket that didn't fit with the approach we were taking.
The problem here is that the context around "why" we did what we were doing is lost when simple classifications are used, and as developers we had no reason to believe classification of this sort would even be happening, and as such had no way to preempt the problem. Once we were able to explain to them that we understood the risks and rewards of our approach, and that our skill actually fell into a grey area between classifications, it was much easier to come to a middle ground that satisfied everyone.
Finally, it was good to hear that there is generally a reason behind most of their guidelines. I won't go so far as to say that the reasons are all "good" or "valid", but they at least feel a little bit less arbitrary. There are a lot of cases where the rules make sense within a fairly narrow definition of how the system is to be used, but where they become less meaningful when people start experimenting with the system's boundaries. I was very happy to hear both the Developer Advocate and Business Guy on the call mention that they are in the process of reviewing the guidelines to understand which of them need clarification, which of them need to be standardized, and which of them can hypothetically have gray areas that require a modicum of leniency.
I don't expect this to be the last time we butt heads with the certification team, but after today I'm hopeful that next time we'll butt heads a little bit earlier, and at less of a break-neck pace.
RSS Feed
