3PO-LABS: ALEXA, ECHO AND VOICE INTERFACE
  • Blog
  • Bots
  • CharacterGenerator
  • Giants and Halflings
  • The Pirate's Map
  • Responder
  • Neverwinter City Guide
  • About
  • Contact

3PO-Labs: Alexa, Echo and Voice Interface

How we taught our Echo to speak Star Wars

9/20/2016

32 Comments

 
Currently Alexa only recognizes English words. Amazon is just now starting to roll out support for other languages, with German up first. But, what if you want it to match words that aren't words?! That's crazy you say? Or is it?!! We encountered this problem, and this post is going to explain how we approached and solved it.


What crazy thing were we trying to do?

Almost a year ago this week, during a hackathon sponsored by the company the members of our team collectively used to work for, we wrote a skill that would interface with the Star Wars Wikipedia (Wookieepedia) so you could ask it Star Wars questions. The project was code-named: Holocron.

This was our very first team project - so it holds a special place in our hearts. We never published it (as the code and IP belonged to the company), but we thought we'd recount the interesting challenges we encountered, given that many of the things we faced then have once again become topical today.

    "Alexa, ask Holocron who is Luke Skywalker?"
    "Alexa, ask Holocron who is Chewbacca?"
Picture
"Holocrons were artifacts used by both the Jedi and Sith that contained ancient lessons or valuable information"

What was the problem?

Nearly every single creature/planet/ship is an "alien name" - otherwise known as a random stream of letters. Just look at the some of bounty hunter's names:

    Boba Fett,  Zuckuss,  4-LOM,  Dengar,  IG-88,  Greedo,  Bossk

That's not even mentioning classic alien creatures like:

    Sarlacc,  Bantha,  Wampa,  Dianoga,  Ewok,  Tauntaun,  Ugnaught

Needless to say, this creates a problem where Alexa cannot determine what the user actually said because it's not a word.
Picture
The Wookiee homeworld of Kashyyyk. In 2015 the Echo pronounced it as "Cash-eye-eye-eyek". Now it seems to pronounce it correctly though.

How did we solve it?

We originally tried to solve this in the Interaction Model, but there weren't enough custom slots for the list of possible topics. (This was late 2015, back then I believe custom slots were limited to a few hundred maximum). So we had to solve this problem in software.

We ended up designing our sample utterances to wildcard capture topics from users. It ended up looking something like this:

    "What is {subject}"
    "What is a {subject}"
    "What are {subject}"
    "Who is {subject}"
    "Tell me about {subject}"

And Alexa would force the user input into English words - no matter how badly garbled it'd be.

    "Alexa, ask Holocron what is Sarlacc?"
    subject = "sir lack"   or   "star lack"

    "Alexa, ask Holocron who is Grand Moff Tarkin?"
    subject = "grandma tarquin"

    "Alexa, ask Holocron what is a Bantha?"
    subject = "ban the"   or   "ban thaw"

    "Alexa, ask Holocron who is Zuckuss?"
    subject = "suckas"

By the time the request reached our skill the only thing we had to play with was the subject. We needed to perform a mapping from nonsensical English words into Star Wars subjects.
Picture
Sorry Tarkin, but you probably would make a cool grandmother


Converting Gibberish into Star Wars

Alright, so at this point we have user input which are English words, and we need to determine what Star Wars subject they're talking about.

Our first observation is that these English words do sorta sound like the Star Wars subject, sorta. So let's use that by doing phonetic matching against the list of possible subjects.

After a brief investigation, we settled upon using the Double Metaphone algorithm from the Apache commons-lang library. This follows a set of rules and reduces words down to a "phonetic version" of themselves.

    Death Star = T0STR
    Grand Moff = KRNTMF
    Grand Moff Tarkin = KRNTMFTRKN
    Sarlacc = SRLK
    Boba Fett = PPFT

    Grandma tarquin = KRNTMTRKN

We attempted exact matches, but found it performed poorly. So we cycled through all Star Wars subjects, and calculated a difference measurement between the user input and the subject. From the same Apache library we used the Jaro–Winkler distance measure - which compares two Strings and returns a similarity score.

So for the user input of "grandma tarquin", we got a metaphone value of KRNTMTRKN. And when looping through Star Wars subjects we got values like these:

    Death Star = 0.0
    Grand Moff = 0.88
    Grand Moff Tarkin = 0.98
    Sarlacc = 0.0
    Boba Fett = 0.45

Ah ha! Grand Moff Tarkin scored the highest! The user was probably asking about him!

Subject Conflict Resolution

The above solution works alright for a generic solution. But this still wasn't good enough for our case. We were running into naming resolution issues because too many subjects sounded the same. For example:

    Yavin
    Yavin (star)
    Yavin System
    Yavin 1 up through Yavin 26


The original Star Wars movie ends with the Death Star trying to blow up Yavin 4, a moon orbiting the planet Yavin. When people ask about Yavin, they either mean the planet, or Yavin 4. But Wookieepedia has a page for each of the other moons, even though the pages are basically empty. This is bad because all the unimportant moons match at about the same score,  even though that's not what the user intended.

So we decided to further weight the results by the significance of the article, which we approximated using page size. Our thinking here was that the page on Yavin 4 will be significantly larger than Yavin 12. Thus now when you asked about "Yavin" it would return the page for Yavin 4 as significantly more likely than any of the others.

This in turn led to massive pages eclipsing smaller articles, as nearly every search result became about Luke or Vader because they have massive pages. We had to tune the matching algorithm to find a sweet spot between both behaviors.

We also added in a topic resolution prompt that would ask you to clarify your request if two topics came back with similar scores. This happened with Death Star and Death Star II, or Luke Skywalker and Luuuke Skywalker (I wish I was joking).

Edge cases

It should be noted that this type of solution has more edges than a polygon.

- If there are no words similar to your non-word then this will fail miserably

- This only truly functions if the user input is long enough to actually map into meaningful words. Short words would often never map to an English word, or they'd just come in as blank. The longer the phonetic strings are, the stronger the comparison will be.

- Numbers don't map through the metaphone function. You need to convert them to their word equivalents first.
    Death Star = T0STR
    Death Star 2 = T0STR
    Death Star Two = T0STRT


- Roman numerals like to throw chaos into the blender. We needed to identify roman numerals and convert them to words.
    Yavin = AFN
    Yavin 4 = AFN
    Yavin Four = AFNFR
    Yavin-IV = AFNF

Conclusion

You can let Amazon butcher the phonetic translation of what users say, and then process the results within your skill using phonetic matching. You'll lose phonetic information at each step of the way, but you can still get decent results for certain things.

You can use this to simulate foreign language support in the short-term, but honestly I'd expect poor results. This only functioned for us because we had a very specific problem to solve, and the user input was relatively simple to process. But if you have similar phonetic matching problems (like a product database, or a sci-fi/fantasy corpus), then this could work.

Many Bothans died to bring you this information.
Picture
32 Comments
Robert
9/22/2016 11:23:45 pm

The amount of work, skill, and downright brilliance, that went into this unlikely concept. Am I the only one who sees a stark resemblance to the Silicon Valley dick scene?

Reply
Andrew Herrick link
12/29/2016 09:08:25 pm

Brilliant. Great job guys. Just started digging into Dot dev so appreciate your insights... It sure is fun to hack on!

Reply
Dad
2/12/2017 09:51:04 pm

Goodleo i

Reply
buy ready essay link
2/12/2017 09:51:41 pm

I have read the blog updated taught our echo to speak star war articles. Every point is prepared to write best lab articles and reviews. Then everything will be good to know about the blog introduced services and reviews.

Reply
Leaflet Printing London link
2/25/2017 02:10:34 am

Very efficiently written information. It will be valuable to everyone who uses it, including myself. Thanks a lot!

Reply
automatic sliding door London link
4/12/2017 10:03:11 pm

Great information, you have a wonderful blog and an excellent article.

Reply
sash belt link
4/18/2017 03:38:07 am

This is such a nice great website with lots of good information.

Reply
Teamviewer 12 Crack link
7/4/2017 07:30:18 am

nice post

Reply
Wifi hacker for pc link
8/16/2017 07:13:58 am

Thanks for such a great post and the review....

Reply
Download Free Crack Games link
10/23/2017 12:25:24 am

This is very much a work in progress whenever I find out about one who is more beautiful than any of these I will add her and kick out number ten Thanks for sharing the informative post.

Reply
Homepage link
11/22/2017 07:41:53 am

Thank you very much.

Reply
NWSM Building Group Sheds link
2/14/2018 04:10:49 pm

This is a great blog. Thanks for this.

Reply
Happy chick apk link
12/16/2018 07:54:43 am

Hello! I just would like to give a huge thumbs up for the great info you have here on this post. I will be coming back to your blog for more soon.

Reply
nbbnbnnb
7/26/2019 02:50:33 am

hgghgh

Reply
essay help link
2/10/2020 02:57:08 pm

Its going to be the best thing that people are interested in essay writing.

Reply
dissertation help link
2/10/2020 02:58:01 pm

Essayarsenal has made it easier for students to purchase quality dissertations.

Reply
assignmentglory link
2/10/2020 02:59:47 pm

Assignment writing has never been easy so it is wise to get the work done by your fellow people.

Reply
proofreading help link
2/10/2020 03:01:49 pm

Proofreading and writing is a great thing that we should be interested.

Reply
pakrush link
2/10/2020 03:03:15 pm

Pakrush offers great content into everything that matters.

Reply
greenflag link
2/10/2020 03:04:09 pm

Greenflaggers are the guys that produce quality content for you.

Reply
Biodegradable to go food containers link
4/27/2020 04:23:19 am

Great Post!

Reply
samsung 7.5 kg semi automatic washing machine link
9/9/2020 03:33:58 am

Greetings to each one, it's really a particular for me to visit this website page, it comprises of helpful Information.

Reply
CBD Oil link
6/29/2021 03:17:38 am

I always like to read a quality content having accurate information regarding the subject and the same thing I found in this post. Nice work

Reply
meeting scheduling website link
8/17/2021 04:32:28 am

I was extremely satisfied to discover this site. I needed to thank you for ones time for this especially awesome read!! I unquestionably liked every single piece of it and I have you bookmarked to take a gander at new data in your web.

Reply
Rehabilitation Center in Raipur link
9/10/2021 02:48:27 am

Hi, its good piece of writing about media print, we all be aware of media is a fantastic source of information.

Reply
RRB NTPC mock test 2021 link
9/13/2021 03:37:03 am

I’m not that much of a internet reader to be honest but your sites really nice, keep it up! I’ll go ahead and bookmark your website to come back down the road. Cheers|.

Reply
http://images.google.cm/url?sa=t&url=https://www.thepostzilla.com/%2F link
9/15/2021 09:52:31 am

Hi, by using Google I found your site even as pursuing down a basically indistinguishable subject, your site came up, it looks really fascinating. I bookmarked it.

Reply
cat car seat link
9/16/2021 11:46:12 am

I have recently been searching for information approximately this topic for a long time and yours is the greatest. I see something truly special in this internet site.

Reply
Boost WiFi Speed Of Rockspace AC750 Extender link
9/27/2021 03:57:36 am

I was scrutinizing your article and contemplated whether you had thought about creation an advanced book regarding this matter. You're creating would offer it brisk. You have a lot of making capacity.

Reply
Calgary roof repair link
9/28/2021 04:50:48 am

Thanks for sharing this blog with us. It will be beneficial and interesting.

Reply
Tips to create quality Video Content link
1/13/2022 12:55:06 am

Very elegantly composed. This blog has every one of the necessary parts that I was searching for. The focuses are all around featured, expressing the genuine importance. Straightforward and read.

Reply
technology used in hotels link
1/13/2022 02:42:52 am

Posing inquiries are truly critical thing in case you are not understanding something totally, with the exception of this post offers great agreement yet.

Reply



Leave a Reply.

    Author

    We're 3PO-Labs.  We build things for fun and profit.  Right now we're super bullish on the rise of voice interfaces, and we hope to get you onboard.



    Archives

    May 2020
    March 2020
    November 2019
    October 2019
    May 2019
    October 2018
    August 2018
    February 2018
    November 2017
    September 2017
    July 2017
    June 2017
    May 2017
    April 2017
    February 2017
    January 2017
    December 2016
    October 2016
    September 2016
    August 2016
    June 2016
    May 2016
    April 2016
    March 2016
    February 2016
    January 2016
    December 2015

    RSS Feed

    Categories

    All
    ACCELERATOR
    ALEXA COMPANION APPS
    BOTS
    BUSINESS
    CERTIFICATION
    CHEATERS
    DEEPDIVE
    EASTER EGG
    ECHO
    FEATURE REQUESTS
    MONETIZATION
    RECAP
    RESPONDER
    TESTING
    TOOLS
    VUXcellence
    WALKTHROUGH

Proudly powered by Weebly
  • Blog
  • Bots
  • CharacterGenerator
  • Giants and Halflings
  • The Pirate's Map
  • Responder
  • Neverwinter City Guide
  • About
  • Contact