How we taught our Echo to speak Star Wars

Currently Alexa only recognizes English words. Amazon is just now starting to roll out support for other languages, with German up first. But, what if you want it to match words that aren't words?! That's crazy you say? Or is it?!! We encountered this problem, and this post is going to explain how we approached and solved it.

---

---

What crazy thing were we trying to do?

Almost a year ago this week, during a hackathon sponsored by the company the members of our team collectively used to work for, we wrote a skill that would interface with the Star Wars Wikipedia (Wookieepedia) so you could ask it Star Wars questions. The project was code-named: Holocron.

This was our very first team project - so it holds a special place in our hearts. We never published it (as the code and IP belonged to the company), but we thought we'd recount the interesting challenges we encountered, given that many of the things we faced then have once again become topical today.

    "Alexa, ask Holocron who is Luke Skywalker?"
    "Alexa, ask Holocron who is Chewbacca?"

"Holocrons were artifacts used by both the Jedi and Sith that contained ancient lessons or valuable information"

What was the problem?

Nearly every single creature/planet/ship is an "alien name" - otherwise known as a random stream of letters. Just look at the some of bounty hunter's names:

    Boba Fett,  Zuckuss,  4-LOM,  Dengar,  IG-88,  Greedo,  Bossk

That's not even mentioning classic alien creatures like:

    Sarlacc,  Bantha,  Wampa,  Dianoga,  Ewok,  Tauntaun,  Ugnaught

Needless to say, this creates a problem where Alexa cannot determine what the user actually said because it's not a word.

The Wookiee homeworld of Kashyyyk. In 2015 the Echo pronounced it as "Cash-eye-eye-eyek". Now it seems to pronounce it correctly though.

How did we solve it?

We originally tried to solve this in the Interaction Model, but there weren't enough custom slots for the list of possible topics. (This was late 2015, back then I believe custom slots were limited to a few hundred maximum). So we had to solve this problem in software.

We ended up designing our sample utterances to wildcard capture topics from users. It ended up looking something like this:

    "What is {subject}"
    "What is a {subject}"
    "What are {subject}"
    "Who is {subject}"
    "Tell me about {subject}"

And Alexa would force the user input into English words - no matter how badly garbled it'd be.

    "Alexa, ask Holocron what is Sarlacc?"
    subject = "sir lack"   or   "star lack"

    "Alexa, ask Holocron who is Grand Moff Tarkin?"
    subject = "grandma tarquin"

    "Alexa, ask Holocron what is a Bantha?"
    subject = "ban the"   or   "ban thaw"

    "Alexa, ask Holocron who is Zuckuss?"
    subject = "suckas"

By the time the request reached our skill the only thing we had to play with was the subject. We needed to perform a mapping from nonsensical English words into Star Wars subjects.

Sorry Tarkin, but you probably would make a cool grandmother

---

---

Converting Gibberish into Star Wars

Alright, so at this point we have user input which are English words, and we need to determine what Star Wars subject they're talking about.

Our first observation is that these English words do sorta sound like the Star Wars subject, sorta. So let's use that by doing phonetic matching against the list of possible subjects.

After a brief investigation, we settled upon using the Double Metaphone algorithm from the Apache commons-lang library. This follows a set of rules and reduces words down to a "phonetic version" of themselves.

    Death Star = T0STR
    Grand Moff = KRNTMF
    Grand Moff Tarkin = KRNTMFTRKN
    Sarlacc = SRLK
    Boba Fett = PPFT

    Grandma tarquin = KRNTMTRKN

We attempted exact matches, but found it performed poorly. So we cycled through all Star Wars subjects, and calculated a difference measurement between the user input and the subject. From the same Apache library we used the Jaro–Winkler distance measure - which compares two Strings and returns a similarity score.

So for the user input of "grandma tarquin", we got a metaphone value of KRNTMTRKN. And when looping through Star Wars subjects we got values like these:

    Death Star = 0.0
    Grand Moff = 0.88
    Grand Moff Tarkin = 0.98
    Sarlacc = 0.0
    Boba Fett = 0.45

Ah ha! Grand Moff Tarkin scored the highest! The user was probably asking about him!

Subject Conflict Resolution

The above solution works alright for a generic solution. But this still wasn't good enough for our case. We were running into naming resolution issues because too many subjects sounded the same. For example:

    Yavin
    Yavin (star)
    Yavin System
    Yavin 1 up through Yavin 26

The original Star Wars movie ends with the Death Star trying to blow up Yavin 4, a moon orbiting the planet Yavin. When people ask about Yavin, they either mean the planet, or Yavin 4. But Wookieepedia has a page for each of the other moons, even though the pages are basically empty. This is bad because all the unimportant moons match at about the same score,  even though that's not what the user intended.

So we decided to further weight the results by the significance of the article, which we approximated using page size. Our thinking here was that the page on Yavin 4 will be significantly larger than Yavin 12. Thus now when you asked about "Yavin" it would return the page for Yavin 4 as significantly more likely than any of the others.

This in turn led to massive pages eclipsing smaller articles, as nearly every search result became about Luke or Vader because they have massive pages. We had to tune the matching algorithm to find a sweet spot between both behaviors.

We also added in a topic resolution prompt that would ask you to clarify your request if two topics came back with similar scores. This happened with Death Star and Death Star II, or Luke Skywalker and Luuuke Skywalker (I wish I was joking).

Edge cases

It should be noted that this type of solution has more edges than a polygon.

- If there are no words similar to your non-word then this will fail miserably

- This only truly functions if the user input is long enough to actually map into meaningful words. Short words would often never map to an English word, or they'd just come in as blank. The longer the phonetic strings are, the stronger the comparison will be.

- Numbers don't map through the metaphone function. You need to convert them to their word equivalents first.
    Death Star = T0STR
    Death Star 2 = T0STR
    Death Star Two = T0STRT

- Roman numerals like to throw chaos into the blender. We needed to identify roman numerals and convert them to words.
    Yavin = AFN
    Yavin 4 = AFN
    Yavin Four = AFNFR
    Yavin-IV = AFNF

Conclusion

You can let Amazon butcher the phonetic translation of what users say, and then process the results within your skill using phonetic matching. You'll lose phonetic information at each step of the way, but you can still get decent results for certain things.

You can use this to simulate foreign language support in the short-term, but honestly I'd expect poor results. This only functioned for us because we had a very specific problem to solve, and the user input was relatively simple to process. But if you have similar phonetic matching problems (like a product database, or a sci-fi/fantasy corpus), then this could work.

Many Bothans died to bring you this information.