3PO-LABS: ALEXA, ECHO AND VOICE INTERFACE
  • Blog
  • Bots
  • CharacterGenerator
  • Giants and Halflings
  • The Pirate's Map
  • Responder
  • Neverwinter City Guide
  • About
  • Contact

3PO-Labs: Alexa, Echo and Voice Interface

A hidden (and mildly interesting) feature

10/26/2016

4 Comments

 
We here at 3PO-Labs are very curious folk. It's our drive to reverse engineer and deconstruct that initially pushed us to build Responder, so we could poke and prod at the system. This same nature is what led us down the path of discovering the third wake word a few months before it was released. In that same vein, today we want to talk about another secret (but admittedly limited) aspect of the platform that we believe has not previously been discovered...


Alright, before we move on, lets set some expectations. What we found is not going to change much of anything for most developers. It's a couple neat little tricks, surrounded by some code that looks more intriguing than it actually is. There's a slight possibility that people more skilled than we are could find a way to go one step further than us and open some doors that would be game-changing; our failure to do so was certainly not for lack of trying. Further, we should be very clear that this is not an "official" feature, and it likely never will be.

Just tell us what it is, already!

Alright, getting straight to the point, what we've found is that the Alexa companion app is built partially on the Handlebars Javascript framework, and the Alexa team has pre-registered a bunch of Handlebars helpers. The Handlebars helper code is applied to certain bits and pieces of content that we, as skill developers, provide them for the cards, and as such we can execute these functions. We'll talk through how we found this, list out all the helpers (and describe which ones can actually be used), and talk a bit about the implications of this all.

In search of links

So, this all came about one evening while we were trying to figure out how to implement card links. This has been sort of a white whale for us - we've spent countless (fruitless) hours trying to find a way to emulate the links that exist in some of the built-in functions (like sports or weather).

On this occasion we were using ASK Responder to try emulating in a custom skill the link fields used in flash briefing skills. In so doing, we were modifying Responder on the backend, while using Chrome's developer utilities to watch the HTTP transactions on the companion apps. In inspecting the flash briefing content for another skill (NPR on TuneIn) we noticed something curious - the text did not match exactly what we were seeing rendered. Specifically, we were seeing this strange looking pattern in the "subtitle" field:
Partial JSON response for NPR Flash Briefing

    
(As an aside, if you're a developer who has built a flash briefing skill, you're probably saying to yourself "wtf, subtitle isn't a field on that API". Apparently if you're TuneIn, you inexplicably get an extra line of text to display. Forum rant about that is here.)
After a (considerable) bit of investigation, we learned that the funny double-curly syntax is Handlebars, and inside it's calling a function named "formatLongDateTime" with the millis-since-epoch value as a parameter to the function. After crawling through a bunch of Javascript that looked like nonsense  to our backend-developer eyes, we eventually stumbled our way to where the code was being triggered. (On that note, JS minification is the invention of a devious monster who I will forever despise) We found that for certain pieces of returned content, that content was being parsed for references to these functions. In the case of the TuneIn feed item, here's how the conversion looked:
Picture

All the functions we found

So, to go along with "formatLongDateTime" above, there were a few other date/time functions, and then a whole assortment of other seemingly random helper functions. Here is that section of the code in its pretty-printed but still minified form:
Handlebars registered helpers in Alexa Companion apps

    
So, a few notes on these functions:
  • None of these work in the card's title, only in the body.
  • They all work in both Standard and Simple cards. (Note: "flash briefing cards" are actually just Standard Cards with a few specific fields).
  • The date functions all work pretty similarly, and are all valid for both card types.
  • We couldn't get any of the logic functions (if_eq, if_num, etc) to work. They all take an object (with a kind of funny format, actually) as input . We tried hard to make this work. Like, really hard. Our hope was that these functions would be the key to opening up more functionality, but we could never get a well-formed object to make it that far into the stack.
  • The "getImage" function initially caught our eyes as promising. It turns out they're not, though. It's nothing more than a map of image name -> image path.
  • There are a bunch of functions (icon, check box, etc) that will give you back some markup, which you can proudly display as text in your card. Not helpful in any way.
  • Speaking of icon, it's pretty heavily used in other parts of the UI to build links and such.

Uses, open questions, and whatnot

So, we've dug into this code, and found a way to piggy back on some fairly innocuous code. We are not front-end developers, though, and JS is not our cup of tea. There may be more to uncover here that we're just not seeing due to our inability to read this terrible, weakly-typed demon's-code. We'd love for people to help us dig deeper here, and Responder is always available for this sort of trial-and-error. 

Our primary thoughts on things at present are:
  1. There are only a few functions that could be of any use. The datetime functions allow you to use unix epoch and still have content render in the timezone of the user's device. There are also a few functions (pluralize, escape, etc) which could definitely be used if you were too lazy to handle things server-side.
  2. "if_num" (and its brethren) still feels like the key here. If, somehow, you can describe an object with a .inverse or .fn function on it, and get that object to survive all the way to the point if_num triggers, you will have gained the ability to execute arbitrary code. This would give us card links, and much more. (It's worth noting that, as this is JS injection, it could also be used for evil). Unfortunately there's a rather infuriating regular expression that keeps breaking our attempts at this.
  3. What's going on with all the constants? There's a ton of stuff in there that we don't see used. What are "salmon", "amber", "doppler", etc. (We do know that Doppler is the name of Amazon's fancy new headquarters in Seattle, but we're assuming that is coincidence and not related)
Let us know (in the comments below, or via email/twitter) if you discover anything else, have ideas about other things we should investigate, have questions about our methodology, or anything else!
4 Comments
is bestessays legit link
8/3/2019 05:14:39 am

I am happy to know that 3PO-Labs is full of many efficient people. Well, a company is supposed got have productive people so the company can progress. If the people you have in your company aren't good enough in terms of doing their job, I don't think that the company will achieve the success that they want to attain. Your engineers are all efficient, that's why I am sure that you will reach your goals!

Reply
instastalker link
10/9/2019 02:38:01 am

your blog is appreciated. your blog is informative me and cool.thanks for sharing the nice and cool post. there are many nice blogs waiting you on my site, please go to

Reply
mobile phone repair in Strongsville link
2/21/2020 10:57:49 pm


Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work.

Reply
NJ traffic ticket link
8/15/2020 05:24:36 am

Thank you very much for publishing this kind of article. I like your article very much.
I want to share my website details with you please give
me some <a https://www.steinberg.net/forums/memberlist.php?mode=viewprofile&u=219198"> information </a> to increase
performance like as your website.

Reply



Leave a Reply.

    Author

    We're 3PO-Labs.  We build things for fun and profit.  Right now we're super bullish on the rise of voice interfaces, and we hope to get you onboard.



    Archives

    May 2020
    March 2020
    November 2019
    October 2019
    May 2019
    October 2018
    August 2018
    February 2018
    November 2017
    September 2017
    July 2017
    June 2017
    May 2017
    April 2017
    February 2017
    January 2017
    December 2016
    October 2016
    September 2016
    August 2016
    June 2016
    May 2016
    April 2016
    March 2016
    February 2016
    January 2016
    December 2015

    RSS Feed

    Categories

    All
    ACCELERATOR
    ALEXA COMPANION APPS
    BOTS
    BUSINESS
    CERTIFICATION
    CHEATERS
    DEEPDIVE
    EASTER EGG
    ECHO
    FEATURE REQUESTS
    MONETIZATION
    RECAP
    RESPONDER
    TESTING
    TOOLS
    VUXcellence
    WALKTHROUGH

Proudly powered by Weebly
  • Blog
  • Bots
  • CharacterGenerator
  • Giants and Halflings
  • The Pirate's Map
  • Responder
  • Neverwinter City Guide
  • About
  • Contact