Write an SMS with Your Eyes Closed: Interview with Yosun Chang, Creator of Tap Shake Messenger
One of the great apps created at the Muther! was Yosun Chang's Tap Shake Messenger. It's essentially a messaging client that translates SMS messages into "haptic Braille", allowing visually impaired people a way to read use messages apart from speech-to-text or text-to-speech. I was really impressed by its utility and clever use of technology -- and so were the judges, as Yosun took home several prizes! I followed up with her this week to find out a little bit more about her app and what else she's working on.
WIP: Hi Yosun, tell us a little bit about yourself.
Yosun Chang: So, I grew up in the SF Bay Area, went away for a while, then came back again. My background is in theoretical physics. My research interests are currently a bit far from that -- mostly in mobile apps, particularly, augmented reality, procedural graphics, AI, as well as immersive virtual simulations. I also explore new UX ideas design that fully utilize unique mobile device capabilities -- beyond just touchscreen, but also accelerometer, gyroscope, mic, and even haptics. I am currently an independent developer, but I also do a bit of freelancing and some consulting on the side.
WIP: The Tap Shake Messenger app you showed off at the Muther! was pretty amazing! Can you tell us what it does?
YC: A working prototype of a cross-platform mechanism that utilizes the touchscreen and haptics on a regular smartphone to let visually-impaired people communicate with accuracy on regular smartphone devices. Using combinations of long vibes and short vibes for output, and long-touch and short touch for input, the app lets you enter each character bit sequentially in "haptic Braille."
The use case envisioned goes like this: A visually-impaired person is lost, and needs a ride back. They might be somewhere noisy, where speech to text is impossible. It's standard, but also limiting, to just ask a passerby for help. But, some of us get smartphone's because we'd rather not have to ask a stranger for directions -- similarly, what if a visually impaired person could gain that degree of independence through such devices? So, here goes: When they first open the app, the app vibrates back the address via reverse geolocation. They can then choose to send a default message to the default contact. They can also choose other contacts to communicate with. Input is through a series of long and short touches that make up the combination of the raised and lowered dots of a Braille character. Output is through long and short haptic vibration's. The entire process is both non-visual and completely discreet. Touch to communicate your words with your eyes closed, then feel these words and other people's words vibrated back to you.
WIP: How did you come up with the idea for it?
YC: I first found out about Immersion's Android SDK at AnDevCon. Everyone always thinks of using Immersion to enhance existing games, but I have a thing for utilizing mobile device features to the max to try creating new user experiences. (I like extreme UX!) Trying out a UI with I/O completely based on haptic and touch input has been something on my long list of to-try's since then! I'm glad I finally got the time to make it happen at Muther!
Suzanne from Immersion had an idea in one of the sessions that went something along the lines of, "What if you could feel a Skype message?" That got me thinking of the app's potential beyond just the visually impaired sector. Back before there were touchtone or even dialtone phone's, kids and hobo's were trained to be morse code operator's (the point being, Morse Code is totally learn-able -- and haptic braille, too!). Now that we'll have Android device's integrated into clothing, what if they could vibrate back the message's in a similar code? I mean, sure, the uber futurists will want brain-Android integration, but that's a tad bit too invasive for me.
The idea of also integrating BlueVia and deCarta was inspired by the categorized prizes at Muther (no surprise!).
And, no - I made what turned out to be an accessibility app not because of the ATT prize - but, because it's nice change to try making a weekend hackathon app that might be able to save the world (as opposed to yet another quirky or mere visualization app)!
WIP: What were the tools you used to create it?
Immersion Haptics SDK in Eclipse/Android platform, deCarta reverse geocoding XML API, BlueVia SMS
WIP: What are your plans for the app in the future? Are you going to continue to develop it?
YC: I'm working on an inline "haptic braille tutorial," that could hopefully get the user familiar with this sort of input and output sequence. And then it's QA. I might put it on Kickstarter for PR.
I think I might also make a receiver-only version for use by office and everyday folks, who receive text messages in the middle of something important -- or late at night -- and want to find out what it is, but might not have convenient (or polite) access to instantly whip out their phone to read the latest rage. Once they learn to feel the vibe's in the "haptic Braille" alphabet, they can just "hear" the vibe's to decipher the message.
As a fun test, I'm planning to release next week a single-purpose "reduced version" of the app... A shake-to-vibe-reverse-geolocation app -- just for kicks and to share a nice party tricks app for the few party psychic-wannabe showoff dork's out there who like that kind of app. The code name is the "deCarta Shaker," but follow me on @yosunchang for updates! (Use case: Make sure your device is fully charged, and the program's on in your pocket. Make a scene -- be like you're gifted with omniscience of location.. Ooh la la. Get your friends to blindfold you, then kidnap you. At any point, you can psychically tell them the exact address or nearest cross street as you shake your bum to get your device to spit back the reverse geolocation in haptic braille. And, when you're bored of this, you can excuse yourself by saying your phone's been vibrating, and there's loads of text messages that you've got to catch up on 
. Disclaimer - Some degree of inebriation may be required for peers of certain tier, and as with any magic trick, the magician must learn the alphabet!)
One thing that will take a while to tweak is actually the haptics. A strong, solid pulse on a Galaxy Tab feels like a jagged pulse on the HTC Evo. This is because their haptic motors are based on different mechanisms. With the extreme device fragmentation in Android, it's going to be interesting QA testing to make sure the haptic feedback is "felt right" on each device. It's a pity that Immersion could not work out a deal with Apple -- it would be so much less device-test-insane if it were on just iOS devices.
This is probably going to be my once-a-month weekend baby for a while. We'll see what happens.
WIP: What were some of the lessons you learned or obstacles you overcame in developing the app?
YC: While it's not hard to do once you are more familiar with the API, it'd be nice if there were an "out of the box" serialization function for playing separate haptic effects back to back.
WIP: What are some of the other projects you're working on?
YC: I have a penchant for making your typical ADHD mobile social games. But, here are the more interesting projects: a dropship-able chocolate design simulation "game", a 3d sculpting app like zBrush / Mudbox but with AR on mobile, and a Shakespearean theatre MMO that fits in your pocket.
I also told Navteq that I'd like to build a real world foursquare-checkins-based first person shooter based on their 3d city data. They don't yet have 3d city data for San Francisco, so until I get the chance to visit Chicago to test out how the gameplay might be, that might be a long time coming!