When communicating via written text with a computer, the auditory and visual cues that normally convey expressions of emotion are absent. This removes important context from conversations and often leads to misinterpretations of the author’s original intent. Studies have shown that, overall, we are much less adept at expressing our emotions in writing and interpreting the emotions of others than we think. So not only are we doing it wrong, but we are pretty sure that our incorrect interpretations are right. This is a formula that often leads to misunderstandings, hurt feelings, or bad business decisions.
Paying attention to the language you use and proofreading the documents before sending them can help, but it does not go any further. Emoji in one form or another have been with us for a long time, but they have really come into common use with the growing popularity of smartphones. These little icons that can be embedded into text have the power to change the perception of a sarcastic comment from a cruel dig to a good-natured joke between friends. On some devices, like smartphones, they’re pretty easy to insert into a conversation. But on laptops and desktops with standard keyboards, they’re still a bit of a pain to use, requiring memorizing complex keyboard shortcuts or launching on-screen selectors that break the normal flow of typing. A frictionless solution to including emoticons in text could go a long way to solving the communication problems so often encountered through this medium.
OpenMV Cam H7 R2 (📷: S. Mistry)
Sandeep Mistry of Arm recently blogged about a project he did that serves exactly that purpose. It used computer vision and tinyML to detect when the person using a computer performs a gesture, such as a thumbs-up, then automatically “types” the corresponding emoji code for it to appear on the screen. The project is based on an easy-to-use and inexpensive microcontroller development board with an integrated image sensor that serves as an all-in-one hardware solution. It was programmed with commonly used open source tools such as TensorFlow Lite for microcontrollers, allowing almost anyone to replicate this project on their own.
The release is powered by the OpenMV Cam H7 R2 with an STM32H743VI ARM Cortex M7 processor running at 480 MHz and 1 MB of SRAM. That might not seem like much if you’re used to running machine learning algorithms in the cloud, but it’s more than enough for this application when pairing TensorFlow Lite for Microcontrollers with the CMSIS-NN software library which maximizes performance and minimizes memory. footprint, neural networks running on Arm processors. Mistry found that this setup could process 96 x 96 pixel grayscale images at 20 frames per second.
Overview of processing pipeline (📷: S. Mistry)
Example gesture images would be needed to train a machine learning classifier, so a gesture recognition dataset was downloaded from Kaggle, then the images were reduced to a set that represents thumbs up, thumbs down, hand up, fist and no gesture. In total, about 14,000 images made up this training dataset. Once the data collection was complete, Mistry created a MobileNetV1 model in TensorFlow Lite and then started the training process. Once completed and the model deployed to the OpenMV Cam H7 R2, Mistry demonstrated that he was able to perform a hand gesture in front of his computer and have the emoji appear automatically on screen in a text editor. . This was accomplished by using the OpenMV Cam as a USB user interface device to simulate keystrokes when a particular gesture was recognized with a high degree of confidence.
Right now, the device is limited to just four simple hand gesture emoticons. As you can imagine, extending this method to a much larger set can be difficult, if not undesirable. Who wants a smiley face tapped every time they manage to smile? And how useful would it be if you had to shed fake tears to get a sad face? This device may not be the optimal solution for all cases, but it’s a great way to learn about computer vision and tinyML, so be sure to check out the blog post so you can follow at home.
#Alt #Emoji #input #device