Using artificial intelligence and a thermal imaging camera, two master’s students and a visiting student have taught computers to interpret hand gestures. The results are published in the prestigious IEEE Sensors Journal.
Getting computers to understand hand gestures can be helpful in many contexts. It can help people with speech impairments communicate with digital assistants and in interaction between human and computer, for example.
One of the challenges for machines has been to interpret hand movements in varying light conditions.
Now the master’s students Daniel S. Breland and Simen B. Skriubakken, together with exchange student Aveen Dayal, have developed a method that solves this problem. They use a thermal camera and artificial intelligence.
“We started by building a database of thermal images of different hand movements. We had 320 images for every hand gesture for the numbers from zero to ten, a total of 3200 images. It is not a large sample, but it was enough to make it work”, says Breland.
The technique has been named Deep Learning-based Sign Language Digits Recognition from Thermal Images with Edge Computing System. The students’ paper on the findings was recently published in one of the leading journals for sensors and signal processing, the IEEE Sensors Journal.
The journal is ranked as a level 2 journal, and thus one of the most significant in the field.
“We were delighted to have our paper published in such a prestigious journal, also because they didn’t request many changes in the text”, says Breland.
The system they developed has a handy size. The actual computing power came from a Raspberry Pi, a computer that fits in the palm of your hand. The images were taken by a thermal camera that takes low-resolution images at 32x32 pixels.
Despite the simple tools, the system has an accuracy of 99.52 percent on the test data.
The students have worked together with the research group Autonomous and Cyber-Physical Systems, led by Associate Professor Linga Reddy Cenkeramaddi at the Department of Information and Communication Technology.
“It is nice to work with such highly motivated, sincere and talented students”, says Cenkeramaddi.
He says it is fascinating to see how thermal imaging works even when the room is in complete darkness.
Breland is also very pleased with the collaboration with the research group, which has contributed with equipment and guidance during the project.
“There is broad knowledge within the group, and a good working environment. Everyone wants to support each other, and that helps with the motivation and progress in projects like this”, says Breland.
He completed his master’s degree this spring and will start a job at the Norwegian Communications Authority where he will work with 5G and the Internet of Things (IoT).
“Unfortunately, I will not continue with the research, but I will keep working on the same things”, he says.