Design responds to new changing world: ML based interaction design

How can you tell if someone is smiling behind a face mask? COVID-19 has changed everything, and the most significant change we experience is wearing masks in public on a daily basis (before vaccination). Wearing masks is not only crucial but also a life-saving act that helps to prevent the spread of the coronavirus and protect everyone. However, in the early stages of the COVID-19 outbreak, the response to the mask mandate reached an extreme level in the West. It engendered strong backlash amongst the anti-maskers who believed that the mandate violated their constitutional rights. 

Anti-mask backlash rose not only in Western countries but also here in Korea.  Some people responded strongly to perceived threats to their freedom and ignored social distancing rules, putting others at risk. They expressed their refusal to adhere to the new social guidelines via various forms of violence and abuse. These incidents demonstrated that people seemed to be confused about the difference between personal freedom and social responsibility. So how could we overcome this misconception regardless of cultural, political, or ethical differences?

Credit: NBC News

Acknowledging individual freedom while prioritising social responsibility
To nudge people towards better behaviours, we had to deliver a clear and effective message convincingly. I currently run iiD Lab (Intelligent Interaction Design Lab) at Hongik University. We have been working on prototypes using Machine Learning to gently convince people to adopt simple behaviours such as wearing masks. Imagine a stranger passing by without a mask on. How can we take a soft approach to get his or her cooperation without forcing him or her to comply? We focus on research and practice of social interaction using Machine Learning as a solution at the iiD Lab. Here are some prototypes we are working on.

Image-based model from Teachable Machine detecting a face mask. A zombie will spook you if you are not wearing a mask.

We went out in public with this zombie prototype and conducted social studies to observe how people responded. We installed a table PC at a bus stop and waited to see if people reacted to the interaction. Surprisingly, almost everyone was wearing a mask, so it was difficult to experiment. Instead, something unexpected happened. People began to pull their masks down out of curiosity to see a zombie on the screen. Although the experiment went contrary to our intention, we gained lessons from failure and direction for further development.

Using augmented reality (AR), a 3D text animation pops up in your face as a nudge to wear a mask.

Early this year, iiD Lab held a Machine Learning-based interaction design workshop “UNTOUCHED” with Professor Serena Cangiano from Visual Culture Lab at SUPSI, Switzerland. It was a 4-day workshop organised by the Swiss-Korean Science Club, Science & Technology Office Seoul. Creatives from a group of undergraduate and Ph.D students, researchers, and designers participated to share post-pandemic experiences and create touchless interaction prototypes needed in public spaces and sharing services. The goal of this workshop was to activate more socially and culturally acceptable interactions with technologies and services in public spaces.

I want to share some design practices that reflect our cultural background during the COVID-19 pandemic. As of now, the Korean government has extended its current social distancing level for three more weeks: Level 2 in the greater Seoul area and Level 1.5 in other regions. As the social distancing level goes up, limited services at restaurants and cafes are applied, and customers need to follow different guidelines for each level. (e.g. take-out and delivery are only available from cafes in Level 2.5).

Image from Mask traffic light.

At Level 1.5 and 2, customers can stay in cafes but must wear masks when not drinking coffee. “Mask traffic light” is a traffic light system that manages how groups of people consume drinks in bars or cafes. If the light is red, you have to wear the mask while someone else in your group drinks with their face uncovered. When it turns green, you can pull the mask down and consume your drink.

Singing at Noraebang ( “singing room” in Korean) is one of the most popular recreational activities for all ages in Korea. High school and college students go to Noraebang with friends after school, and adults go with coworkers as part of an after-work drinking culture. The pandemic didn’t stop people from going to Noraebang, and some would even go to sing along.

Unlike karaoke bars in America, where people sing in open spaces or private rooms, Noraebang only has private rooms so people can enjoy singing with their acquaintances. Due to lack of ventilation and people sharing mics and remote controls, the surroundings become a health risk during a pandemic. The Korean government imposed social distancing measures for everyone’s safety and all Noraebang venues had to halt operations during Level 2.5.

But what if there’s a way to ensure social distancing in Noraebang? Singing generates respiratory droplets in the air at higher rates, so new safety guidelines need to be implemented. “Untouched Noraebang” came up with the idea of detecting social distancing using machine learning. The prototype was made with two machine learning platforms during the “UNTOUCHED” workshop. Using Teachable Machine, they recorded two sets of images to train a model – a singer holding a mic with a mask on and without a mask on. Then, using a PoseNet model on the runway, they recorded two different poses – one holding the mic closer and the other holding the mic farther from the mouth.

Image from Untouched Noraebang.

As a result, the song would pause when a singer’ took their mask off, and the volume would change depending on the distance between a mic and a singer. The volume would go down as the mic got closer to your mouth and go back to normal if you maintained a distance from the mic.

These design practices aim to encourage compliance with protective measures, such as wearing a mask, keeping your distance, and reinventing public spaces.

An empathetic approach to embracing diversity using new technologies

Korea absorbs and implements new technologies quickly, recognising the need for developing innovative ideas. In the design industry, many designers are trying to broaden the spectrum of design with artificial intelligence (AI).

In the post-pandemic era, we will need to adhere to new international guidelines and social rules to improve our socio-cultural norms. With these design approaches and the advancement of technologies, we will learn how to co-exist and adapt to new changes. In this regard, iiD Lab at Hongik University aims to face challenges and come up with design fused with technology that can make people more comfortable, healthier, and happier.