Social Interaction

Projects | | Links:

From Bimodal to Multimodal: Augmented MELD for Emotion Recognition in Conversations (Independent Undergraduate Research)

Problem: Multimodal emotion recognition in conversation
Challenge: Lack of a large multimodal emotion recognition dataset in multiparty conversations
Solution: Augmentation of the MELD dataset with facial expression data and implementation of a multimodal model with a fusion of bcLSTM and VGG16

Data Filtering: Restricted dialogues to those of the 6 main characters to enhance performance in challenging video-based speaker identification and localization
Data Augmentation: Augmented the MELD dataset with visual features using DLib-based face detection and recognition
Multimodal Model: Implemented a multimodal model with a late fusion of bcLSTM and VGG16

Statistical evaluation: Improvement of emotion recognition accuracy by up to 10% compared to the baselines without visual features
Poster Presentation: Delivered a poster presentation at the “Brain-Mind-Behavior Individual Study Symposium.”

Problem: Developing engaging open-domain conversational agents
Challenge: Large language models (LLMs) struggle to produce diverse empathetic responses and maintain self-concept in long-term conversations
Solution: An interdisciplinary study of 2,300 human open-domain dialogs with self-labeled psychological variables to identify key factors for engagement and propose a new approach to enhance LLM-based conversational agents’ social characteristics

Dataset Collection: Identified reciprocity of self-disclosure, counter-questions, and empathy as key engagement factors through a correlation analysis between dialogue patterns and participant ratings
Pilot Experiment: Shaped crowdsourcing guidelines for the dataset by conducting a pilot experiment and incorporating participant feedback
Data Construction: Processed the unstructured, crowd-sourced dialogue data into a structured schema for subsequent analysis

Construction of an open-sourced OPELA dataset, consisting of 2.3K dialogues with psychological attributes (e.g., empathy, connectedness, fun, and engaging)
Published the research paper at CogSci 2022, incorporating key findings from the correlation analysis
We discovered some useful data for the main experiment. (…)
Second, users rated conversations with a nice balance of self-disclosure, counter-questions, and empathy as the best.
In other words, if only one side of the conversation showed empathy, told personal stories, or asked questions, the conversation was deemed uninteresting.

Problem: (1) Lowest employment rate among visually impaired individuals compared to other disabled groups (2) South Korea’s highest suicide rate in the OECD, with a significantly low depression diagnosis rate
Challenge: (1) Creating sustainable and specialized job opportunities for the visually impaired (2) Overcoming economic and psychological barriers to accessing professional mental health services
Solution: A unique psychotherapy service where visually impaired individuals provide empathetic counseling sessions in completely dark environments, making mental health support more affordable and accessible for depressed and lonely individuals.

Laid the foundation for a mental healthcare service by conducting extensive interviews and surveys with mental health professionals and visually impaired individuals to design a service
Recruited, trained, and employed 6 visually impaired individuals as counselors with a tailored curriculum co-developed with a psychotherapist

Empowered total 10 blind individuals with sustainable income and professional skills
Selected as a certified profession by Korea Employment Agency for the Disabled (Link)
Served 850+ customers, secured $105,000 in funding, and received coverage from nine major media outlets: