Affect Recognition Project
Overview
This project is in collaboration with an industry partner.
The project aim is to build an affect recognition model with state-of-the-arts AI architectures. The model will be integrated into various wrist-worn / HMD wearable devices for affect-aware recommendation systems.
We have published some intermediary results as part of a workshop paper for NeurIPS '25 presented this year! In this work, we investigate the role of fine-grained sleep physiological data in next-day affect prediction, particularly stress. In our custom, in-the-wild dataset dataset, we collected physiological and activity data from 44 participants over 28 days, 24 hours per day; the Garmin Venu 3S smartwatches were used. Features were extracted from a variety of signals including heart rate variability (HRV), respiration rate, heart rate, SpO2 (Oxygen saturation level), Beat-to-Beat Intervals (BBI), skin temperature, and step count. Using the extracted time-series feature data, we train an XGBoost model and a custom multimodal encoder network based on a CNN architecture. We achieve a mean AUROC score of over 67%, with the combination of fine-grained sleep data and current physiological data outperforming the case where only the current physiological data is used to predict stress. More details can be found in our paper.
For the intermediary work, my role was on:
- Data analysis and modeling. After brainstorming on several ways to compare the effectiveness of fine-grained sleep data, we came up with our custom multimodal encoder network, which I implemented, trained, and tested on our custom dataset. We are now looking into other state-of-the-arts models and methods on personalization, domain generalization, contrastive learning, and self-supervised learning.
- Some involvement in signal processing and feature extraction from the collected watch data.
- Preliminary preparation work for the data collection (e.g. IRB ethics applications, questionnaire design, research protocol development).
(More details will be included at the completion of the project.)