The Good, the Bad, and the Facts: Multimodal Representation of Medical Conversations for Patient Understanding

Advisors: Prof. Terry Knight and Prof. Randall Davis, EECS

In this thesis, I address the challenge medical patients face for managing their health information. In particular, I focus on the challenge of capturing, reviewing and extracting information from medical appointments for patients enduring serious and often emotionally demanding health conditions, such as cancer. In response to this problem, I first developed a novel multimodal web interface that captures content from medical appointments as text and audio with highlighted important positive and negative information. I conducted 25 user studies where I enacted fictional conversations between a doctor and a patient to evaluate whether this method of representing information would help patients review and understand their appointments. In these user studies, I assumed the role of doctor and participants assumed the role of patient. Results from the user studies show that the web interface does serve as a useful tool for reviewing the content of the conversations, however its effect on patient understanding cannot yet be determined. Second, I created a machine learning algorithm to automatically classify the positive and negative information in medical conversations based on analysis of the text and prosody in speech. The model with the highest performance on my dataset achieved an accuracy of 0.906 and F1-score of 0.888. While this study focuses on challenges within the medical field, findings from this study are relevant to emotional conversations in any setting such as sportscasting, psychotherapy, political debates and more.