Louis-Philippe Morency

Leonardo Associate Professor of Computer Science,

Language Technology Institute,

School of Computer Science, Carnegie Mellon University

Director, MultiComp Lab

Gates-Hillman Center (GHC) Office 5411,

5000 Forbes Avenue, Pittsburgh, PA 15213

Email: morency@cs.cmu.edu

Phone: (412) 268-5508

 

I am tenure-track Faculty at CMU Language Technology Institute where I lead the Multimodal Communication and Machine Learning Laboratory (MultiComp Lab). I was previously Research Faculty at USC Computer Science Department. I received my Ph.D. in Computer Science from MIT Computer Science and Artificial Intelligence Laboratory.

My research focuses on building the computational foundations to enable computers with the abilities to analyze, recognize and predict subtle human communicative behaviors during social interactions. Central to this research effort is the technical challenge of multimodal machine learning: mathematical foundation to study heterogeneous multimodal data and the contingency often found between modalities. This multi-disciplinary research topic overlaps the fields of multimodal interaction, social psychology, computer vision, machine learning and artificial intelligence, and has many applications in areas as diverse as medicine, robotics and education.

Multimodal Machine Learning

·       Probabilistic modeling of acoustic, visual and verbal modalities

·       Learning the temporal contingency between modalities

Human Communication Dynamics

·       Analyze, recognize and predict subtle human communicative behaviors during social interactions.

Health Behavior Informatics

·       Technologies to support clinical practice during diagnosis and treatment of mental health disorders

Selected Publications (see full list at MultiComp Lab or Google Scholar)

§  T. Baltrusaitis, C. Ahuja, and L.-P. Morency. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Volume: 41, Issue: 2, February 2019

§  Y.-H.H. Tsai, S.K. Divvala, L.-P. Morency, R. Salakhutdinov and A. Farhadi. Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

§  A. Zadeh, M. Chan, P.P. Liang, E. Tong and L.-P. Morency. Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

§  Y.-H.H. Tsai, P.P. Liang, A. Zadeh, L.-P. Morency and R. Salakhutdinov. Learning Factorized Multimodal Representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2019

§  P. P. Liang, Z. Li, Y.-H. H. Tsai, Q. Zhao, R. Salakhutdinov and L.-P. Morency, Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019

§  Y-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency and R. Salakhutdinov, Multimodal Transformer for Unaligned Multimodal Language Sequences, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019

§  A. Zadeh, P. Liang, J. Vanbriesen, S. Poria, E. Tong, E. Cambria, M. Chen and L.-P. Morency, Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018

§  Z. Liu, Y. Shen, V. Lakshminarasimhan, P. Liang, A. Zadeh and L.-P. Morency, Efficient Low-rank Multimodal Fusion with Modality-Specific Factors, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018

§  D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein and T. Darrell, Speaker-Follower Models for Vision-and-Language Navigation. In Proceedings of the Thirty-Second Annual Conference on Neural Information Processing Systems (NIPS), 2018

§  P. Liang, Z. Liu, A. Zadeh and L.-P. Morency. Multimodal Language Analysis with Recurrent Multistage Fusion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018

§  C. Ahuja and L.-P. Morency, Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling, In Proceedings of the Thirty-Second AAAI Annual Conference on Artificial Intelligence (AAAI), 2018

§  V. Cirik, T. Berg-Kirkpatrick and L.-P. Morency, Using Syntax to Ground Referring Expressions in Natural Images, In Proceedings of the Thirty-Second AAAI Annual Conference on Artificial Intelligence (AAAI), 2018

§  A. Zadeh, P. Liang, S. Poria, P. Vij, E. Cambria, and L.-P. Morency, Multi-attention Recurrent Network for Human Communication Comprehension, In Proceedings of the Thirty-Second AAAI Annual Conference on Artificial Intelligence (AAAI), 2018

§  A. Zadeh, P. Liang, N. Mazumder, S. Poria, E. Cambria and L.-P. Morency, Memory Fusion Network for Multi-view Sequential Learning, In Proceedings of the Thirty-Second AAAI Annual Conference on Artificial Intelligence (AAAI), 2018

§  E. Tong, A. Zadeh, C. Jones and L.-P. Morency, Combating Human Trafficking with Very Deep Multimodal Models, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  S. Pouria, E. Cambria, D. Hazarika, N. Mazumder, A. Zadeh and L.-P. Morency, Context-Dependent Sentiment Analysis in User-Generated Videos, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  S. Ghosh, M. Chollet, E. Laksana, L.-P. Morency and S. Scherer, Affect-LM: A Neural Language Model for Customizable Affective Text Generation, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  W. Pei, T. Baltrušaitis, D. Tax and L.-P. Morency. Temporal Attention-Gated Model for Robust Sequence Classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

§  A. Zadeh, M. Chen, S. Poria, E. Cambria and L.-P. Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017

§  S. Rajagopalan, L.-P. Morency, T. Baltrus̆aitis and R. Goecke, Extending Long Short-Term Memory for Multi-View Structured Learning, In Proceedings of the European Conference on Computer Vision (ECCV), 2016

§  E. Wood, T. Baltrušaitis, L.-P. Morency, P. Robinson and Andreas Bulling, A 3D Morphable Eye Region Model for Gaze Estimation, In Proceedings of the European Conference on Computer Vision (ECCV), 2016

§  H. Yu, S. Zhang and L.-P. Morency, Unsupervised Text Recap Extraction for TV Series, In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016

Graduate Students Advising (see all group members at MultiComp Lab website)

Amir Ali Bagherzade, Ph.D. program (LTI)

Chaitanya Ahuja, Ph.D. program (LTI)

Volkan Cirik, Ph.D. program (LTI co-supervised with Taylor Berg-Kirkpatrick)

Alexandria Vail, Ph.D. program (HCII)

Paul Liang, Ph.D. program (MLD, co-supervised with Ruslan Salakhutdinov)

Hubert Tsai, Ph.D. program (MLD, co-supervised with Ruslan Salakhutdinov)

Torsten Wörtwein, Ph.D. program (LTI)

Teaching

       CMU-11777: Advanced Multimodal Machine Learning, Spring 2016, Spring 2017, Fall 2017, Fall 2018, Fall 2019

Fundamental mathematical concepts related to multimodal machine learning including multimodal alignment and fusion, heterogeneous representation learning and multi-stream temporal modeling.

·       Webpage of Fall 2019 semester which includes syllabus and lecture slides

https://piazza.com/cmu/fall2019/11777/home

       CMU-11776: Multimodal Affective Computing, Fall 2015, Fall 2016, Spring 2018

Recent computational techniques to analyze, recognize and predict human communication behaviors during social interactions.

·       Webpage of Spring 2019 semester which includes syllabus and lecture slides

https://piazza.com/cmu/spring2019/11776/home

       USC-CS599: Human Communication and Machine Learning, Fall 2010, 2012, 2014

Selected Press Coverage