跳转至

01 introduction

Multimodal Communicative Behaviors

  • Verbal
  • Vocal
  • Visual

Prior Research on Multimodal

  • behavioral
  • computational
  • interaction
  • deep learning

Two core challenges

  • Representation
  • Joint representations
  • Coordinated representations
  • Alignment -- The very core
  • Can be seen as a specialization of multi-view learning
  • Explicit Alignment
  • Implicit Alignment
  • Translations
  • Translating from one modality to another
  • Example-based
  • Model-driven
  • Fusion
  • Model-Agnostic Approaches
    • Early Fusion
    • Late Fusion
  • Model-Based (Intermediate) Approaches
    • Deep neural networks
    • Kernel-based methods
    • Graph models
  • Co-Learning
  • Parallel
  • Weakly-Parallel

Real world tasks tackled by MMML

Affective Computing