Explaining Active Speaker Detection (ASD)

Example of Active Speaker Detection (Ground Truth Labeled Image). Credit: "LoCoNet: Long-Short Context Network for Active Speaker Detection" (Link)

Introduction

Imagine a system that can model how people work and interact. Well, one interaction to model is detecting who spoke in a given scene/frame. This way, not only do models learn how people interact (e.g. two people talking, one waiting for the other, or both talking on top of each other), but can then be used for human-model interactions, and speech diarization (segmenting audio + identifying who spoke when), and much more.

Model Architectures

In this section, we will discuss two models. 
TalkNet's goal is to capture long-term context features, as previous models only focused on short term context features, and small time/temporal segments.

Their solution utilizes two cross-attention mechanisms, swapping the query values in between the audio and visual features. Then, you concatenate the cross-attention values and pass it through a self-attention mechanism. Finally, they apply a linear layer to get the outputs. 
Diagram of the multimodal fusion technique
Cite: Tao et. al. 2021

LoCoNet (Wang et. al. 2024)

LoCoNet, or Long-Short Context Network, is a transformer-based model that incorporates audio and visual cues to detect whether somebody is speaking or not. 

Their proposed solution is two-fold: use a self-attention mechanism to detect how a single face changes over time (Long-term Intra-Speaker Modeling), and use cues from other people in frames as context to detect whether or not a person is speaking (Short-Term Inter-Speaker Modeling). 
Overview of the LoCoNet model
Cite: Wang et. al. 2024

The network encodes audio (Mel-Spectrogram) through a VGGish network, the ish coming from the fact they remove the pooling layer from the fourth block, and instead use a deconvolution layer to upsample the features. 

Then, they are passed into the Long-term Intra-Speaker Modeling (LIM) block. Here, the audio and visual features go through self-attention mechanisms and a linear layer, where subscript v stands for video.
Cite: Wang et. al. 2024
And, this can be used for audio as well, which will be represented by subscript a. 

Then, both audio and visual features go through cross-attention mechanisms to fuse both modalities, 
Cite: Wang et. al. 2024
(Also done vice-versa for audio). This is the audio-enchanced visual embeddings.

Next is the Short-term Inter-Speaker Modeling (SIM) block. Here, the authors utilized a CNN to extract features to see how Visual Cues/Audio Cues change over time (both separately). Then, it is passed through a linear layer to get outputs/predictions.
Cite: Wang et. al. 2024

The equation would be, 
Cite: Wang et. al. 2024

Then, audio and visual features are concatenated to get one vector with the outputs. 

Demo

Here's a demo of the LocoNet model. In this video, there are two people in this scene, and one person has a bounding box (red = not speaking, green = speaking). 


Link to YouTube video of TalkNet: https://www.youtube.com/watch?v=U-eiMA0STY4

Works Referenced

Here are works I used to understand the previous materials above. 
"Self-Supervised Multimodal Learning: A Survey" by Zong et. al. 2024 (arxiv; accepted to IEEE T-PAMI).
"Why Cross-Attention is the Secret Sauce of Multimodal Models" by Jakub Strawa 2025 (Medium).
  


Comments

Post a Comment

Popular posts from this blog

What is a Multimodal LLM?

Image
Introduction Multimodal is the ability for a model to process several different types of inputs. For example, ChatGPT can support text input as well as image input as one combined input. You could also have a model take in .pdf, .md, and input text as inputs (yes, these count as different modalities).  This allows for users to give represent different ideas in its rawest form possible, such as an image, instead of describing it. If you try to describe it, you could hit hurdles in trying to properly explain it and making sure it interpreted the description correctly.  How do Multimodal LLMs work?  Typically, these modalities are "fused" together. Here's an architecture diagram,  Image from NVIDIA In this case, we have an encoder for a specific modality (e.g. video encoder, image encoder, audio encoder). This encodes them into embeddings the LLM can understand and use. After encoding each specific modality, we then fuse them together. Then, the output is projected and ...
Read more

A Brief Introduction into Model Quantization

Image
  Llama. Lots of popular packages and models have "llama" in the name (I'll be using a model and package that have llama in their name!) ( Flickr user Katherine ).   Introduction Have you ever wondered how people get 20B parameter models working on their laptops? Well, they have to quantize a model. Quantization is when you lower the accuracy of the model weights (from floating point 32 bit to floating point 16 bit). Although you lose some precision, researchers have found that this loss is minimal. In fact, accuracy still rivals the pre-quantized model. There also comes an added bonus of less storage.  Researchers are even pushing the limit to 1 bit ( Wang et. al. ) and 1.58 bit ( Ma et. al. ) LLMs.  Different Quantizations Terminology Q1, Q2, Q3, etc.  The number next to the Q refers to how many bits the model will be quantized to. For example, Q6 means 6 bit, Q8 is 8 bit, and Q1 is 1 bit.  _K, _0, _1 This tells us how the model rounds.  Type-0 ...
Read more

A Mathematical Explanation of Gradient Descent

Image
Example of a close-to-optimal Gradient descent for a quadratic function. Credit: Dylan Introduction Although I wrote an article recently explaining Gradient Descent, I feel like a mathematical explanation of how it works would be beneficial not only to understand how it works, but also how you derive the functions.  Explanation First, we need a way to calculate how far off the model is. For our example, we will use the L2 Loss Function,  Let N represent the total amount of data points we have, i the iteration we are on, yᵢ represent the expected output, and ŷ represent the output predicted by the model.  Let's look more into ŷ. Since ŷ is the predicted value, this is the function that needs to have their parameters changed. Our goal is to change these parameters enough to minimize the loss of ŷ. For our example, let's set ŷ to be a linear line.  The values in "a" and "b" are our parameters. These will be the values we will have to change within f(x) to chang...
Read more