Orthoganality in Linear Attention Issue (Delta Rules!)

Image of Kimi Linear Attention (different than the attention described here).
A Quick Note, 

This is an explainer series to my daily X posts on "becoming a kernel engineer." I'll usually post daily. You can find them here: https://x.com/dj_sidequests.

I'll keep these "blog posts" mostly unrefined. That is, I will be publishing my notes here on my blog. Feel free to message me asking to clarify something!

The Problem + Brief Intro

1. Useful background intuition: 

      1) Information reduction. You reduced a 1million nxn, and n >> d, dxd matrix causes information collapse by compressing that much information. We must solve this issue somehow. 

      2) Orthogonality: the vectors that make up our matrix are perpendicular (90deg separation). We need to look at our key values. And, if 0 => then orthogonal (else not orthogonal). 

      3) One way to think about this: take the information reduction thing I just said. By compressing several distinct values into a smaller matrix, you lose a lot of information that differentiates the values (e.g. if they are not orthogonal, then your values “collapse” when you multiply it by the query matrix, resulting in you getting the incorrect answer) 

2. Issue, 
      1) By multiplying KV first, you match K and V together. The issue: originally, you run softmax(QK^T) and allow V to adaptively “weigh in” (change the distribution) on the softmax(QK^T). We don’t do that anymore, so V isn’t really “weighing in” anymore. 

      2) This results in you contributing to the same dxd matrix instead of a NEW n x n matrix that turns into a n x d matrix. 


Delta Rule Solution

Compare the old and new value of the key * value matrix =  state_matrix * key. then, 

State = State_old + beta * (key (value_vector - state_matrix*key)) 

the beta is a parameter you can optimize. this parameter will tell you how much of the new key-value matrix you should add. 

note: we derive state_matrix * key by, 

state_matrix = vector * key^T
=> state_matrix * key = vector (since we assume the values are orthogonal, then key^T * key = 1 for the keys of same index, but 0 for keys of differing index, allowing us to remove them from the calculation). What I said could probably didn't make sense, so here's the worked out math by ChatGPT, 


Comments

Post a Comment

Popular posts from this blog

What is a Multimodal LLM?

Image
Introduction Multimodal is the ability for a model to process several different types of inputs. For example, ChatGPT can support text input as well as image input as one combined input. You could also have a model take in .pdf, .md, and input text as inputs (yes, these count as different modalities).  This allows for users to give represent different ideas in its rawest form possible, such as an image, instead of describing it. If you try to describe it, you could hit hurdles in trying to properly explain it and making sure it interpreted the description correctly.  How do Multimodal LLMs work?  Typically, these modalities are "fused" together. Here's an architecture diagram,  Image from NVIDIA In this case, we have an encoder for a specific modality (e.g. video encoder, image encoder, audio encoder). This encodes them into embeddings the LLM can understand and use. After encoding each specific modality, we then fuse them together. Then, the output is projected and ...
Read more

A Brief Introduction into Model Quantization

Image
  Llama. Lots of popular packages and models have "llama" in the name (I'll be using a model and package that have llama in their name!) ( Flickr user Katherine ).   Introduction Have you ever wondered how people get 20B parameter models working on their laptops? Well, they have to quantize a model. Quantization is when you lower the accuracy of the model weights (from floating point 32 bit to floating point 16 bit). Although you lose some precision, researchers have found that this loss is minimal. In fact, accuracy still rivals the pre-quantized model. There also comes an added bonus of less storage.  Researchers are even pushing the limit to 1 bit ( Wang et. al. ) and 1.58 bit ( Ma et. al. ) LLMs.  Different Quantizations Terminology Q1, Q2, Q3, etc.  The number next to the Q refers to how many bits the model will be quantized to. For example, Q6 means 6 bit, Q8 is 8 bit, and Q1 is 1 bit.  _K, _0, _1 This tells us how the model rounds.  Type-0 ...
Read more

Explaining Active Speaker Detection (ASD)

Image
Example of Active Speaker Detection (Ground Truth Labeled Image). Credit: " LoCoNet: Long-Short Context Network for Active Speaker Detection " ( Link ) Introduction Imagine a system that can model how people work and interact. Well, one interaction to model is detecting who spoke in a given scene/frame. This way, not only do models learn how people interact (e.g. two people talking, one waiting for the other, or both talking on top of each other), but can then be used for human-model interactions, and speech diarization (segmenting audio + identifying who spoke when), and much more. Model Architectures In this section, we will discuss two models.  TalkNet ( Tao et. al. 2021 ) TalkNet's goal is to capture long-term context features, as previous models only focused on short term context features, and small time/temporal segments. Their solution utilizes two cross-attention mechanisms, swapping the query values in between the audio and visual features. Then, you concatenate ...
Read more