Wacky Idea #1: A Look-up Table for CV models


Engram architecture from Deepseek's paper. Image Link: https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf (page 3)

Intro

I’ve had a wack idea recently: could a small “hashed-memory module” add useful info to an existing computer vision model? This idea was inspired after DeepSeek’s Engram work (paper: https://arxiv.org/abs/2601.07372).

Idea Explained

TLDR; Engram uses hashed token sequences to look up learned embeddings to combine the important learned tokens with the model’s hidden states. 


What I want to do with a learned “look-up table” is see if I can add useful corrections to the final layer of a network to bump up accuracy? I want to first experiment with the last layer of a network to keep my experiment small and controlled at first. Adding corrections to all hidden layers would be useful later on to prevent full hallucinations in tasks such as segmentation (e.g. a model forgot to segment out a whole portion of an object because it did not recognize it was a certain object).


But, the problem gets complicated with images. Language has discrete token IDs, which results in natural keys. But, a vision model’s patch embeddings are continuous vectors, where each vector’s representation changes ever so slightly depending on small changes. Hashing these very, very small floating point changes would actually cause these slightly different vectors to be mashed into “one vector,” removing the vector’s representation of the small changes in its patch. This would make the look-up table “feature-poor.”


One approach could be to use fixed random projections and turn the final projection of the vector into bits, becoming the “address.” Similar feature vectors are more likely to share an address, though collisions can also group unrelated features. But, multiple hash tables could make the module less dependent on any single collision. Still, I’m not a big fan of this approach because of the potential of collisions and the need to do multiple work arounds.


I’ll keep y’all updated with the results! I still have some more reading to do, as I’m sure someone else has implemented a similar idea. Here are a few things I would search up, 

  1. Has anyone used look-up tables before? It for sure has, so this leads me to my next question: how did they implement the look-up table?

  2. How does the CV community deal with feature vectors that are very similar? Do they just group them together? How do they differentiate them?

  3. What are the benefits of just using a trained encoder backbone to turn the values into discrete tokens?

Comments

Post a Comment

Popular posts from this blog

What is a Multimodal LLM?

Image
Introduction Multimodal is the ability for a model to process several different types of inputs. For example, ChatGPT can support text input as well as image input as one combined input. You could also have a model take in .pdf, .md, and input text as inputs (yes, these count as different modalities).  This allows for users to give represent different ideas in its rawest form possible, such as an image, instead of describing it. If you try to describe it, you could hit hurdles in trying to properly explain it and making sure it interpreted the description correctly.  How do Multimodal LLMs work?  Typically, these modalities are "fused" together. Here's an architecture diagram,  Image from NVIDIA In this case, we have an encoder for a specific modality (e.g. video encoder, image encoder, audio encoder). This encodes them into embeddings the LLM can understand and use. After encoding each specific modality, we then fuse them together. Then, the output is projected and ...
Read more

A Brief Introduction into Model Quantization

Image
  Llama. Lots of popular packages and models have "llama" in the name (I'll be using a model and package that have llama in their name!) ( Flickr user Katherine ).   Introduction Have you ever wondered how people get 20B parameter models working on their laptops? Well, they have to quantize a model. Quantization is when you lower the accuracy of the model weights (from floating point 32 bit to floating point 16 bit). Although you lose some precision, researchers have found that this loss is minimal. In fact, accuracy still rivals the pre-quantized model. There also comes an added bonus of less storage.  Researchers are even pushing the limit to 1 bit ( Wang et. al. ) and 1.58 bit ( Ma et. al. ) LLMs.  Different Quantizations Terminology Q1, Q2, Q3, etc.  The number next to the Q refers to how many bits the model will be quantized to. For example, Q6 means 6 bit, Q8 is 8 bit, and Q1 is 1 bit.  _K, _0, _1 This tells us how the model rounds.  Type-0 ...
Read more

Explaining Active Speaker Detection (ASD)

Image
Example of Active Speaker Detection (Ground Truth Labeled Image). Credit: " LoCoNet: Long-Short Context Network for Active Speaker Detection " ( Link ) Introduction Imagine a system that can model how people work and interact. Well, one interaction to model is detecting who spoke in a given scene/frame. This way, not only do models learn how people interact (e.g. two people talking, one waiting for the other, or both talking on top of each other), but can then be used for human-model interactions, and speech diarization (segmenting audio + identifying who spoke when), and much more. Model Architectures In this section, we will discuss two models.  TalkNet ( Tao et. al. 2021 ) TalkNet's goal is to capture long-term context features, as previous models only focused on short term context features, and small time/temporal segments. Their solution utilizes two cross-attention mechanisms, swapping the query values in between the audio and visual features. Then, you concatenate ...
Read more