A Beginner's look into Artificial Intelligence

Image of Kimi Linear Attention (different than the attention described here). Cite:  https://arxiv.org/abs/2510.26692 A Quick Note,  This is an explainer series to my daily X posts on "becoming a kernel engineer." I'll usually post daily. You can find them here:  https://x.com/dj_sidequests . I'll keep these "blog posts" mostly unrefined. That is, I will be publishing my notes here on my blog. Feel free to message me asking to clarify something! The Problem + Brief Intro 1. Useful background intuition:         1) Information reduction. You reduced a 1million nxn, and n >> d, dxd matrix causes information collapse by compressing that much information. We must solve this issue somehow.         2) Orthogonality: the vectors that make up our matrix are perpendicular (90deg separation). We need to look at our key values. And, if 0 => then orthogonal (else not orthogonal).         3) One way to think about thi...

Engram architecture from Deepseek's paper. Image Link: https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf (page 3) Intro I’ve had a wack idea recently: could a small “hashed-memory module” add useful info to an existing computer vision model? This idea was inspired after DeepSeek’s Engram work (paper: https://arxiv.org/abs/2601.07372 ). Idea Explained TLDR; Engram uses hashed token sequences to look up learned embeddings to combine the important learned tokens with the model’s hidden states.  What I want to do with a learned “look-up table” is see if I can add useful corrections to the final layer of a network to bump up accuracy? I want to first experiment with the last layer of a network to keep my experiment small and controlled at first. Adding corrections to all hidden layers would be useful later on to prevent full hallucinations in tasks such as segmentation (e.g. a model forgot to segment out a whole portion of an object because it did not recognize it was a c...

  Llama. Lots of popular packages and models have "llama" in the name (I'll be using a model and package that have llama in their name!) ( Flickr user Katherine ).   Introduction Have you ever wondered how people get 20B parameter models working on their laptops? Well, they have to quantize a model. Quantization is when you lower the accuracy of the model weights (from floating point 32 bit to floating point 16 bit). Although you lose some precision, researchers have found that this loss is minimal. In fact, accuracy still rivals the pre-quantized model. There also comes an added bonus of less storage.  Researchers are even pushing the limit to 1 bit ( Wang et. al. ) and 1.58 bit ( Ma et. al. ) LLMs.  Different Quantizations Terminology Q1, Q2, Q3, etc.  The number next to the Q refers to how many bits the model will be quantized to. For example, Q6 means 6 bit, Q8 is 8 bit, and Q1 is 1 bit.  _K, _0, _1 This tells us how the model rounds.  Type-0 ...

Introduction Multimodal is the ability for a model to process several different types of inputs. For example, ChatGPT can support text input as well as image input as one combined input. You could also have a model take in .pdf, .md, and input text as inputs (yes, these count as different modalities).  This allows for users to give represent different ideas in its rawest form possible, such as an image, instead of describing it. If you try to describe it, you could hit hurdles in trying to properly explain it and making sure it interpreted the description correctly.  How do Multimodal LLMs work?  Typically, these modalities are "fused" together. Here's an architecture diagram,  Image from NVIDIA In this case, we have an encoder for a specific modality (e.g. video encoder, image encoder, audio encoder). This encodes them into embeddings the LLM can understand and use. After encoding each specific modality, we then fuse them together. Then, the output is projected and ...

  Image created by FLUX.1 Kontext by Black Forest Labs   Introduction As AI models grow larger and larger, companies have to find different ways to get more training data. The more diverse, rich, human content these companies can find, the better these models can get in imitating us.  What do these companies do with these datasets? Well, they feed them to huge Multimodal Models (models that can support several different types of inputs) to make art, music, text, and much more.  But, what does this mean for the creator? Sometimes, companies will copyright strike music from artists despite it being in the public domain because the company has a recording of it.  In this case, can Artists and Musicians copyright strike AI for stealing their art? How does AI Art work and how do you detect it?  In essence, a model takes in a text input and creates a noisy image, and then tries decoding it. This is called Stable Diffusion.  Take Flux K.1 Kontext for example...

Example of Active Speaker Detection (Ground Truth Labeled Image). Credit: " LoCoNet: Long-Short Context Network for Active Speaker Detection " ( Link ) Introduction Imagine a system that can model how people work and interact. Well, one interaction to model is detecting who spoke in a given scene/frame. This way, not only do models learn how people interact (e.g. two people talking, one waiting for the other, or both talking on top of each other), but can then be used for human-model interactions, and speech diarization (segmenting audio + identifying who spoke when), and much more. Model Architectures In this section, we will discuss two models.  TalkNet ( Tao et. al. 2021 ) TalkNet's goal is to capture long-term context features, as previous models only focused on short term context features, and small time/temporal segments. Their solution utilizes two cross-attention mechanisms, swapping the query values in between the audio and visual features. Then, you concatenate ...