📝 Lecture Note: Generative Modeling

Lecture note for course Generative Modeling given by Kaiming He at MIT Schwarzman College of Computing Deep Learning Day

• Main Themes and Important Ideas
• Quotes
• Applications and Future Directions
• Key Takeaways
• Key Concepts

Briefing Document: Deep Learning Day - Generative Modeling
Source: Excerpts from "Deep Learning Day: Generative Modeling" presentation.

Main Themes and Important Ideas

1. The Rise of Generative AI

• The current era is characterized as the "gen AI area," marked by the introduction of powerful tools like chatbots (e.g., ChatGPT) and text-to-image generators.
• These models allow users to interact with computers in natural language and generate novel content based on prompts.
• Examples of impressive generative capabilities include text generation, image creation (e.g., a "Teddy bear teaching a course with generative model written on blackboard"), and even video generation (e.g., Sora's realistic and imaginative scenarios like "so many um um paper planes flying on top of trees or or or Forest").

2. Generative Models as Powerful Tools

• Generative models are not limited to creative applications but also serve as productive tools in daily life, such as AI code assistants that can understand and fix code through natural language interaction.
• The speaker suggests that "perhaps the previous um programming language is C++ python or Java the next level of programming language would just be English or human language."
• Beyond computer science, generative models are impacting various scientific domains, including protein design and generation (e.g., RF diffusion, linked to Nobel Prize-winning work) and weather prediction (DeepMind's work on qualitative weather behavior).

3. Historical Context and Evolution

• The concept of generative modeling is not entirely new, with roots in classical computer vision algorithms like "patch match" (content-aware fill in Photoshop) and texture synthesis techniques.
• Texture synthesis, which aimed to extend textures pixel by pixel based on existing examples, is presented as a conceptual precursor to today's autoregressive models.

4. Defining Generative Models

• Defining generative models is challenging as their capabilities and scope are constantly expanding.
• However, common characteristics of tasks addressed by generative models include:
  • Multiple/Infinite Predictions for a Single Input: For example, generating countless possible images of a cat from the prompt "cat."
  • Varying Plausibility of Predictions: Some generated outputs are more likely or realistic than others (e.g., a lion is more plausible than a dog when prompted for a cat).
  • Out-of-Distribution Generation: The model can generate novel outputs not explicitly present in the training data (e.g., the teddy bear scenario or the unique video of paper planes). The speaker states, "your training data may not contain the exact solution."
  • Complex and Higher-Dimensional Outputs: Text prompts, which are low-dimensional, can generate high-dimensional images with millions of pixels.

5. Generative vs. Discriminative Models

• Discriminative Models: Learn to map an input (X) to a label (Y), focusing on finding boundaries between classes (e.g., image classification). The speaker illustrates this with green and orange dots and a separating boundary, aiming to estimate the conditional probability P(Y|X).
• Generative Models: Aim to learn the underlying probability distribution of the data itself (P(X)) or the conditional probability of generating data given some condition (P(X|Y)). The goal is to understand how the data is generated and be able to sample new, plausible data points. The speaker explains, "conceptually in a generative model we care about probabilistic modeling."

6. The Role of Probabilistic Modeling

• Generative models rely on probabilistic modeling to capture the complex distributions of real-world data.
• The underlying assumption is that data is generated by some (often unknown) complex "world model" with latent factors (e.g., pose, lighting, identity for faces) and associated distributions.
• Generative modeling seeks to learn a model (often a neural network) that can approximate this underlying data distribution. The goal is to "minimize um the distance between the data distribution and the distribution you estimate."
• Once a good approximation is achieved, the model can sample new data points that resemble the original data distribution.

7. Deep Learning for Generative Modeling

• Deep neural networks are the "most powerful tool today" for generative modeling.
• Unlike discriminative learning that learns representations of individual data instances, generative modeling uses deep learning to learn the representation of entire probability distributions.
• Conceptually, this can be seen as mapping a simple distribution (e.g., Gaussian noise) to a complex data distribution (e.g., images of dogs) through a neural network. The speaker describes this as "a mapping between distributions."

8. Key Elements of Generative Models

The speaker briefly outlines the fundamental elements required to build a generative model:

• Problem Formulation: Defining the real-world problem as a probabilistic or generative modeling task.
• Representations: Using neural networks to represent data and their distributions.
• Objective Functions: Measuring the difference between the learned and target distributions.
• Optimizers: Solving the complex optimization problems involved in training.
• Inference Algorithms (Samplers): Generating new samples from the learned distribution.

9. Modern Approaches to Generative Modeling

The speaker provides a high-level overview of several popular and modern generative modeling techniques:

• Variational Autoencoders (VAEs): Extend the concept of autoencoding to distributions, mapping the data distribution to a simpler distribution and back.
• Generative Adversarial Networks (GANs): Employ a generator network to produce data and a discriminator network to distinguish between real and generated data, leading to a competitive learning process. The speaker notes that GANs were "the most uh um popular and most powerful generative models um over the last um decades."
• Autoregressive Models: Decompose the joint probability of data into a sequence of conditional probabilities, predicting each element based on the preceding ones (e.g., next token prediction in NLP). This breaks down a complex problem into smaller, simpler predictions.
• Diffusion Models (Denoising Diffusion): Inspired by thermodynamics, these models learn to reverse a process of progressively adding noise to clean data. Generation involves starting with noise and iteratively denoising it back to a realistic sample. The speaker highlights their recent emergence and power, especially in image generation.
• Flow Matching: An emerging idea inspired by computer graphics and the concept of morphing shapes. It aims to learn a continuous flow field that transforms a simple probability distribution into a more complex target distribution.

10. Formulating Real-World Problems as Generative Models

The core of applying generative models lies in defining the conditional distribution P(X|Y), where Y represents the conditions or constraints and X is the desired output data.
Examples include:

• Chatbots: Y = user prompt, X = chatbot response.
• Text-to-Image/Video: Y = text prompt, X = generated visual content.
• 3D Generation: Y = text prompt, X = 3D structure.
• Protein Generation: Y = desired properties (abstract), X = protein structure.
• Image Generation (Class-Conditional): Y = class label, X = generated image.
• Unconditional Generation: Y = implicit condition (data distribution), X = generated data following that distribution.
• Image Classification (Reformulated): Y = image, X = class label (leading to open vocabulary recognition).
• Image Captioning: Y = image, X = descriptive sentence/paragraph.
• Visual Dialogue: Y = image + text prompt, X = chatbot response.
• Robotics Policy Learning: Y = task/goal, X = plausible robot trajectories/policies. The speaker notes that for tasks with multiple valid solutions, generative models are well-suited.

11. Generative Models as the Next Level of Abstraction

The speaker concludes by emphasizing the hierarchical nature of AI model development:

• Layers: Basic building blocks of deep neural networks (convolutions, activations, etc.).
• Deep Neural Networks: Built from layers, used for various tasks.
• Generative Models: Built from deep neural networks, representing a higher level of abstraction for complex tasks.
• Large Language Models, Reasoning, Agentic Machine Learning: Future systems that will likely leverage generative models as fundamental building blocks.

This progression mirrors the advancement of computer science through the creation of increasingly sophisticated levels of abstraction, unlocking new possibilities.

Quotes

• "perhaps the previous um programming language is C++ python or Java the next level of programming language would just be English or human language" (regarding AI code assistants).
• "your training data may not contain the exact solution" (highlighting the out-of-distribution generation capability).
• "conceptually in a generative model we care about probabilistic modeling" (emphasizing the core principle).
• "a mapping between distributions" (describing the fundamental operation of generative models).
• "the most uh um popular and most powerful generative models um over the last um decades" (referring to GANs).

Applications and Future Directions

• Generative models are transforming various fields, from image generation to natural language understanding and even robotics.
• Future developments are expected to lead to more sophisticated systems capable of generalizing across tasks with fewer data and greater flexibility.
• Continued research will focus on refining existing models, such as GANs and VAEs, and exploring new paradigms like diffusion models and flow matching.

Key Takeaways

• Generative models represent a profound shift in deep learning, offering powerful tools for creative and scientific tasks.
• They require a deep understanding of probabilistic modeling and are increasingly built using advanced deep neural networks.
• Generative models are positioned to play a crucial role in the next era of AI, serving as foundational components for more sophisticated systems capable of diverse and complex tasks.

Key Concepts

• Generative Model: A type of artificial intelligence model that learns the underlying probability distribution of a dataset and can generate new data points that resemble the training data.
• Discriminative Model: A type of AI model that learns the boundary between different classes of data, typically used for classification tasks (e.g., identifying if an image contains a cat or a dog).
• Prompt: Textual input provided to a generative model (especially text-to-image or chatbots) to guide the generation of the desired output.
• Out-of-Distribution Generation: The ability of a generative model to create new data that is different from the exact examples seen in the training data.
• Latent Factors: Underlying, often unobservable variables that are assumed to influence the observed data. Generative models often try to learn these factors.
• Probability Distribution: A mathematical function that describes the likelihood of different outcomes or values for a random variable. Generative models aim to learn and sample from these distributions.
• Deep Learning: A subfield of machine learning that uses artificial neural networks with multiple layers to learn complex patterns from data.
• Representation Learning: The process of automatically discovering useful ways to represent data, often using deep learning models.
• Unsupervised Learning: A type of machine learning where the model learns patterns from unlabeled data, as opposed to supervised learning where labeled input-output pairs are provided.
• Autoencoder: A neural network architecture used for unsupervised learning that aims to learn a compressed representation (encoding) of the input data and then reconstruct the original data from this representation (decoding).
• Variational Autoencoder (VAE): A probabilistic extension of the autoencoder that learns a probability distribution in the latent space, enabling generative capabilities.
• Generative Adversarial Network (GAN): A generative model consisting of two neural networks, a generator that creates synthetic data and a discriminator that tries to distinguish between real and synthetic data. These two networks are trained in an adversarial manner.
• Auto-Regressive Model: A type of generative model that predicts each element of a sequence (e.g., text, image pixels) based on the previously generated elements.
• Next Token Prediction: A common application of auto-regressive models in natural language processing, where the model predicts the next word in a sequence given the preceding words.
• Diffusion Model (Denoising Diffusion): A generative model inspired by thermodynamics that learns to reverse a process of gradually adding noise to data. Generation involves starting with noise and iteratively removing it to obtain a realistic sample.
• Flow Matching: A recent approach in generative modeling that aims to learn a continuous transformation (a "flow") between a simple probability distribution (e.g., Gaussian) and the target data distribution.
• Conditional Distribution: The probability distribution of a random variable given the occurrence of another event or the value of another variable. Generative models often learn conditional distributions (e.g., generating an image given a text prompt).
• Policy Learning: In the context of robotics and reinforcement learning, the process of learning a strategy (policy) that dictates the actions an agent should take to achieve a goal.