# NVIDIA DLI: Rapid Application Development Using LLMs — Complete Lecture Notes > **Course ID**: `course-v1:DLI+C-FX-09+V2` > **Total Slides**: 193 > **Extracted from**: `DLI-RAD-Slides-v2.pptx` --- ### Slide 1: Rapid Application Development using Large Language Models - "Hello and welcome to the NVIDIA Deep Learning Institute (DLI) course on rapid application development using large language models (LLMs)." - We will explore LLMs, generative AI, and software paradigms built around these technologies. - Expect hands-on experience, practical learning, and insights into advanced AI concepts. - Let’s dive into the course and the exciting content ahead. --- ### Slide 2: Large Language Models - "LLMs enable computers to understand language without human intervention." - LLMs can handle various tasks: text classification, text generation, and even image-to-text conversion. - We’ll work with multiple models like semantic classification models, question-answering models, and more. - We will narrow down on text generation towards the end of the course. --- ### Slide 3: Language-Reasoning Systems - "We distinguish between generative AI models and orchestration." - **Generative AI models**handle single tasks such as text understanding and generation. - **Orchestration**combines multiple models to create integrated AI systems. - By the end of the course, you'll see how to develop such systems easily. --- ### Slide 4: Sequence Modeling - "Our focus will be on sequence modeling, particularly transformers, but our exploration will lead us to other domains as our understanding continues to evolve." - Sequence models are central to many AI tasks: from text understanding to generating images and animations. - We’ll also touch on complementary architectures like autoencoders and diffusion models to help us reason with more complex input formats like images. - You’ll learn how these models connect across various tasks and domains, and will approach an understanding of more general generative AI principles along the way. --- ### Slide 5: Course Objectives - "This course is structured to help you understand and build LLM-based applications." - **Understand LLM architecture**: Deep dive into model architectures and design paradigms. - **Apply LLMs to problems**: Language reasoning, text generation, chatbot development, etc. - **Deploy and scale**: Learn how to deploy LLM servers and optimize them. --- ### Slide 6: Prerequisites - "Before starting, it's crucial you have a background in deep learning and Python." - Python knowledge is required, and familiarity with libraries like NumPy,Keras, andPyTorchis recommended. - Prior experience in deep learning frameworks is helpful. - We will build on machine learning and deep learning fundamentals throughout the course. --- ### Slide 7: Core Tools - "Our main tools for this course will be Transformers andLangChain." - **Transformers**: The foundation for model specification and inference. - **LangChain**: The orchestration framework for integrating different AI models. - These tools will be our primary focus, and we’ll explore their usage in depth. --- ### Slide 8: Course Schedule - "The course will cover the following main topics:" - **Getting Started**: Introduction to the tools and setup. - **Transformer Intuitions**: Understanding how transformers work. - **Encoder Tasks**: Text classification and feature extraction. - **Seq2Seq Models**: Sequence-to-sequence modeling. - **Multimodality**: Integrating different data types (text, images, audio). - **Inference Servers**: How to deploy models. - **Agentics**: Introduction to agent-based systems. - **Final Assessment**: Testing your understanding of the course. --- ### Slide 9: Part 1: From DL To LLMs - "Welcome to Part 1: From Deep Learning to Language Models (LLMs). We’ll start by laying the foundation with a brief overview of deep learning concepts, then move towards applying them in language modeling." --- ### Slide 10: Deep Learning - "At the heart of machine learning is deep learning, a paradigm shift in how we approach solving problems with complex data like images and text." - "The process typically begins with raw data, which we extract, transform, and load into a system of deep learning models." - "The goal is to iteratively train models using this data, continually refining their predictions until we have a model capable of generalizing well on unseen data." --- ### Slide 11: Deep Learning - "Let’s look at a simple example of deep learning applied to image classification. Here, we’re asking the model to classify whether an image is a cat, dog, or bird." - "The model takes in an image, represented as a grid of pixels, and processes it through multiple layers, where each layer transforms the data step-by-step, from a highly complex image to a final prediction like 'Cat: 80%, Dog: 15%, Bird: 5%'." --- ### Slide 12: Deep Learning - "Deep learning models can have different types of connections between neurons. Two common types are**Sparse**and**Dense**." - "**Sparse connections**only interact with specific parts of the input, like certain regions of an image, while**Dense connections**work on the entire input at once." - "You may have encountered sparse connections in**convolutional layers**, which focus on small regions of the input, while dense layers aggregate all the data." --- ### Slide 13: Deep Learning - "Deep learning models often break down tasks into stages. The first part of the model, known as the**feature extractor**, learns to recognize general patterns in data." - "For instance, the feature extractor might focus on identifying the edges and shapes in an image." - "The second part, known as the**task head**, focuses on the specific problem, such as classifying the image into one of the three categories: cat, dog, or bird.” --- ### Slide 14: Deep Learning - "A crucial feature of deep learning models is that they are**differentiable**. This allows us to calculate how small changes to the model’s parameters (its weights) impact its predictions." - "This process, called**backpropagation**, is what enables the model to learn from its mistakes and optimize itself." - "Through backpropagation, we calculate the gradient of the loss function with respect to the model’s parameters and use this to adjust the model, making it perform better." --- ### Slide 15: Deep Learning - "The model’s prediction is called a hypothesis. For example, it might predict that an image is 80% likely to be a cat and 20% likely to be a dog." - "We compare this hypothesis to the true label—what we know the image actually is.” (next slide) - "The difference between the model’s prediction and the true label is called the**loss**. The goal of training is to minimize this loss. --- ### Slide 16: Deep Learning - "The difference between the model’s prediction and the true label is called the**loss**. The goal of training is to minimize this loss.” - (Next Slide) - "To reduce the loss, we use a process called**gradient descent**. Essentially, we calculate how the loss changes with respect to the model’s parameters, then adjust those parameters in the direction that reduces the loss." - "This allows the model to improve with each iteration, gradually finding better parameters that make better predictions." --- ### Slide 17: Deep Learning - "To reduce the loss, we use a process called**gradient descent**. Essentially, we calculate how the loss changes with respect to the model’s parameters, then adjust those parameters in the direction that reduces the loss." - "This allows the model to improve with each iteration, gradually finding better parameters that make better predictions - "By repeatedly applying gradient descent, the model gets better and better at making accurate predictions. With each iteration, the loss decreases, and the model’s performance improves." - "Eventually, the model reaches a point where its predictions are consistently very close to the true values, which may hopefully indicate that it can generalize well to new data - assuming everything is set up nicely." --- ### Slide 18: From Deep Learning to LLMs - "At the core of this course, we aim to transition from understanding**deep learning fundamentals**to tackling more complex language tasks with**large language models (LLMs)**." --- ### Slide 19: What Is Language? - "So, what exactly is**language**? It’s not just a string of letters or characters but a**sequence**of information that carries meaning." - "We start with raw sequences of letters, as in this sentence: 'Hello! How is it going?!'." - "Using a process called**tokenization**, we break these sequences into smaller, understandable units—like words, punctuation marks, or even sub-word components." - "These tokens form an**ordered sequence of classes**that a model can process and understand. The goal is not just to recognize words but also to capture the**relationships**between them." --- ### Slide 20: The Base Formulation - "When dealing with language, we can approach it from different angles. Consider the phrase 'Hello! How is it going?!'." - "Some models, like**BERT**, treat language as a sequence that can be encoded. The encoder processes the entire sequence of tokens and extracts useful representations." - "Other models, like**GPT-2**, use decoders to generate new sequences, predicting what comes next or generating responses." - "Depending on the task, we might use**sequence prediction**(e.g., predicting the sentiment of a sentence: 90% positive, 10% negative),**pre-token prediction**(like understanding each token’s role in the sentence), or**forecasting predictions**(e.g., generating a response like 'It’s going great!’).” In the next notebook, we’ll explore some of the easier flavors of these models and will learn how we can start using them in our environment. --- ### Slide 21: Part 1: From DL To LLMs **NOTE: Ask people to start their notebooks now.** - Welcome back to the course! In the previous section, we introduced deep learning and its fundamentals to set the stage for more advanced topics. - Now, we’re going to dive deeper into language models and start pulling in models from the Hugging Face platform. --- ### Slide 22: Hugging Face - To kick things off, we’ll be working with**Hugging Face**, which is a large open-source community dedicated to sharing machine learning models, datasets, and tools. - Hugging Face offers a broad range of pre-trained models for various tasks, from text generation to image classification. - The platform allows you to pull in models, fine-tune them, or even deploy them directly in production. --- ### Slide 23: Hugging Face - Hugging Face’s repository is particularly useful for experimentation. You can browse thousands of models uploaded by everyone from large corporations to individual developers. - The platform also offers an intuitive way to search, upload, and experiment with models, making it accessible even for newcomers. - (Next slide talks about EULAs and restrictions) --- ### Slide 24: Hugging Face - One thing to keep in mind when working with Hugging Face models is licensing. - Not every model is suitable for every purpose, so it’s essential to check the End User License Agreements (EULA) for models and datasets to ensure they align with your project needs. - Some models are only intended for research or have specific restrictions, so always do your due diligence. --- ### Slide 25: Pulling In A Simple Model - In this section, we’ll show how easy it is to pull in a simple**LLM**from Hugging Face. - The first step is searching for a relevant model by filtering in the Hugging Face model directory for what you want. --- ### Slide 26: Pulling In A Simple Model - As an example, we can selected a**lightweight BERT model**for multilingual sentiment analysis. - You’ll see some key details like the model’s architecture (BERT), the licensing (MIT), and what frameworks it supports (e.g.,PyTorch, TensorFlow, etc.). - It’s always a good practice to review this information before proceeding. --- ### Slide 27: Pulling In A Simple Model - Once you’ve selected a model, you can test it using Hugging Face’s**hosted inference API**. - This allows you to experiment with the model directly in the browser without needing to download it. - For instance, you can input a sentence like “I like you. I love you” and see that it returns a positive sentiment score. --- ### Slide 28: Pulling In A Simple Model - Next, let’s walk through how to import the model into your environment. - Using Hugging Face’s**Transformers library**, you can import the model with just a few lines of code. - The pipeline function simplifies this by abstracting away the complex logic behind pre-processing, model loading, and post-processing. --- ### Slide 29: Pulling In A Simple Model - Now that we've seen how easy it is to navigate Hugging Face and find models, let's pull one in and try it out. - Here, we use the transformers library to import a BERT-based sentiment classification model. - You can see how it downloads everything necessary to run the model – the configuration, tokenizer, model weights, and more. - This all comes from the model repository, which you can think of like a git repo associated with eachhuggingfacemodel. --- ### Slide 30: Running Your Simple Model - We can now feed simple words into the model and observe how it classifies them. - For example, common positive words like 'hello' or 'awesome' are rated with 5 stars, while words like 'uh' and 'eh' get lower ratings. - The model is making these classifications based on patterns it has learned during training, and each result comes with a confidence score that tells us how strongly the model associates the input with the corresponding label. --- ### Slide 31: Running Your Simple Model - Next, we test the model with some more complex inputs. - Sentences like 'I loved my old pillow more than anything!' are still classified as 5-star reviews with high confidence, while more negative-sounding inputs, such as 'I hated my old pillow so much,' receive 1 star. - You’ll notice that the model isn’t really great at handling tricky inputs, but you can see that it is trying to leverage semantic meanings to take guesses at the overall sentiment. --- ### Slide 32: Pipeline Components - To break down the process a bit more, the pipeline is divided into three major stages: preprocessing, forward pass, and postprocessing. - In preprocessing, the input is tokenized and converted into tensors, which are the numerical representations the model can work with. - The forward pass sends these tensors through the model, where the actual predictions are made. - Finally, postprocessing takes the model's output and turns it back into a format that's understandable for us—such as star ratings or sentiment labels. - This structure makes it easy to work with language models, hiding the complexity of what's happening under the hood. --- ### Slide 33: Pipeline Components - Let's see the pipeline in action. - The input is the sentence, 'It’s so fluffy I’mgonnadie.’ - First, it goes through preprocessing, where the sentence is tokenized into input IDs and tensors. - Then, the forward pass sends it through the model, generating logits—the raw predictions. - Finally, in the postprocessing step, the logits are converted back into human-readable output, in this case a star rating of 1 star and a score of around 0.49. - This process happens seamlessly when using the pipeline, and shows the overall flow of data in such a pipeline; from text to tensor, from tensor to tensor, and from tensor to output. --- ### Slide 34: Pipeline Components - This diagram shows how each stage of the pipeline transforms data. - The message goes from a string to a dictionary of tensors, back to a dictionary of logits, and finally back to a string output. - Each step is tied to a specific component of the pipeline, be it the tokenizer, the model, or the pipeline itself. - All of these components are constructed in your environment from the config files which are present in the model repo combined with the settings you change in constructing your pipeline. --- ### Slide 35: Pipeline Components **Now that we know the specific components, let’s look a little bit more closely about how they were constructed for our specific model.** **(Subsequent sections)** - For the preprocessing phase, we can see that the config settings specify various tokenization properties and also point out to BERT, which loads in a default tokenizer if one is not specified. - For the forward, we can see that some pipeline details are also provided in the config andreguardingwhat kind of forward pipeline to use along with some additional architectural settings. - And then lastly, the postprocessing comes naturally out of the pipeline definition, which is also selected in the config file and is aliased and directed to appropriate python objects by the transformers library. --- ### Slide 36: Pipeline Components Now that we know the specific components, let’s look a little bit more closely about how they were constructed for our specific model. - **For the preprocessing phase, we can see that the config settings specify various tokenization properties and also point out to BERT, which loads in a default tokenizer if one is not specified.** - (Subsequent sections) - For the forward, we can see that some pipeline details are also provided in the config andreguardingwhat kind of forward pipeline to use along with some additional architectural settings. - And then lastly, the postprocessing comes naturally out of the pipeline definition, which is also selected in the config file and is aliased and directed to appropriate python objects by the transformers library. --- ### Slide 37: Pipeline Components Now that we know the specific components, let’s look a little bit more closely about how they were constructed for our specific model. - For the preprocessing phase, we can see that the config settings specify various tokenization properties and also point out to BERT, which loads in a default tokenizer if one is not specified. - **For the forward, we can see that some pipeline details are also provided in the config and****reguarding****what kind of forward pipeline to use along with some additional architectural settings.** - (Subsequent sections) - And then lastly, the postprocessing comes naturally out of the pipeline definition, which is also selected in the config file and is aliased and directed to appropriate python objects by the transformers library. --- ### Slide 38: Pipeline Components Now that we know the specific components, let’s look a little bit more closely about how they were constructed for our specific model. - For the preprocessing phase, we can see that the config settings specify various tokenization properties and also point out to BERT, which loads in a default tokenizer if one is not specified. - For the forward, we can see that some pipeline details are also provided in the config andreguardingwhat kind of forward pipeline to use along with some additional architectural settings. - **And then lastly, the postprocessing comes naturally out of the pipeline definition, which is also selected in the config file and is aliased and directed to appropriate python objects by the transformers library.** --- ### Slide 39: HF Pipeline Options In most context, theHuggingFaceecosystem can pick default models based on the task at hand. - Based on the task you specify—whether it’s text classification, token classification, or even audio processing—HuggingFaceautomatically pulls in the appropriate model, tokenizer, and postprocessing steps if one is not specified. - This also includes flexibility between backends likePyTorchand TensorFlow. The task support abstracts much of the complexity away, enabling rapid experimentation and integration of state-of-the-art models." --- ### Slide 40: HF Pipeline Options HuggingFacesupports a variety of models, and we’ll only be experimenting with a subset of its capabilities throughout this course. - Beyond NLP tasks like text classification, token classification, and question answering,HuggingFacealso provides pipelines for audio tasks like speech recognition, as well as vision tasks like object detection and image segmentation. - It also encompasses diffusers which we will use a bit later for image generation. **All-in-all, this should be enough to help you get started with Notebook 1, which should involve just making sure the environment is up and doing a simple model test to confirm that everything is working as intended.** --- ### Slide 41: Part 1: From DL To LLMs - Welcome back to the course! - In the last section, we introduced you to Hugging Face and the Transformers library, helping you get comfortable with the environment. - Now, we’re going to dive deeper under the hood to explore sequence reasoning using transformers. - We’ll connect concepts from deep learning fundamentals to transformers, and show how they handle ordered sequences of classes. - Let’s get started! --- ### Slide 42: Our Problem Space - **Let's recall the problem space.** - **We are transitioning from general sequence models to handling ordered sequences of classes.** - We know that models like BERT are effective for tasks like masked language modeling and sentence classification. But how are they doing this? - We need to dive into the architecture and see how transformers handle tokens and classes.” --- ### Slide 43: The Base Formulation - Let's recall the problem space. - We are transitioning from general sequence models to handling ordered sequences of classes. - **We know that models like BERT are effective for tasks like masked language modeling and sentence classification. But how are they doing this?** - **We need to dive into the architecture and see how transformers handle tokens and classes.** --- ### Slide 44: Dealing With Classes We can predict classes from a variety of inputs, just like in standard deep learning. - The key is to map input data to classes and train the network to reinforce this mapping. - When generating tokens, we can extend this same principle. - Essentially, the transformer processes tokens as classes, predicting probabilities for each token. - This is the foundation of token classification. --- ### Slide 45: Dealing With Classes - **While we know that this class-based approach works for outputting tokens, does it work as well for taking them in?** - **It turns out that it still works, but there may be better ways to take in high-dimensional vocabulary vectors.** - Instead of dense layers, which become problematic with large vocabularies, we use embedding layers. - Embedding layers map tokens to dense vector representations, reducing computational overhead. - This is more efficient for handling the large number of possible tokens in tasks like natural language processing, where we may have tens or hundreds of thousands of unique tokens. --- ### Slide 46: Dealing With Classes - While we know that this class-based approach works for outputting tokens, does it work as well for taking them in? - It turns out that it still works, but there may be better ways to take in high-dimensional vocabulary vectors. - **Instead of dense layers, which become problematic with large vocabularies, we use embedding layers.** - **Embedding layers map tokens to dense vector representations, reducing computational overhead.** - **This is more efficient for handling the large number of possible tokens in tasks like natural language processing, where we may have tens or hundreds of thousands of unique tokens.** --- ### Slide 47: Dealing With Classes - **We can think of sequences as many classes linked together. A good analogy is how convolutional layers process grids of pixels in images.** - **We know from deep learning that treating a large image as a single block is not a good idea, and the intuition persists here:** - **we probably don’t want to reason about text using densely-connected architectures.** - Instead, position-wise feed-forward networks in transformers process sequences token by token. This helps break down the reasoning process at the token level, which will be much easier to train. --- ### Slide 48: Dealing With Classes - We can think of sequences as many classes linked together. A good analogy is how convolutional layers process grids of pixels in images. - We know from deep learning that treating a large image as a single block is not a good idea, and the intuition persists here: - we probably don’t want to reason about text using densely-connected architectures. - **Instead, position-wise feed-forward networks in transformers process sequences token by token. This helps break down the reasoning process at the token level, which will be much easier to train.** --- ### Slide 49: Dealing With Ordered Classes **Token-level prediction is easy enough to train, but it won’t give us sequence-level reasoning.** **A step in the right direction may be to provide more information, like information about word ordering and maybe other info.** But it still isn’t quite enough… --- ### Slide 50: Dealing With Ordered Classes Token-level prediction is easy enough to train, but it won’t give us sequence-level reasoning. A step in the right direction may be to provide more information, like information about word ordering and maybe other info. **But it still isn’t quite enough…** --- ### Slide 51: Transformers for Sequence Reasonings **To reason with ordered SEQUENCES of classes:** - **Start with your per-sequence-entry feed-forward. Good for per-entry logic, but insufficient.** - You really want per-entry logic to persist, so why not add it as a residual connection? - From your input embeddings, derive 3 matrices, each being per-entry. - Use the key and value to compute attention (more details in notebook) - Lightweight interface produces a square matrix; self-attention - Feed the sequence-level logic back into the per-token pathway by multiplying with softmax-ed attention. - These components are known as transformer encoder blocks, and they generate per-token encodings with sequence-level knowledge --- ### Slide 52: Transformers for Sequence Reasonings **To reason with ordered SEQUENCES of classes:** - Start with your per-sequence-entry feed-forward. Good for per-entry logic, but insufficient. - **You really want per-entry logic to persist, so why not add it as a residual connection?** - From your input embeddings, derive 3 matrices, each being per-entry. - Use the key and value to compute attention (more details in notebook) - Lightweight interface produces a square matrix; self-attention - Feed the sequence-level logic back into the per-token pathway by multiplying with softmax-ed attention. - These components are known as transformer encoder blocks, and they generate per-token encodings with sequence-level knowledge --- ### Slide 53: Transformers for Sequence Reasonings **To reason with ordered SEQUENCES of classes:** - Start with your per-sequence-entry feed-forward. Good for per-entry logic, but insufficient. - **From your input embeddings, derive 3 matrices, each being per-entry.** - **Key and Query extracted to a latent dimension of choice** - **Value extracted to whatever dimension (why not keep it the same as input dimension for simplicity)** - **Move some of your per-token dense logic after the value computation. Why not?** - Use the key and value to compute attention (more details in notebook) - Lightweight interface produces a square matrix; self-attention - Feed the sequence-level logic back into the per-token pathway by multiplying with softmax-ed attention. - These components are known as transformer encoder blocks, and they generate per-token encodings with sequence-level knowledge --- ### Slide 54: Transformers for Sequence Reasonings **To reason with ordered SEQUENCES of classes:** - Start with your per-sequence-entry feed-forward. Good for per-entry logic, but insufficient. - From your input embeddings, derive 3 matrices, each being per-entry. - **Use the key and value to compute attention (more details in notebook)** - **Lightweight interface produces a square matrix; self-attention** - Feed the sequence-level logic back into the per-token pathway by multiplying with softmax-ed attention. - These components are known as transformer encoder blocks, and they generate per-token encodings with sequence-level knowledge --- ### Slide 55: Transformers for Sequence Reasonings **To reason with ordered SEQUENCES of classes:** - Start with your per-sequence-entry feed-forward. Good for per-entry logic, but insufficient. - From your input embeddings, derive 3 matrices, each being per-entry. - Use the key and value to compute attention (more details in notebook) - Lightweight interface produces a square matrix; self-attention - **Feed the sequence-level logic back into the per-token pathway by multiplying with softmax-ed attention.** - These components are known as transformer encoder blocks, and they generate per-token encodings with sequence-level knowledge --- ### Slide 56: Transformers for Sequence Reasonings **To reason with ordered SEQUENCES of classes:** - Start with your per-sequence-entry feed-forward. Good for per-entry logic, but insufficient. - From your input embeddings, derive 3 matrices, each being per-entry. - Use the key and value to compute attention (more details in notebook) - Lightweight interface produces a square matrix; self-attention - **Feed the sequence-level logic back into the per-token pathway by multiplying with softmax-ed attention.** - These components are known as transformer encoder blocks, and they generate per-token encodings with sequence-level knowledge --- ### Slide 57: Transformers for Sequence Reasonings **To reason with ordered SEQUENCES of classes:** - Start with your per-sequence-entry feed-forward. Good for per-entry logic, but insufficient. - From your input embeddings, derive 3 matrices, each being per-entry. - Use the key and value to compute attention (more details in notebook) - Lightweight interface produces a square matrix; self-attention - Feed the sequence-level logic back into the per-token pathway by multiplying withsoftmax-ed attention. - **These components are known as transformer encoder blocks, and they generate per-token encodings with sequence-level knowledge** (Next slide): This leads to entry-on-entry attention mechanism! --- ### Slide 58: Transformers Intuition **This leads to entry-on-entry attention mechanism!** (Next slide: why is this good) --- ### Slide 59: Transformer Value Add **Why is it good? Read off the slide.** (Next slide: How do the big models use attention?) --- ### Slide 60: “Encoder” Transformers In Practice **Big models don’t have one attention mechanism, but many multi-headed attention mechanisms, and they’re trained for bigger things.** - Encoder-style architectures like BERT are giant, trained on large datasets to predict masked out tokens. - They are used as backbones to perform more complex reasoning. --- ### Slide 61: BERT Model Big models don’t have one attention mechanism, but many multi-headed attention mechanisms, and they’re trained for bigger things. - **Encoder-style architectures like BERT are giant, trained on large datasets to predict masked out tokens.** - **They are used as backbones to perform more complex reasoning.** What do they do that makes them work well for such complex tasks? --- ### Slide 62: From Transformer to BERT **What do they do that makes them work well for such complex tasks?** - Grow the network width by adding more attention mechanisms per layer. - Add more transformer layers to grow deep network depth. - Add regularization efforts to make training smoother and overfitting less likely. - Add residual connections to help enforce per-token reasoning and reduce vanishing gradient. --- ### Slide 63: From Transformer to BERT What do they do that makes them work well for such complex tasks? - **Grow the network width by adding more attention mechanisms per layer.** - Add more transformer layers to grow deep network depth. - Add regularization efforts to make training smoother and overfitting less likely. - Add residual connections to help enforce per-token reasoning and reduce vanishing gradient. --- ### Slide 64: From Transformer to BERT What do they do that makes them work well for such complex tasks? - Grow the network width by adding more attention mechanisms per layer. - **Add more transformer layers to grow deep network depth.** - Add regularization efforts to make training smoother and overfitting less likely. - Add residual connections to help enforce per-token reasoning and reduce vanishing gradient. --- ### Slide 65: From Transformer to BERT What do they do that makes them work well for such complex tasks? - Grow the network width by adding more attention mechanisms per layer. - Add more transformer layers to grow deep network depth. - **Add regularization efforts to make training smoother and overfitting less likely.** - Add residual connections to help enforce per-token reasoning and reduce vanishing gradient. --- ### Slide 66: From Transformer to BERT What do they do that makes them work well for such complex tasks? - Grow the network width by adding more attention mechanisms per layer. - Add more transformer layers to grow deep network depth. - Add regularization efforts to make training smoother and overfitting less likely. - **Add residual connections to help enforce per-token reasoning and reduce vanishing gradient.** Now you have a BERT-like model! --- ### Slide 67: BERT Constructed **Now you have a BERT-like model!** **So please check it out in the Notebook 2!** --- ### Slide 68: Part 1: From DL To LLMs We’ve pulling in some models, experimented with them, and now have an understanding of the architecture and intuitions! **Now let’s see how this backbone is used to solve other tasks aside from Masked Language Modeling!** For this part, we’re going to start off by considering the basic NLP tasks listed in the graphic! --- ### Slide 69: HF Pipeline Options We’ve pulling in some models, experimented with them, and now have an understanding of the architecture and intuitions! Now let’s see how this backbone is used to solve other tasks aside from Masked Language Modeling! **For this part, we’re going to start off by considering the basic NLP tasks!** - Feature extraction: Just semantically-dense embeddings of the text that comes in. Sequence-in sequence-out, not human-interpretable. - Token classification: From a semantically-rich embedding, just predict something for each token, and we have token classification. - If the output space is the space of tokens and we’re trying to predict unknown words, then this is mask filling (like most encoder cores) - Text classification: Alternatively, query the model out of just a single sequence entry, and perhaps you can derive insight about the whole passage? --- ### Slide 70: Natural Encoder Tasks Now let’s see how this backbone is used to solve other tasks aside from Masked Language Modeling! **For this part, we’re going to start off by considering the basic NLP tasks!** - **Feature extraction: Just semantically-dense embeddings of the text that comes in. Sequence-in sequence-out, not human-interpretable.** - Token classification: From a semantically-rich embedding, just predict something for each token, and we have token classification. - If the output space is the space of tokens and we’re trying to predict unknown words, then this is mask filling (like most encoder cores) - Text classification: Alternatively, query the model out of just a single sequence entry, and perhaps you can derive insight about the whole passage? --- ### Slide 71: Natural Encoder Tasks Now let’s see how this backbone is used to solve other tasks aside from Masked Language Modeling! **For this part, we’re going to start off by considering the basic NLP tasks!** - Feature extraction: Just semantically-dense embeddings of the text that comes in. Sequence-in sequence-out, not human-interpretable. - **Token classification: From a semantically-rich embedding, just predict something for each token, and we have token classification.** - If the output space is the space of tokens and we’re trying to predict unknown words, then this is mask filling (like most encoder cores) - Text classification: Alternatively, query the model out of just a single sequence entry, and perhaps you can derive insight about the whole passage? --- ### Slide 72: Natural Encoder Tasks Now let’s see how this backbone is used to solve other tasks aside from Masked Language Modeling! **For this part, we’re going to start off by considering the basic NLP tasks!** - Feature extraction: Just semantically-dense embeddings of the text that comes in. Sequence-in sequence-out, not human-interpretable. - Token classification: From a semantically-rich embedding, just predict something for each token, and we have token classification. - **If the output space is the space of tokens and we’re trying to predict unknown words, then this is mask filling (like most encoder cores)** - Text classification: Alternatively, query the model out of just a single sequence entry, and perhaps you can derive insight about the whole passage? --- ### Slide 73: Natural Encoder Tasks Now let’s see how this backbone is used to solve other tasks aside from Masked Language Modeling! **For this part, we’re going to start off by considering the basic NLP tasks!** - Feature extraction: Just semantically-dense embeddings of the text that comes in. Sequence-in sequence-out, not human-interpretable. - Token classification: From a semantically-rich embedding, just predict something for each token, and we have token classification. - If the output space is the space of tokens and we’re trying to predict unknown words, then this is mask filling (like most encoder cores) - **Text classification: Alternatively, query the model out of just a single sequence entry, and perhaps you can derive insight about the whole passage?** Turns out, all of this can be achieved from the same underlying architectural backbone! --- ### Slide 74: Common Encoder Backbone **Turns out, all of this can be achieved from the same underlying architectural backbone!** **It’s just either a subset (feature ext), entirety of (mask fill), or added layer on top of (text/token class) a single pretrained LLM**(or maybe derivatives of one) --- ### Slide 75: Common Encoder Backbone Turns out, all of this can be achieved from the same underlying architectural backbone! It’s just either a subset (feature ext), entirety of (mask fill), or added layer on top of (text/token class) a single pretrained LLM**(or maybe derivatives of one)** **Let’s talk about some of the derivatives!** --- ### Slide 76: Encoder Model Options **Let’s talk about some of the derivatives!** (See slides) --- ### Slide 77: Encoder Model Options **Let’s talk about some of the derivatives!** (See slides) --- ### Slide 78: Encoder Model Options **Let’s talk about some of the derivatives!** (See slides) NEXT: Now let’s talk about what they all have in common: What they’re good for! --- ### Slide 79: BERT Benefits **Now let’s talk about what they all have in common: What they’re good for!** (in slides) NEXT: But what about when you want to predict multiple things, but you don’t want them to be n->n or n->1? You can do zero-shot classification! --- ### Slide 80: Multiple-Query Pipelines But what about when you want to predict multiple things, but you don’t want them to be n->n or n->1? **You can do zero-shot classification! Derived from contradiction/entailment task.** **Great for pulling m classes, one at a time, from an n-entry sequence…** but doesn’t generate sequences… --- ### Slide 81: Variable-Length Outputs What aboutsubsettingsequences? **Sure, you can do that with range prediction. Predict for each sequence entry:** **- Whether the token is a start** **- Whether the token is an end** But what about when that’s not enough and we want more than a subset? --- ### Slide 82: BERT Benefits Still, we’re running up on some limitations. In the next notebook, we’ll explore these limitations in more detail and will propose a better solution for novel text generation. **Start Notebook 3!!** --- ### Slide 83: Part 2: GenAI Models - Welcome back to the course! - This section covers decoder models, which generate language sequences. - We'll contrast decoders with encoders and dive into language generation. --- ### Slide 84: Variable-Length Outputs We’re going from very natural n->n and n->1 tasks, and even a bit of n->m tasks… but let’s look at some new tasks that might not be so good for our architecture. **Question-answering is our first such task, but we said earlier that it’s actually partially solvable by range prediction.** **Added structure is good for small models!** --- ### Slide 85: Translation Tasks We’re going from very natural n->n and n->1 tasks, and even a bit of n->m tasks… but let’s look at some new tasks that might not be so good for our architecture. **How about translation though:** - **Not necessarily 1-to-1 input-token-to-output-token correspondence.** - **n->n is a poor constraint. You can do it, but it’s complicated and we won’t talk about it.** **We need a new formulation!** --- ### Slide 86: From Encoding to Decoding **We’ve been dealing with encoders, so going from a sequence to a latent sequence.** Now we go to Decoder: From a sequence, predict the next entry - Go back to zero-shot classification’s n->1 formulation - Train the model to predict a single token; the next token! --- ### Slide 87: From Encoding to Decoding We’ve been dealing with encoders, so going from a sequence to a latent sequence. **Now we go to Decoder: From a sequence, predict the next entry** - **Go back to zero-shot classification’s n->1 formulation** - **Train the model to predict a single token; the next token!** Given this kind of formulation, we can generate text by predicting one word after another after another. Thereby, n->m. --- ### Slide 88: From Encoding to Decoding **Given this kind of formulation, we can generate text by predicting one word after another after another.** **Thereby, n->m.** Great, but isn’t this just n->1? Isn’t this just an encoder? --- ### Slide 89: Decoder Training Parallelization **Great, but isn’t this just n->1? Isn’t this just an encoder?** **We can just do it with an encoder, but it’s not a good idea:** - **For a dataset of strings, we’d be training the network for a lot of substrings to do 1->1, 2->1, 3->1 etc.** - **Otherwise, a large number of inputs would be OOD (out of domain) for the model. Same with short, medium, etc.** To solve this, we can restrict our encoder model to only reason about the previous sequence entries. --- ### Slide 90: Decoder Training Parallelization **Great, but isn’t this just n->1? Isn’t this just an encoder?** We can just do it with an encoder, but it’s not a good idea: - For a dataset of strings, we’d be training the network for a lot of substrings to do 1->1, 2->1, 3->1 etc. - Otherwise, a large number of inputs would be OOD (out of domain) for the model. Same with short, medium, etc. **To solve this, we can restrict our encoder model to only reason about the previous sequence entries.** - **Means single pass on long string is enough to train for all subsets.** - **All we need to do is mask the attention!** **This solves a lot of practical problems, though it makes our decoder weaker. Still, doesn’t solve all of our problems.** **Going from encoder to decoder solves our short-medium-long-range training domain problem, but what about super-long domain?** --- ### Slide 91: Context Limits To solve this, we can restrict our encoder model to only reason about the previous sequence entries. **This solves a lot of practical problems, though it makes our decoder weaker. Still, doesn’t solve all of our problems.** **Going from encoder to decoder solves our short-medium-long-range training domain problem, but what about super-long domain?** - **Can’t train for super-long because that requires a lot of compute and time.** - **This manifests in settling for training data length maximums, which creates context limits.** - **After context limits, generation goes beyond distribution and deteriorates rapidly.** --- ### Slide 92: Decoder-Guided Generation - Pros: Decoders process everything in one path, gaining more training efficiency. - Cons: Requires more data and can lead to derailment if early predictions are off. - More training improves results, but the need for robust handling of inputs remains. --- ### Slide 93: Encoders vs Decoders **So now we have out encoders and our decoders (read off the slide).** Both have limitations, so why not combine them? --- ### Slide 94: Encoders vs Decoders **So now we have out encoders and our decoders (read off the slide).** **Both have limitations, so why not combine them?** --- ### Slide 95: Encoder-Decoder **Encoder-Decoder from Attention Is All You Need (2017)** - **Combines both encoder and decoder components together via an interface called cross-attention.** - **In other words, use an n-element sequence to condition an m-element sequence, resolving sequence length discrepancy.** - Feed an n-sequence key and value and an m-sequence query, and the math will work out! (N+M)->1 conditioning! --- ### Slide 96: Encoder-Decoder **Encoder-Decoder from Attention Is All You Need (2017)** - Combines both encoder and decoder components together via an interface called cross-attention. - In other words, use an n-element sequence to condition an m-element sequence, resolving sequence length discrepancy. - **Feed an n-sequence key and value and an m-sequence query, and the math will work out!** - Q is from encoder (n), so attn row is n. - K is transposed and from decoder (m), so attn column is m. - V is also from encoder (n), so row of V is n. - MultiplyMxNandNxKleadsMxK, where k is some latent value. - **Math Works! (N+M)->1 Conditioning** --- ### Slide 97: Encoder-Decoder Benefits Overall Encoder-Decoder benefits. - Decoder supervised with context from encoder which is held constant. - (Good time to note KV caching, where the encoder outputs are often-times not recomputed) - Takes advantage of decoder to generate sequences autoregressively. Best when context and response are strongly separated (i.e. vocabulary gap, semantic gap,etc) Not ideal when there is little-to-no context/response gap (i.e. same vocabulary, fuzzy boundaries) --- ### Slide 98: Conversational Tasks **Text-generation and conversation are both forecasting tasks, one on text generation and the other on response.** **This is the area that has seen a lot of interest, so it’s a great one to finish off on.** --- ### Slide 99: Progressing Towards Zero-Shot We can train a variety of pipelines to do whatever seq2seq task we want. **However, this isn’t really how the modern seq2seq landscape has evolved.** **Instead of training single-task seq2seq pipelines, many modern formulations train a single model to do many tasks.** - **More language understanding in both the encoder and decoder.** - **Potential transferability of intuition from one task to another.** T5: An encoder-decoder that’s commonly trained on many tasks, both useful on their own and for core language understanding. --- ### Slide 100: Question-Understanding Seq2Seq We can train a variety of pipelines to do whatever seq2seq task we want. **However, this isn’t really how the modern seq2seq landscape has evolved.** **Instead of training single-task seq2seq pipelines, many modern formulations train a single model to do many tasks.** **T5: An encoder-decoder that’s commonly trained on many tasks, both useful on their own and for core language understanding.** **The goal is for eventual understanding of the language and nuanced meaning of encoder inputs to generate decoder outputs.** (But the T5 on its own is insufficient. It does not transfer well beyond training tasks, but is compact at least) --- ### Slide 101: Instruction-Following Seq2Seq **The goal is for eventual understanding of the language and nuanced meaning of encoder inputs to generate decoder outputs.** But the T5 on its own is insufficient. It does not transfer well beyond training tasks, but is compact at least. **Flan, on the other hand, has shown to be pretty good.** **Leverages the same multi-task formulation but trains on even more complicated tasks that require nuanced understanding.** **Transfers well to new tasks based on language instruction alone.** **You will investigate this formulation and more in the next notebook, so good luck!** --- ### Slide 102: Part 2: GenAI Models Hello and welcome back to the course! - If you haven't taken a break yet, we highly recommend you do so before continuing. - But if you're ready, let's dive into the next section. --- ### Slide 103: Large Language Models We promised earlier that we'd discuss how data is processed, the intuitions at play, and what the architectures are actually doing as the pipeline progresses. --- ### Slide 104: Encoders vs Decoders Now, we have our components: encoders for taking in sequences and decoders for outputting sequences, using transformers as our architecture. But what about other modalities other than language? --- ### Slide 105: Modular Modalities - At the beginning, we mentioned that natural language isn't just for reasoning with language in isolation—it's useful to incorporate other modalities like images, audio, 3D video, DNA, protein molecules, etc. - We've been working with text input and text output, but what about these other things? Before diving into each individually, let's discuss what we mean by 'different modalities.'" --- ### Slide 106: Information Modalities An information modality is data in a particular format. You're familiar with classes, images, and text. - We've reasoned with images as two-dimensional grids of pixels with spatial locality—hence why convolutional networks are common. - For natural language, while you could use convolutional neural networks, people often prefer transformers for their attention mechanisms. - Other modalities exist, and as far as processing is concerned, these are just different data distributions with different modeling assumptions. But let’s define them a bit more rigorously… --- ### Slide 107: What Makes Modalities Different? Modalities can be communicated in different ways—different file types, sensory types, etc. - Whether they're explicit files processed differently by an operating system or data frequencies processed internally by humans through senses like sight, hearing, touch—they're different. - That's the surface-level definition. Technically, all this data is just data. We can think of modalities as different data distributions—maybe Gaussian, Poisson, or the distribution of images or employee data.” - But we also have a narrower technical definition when motivating architectures. When we think about images or natural language, we consider how to understand the data—what structure does it elicit, and how do we process it effectively? - For images, we used convolutional neural networks to extract features, then dense layers to reason about them. For language, we use encoder-decoder structures. - Depending on the data's structure and important intuitions, we choose different models. This can define a different modality or data distribution, radically changing our approach to processing the domain." --- ### Slide 108: Multimodal Connections - So all this data—text, audio, images, 3D, etc.— may be modeled differently depending on how it's captured and represented. - Text can be modeled as a sequence of letters or classes—that's an engineering decision that works well. - Audio—is it a sequence of frequencies? Frequencies can be considered as vectors; maybe we can model it as images if we window it. - For images, we've thought of grids of pixels with spatial locality, but perhaps there are other ways—chunking into patches, for example. - 3D data can be modeled as volumes, point clouds, or collections of faces (meshes). - Video is a sequence of images—but should we model it that way? Maybe as sequences of windows or 3D blocks. - DNA is also a sequence of classes with a limited vocabulary—it can be modeled similarly. - So, how do we link these modalities together?" --- ### Slide 109: Multimodal Connections At an architectural level, we can link modalities by encoding them through intuitive architectures and combining them if their dimensionalities align. We've seen in previous notebooks that we can pass from natural language representations into latent sequences repeatedly. Is it possible to combine information from one sequence with another? - Mathematically, yes—if we have vectors of lengths N and M, we can perform operations on them. - So, if we can encode multiple modalities in a way that they can be connected mathematically, or adjust their structures to align, that works. - In practice, creating multimodal connections involves modeling different input pathways, different decoder or output pathways, and linking them together." --- ### Slide 110: Multimodal Connections This sets a natural definition for encoders and decoders. - The bidirectional transformer architecture is great for encoding language but not for decoding it—encoders aren't well-suited for generating sequences one token at a time. - In contrast, decoder-style architectures are great at generating text. - The same applies to other modalities—for images, convolutional neural networks are great encoders, and transpose convolutions can serve as decoders. Linking them creates encoder-decoder architectures, which we'll see in image generation. So, multimodality involves linking components suited for their domains." --- ### Slide 111: Multimodal Connections Now that we understand we have these components and can link them, let's focus on the ones we'll discuss in the course and notebooks: audio and images, in addition to text. - Text, as a sequence of letters or classes, makes sense. - Audio has similar assumptions—it's a sequence of frequencies. - Images are also relatively easy to reason about as sequences but will present more issues when we discuss generation. Other modalities like 3D, video, DNA, robotics—you can explore on your own, but they also involve modeling assumptions and architectures suited to process that data efficiently. --- ### Slide 112: Transformer Benefits **Reiterating the value proposition of transformers—they're great for text because text is an ordered sequence of classes. So, if we can think of data as a sequence and reason about it as a sequence, transformers might be a good fit.** Moving to audio, people use transformers a lot for reasoning with audio and speech because audio is a sequence over time—a time series of frequencies. You sample audio over time, getting frequency samples at each timestep. You can normalize them tomel-frequency spectrograms, but you’re inherently reasoning with a time series. So, transformers can be applied here. "What about images? We've learned that convolution is great for encoding images. Can we use transformers? Yes—Vision Transformers (ViTs) take an image, split it into patches, and reason about the embeddings of patches. This provides benefits, allowing us to synergize with existing transformer architectures.ViTsare still used today for image processing." --- ### Slide 113: Speech-Guided Encoders Reiterating the value proposition of transformers—they're great for text because text is an ordered sequence of classes. So, if we can think of data as a sequence and reason about it as a sequence, transformers might be a good fit. **Moving to audio, people use transformers a lot for reasoning with audio and speech because audio is a sequence over time—a time series of frequencies. You sample audio over time, getting frequency samples at each timestep. You can normalize them to****mel****-frequency spectrograms, but you’re inherently reasoning with a time series. So, transformers can be applied here.** "What about images? We've learned that convolution is great for encoding images. Can we use transformers? Yes—Vision Transformers (ViTs) take an image, split it into patches, and reason about the embeddings of patches. This provides benefits, allowing us to synergize with existing transformer architectures.ViTsare still used today for image processing. --- ### Slide 114: Image-Guided Encoders Reiterating the value proposition of transformers—they're great for text because text is an ordered sequence of classes. So, if we can think of data as a sequence and reason about it as a sequence, transformers might be a good fit. Moving to audio, people use transformers a lot for reasoning with audio and speech because audio is a sequence over time—a time series of frequencies. You sample audio over time, getting frequency samples at each timestep. You can normalize them tomel-frequency spectrograms, but you’re inherently reasoning with a time series. So, transformers can be applied here. **"What about images? We've learned that convolution is great for encoding images. Can we use transformers? Yes—Vision Transformers (****ViTs****) take an image, split it into patches, and reason about the embeddings of patches. This provides benefits, allowing us to synergize with existing transformer architectures.****ViTs****are still used today for image processing.** --- ### Slide 115: Synergizing Encoders - Given that we can create different encoders based on the transformer architecture, can we get use out of them without an explicit decoder? - Yes—Contrastive Language-Image Pre-training (CLIP) shows this. CLIP uses multiple encoders for different modalities and synergizes their outputs. - When we have a pair of an image and a caption, we train the network to produce embeddings that are close together. This synergizes a text encoder and an image encoder. - You'll see this in the notebook—the embeddings become useful representations for images and text, allowing for tasks like retrieval or swapping between images and text in analysis. --- ### Slide 116: Multimodal Retrievers - Going further into retrieval, synergized encoders are useful even within natural language. - If you have two types of text or documents, you can use encoders to retrieve what's relevant, similar, or useful in a context. This is common in retrieval-augmented generation. - For example, with a query 'Labrador sitting on bench near water,' if you have synergized image and text encoders, you can embed the query and retrieve images from a pool that match. - Similarly, with text documents, you can use different encoders to retrieve longer passages or summaries. --- ### Slide 117: Synergizing Encoders This idea of synergizing encoders can be pushed to extremes. TheImageBindresearch from Meta trained encoders mapping various modalities (depth, heatmaps, IMUs, audio, text) into a joint embedding space. They could swap out modalities, generate new data, retrieve from pools, and explore emergent properties. --- ### Slide 118: Speech-Guided Text Generation - Let's step back to not only taking in interesting modalities but also creating interesting information. - **Is speech-guided text generation possible? Yes—if we have transformer components for ingesting audio and creating text, we can combine them in a classic encoder-decoder architecture. This makes sense due to the clear modality gap: input audio, output text. Encoder-decoder architectures work well here—OpenAI's Whisper model is an example, using encoder plus decoder with cross-attention. You'll get to play with that in the notebook.** Similarly, for image-guided text generation, we can take an image encoder and combine it with a text decoder using cross-attention. This works well in practice, and you can experiment with it. More recently, people have been training decoder-only models for the same objective. --- ### Slide 119: Image-Guided Text Generation - Let's step back to not only taking in interesting modalities but also creating interesting information. - Is speech-guided text generation possible? Yes—if we have transformer components for ingesting audio and creating text, we can combine them in a classic encoder-decoder architecture. This makes sense due to the clear modality gap: input audio, output text. Encoder-decoder architectures work well here—OpenAI's Whisper model is an example, using encoder plus decoder with cross-attention. You'll get to play with that in the notebook. **Similarly, for image-guided text generation, we can take an image encoder and combine it with a text decoder using cross-attention. This works well in practice, and you can experiment with it.**More recently, people have been training decoder-only models for the same objective. --- ### Slide 120: Multi-Image Guided Text Generation Similarly, for image-guided text generation, we can take an image encoder and combine it with a text decoder using cross-attention. This works well in practice, and you can experiment with it. **More recently, people have been training decoder-only models for the same objective.** **If you look into models like****LLaVA****, VILA, or****InstructBLIP****, you'll notice that models taking both text and images often put them through the same input pathway. There's a text decoder running, and you can input natural language or train an auxiliary layer to embed images into the same input space as text. This allows for rich and even multi-image prompts. Of course, this requires different training formulations, but there's no reason you can't use a one-size-fits-all decoder taking in both text and images, as long as images are projected appropriately (e.g., using a****ViT****).** --- ### Slide 121: NVLM: Open Frontier-Class Multimodal LLMS (2024) We've solidified that both encoder-decoder and decoder-only models can be used for multimodality. Models like NVLM exemplify this with three architectures for the same task of visual-language modeling: cross-attention-based, hybrid, and decoder-only schemes. Encoding is processed at the beginning, and inputs of images and text are fed through the same pathway. This validates that encoder-decoder vs. decoder-only is up to you, with pros and cons for each." --- ### Slide 122: Introducing Llama 3.1: Our most capable models to date | Meta (2024) Going further, people have been experimenting with creating complex pathways with arbitrary numbers of encoders and decoders to build the systems they want. This can be prohibitively expensive if starting from scratch due to the need to synergize components. But if you have the resources, you can create custom pipelines for video, image, speech, etc., and combine them. Most people benefit from these architectures' emergence. Conceptually, when reading papers, this is what's happening—experimenting with different modalities, input representations, and combining them into shared fused architectures. --- ### Slide 123: Core Transformer Benefits - To wrap up this multimodal encoding discussion, what are the core benefits of transformers? It's not just language modeling—it's sequence modeling. If we're good at modeling sequences, we can model audio and image ingestion as well. We can combine them, use fusion formulations, and train mappings from one latent representation to another. - That's what you saw with decoder-only language modeling where there's an image encoder at the beginning mapping to the decoder's input. Lots of good stuff going on. --- ### Slide 124: Image Generation? - **It looks like transformers can reason with a lot of data… so does that mean they can generate images? Unfortunately, sort of, but not really.** That's why, when we looked at diagrams connecting modalities, there wasn't just the transformer—there was another structure, like an autoencoder or a U-Net, which we'll discuss. Generating images isn't the transformer's strong suit on its own. This is partly because we don't model images as sequences when generating them. Generating images as sequences is problematic—you miss high-frequency, pixel-level information. What's your sequence resolution—one pixel? Autoregressive image generation is inefficient." --- ### Slide 125: Image Generation? - It looks like transformers can reason with a lot of data… so does that mean they can generate images? Unfortunately, sort of, but not really. **That's why, when we looked at diagrams connecting modalities, there wasn't just the transformer—there was another structure, like an autoencoder or a U-Net, which we'll discuss. Generating images isn't the transformer's strong suit on its own.** This is partly because we don't model images as sequences when generating them. Generating images as sequences is problematic—you miss high-frequency, pixel-level information. What's your sequence resolution—one pixel? Autoregressive image generation is inefficient." --- ### Slide 126: Image Generation? - It looks like transformers can reason with a lot of data… so does that mean they can generate images? Unfortunately, sort of, but not really. That's why, when we looked at diagrams connecting modalities, there wasn't just the transformer—there was another structure, like an autoencoder or a U-Net, which we'll discuss. Generating images isn't the transformer's strong suit on its own. **This is partly because we don't model images as sequences when generating them. Generating images as sequences is problematic—you miss high-frequency, pixel-level information. What's your sequence resolution—one pixel? Autoregressive image generation is inefficient."** --- ### Slide 127: Making An Autoencoder So, what do we do instead? - **We often use bottlenecking architectures where we go down in encoding and up in decoding.** - **Transpose convolutions can add details, maintaining spatial locality, allowing the decoder to learn how to add pixel information. This is useful for denoising or super-resolution.** - U-Nets are similar to autoencoders but include skip connections between encoder and decoder layers, allowing information at different resolutions to be communicated as the decoder progresses." --- ### Slide 128: Making A U-Net So, what do we do instead? - We often use bottlenecking architectures where we go down in encoding and up in decoding. - Transpose convolutions can add details, maintaining spatial locality, allowing the decoder to learn how to add pixel information. This is useful for denoising or super-resolution. - **U-Nets are similar to autoencoders but include skip connections between encoder and decoder layers, allowing information at different resolutions to be communicated as the decoder progresses."** --- ### Slide 129: Making A Denoiser Another strategy is denoising or diffusion. Instead of going from implicit representations to explicit output suddenly, we progress the representation in a natural space, like image space or compressed image space. We go from noise to slightly less noise progressively until we get a clear image. This allows details to be added progressively, enforcing spatial locality and refining high-precision information. It's an all-in-one approach for modeling complex, high-frequency details progressively. --- ### Slide 130: Policy Networks Considering that we have diffusion, autoregression, and other methods, there are situations where all are viable. For example, policy networks in robotics predict what a robot should do at a given time. You can model the entire trajectory directly, which works but is restrictive. Alternatively, you can model it progressively, predicting the next action step by step. This can be autoregressive. In some cases, diffusion is used to initialize possible paths and progressively denoise them into the final pathway. Despite treating it less like a sequence problem, this works fine. --- ### Slide 131: Helping Image Generation? - That was a deep dive, but hopefully you're still with us. - We’ve confirmed that transformers might not be ideal for generating complex images. - But, can transformers help generate images? The answer is very much yes. --- ### Slide 132: Image Generation? - This is the architecture for a common image-generating model called Stable Diffusion. - As you can see, we can guide the denoising process with textual information or other representations using attention mechanisms. We inject context into the generation, impacting the output. - There's also an additional encoder-decoder on the far left, converting raw pixel space into a smaller representation before denoising. It's engineering, but it follows the logic of giving enough information, simplifying representations, modeling important aspects, and with proper training, it works." --- ### Slide 133: Synergized Multi-Domain Encoders Pushing this further, models like X-LAN have multiple embedding and encoding strategies optimized for various modalities, creating a visual embedding highway. Representations communicate with each other, and as long as we have something like a language model and the highway connecting information without decimating structure, we can learn effective relationships. We can have diffusion-guided image decoding, video decoding, audio decoding, language generation—all happening with shared representations synergized together in an end-to-end deep learning pipeline." --- ### Slide 134: Modular Modalities - Okay, that's it! This was probably more detailed than you wanted about the magic of generative AI, but hopefully you learned a lot. - You're now ready to tackle Notebook 5, which will showcase synergized encoders, decoders, image generation, and a few other things. - Good luck, and I'll see you in the next section for generative AI services. --- ### Slide 135: Part 2: GenAI Models - Welcome back! - Last section:**Multimodality**—synergizing encoders and decoders. - Now, shifting focus to**scaling, optimization, and building production-level systems**for real applications. --- ### Slide 136: Decoder-Guided Generation Main focus:**Decoder models for natural language**. We can build multitask systems with decoders that take in context/questions and generate responses. This is great, but you’ve also definitely seen this in action before… --- ### Slide 137: LLMs in the Wild: ChatGPT **ChatGPT**is a prime example. A decoder-style model wrapped with a user interface that "remembers" (feeds previous info as context). --- ### Slide 138: Productized LLM Application **GPT-4**is the powerhouse behind ChatGPT. - Handles various tasks and text generation options. - Supports conversations, tasks, continuations—generates until it stops. - It's helpful, reasons well, doesn't derail easily. - High expectations—we want it to be correct and reliable. - Techniques beyond training help ensure this. - Such models are powerful, solving many language problems. --- ### Slide 139: Productized LLM Application GPT-4 is closed-source, so you can't see its internals. - But there are open models like**Llama 3.1 40B**. - Huge models—you probably can't run them locally due to resource limits. - In theory, you can use the same pipeline APIs. --- ### Slide 140: Productized LLM Application Despite not being able to host it locally, you can still access giant and/or closed models through an**API interface**using POST requests. Set up inputs as system and user messages and pass them in, and the server will format the input correctly for the model and send back a generated result. --- ### Slide 141: Productized LLM Application - Using such an API which can scale and stay resilient against load, systems like ChatGPT can scale to many concurrent users. - This is the penultimate example of a generative AI application scaled to the masses without requiring the masses to be able to host it themselves. --- ### Slide 142: The Problem Landscape Challenge: Going from a local decoder model to an app like ChatGPT. - **Limited resources**—hard to load large models locally, or even fit big models on typical systems. - Lacking chat interfaces and functionalities, so need**software and orchestration**to bridge the gap. (NEXT SLIDE) Also hard to scale to many users and many instances. --- ### Slide 143: The Problem Landscape Challenge: Going from a local decoder model to an app like ChatGPT. - **Limited resources**—hard to load large models locally, or even fit big models on typical systems. - Lacking chat interfaces and functionalities, so need**software and orchestration**to bridge the gap. **Also hard to scale to many users and many instances.** - Even if you have access to big models, sharing them is tough. Can't expect everyone to host massive models, and doing so would be grossly inefficient and expensive. - Need solutions beyond individual environments to provide access. --- ### Slide 144: Model Deployment **Model optimization is necessary for deployment.** - Make models smaller, optimize them for specific hardware. - Fuse components to improve efficiency. - Sometimes need to**distribute models over multiple GPUs**. - Various routines and tools can help with this. --- ### Slide 145: Model Deployment **Model scaling is necessary for deployment.** - Allow models to grow, get and prioritize requests from many users, and replicate. - The goal is to use some access path to the model as a primitive to build more interesting applications. - (NEXT) How OpenAI does this. --- ### Slide 146: Large Model Hosting Platforms - OpenAI-Style - **lide****146: OpenAI's Solution** - OpenAI uses an**OpenAI gateway system using a standard format**. - Models deployed in data centers, replicating as needed. - Users access models via**endpoints**—submit requests, specify models, use API keys. - Abstracts complexity, allows for scaling and monitoring automatically so they can focus on building applications. --- ### Slide 147: Query Router Access Review of sending a request: - Users send messages and settings to an endpoint. - Endpoint handles transaction facilitation, monitoring, optimization. - Inputs formatted for the model, processed in batches. - Load balancing and other optimizations occur behind the scenes. --- ### Slide 148: NVIDIA NIM Optimized Inference Microservices In this course, we'll use**NVIDIA NIM**Inference Microservices to deploy large models efficiently. - Offers prebuilt containers, scaling tools like Helm charts. - Supports**in-flight batching**—smoothly integrates new requests. - Allows deploying models on your cloud infrastructure. - Creates endpoints for client-side processing or integration into larger applications. --- ### Slide 149: NIM In Your Course This course will have you interact with 3 levels of deployment: - VLLM-based; non-quantizing deployment of a small model manually. Kickstarted in complementary notebook. - NIM-based: Model kickstarted in the background which is optimized for our A100-backed environment. Accessed through nim microservice. - Build.nvidia.com: Also NIM-backed, but hosted externally. Check outbuild.nvidia.comfor more info. Accessed throughllm_clientmicroservice. --- ### Slide 150: Notebook Objective: Endpoint Orchestration The goal for Notebook 6 is to explore**model deployment solutions**and introduce you to**endpoint orchestration.**More details on orchestration in the coming notebooks. --- ### Slide 151: Part 3: Stateful Systems Welcome back! Now that we have our AI primitives deployed, we're moving into**Stateful Systems**. Focus:**Orchestration and****Agentics**. We'll explore how to add interesting state to our AI systems and do some very cool things with them --- ### Slide 152: Endpoint Orchestration We've introduced**Endpoint Orchestration**and**LLM Orchestration**. - Learned about how endpoints are created, and how they can be used to build interesting systems. - We've done basic document Q&A, tested our deployed models, and even created a chatbot interaction. - Let's delve deeper. --- ### Slide 153: Chatbot Workflow - **Chatbot Workflow (How Things Started)** - The classic**Chatbot Workflow**is an iterative process. - We feed the model dialogue exchanges and ask it to generate the next agent response. - It's a simple loop and forms the foundation of many chatbots today. --- ### Slide 154: General Chat Application - **General Chat Application (How Things Progressed)** - Chatbots have evolved. Beyond accumulating correspondence, we need**state management**. - This includes: - Scale deployment aspects. - Multiple endpoints for different use cases. - Support for calling tools. - Finding more context and stateful information over time. - Directing or intercepting outputs to create richer interfaces. --- ### Slide 155: Dialog/Retrieval Agents - **Dialog/Retrieval Agents (Demo: ChatGPT)** - **ChatGPT**showcases advanced capabilities. - It can display graphs, write code, and even execute code. - When it writes code, it attempts to run it, providing outputs like plots. - This is possible by combining LLMs with additional software tools. --- ### Slide 156: Dialog/Retrieval Agents - **Dialog/Retrieval Agents (Web Search)** - ChatGPT and similar systems can retrieve real-time information. - They can scrape data from websites, perform web searches, and incorporate that into responses. - Example: Asking about a recent conference and getting up-to-date information. --- ### Slide 157: Dialog/Retrieval Agents - **Dialog/Retrieval Agents (Memory and Context)** - These systems can remember information about you. - Not because the model weights change, but because they maintain a**state**or**memory**. - They can recall past interactions, updating their knowledge base dynamically. --- ### Slide 158: Dialog/Retrieval Agents - **LLM Orchestration with Context and Control** - This leads us to**LLM Orchestration**, which involves combining LLMs with software to route, direct, and influence both the software and the LLM. - Key components: - Retrieving context. - Updating context. - Using classical algorithms or heuristics. - Incorporating state management. --- ### Slide 159: Dialog/Retrieval Agents - Retrieving context. - Beyond just user input, we retrieve additional context relevant to the generation. - This can involve database queries, code execution, or semantic searches. - Helps the LLM provide more informed responses. --- ### Slide 160: Dialog/Retrieval Agents - **Augment Stage**: - We synthesize the retrieved context and prepare it to feed into the LLM. - Enhances the LLM's ability to generate precise responses based on the question. --- ### Slide 161: Dialog/Retrieval Agents - **Generate Stage**: - The LLM takes the augmented context and generates a response. - The process integrates retrieval, augmentation, and generation. - Results in a well-contextualized answer to the user's query. --- ### Slide 162: LLM Orchestration Terminology - **LLM Orchestration Terminology** - Key terms in LLM orchestration: - **Retrieval**: Fetching data from a database or source. - **Tooling**: Allowing the LLM to call external tools or execute code. - **Routing**: Switching between different prompts or instructions. - **Agentics**: Delegating tasks to specialized expert systems or agents. --- ### Slide 163: Semantic Tools - **Similarity Queries** - Using**semantic tools**for similarity queries. - Embedding documents and performing semantic searches. - Example: Retrieving relevant scientific papers based on a user's query. --- ### Slide 164: Semantic Tools - **LLM Orchestration Technique** - LLMs use orchestration to perform tasks like recalling a user's name from a database. - Involves retrieving relevant info (e.g., via a vector database retriever) and synthesizing it into the response. --- ### Slide 165: Semantic Tools - **Routing Systems** - **Routing Systems**decide how the LLM should respond by routing to different prompts, states, and workflows. - They can determine whether to answer a query based on policies or classifiers. - Example: If a question is disallowed, the LLM will refuse to answer by being routed to a prompt that forces it to react negatively and abruptly. --- ### Slide 166: Hybrid Semantic/Structured - **Hybrid Semantic/Structured Agentic Systems** - Combining LLMs with structured data sources. - Allows handling both structured and semantic queries. - Interaction with file systems, databases, or APIs to perform tasks like searching files or reasoning over data. --- ### Slide 167: Structured Retrieval - **Interacting Like a Human** - LLMs can interact with data in a human-like manner by just outputting text. - Generating unstructured text from input queries, which can be fed into other LLMs (or humans) - Good for some interactions, but not necessarily for environment access. --- ### Slide 168: Structured Retrieval - **Structured Retrieval With Direct Environment Queries** - LLMs can retrieve structured data by interacting with databases or programmatic environments. - Example: The LLM generates an SQL query to retrieve data from a database and answers the user's question. --- ### Slide 169: Structured Retrieval - **Guided Structure Generation** - LLMs can be forced to output schema in a particular grammar with sampling-level tricks. More on this later. - Since natural input to functions is a key-value pair (variable + value), this is extremely powerful. --- ### Slide 170: Structured Retrieval - **Agentic Tooling** - If LLMs can be forced to output a specific schema, maybe they can do it twice; once to decide what to do, and another to do it. - Backbone logic of LLMs selecting tools, since LLM can be forced to predict both from the set of tool options and then the arguments for said tool. --- ### Slide 171: How To Predict Valid Schema? - **Schema Prediction via Prompt Engineering** - Making valid schema predictions using prompt engineering. - System fills in missing information (e.g., retrieving a name like "Jane Doe"). - Refines outputs through iterative attempts and algorithmic postprocessing, but likely to be rough. --- ### Slide 172: How To Predict Valid Schema? - **LLM Priming** - Using**LLM Priming**to improve schema predictions. - Providing well-prepared prompts to guide the LLM. - Helps the LLM fill in expected values based on structured schema inputs, but still can derail or miss. --- ### Slide 173: How To Predict Valid Schema? - **Tool Training for Accurate Schema Generation** - Enhancing schema prediction with**Tool Training**. - Integrating tools into the LLM's flow for more accurate outputs. - Ensures consistency in retrieval processes, possibly via external APIs or code functions. --- ### Slide 174: How To Predict Valid Schema? - **Constrained Decoding** - By restricting the output space to valid tokens, the LLM produces outputs that fit expected data formats. - Reduces errors in the prediction pipeline, but still requires other tools in the arsenal (i.e. prompt engineering, function-calling training). - NOTE: sometimes models that do function-calling are called “action models.” --- ### Slide 175: Tool-Calling - **Tool-Calling Mechanism in Action** - **Tools can be introduced to the main event loop directly, and the main system can try to pick them as it goes along.** - (Next) alternatively, each tool can be wrapped with its own agentic loop or some LLM-backed interface which keeps the main loop from getting blasted with complexity/context. --- ### Slide 176: Agentic Tool-Calling - **Tool-Calling Mechanism in Action** - Tools can be introduced to the main event loop directly, and the main system can try to pick them as it goes along. - **Alternatively, each tool can be wrapped with its own agentic loop or some LLM-backed interface which keeps the main loop from getting blasted with complexity/context.** --- ### Slide 177: Tooled ReAct Loop - **Multi-Turn With****ReAct****:** - **- Tools can also be abstracted to include the user, either explicitly or implicitly.** - User here would be a tool returns dialog or supervision as context… just like any other tool. - If all or most tools are properly honored, their results are fed back into the agent loop, and multiple tools can be called between user interactions, it’s often known as the “ReAct(Reason and Act)” paradigm (which is one of many). Notebook 7 will explore these ideas more, and will culminate in aReActloop. --- ### Slide 178: Part 3: Stateful Systems **Optional Section on Multi-Agent Systems** Intended to prime the optionalLangGraphnotebook and discuss more about multi-agent systems. --- ### Slide 179: Tooled ReAct Loop We left off at theReActloop, and previously defended the multi-agent abstraction as useful for maintaining simplicity and modularity. Let’s further investigate this proposition… --- ### Slide 180: Multi-Agent System In contrast to single-agent systems, MAS involve multiple systems that performagentically, which could mean agents that kickstart other agents or multiple concurrent agents that interact through a shared interface. The main thing here is the notion of disjoint states and potentially-disjoint policies, or data pipelines, for dealing with their inputs…. --- ### Slide 181: Data Pipeline The main thing here is the notion of disjoint states and potentially-disjoint policies, or data pipelines, for dealing with their inputs…. - **A particular agent could have a refinement pipeline similar to the dialog management chatbot or the long document processing examples in the previous notebook.** - Another might have more of a baked-in retrieval heuristic where they query an external resource by default as part of their operations. - Another one yet may construct a hierarchy or other abstraction to deal with information. All these have been explored in the previous notebook in isolation, butagenticswould in theory allow any of these processes to be modularized behind an “agent interface” and be called as a sub-process that managed its own state. The only thing we would really need, is a buffer system that facilitates this transfer of information and a way to inform our systems how the other agents function. If we just do that, we can treat agents as tools and make arbitrarily modularized or arbitrarily complex pipelines for dealing with language and interacting with environments. --- ### Slide 182: Data Pipeline The main thing here is the notion of disjoint states and potentially-disjoint policies, or data pipelines, for dealing with their inputs…. - A particular agent could have a refinement pipeline similar to the dialog management chatbot or the long document processing examples in the previous notebook. - **Another might have more of a baked-in retrieval heuristic where they query an external resource by default as part of their operations.** - Another one yet may construct a hierarchy or other abstraction to deal with information. All these have been explored in the previous notebook in isolation, butagenticswould in theory allow any of these processes to be modularized behind an “agent interface” and be called as a sub-process that managed its own state. The only thing we would really need, is a buffer system that facilitates this transfer of information and a way to inform our systems how the other agents function. If we just do that, we can treat agents as tools and make arbitrarily modularized or arbitrarily complex pipelines for dealing with language and interacting with environments. --- ### Slide 183: Data Pipeline The main thing here is the notion of disjoint states and potentially-disjoint policies, or data pipelines, for dealing with their inputs…. - A particular agent could have a refinement pipeline similar to the dialog management chatbot or the long document processing examples in the previous notebook. - Another might have more of a baked-in retrieval heuristic where they query an external resource by default as part of their operations. - **Another one yet may construct a hierarchy or other abstraction to deal with information.** All these have been explored in the previous notebook in isolation, butagenticswould in theory allow any of these processes to be modularized behind an “agent interface” and be called as a sub-process that managed its own state. The only thing we would really need, is a buffer system that facilitates this transfer of information and a way to inform our systems how the other agents function. If we just do that, we can treat agents as tools and make arbitrarily modularized or arbitrarily complex pipelines for dealing with language and interacting with environments. --- ### Slide 184: Local State Accumulation The main thing here is the notion of disjoint states and potentially-disjoint policies, or data pipelines, for dealing with their inputs…. - A particular agent could have a refinement pipeline similar to the dialog management chatbot or the long document processing examples in the previous notebook. - Another might have more of a baked-in retrieval heuristic where they query an external resource by default as part of their operations. - Another one yet may construct a hierarchy or other abstraction to deal with information. **All these have been explored in the previous notebook in isolation, but****agentics****would in theory allow any of these processes to be modularized behind an “agent interface” and be called as a sub-process that managed its own state.** The only thing we would really need, is a buffer system that facilitates this transfer of information and a way to inform our systems how the other agents function. If we just do that, we can treat agents as tools and make arbitrarily modularized or arbitrarily complex pipelines for dealing with language and interacting with environments. --- ### Slide 185: Communication Protocols The main thing here is the notion of disjoint states and potentially-disjoint policies, or data pipelines, for dealing with their inputs…. - A particular agent could have a refinement pipeline similar to the dialog management chatbot or the long document processing examples in the previous notebook. - Another might have more of a baked-in retrieval heuristic where they query an external resource by default as part of their operations. - Another one yet may construct a hierarchy or other abstraction to deal with information. All these have been explored in the previous notebook in isolation, butagenticswould in theory allow any of these processes to be modularized behind an “agent interface” and be called as a sub-process that managed its own state. **The only thing we would really need, is a buffer system that facilitates this transfer of information and a way to inform our systems how the other agents function. If we just do that, we can treat agents as tools and make arbitrarily modularized or arbitrarily complex pipelines for dealing with language and interacting with environments.** --- ### Slide 186: Case For Agents - **Case for Agents (Why Even Bother)** - Modular systems reduce context bottlenecks. - Each agent handles its part—SQL queries, document processing, etc. - Focus on dividing tasks into manageable modules. - Agents reduce the workload on the main loop by handling specialized logic independently. --- ### Slide 187: LangGraph LangGraphwill be used in the notebook. Please check it out to see roughly how it works, and consider checking out examples after the class regarding example implementations and off-the-shelf solutions from theLangChainteam! --- ### Slide 188: Part 3: Stateful Systems **Welcome back—and congratulations!** We've**completed the course content**. - Covered how large language models are developed, implemented, and deployed. - Explored their use in cognitive architectures, multi-agent systems, chatbots, and processing applications. In this section, we'll introduce the**final assessment**—**Notebook 8**. We want you to go through it, finish it, and earn your certification. --- ### Slide 189: Data Pipeline **Recall**: We've talked a lot about**data pipelines**. - Learned how to set up systems that pass from one agent component to another. - It's all software—combining traditional software with**LLM interfaces**. - If you have an LLM predicting actions, you instruct it to predict in a specific structure. - Then, fulfill the request, provide necessary context, or transition to the next step. --- ### Slide 190: Local State Accumulation - **Applying Concepts in Practice** - **You can build a cognitive architecture:** - Queries update states, accumulate context, and provide feedback. - Like a**dialogue-style loop**. - (Next) Or keep it simple: - Transition from input to processed versions sequentially. - Accumulate states, let states drop, or move between states as needed. - Define your pathways—for example: - Natural language input → model → output. - LLM output → algorithm for further handling. - The**final assessment**tests your ability to apply this in practice. --- ### Slide 191: Typical Data Pipeline - **Applying Concepts in Practice** - You can build a cognitive architecture: - Queries update states, accumulate context, and provide feedback. - Like a**dialogue-style loop**. - Or keep it simple: - Transition from input to processed versions sequentially. - Accumulate states, let states drop, or move between states as needed. - Define your pathways—for example: - Natural language input → model → output. - LLM output → algorithm for further handling. - (Up Next) The**final assessment**tests your ability to apply this in practice. --- ### Slide 192: Assessment Diffusion Pipeline - **Final Assessment—Implementing a Diffusion Pipeline** - **Your task**: - Implement a**diffusion pipeline**that: - Takes in an image. - **Summarizes**the image. - Generates prompts to create new images. - Generates those new images. - Use skills from**Notebooks 4 through 7**. - **Instructions are in the notebook**. - Expected outcome: - **Four images per input image**. - Input images are from the file system. - Each is described using a**visual language model**. - Descriptions generate four synthetic prompts for the diffusion pipeline. - The diffusion model generates four images from those prompts. - **Total**: Four input images and**16 output images**(or more if you like). - Follow the instructions to get started. - After finishing and receiving a**success message**: - **Exit the environment**. - Go to the bottom where you set up your environment. - Click the**"Assess Task"**checkbox. - You'll receive a success message if everything works correctly. - **Hope you enjoyed the course and the assignment**. ---