CertBrain
Courses
Home
Blog
Sign in
← Back to decks
Sign in to save your progress, vote, and build your own decks.Sign in

Transformers, Foundation Models and Diffusion Models

Most generative AI today runs on a handful of model designs. Transformer-based large language models produce text, multimodal models handle more than one kind of input or output, and diffusion models produce images. This lesson explains how each works at a high level, what a foundation model is…

10 cards
Study this deck

Transformers, Foundation Models and Diffusion Models

Most generative AI today runs on a handful of model designs. Transformer-based large language models produce text, multimodal models handle more than one kind of input or output, and diffusion models produce images. This lesson explains how each works at a high level, what a foundation model is, and where prompt engineering fits.

Transformers and large language models

A transformer is a type of neural network designed for sequences such as sentences. Its key idea is self-attention: when processing each token, the model weighs every other token in the input to decide which ones matter for its meaning. In "The bank raised its rates", attention links "bank" to "rates", so the model reads it as a financial bank, not a riverbank.

Older sequence models, such as recurrent neural networks, read text one token at a time and tended to forget details from far back. Transformers process the whole sequence at once, which makes training much faster on modern hardware and handles long passages better. That speed is what made it practical to train the very large models behind today's large language models (LLMs).

An LLM generates text by repeatedly predicting the most suitable next token, adding it, and predicting again, until the response is complete.

Foundation models

A foundation model (FM) is a large model pre-trained on a broad, general dataset, so it can be adapted to many tasks instead of being built for just one. The same FM can summarize a report, answer questions and draft an email. Building one costs a great deal of data and computing power, so most organizations use an existing FM, through a service such as Amazon Bedrock, and adapt it with prompts, retrieval or fine-tuning rather than training their own.

Multimodal models

A multimodal model works with more than one type of data, such as text, images, audio or video. It might take a photo and a question as input and answer in text, or take a text description and produce an image. Amazon Nova, for example, includes models that understand text, images and video, as well as models that generate images and video.

Diffusion models

Diffusion models are the main technique behind AI image generation. During training, the model is shown images that have had random noise added step by step until nothing recognizable is left, and it learns to reverse that process, removing a little noise at a time. To generate a new image, it starts from pure noise and removes noise over many steps, guided by a text prompt that steers the result toward the description. Stable Diffusion is a well-known example. It also works in a compressed version of the image, called latent space, which makes generation far less demanding on computing power.

Prompt engineering

Prompt engineering is the practice of designing the input you give a model, its instructions, context and examples, so the model produces the output you want. It's usually the first and cheapest way to improve results, because it needs no retraining. Domain 3 covers the techniques in detail.

On the exam

Match the model type to the output. Text generation and reasoning point to transformer-based LLMs, and image generation from a description points to diffusion models. Input that mixes images and text points to a multimodal model. If a question asks why transformers replaced older sequence models, the answer is parallel processing through attention, not a bigger dataset.

Quiz me on this
Which feature of the transformer architecture lets it weigh how every word in a sentence relates to every other word?
  1. Convolution filters that slide across the text
  2. ✓ Self-attention
  3. Decision trees that split the text into branches
  4. Gradient boosting across many small models
A furniture retailer wants shoppers to type a description such as "a mid-century armchair in mustard velvet" and see a realistic, newly generated image of it. Which type of model is designed for this?
  1. A clustering model trained on the product catalog
  2. A regression model that predicts image dimensions
  3. A text-only transformer language model
  4. ✓ A diffusion model guided by the text prompt
An insurance company wants an assistant that accepts a photo of a damaged car together with the customer's written description, then drafts a summary of the claim. Which kind of model should it choose?
  1. ✓ A multimodal model that accepts both images and text
  2. A text-only model, with the photo's file name included in the prompt
  3. A diffusion model trained to generate car images
  4. A speech recognition model that transcribes the description
A startup wants to add a document summarizer to its app within a month. A consultant suggests training a new language model from scratch. Why is using an existing foundation model the better choice?
  1. Foundation models never make mistakes, unlike custom models
  2. Training from scratch is forbidden by AWS service terms
  3. ✓ An FM is already pre-trained and can be adapted at far lower cost
  4. Foundation models only work for summarization, so they fit this project exactly
A team is comparing a recurrent neural network with a transformer for a model that must read very long documents. Training time is a major constraint. Why does the transformer suit this better?
  1. ✓ It processes tokens in parallel and links distant words with attention
  2. It reads one token at a time, which keeps memory use low
  3. It doesn't need any training data to understand documents
  4. It stores each document in a database to look up, instead of learning from it
Transformer
A neural network design that uses self-attention to process a whole sequence at once. The basis of most LLMs.
Self-attention
The transformer mechanism that weighs how much each token relates to every other token in the input.
Multimodal model
A model that works with more than one data type, such as text, images, audio or video, as input or output.
Diffusion model
An image generator that learns to remove noise, then creates new images from random noise, guided by a prompt.
Prompt engineering
Optimizing the text you give a model (instructions, context, examples) to get the responses you want. It changes the input, not the model.