Deep Learning Lab
Guide
Module 6 · 6 lessons · 3.5 hours in class

Transformers and Attention Mechanisms

How does a Transformer process information and learn relationships between elements in a sequence?

You will learn

  • Complete Transformer architecture
  • Tokenisation & positions
  • Q/K/V
  • Self-attention
  • Transformer blocks
  • Causal training
  • Text generation
  • Vision & multimodal Transformers

Hands-on: Build a small character-level Transformer in PyTorch, train it on Tiny Shakespeare, and inspect its attention and generated text.

Learning route: Lessons 6.1–6.6, including architecture history, vision and multimodal applications, and weight tying, are covered in class.

Lessons

  1. Lesson 6.1: Understanding the Transformer Architecture

    What is a Transformer, as a whole, and how does information flow through it?

    Classroom · ~34 min self-study · 11 sections · 12 code cells
    Not started Start

  2. Lesson 6.2: Preparing Inputs for a Transformer

    How does raw text become the input tensors a Transformer learns from?

    Classroom · ~27 min self-study · 8 sections · 13 code cells
    Not started Start

  3. Lesson 6.3: Understanding Attention

    How does attention let each token gather information from the others?

    Classroom · ~46 min self-study · 15 sections · 17 code cells · 1 lab
    Not started Start

  4. Lesson 6.4: The Transformer Block

    What is inside a Transformer block, and why does each part matter?

    Classroom · ~33 min self-study · 10 sections · 12 code cells · 1 lab
    Not started Start

  5. Lesson 6.5: Building a Decoder-Only Transformer

    How do the components assemble into a complete language model, and how do we know it is built correctly?

    Classroom · ~24 min self-study · 8 sections · 10 code cells
    Not started Start

  6. Lesson 6.6: How Transformers Learn and Generate Text

    How does a Transformer learn from text, and how does it turn what it learned into new text?

    Classroom · ~47 min self-study · 13 sections · 18 code cells · 1 lab
    Not started Start

Module challenge

Earns the Module 6 badge

Module 6 challenge: Transformers and Attention Mechanisms

8 scenario questions that mix ideas from all 6 lessons. One question at a time, an explanation after each, and your score at the end. Pass mark 75%, so 6 of 8.

not attempted

Best after you have practised all 6 lessons.