Deep Learning Lab
Guide
Module 2 · PyTorch: Building Neural Networks Lesson 2.1

Tensors and the Shape Contract

149 min · 129 code cells · 5 labs

In this lesson
Time
~149 min
Sections
8
Code cells
129
Labs
5
Figures
2
Start the lesson

Before you start

Every tensor carries a three-part contract: shape, dtype and device. Nearly every PyTorch error you will ever see breaks one of those three clauses. This lesson teaches you to read shapes like sentences, reshape without lying, broadcast on purpose, see a layer as a matrix multiply, and above all to fear the bugs that don't raise errors.

Goal

Stop treating tensors as "arrays that PyTorch happens to use" and start treating a tensor's shape, dtype and device as a three-part contract that every layer, every loss function and every batch either honours or violates.

Outcome

By the end of this lesson you can look at a shape error, name the cause before you read the traceback, and fix it without guessing. You will also have met the class of bug that produces no error at all — the silent broadcast — and know how to detect it.

You will be able to
  • create tensors, convert to and from NumPy, and explain why deep learning runs in float32;
  • read a shape as a sentence: which axis is the batch, which is the feature, which is the channel;
  • reshape safely, and explain when view fails and reshape does not;
  • state the broadcasting rules and use them deliberately for per-feature standardisation;
  • recognise the (N,) vs (N, 1) broadcast bug that inflates a loss without raising an error;
  • reproduce nn.Linear by hand with a matrix multiply and verify the result matches;
  • explain why classification labels must be long for CrossEntropyLoss;
  • detect and resolve device mismatches across cpu, cuda and mps;
  • debug five broken cells by predicting the failure before running them.

Estimated runtime: every cell in this lesson completes in under two seconds. Total execution time is well under one minute; the time you spend here should be thinking time, not waiting time.

Setup — imports, versions and device detection used by the notebook (2 cells)

Run this first. It pins the versions this lesson was written against and reports which compute device you have. Everything in Lesson 2.1 runs comfortably on CPU — the device detection matters because Section 6 demonstrates a device-mismatch error, and the code needs to know what hardware is actually present.

Setup 10 lines 
# Pinned for reproducibility. Uncomment to install in a fresh environment (Colab / Kaggle).# !pip install -q torch==2.14.0 numpy==2.4.4 matplotlib==3.10.8import sysimport numpy as npimport torchprint(f"Python      : {sys.version.split()[0]}")print(f"torch       : {torch.__version__}")print(f"numpy       : {np.__version__}")
Output 
Python      : 3.13.9
torch       : 2.14.0
numpy       : 2.3.5
Setup 31 lines 
def get_device() -> torch.device:    """Return the best available device: CUDA, then Apple MPS, then CPU.    MPS is used only if PyTorch was built with it *and* the hardware supports it.    Detect the device once and pass it everywhere, instead of hard-coding ``"cuda"``.    Returns:        A ``torch.device`` for the fastest backend on this machine.    """    if torch.cuda.is_available():        return torch.device("cuda")    if torch.backends.mps.is_available() and torch.backends.mps.is_built():        return torch.device("mps")    return torch.device("cpu")DEVICE = get_device()print(f"Selected device: {DEVICE}")if DEVICE.type == "cuda":    print(f"  GPU          : {torch.cuda.get_device_name(0)}")    print(f"  VRAM         : {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")elif DEVICE.type == "mps":    print("  Apple Silicon GPU via Metal Performance Shaders")else:    print("  Running on CPU. Every cell in this note is sized for that.")# One seed, set once, used everywhere in this lesson.SEED = 42torch.manual_seed(SEED)np.random.seed(SEED)
Output 
Selected device: mps
  Apple Silicon GPU via Metal Performance Shaders
1

The one object everything else is made of

In Lesson 1.3 you called torch.tensor(...) and moved on, because the goal then was to see a network train. Now the tensor itself is the subject.

A tensor is a block of numbers in memory plus some metadata describing how to interpret that block. The three pieces of metadata that will govern the rest of this course are:

Attribute Question it answers Typical failure when wrong
shape How is the block arranged into axes? RuntimeError: mat1 and mat2 shapes cannot be multiplied
dtype What kind of number is each element? RuntimeError: expected scalar type Long but found Float
device Which processor's memory holds it? RuntimeError: Expected all tensors to be on the same device

Almost every error you will hit for the next nine modules is one of those three. Learning to interrogate a tensor is therefore not housekeeping — it is the debugging skill.

The contract, and what breaks it

shape

How the numbers are arranged into axes. Broken → mat1 and mat2 shapes cannot be multiplied, or, worse, a silent broadcast.

dtype

What kind of number each element is. Broken → expected scalar type Long but found Float.

device

Whose memory holds it: CPU, CUDA or MPS. Broken → Expected all tensors to be on the same device.

Python 19 lines 
def describe(name: str, t: torch.Tensor) -> None:    """Print a tensor's three-part contract (shape, dtype, device) on one line.    Also prints ``ndim`` (the number of axes) and ``numel`` (the total number of    elements). Call it whenever you are unsure what a tensor is: it is faster than    reasoning about it.    Args:        name: Label printed in front of the description.        t: The tensor to describe.    """    print(f"{name:<12} shape={tuple(t.shape)!s:<16} dtype={str(t.dtype):<16} "          f"device={str(t.device):<6} ndim={t.ndim}  numel={t.numel()}")scalar = torch.tensor(3.14)vector = torch.tensor([1.0, 2.0, 3.0])matrix = torch.tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])cube   = torch.zeros(2, 3, 4)
What this code does

A one-line inspector for the three-part contract. Call it whenever you are unsure what a tensor is: it's faster than reasoning.

  1. 1

    !s:<16 formats the shape as a string, left-aligned in 16 characters, so rows line up.

  2. 2

    ndim = number of axes (rank); numel = total number of elements (the product of the shape).

Inspect a value 1 line 
scalar
Output 
tensor(3.1400)
Inspect a value 1 line 
vector
Output 
tensor([1., 2., 3.])
Inspect a value 1 line 
matrix
Output 
tensor([[1., 2.],
        [3., 4.],
        [5., 6.]])
Inspect a value 1 line 
cube
Output 
tensor([[[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]],

        [[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]]])
Python 4 lines 
describe("scalar", scalar)describe("vector", vector)describe("matrix", matrix)describe("cube", cube)
Output 
scalar       shape=()               dtype=torch.float32    device=cpu    ndim=0  numel=1
vector       shape=(3,)             dtype=torch.float32    device=cpu    ndim=1  numel=3
matrix       shape=(3, 2)           dtype=torch.float32    device=cpu    ndim=2  numel=6
cube         shape=(2, 3, 4)        dtype=torch.float32    device=cpu    ndim=3  numel=24

Notice that the scalar has shape=() — an empty tuple, zero axes. That is not a quirk. Your loss value is a zero-dimensional tensor, and loss.backward() only works on one. When you eventually see grad can be implicitly created only for scalar outputs, this is the reason: something you expected to be a single number still had axes attached.

The construction helpers you will actually reach for:

Inspect a value 1 line 
np.zeros([2,3])
Output 
array([[0., 0., 0.],
       [0., 0., 0.]])
Inspect a value 1 line 
torch.zeros(2, 3)
Output 
tensor([[0., 0., 0.],
        [0., 0., 0.]])
Python 12 lines 
print("zeros(2,3):\n", torch.zeros(2, 3))print("\nones(2,3):\n", torch.ones(2, 3))print("\narange(6):", torch.arange(6))print("\nlinspace(0,1,5):", torch.linspace(0, 1, 5))print("\nrandn(2,3)  — standard normal, the default for weight init experiments:\n",      torch.randn(2, 3))print("\nrand(2,3)   — uniform on [0,1):\n", torch.rand(2, 3))print("\neye(3)      — identity, useful for sanity-checking matmuls:\n", torch.eye(3))# Shape-matching constructors: 'give me zeros shaped like this other thing'.template = torch.randn(3, 4)print("\nzeros_like(template).shape:", tuple(torch.zeros_like(template).shape))
Output 
zeros(2,3):
 tensor([[0., 0., 0.],
        [0., 0., 0.]])

ones(2,3):
 tensor([[1., 1., 1.],
        [1., 1., 1.]])

arange(6): tensor([0, 1, 2, 3, 4, 5])

linspace(0,1,5): tensor([0.0000, 0.2500, 0.5000, 0.7500, 1.0000])

randn(2,3)  — standard normal, the default for weight init experiments:
 tensor([[ 0.3367,  0.1288,  0.2345],
        [ 0.2303, -1.1229, -0.1863]])

rand(2,3)   — uniform on [0,1):
 tensor([[0.8694, 0.5677, 0.7411],
        [0.4294, 0.8854, 0.5739]])

eye(3)      — identity, useful for sanity-checking matmuls:
 tensor([[1., 0., 0.],
        [0., 1., 0.],
        [0., 0., 1.]])

zeros_like(template).shape: (3, 4)

Crossing the NumPy border

Your data will arrive from pandas or scikit-learn as NumPy arrays. The bridge runs both ways, and one direction has a sharp edge.

Python 3 lines 
arr = np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32)arr
Output 
array([[1., 2.],
       [3., 4.]], dtype=float32)
Python 3 lines 
# from_numpy SHARES memory with the array. Cheap, but mutations propagate.shared = torch.from_numpy(arr)shared
Output 
tensor([[1., 2.],
        [3., 4.]])
Python 2 lines 
arr[0, 0] = 99.0arr
Output 
array([[99.,  2.],
       [ 3.,  4.]], dtype=float32)
Inspect a value 1 line 
shared
Output 
tensor([[99.,  2.],
        [ 3.,  4.]])
Python 1 line 
print("after mutating the numpy array, the tensor sees it:\n", shared)
Output 
after mutating the numpy array, the tensor sees it:
 tensor([[99.,  2.],
        [ 3.,  4.]])
Python 3 lines 
# torch.tensor(...) COPIES. Safer default when you are unsure.arr2 = np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32)arr2
Output 
array([[1., 2.],
       [3., 4.]], dtype=float32)
Python 2 lines 
copied = torch.tensor(arr2)copied
Output 
tensor([[1., 2.],
        [3., 4.]])
Python 2 lines 
arr2[0, 0] = 99.0arr2
Output 
array([[99.,  2.],
       [ 3.,  4.]], dtype=float32)
Inspect a value 1 line 
copied
Output 
tensor([[1., 2.],
        [3., 4.]])
Python 1 line 
print("\nafter mutating, the copy is unaffected:\n", copied)
Output 
after mutating, the copy is unaffected:
 tensor([[1., 2.],
        [3., 4.]])
Python 4 lines 
# Going back to numpy. .numpy() also shares memory, and fails on a tensor# that requires grad — you must .detach() first.t = torch.randn(2, 2, requires_grad=True)t
Output 
tensor([[-0.3514, -0.7906],
        [-0.0915,  0.2352]], requires_grad=True)
Inspect a value 1 line 
t.numpy()
Error raised
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[23], line 1
----> 1 t.numpy()

RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
Inspect a value 1 line 
t.detach().numpy()
Output 
array([[-0.35144043, -0.7906431 ],
       [-0.09150922,  0.23517996]], dtype=float32)
Python 5 lines 
try:    t.numpy()except RuntimeError as e:    print("\nExpected error:", str(e)[:80])print("Correct:", t.detach().numpy().shape)
Output 
Expected error: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() ins
Correct: (2, 2)

Why float32 and not float64

NumPy defaults to float64. PyTorch defaults to float32, and this mismatch is the single most common surprise when converting a pandas DataFrame.

Python 2 lines 
np_default = np.array([1.0, 2.0, 3.0])          # float64torch_default = torch.tensor([1.0, 2.0, 3.0])   # float32
Inspect a value 1 line 
np_default
Output 
array([1., 2., 3.])
Inspect a value 1 line 
torch_default
Output 
tensor([1., 2., 3.])
Python 2 lines 
print("numpy default dtype:", np_default.dtype)print("torch default dtype:", torch_default.dtype)
Output 
numpy default dtype: float64
torch default dtype: torch.float32
Python 2 lines 
converted = torch.from_numpy(np_default)print("\nfrom_numpy on a float64 array gives:", converted.dtype)
Output 
from_numpy on a float64 array gives: torch.float64
Python 1 line 
converted = torch.tensor(np_default, dtype=torch.float32)
Inspect a value 1 line 
converted
Output 
tensor([1., 2., 3.])
Python 3 lines 
# A float32 layer refuses a float64 input.import torch.nn as nnlayer = nn.Linear(3, 2)
Inspect a value 1 line 
layer
Output 
Linear(in_features=3, out_features=2, bias=True)
Inspect a value 1 line 
layer(converted)
Output 
tensor([-2.1636,  1.5137], grad_fn=<ViewBackward0>)
Python 6 lines 
try:    layer(converted)except RuntimeError as e:    print("\nExpected error:", str(e)[:110])print("\nFixed:", layer(converted.float()).shape)
Output 
Fixed: torch.Size([2])

Deep learning runs in float32 because halving the bits per number halves memory traffic and roughly doubles arithmetic throughput, and the extra precision of float64 buys nothing: gradient descent is a noisy, approximate procedure whose errors dwarf the difference between the seventh and sixteenth significant digit. Module 4 pushes this further, to float16 and bfloat16.

The practical rule, which will save you an hour at some point this week:

Cast to float32 at the boundary where NumPy becomes PyTorch, not later. X.astype(np.float32) on the DataFrame side, once, and the problem never recurs.

Check your understandingQuestions to sit with
  1. torch.tensor(3.14) has ndim=0 and torch.tensor([3.14]) has ndim=1. Both hold one number. Which one can you call .backward() on, and why does the distinction exist?
  2. from_numpy shares memory and torch.tensor copies. Name a situation where the sharing is a feature, and one where it is a bug waiting to happen.
Checkpoint Before you move on from §1
1

loss.backward() complains grad can be implicitly created only for scalar outputs. What is wrong?

2

You build a tensor with torch.from_numpy(arr) and then change arr[0, 0]. What happens to the tensor?

Explain it back · two sentences, in your own words

Where should you cast to float32 when data goes from pandas/NumPy into PyTorch, and why there?

2

Reading a shape like a sentence

A shape is not a list of numbers. It is a sentence describing how your data is laid out, and the convention is stable enough across the whole field that you can read an unfamiliar codebase from its shapes alone.

The universal rule: axis 0 is the batch. Whatever else changes, the first number is how many independent examples are stacked together. Every layer in PyTorch assumes this without asking.

Data type Shape convention Read it as
Tabular (N, F) N rows, F features per row
Image (N, C, H, W) N images, C channels, H high, W wide
Token IDs (N, L) N sequences, L tokens each
Token embeddings (N, L, E) N sequences, L tokens, E numbers per token

Note that PyTorch puts channels before spatial dimensions — (N, C, H, W), not the (N, H, W, C) you may have seen in TensorFlow or in a raw PIL image. This single difference causes a large fraction of first-week image bugs. Module 10 revisits it when we cross frameworks.

Four panels label tabular (N,F), images (N,C,H,W), token IDs (N,L), and token embeddings (N,L,E), explaining what each axis holds.
Figure 2.1.1. Examples of the four tensor layouts used in this lesson. Drawn grids illustrate axes; token numbers are examples rather than a real vocabulary encoding.
Interactive

What each rank means

Step through ranks 0–4. Axis 0 is always the batch; the other axes are features, tokens, channels or pixels depending on the data type.

Loading interactive…
Python 6 lines 
N, F = 8, 5                     # 8 rows, 5 tabular featurestabular = torch.randn(N, F)images = torch.randn(4, 3, 32, 32)     # 4 RGB images, 32x32tokens = torch.randint(0, 1000, (2, 10))       # 2 sequences of 10 token idsembeds = torch.randn(2, 10, 64)                # same, each token -> 64 numbers
Python 1 line 
print(images[0][0])
Output 
tensor([[-0.5992,  0.4771,  0.7262,  ...,  0.6368,  0.2176, -0.0467],
        [ 1.6192,  1.4506,  0.2695,  ...,  2.0265,  0.9812, -0.6401],
        [-0.8057, -0.2076, -0.9319,  ...,  0.4598,  0.2618, -0.7599],
        ...,
        [ 1.1862, -1.2203,  0.2910,  ..., -2.5380, -0.0334, -1.7017],
        [ 0.3712,  1.5118, -0.8915,  ..., -0.0406,  0.6467, -2.0813],
        [-0.9167, -0.7794, -0.4175,  ...,  3.4456,  0.3673, -0.3919]])
Python 10 lines 
describe("tabular", tabular)describe("images", images)describe("tokens", tokens)describe("embeds", embeds)print()print("How many examples in the image batch?  ->", images.shape[0])print("How many channels per image?           ->", images.shape[1])print("Pixels per image (H*W)                 ->", images.shape[2] * images.shape[3])print("Numbers in the whole image batch       ->", images.numel())
Output 
tabular      shape=(8, 5)           dtype=torch.float32    device=cpu    ndim=2  numel=40
images       shape=(4, 3, 32, 32)   dtype=torch.float32    device=cpu    ndim=4  numel=12288
tokens       shape=(2, 10)          dtype=torch.int64      device=cpu    ndim=2  numel=20
embeds       shape=(2, 10, 64)      dtype=torch.float32    device=cpu    ndim=3  numel=1280

How many examples in the image batch?  -> 4
How many channels per image?           -> 3
Pixels per image (H*W)                 -> 1024
Numbers in the whole image batch       -> 12288

Indexing preserves or drops axes, and it matters

The difference between an index and a slice is the difference between losing an axis and keeping it. This is the origin of a very large share of (N,) vs (N, 1) confusion.

Python 2 lines 
batch = torch.randn(8, 5)batch
Output 
tensor([[-0.2514,  1.6124,  0.0593, -0.2723, -0.1372],
        [ 0.7022,  1.5885,  1.0206, -1.0839,  1.0406],
        [-1.2676,  1.3164,  0.0458, -0.1287, -2.0694],
        [ 0.4159,  0.6327,  1.3299, -0.2185, -1.3713],
        [-0.8273,  0.9662, -0.5898, -0.8523, -0.1236],
        [-1.2720, -1.0702,  0.0196,  0.4001, -0.2221],
        [ 0.1648, -0.2799,  0.7053,  0.4979,  1.1845],
        [ 0.4461, -0.2101,  0.4610, -0.4705,  2.3401]])
Inspect a value 1 line 
describe("batch", batch)
Output 
batch        shape=(8, 5)           dtype=torch.float32    device=cpu    ndim=2  numel=40
Inspect a value 1 line 
batch[0]
Output 
tensor([-0.2514,  1.6124,  0.0593, -0.2723, -0.1372])
Python 1 line 
describe("batch[0]", batch[0])        # integer index -> axis 0 disappears
Output 
batch[0]     shape=(5,)             dtype=torch.float32    device=cpu    ndim=1  numel=5
Inspect a value 1 line 
batch[0:1]
Output 
tensor([[-0.2514,  1.6124,  0.0593, -0.2723, -0.1372]])
Python 1 line 
describe("batch[0:1]", batch[0:1])    # slice        -> axis 0 survives with length 1
Output 
batch[0:1]   shape=(1, 5)           dtype=torch.float32    device=cpu    ndim=2  numel=5
Inspect a value 1 line 
batch[:, 2]
Output 
tensor([ 0.0593,  1.0206,  0.0458,  1.3299, -0.5898,  0.0196,  0.7053,  0.4610])
Python 1 line 
describe("batch[:, 2]", batch[:, 2])  # one feature, as a flat vector
Output 
batch[:, 2]  shape=(8,)             dtype=torch.float32    device=cpu    ndim=1  numel=8
Inspect a value 1 line 
batch[:, 2:3]
Output 
tensor([[ 0.0593],
        [ 1.0206],
        [ 0.0458],
        [ 1.3299],
        [-0.5898],
        [ 0.0196],
        [ 0.7053],
        [ 0.4610]])
Python 1 line 
describe("batch[:, 2:3]", batch[:, 2:3])  # one feature, as a column
Output 
batch[:, 2:3] shape=(8, 1)           dtype=torch.float32    device=cpu    ndim=2  numel=8
Python 4 lines 
print()print("A single example fed to a model must still look like a batch of one.")print("batch[0].shape    =", tuple(batch[0].shape), " <- a model will reject or misread this")print("batch[0:1].shape  =", tuple(batch[0:1].shape), " <- correct: batch of 1")
Output 
A single example fed to a model must still look like a batch of one.
batch[0].shape    = (5,)  <- a model will reject or misread this
batch[0:1].shape  = (1, 5)  <- correct: batch of 1
Inspect a value 1 line 
batch
Output 
tensor([[-0.2514,  1.6124,  0.0593, -0.2723, -0.1372],
        [ 0.7022,  1.5885,  1.0206, -1.0839,  1.0406],
        [-1.2676,  1.3164,  0.0458, -0.1287, -2.0694],
        [ 0.4159,  0.6327,  1.3299, -0.2185, -1.3713],
        [-0.8273,  0.9662, -0.5898, -0.8523, -0.1236],
        [-1.2720, -1.0702,  0.0196,  0.4001, -0.2221],
        [ 0.1648, -0.2799,  0.7053,  0.4979,  1.1845],
        [ 0.4461, -0.2101,  0.4610, -0.4705,  2.3401]])
Inspect a value 1 line 
batch.shape
Output 
torch.Size([8, 5])
Inspect a value 1 line 
batch[0]
Output 
tensor([-0.2514,  1.6124,  0.0593, -0.2723, -0.1372])
Inspect a value 1 line 
batch[0].shape
Output 
torch.Size([5])
Inspect a value 1 line 
batch[0:1]
Output 
tensor([[-0.2514,  1.6124,  0.0593, -0.2723, -0.1372]])
Inspect a value 1 line 
batch[0:1].shape
Output 
torch.Size([1, 5])

When you deploy a model and pass it one customer record, this is the bug you will hit. The model was trained on (N, F); you hand it (F,); nn.Linear happens to accept it and returns something plausible-looking. Section 4 shows what happens next.

Index or slice? Watch the axis

batch[0] → (5,)

An integer index removes axis 0. Feed this to a model and there is no batch axis any more.

batch[0:1] → (1, 5)

A slice keeps the axis (length 1): a correct batch of one. Or use x.unsqueeze(0).

Check your understandingQuestions to sit with
  1. Given images with shape (4, 3, 32, 32), what does images[1, 0] return, and what shape is it? Predict before you run it.
  2. Your model's output has shape (32, 1) and your labels have shape (32,). Which one should change, and does the answer depend on the loss function you chose?
Checkpoint Before you move on from §2
1

batch has shape (8, 5). What is the shape of batch[0], and of batch[0:1]?

2

An image batch loaded from PIL/matplotlib has shape (N, H, W, C). What does PyTorch expect?

Explain it back · two sentences, in your own words

Read the shape (32, 10, 64) aloud as a sentence for a batch of token embeddings.

3

Reshaping without lying to yourself

Reshaping does not move data. It rewrites the metadata that says how to walk through the block of memory. That is why it is nearly free — and why it occasionally refuses.

The number of elements must be conserved: a (4, 3, 32, 32) tensor has 12,288 elements, so it can become (4, 3072) or (12, 1024) or (12288,), but never (4, 3000).

Interactive

Reshape: same numbers, new shape

Loading interactive…
Python 2 lines 
x = torch.arange(24)x
Output 
tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23])
Inspect a value 1 line 
describe("x", x)
Output 
x            shape=(24,)            dtype=torch.int64      device=cpu    ndim=1  numel=24
Inspect a value 1 line 
x.view(4, 6)
Output 
tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11],
        [12, 13, 14, 15, 16, 17],
        [18, 19, 20, 21, 22, 23]])
Python 4 lines 
describe("x.view(4, 6)", x.view(4, 6))describe("x.view(2, 3, 4)", x.view(2, 3, 4))describe("x.view(4, -1)", x.view(4, -1))     # -1 means 'infer this axis'describe("x.view(-1, 8)", x.view(-1, 8))
Output 
x.view(4, 6) shape=(4, 6)           dtype=torch.int64      device=cpu    ndim=2  numel=24
x.view(2, 3, 4) shape=(2, 3, 4)        dtype=torch.int64      device=cpu    ndim=3  numel=24
x.view(4, -1) shape=(4, 6)           dtype=torch.int64      device=cpu    ndim=2  numel=24
x.view(-1, 8) shape=(3, 8)           dtype=torch.int64      device=cpu    ndim=2  numel=24
Inspect a value 1 line 
x.view(5, 5)
Error raised
RuntimeError: shape '[5, 5]' is invalid for input of size 24
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[62], line 1
----> 1 x.view(5, 5)

RuntimeError: shape '[5, 5]' is invalid for input of size 24
Python 5 lines 
# Conservation is enforced.try:    x.view(5, 5)except RuntimeError as e:    print("\nExpected error:", str(e)[:70])
Output 
Expected error: shape '[5, 5]' is invalid for input of size 24

Where view fails and reshape succeeds

A tensor is contiguous when its elements sit in memory in the same order you would read them by walking the axes left to right. Operations like .t() and .permute() do not move data — they just relabel the axes — so the result is a view onto memory that is no longer in reading order.

view requires contiguity, because it only rewrites strides. reshape will silently copy if it has to. That copy is the entire difference, and it is why reshape is the safer default.

Interactive

Why view fails after a transpose

A tensor is a flat strip of memory plus strides that say how far to jump for each axis. Press Walk in reading order for each of the three versions and watch which memory slots are visited.

Loading interactive…
Inspect a value 1 line 
torch.arange(12)
Output 
tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])
Python 2 lines 
x = torch.arange(12).reshape(3, 4)x
Output 
tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]])
Inspect a value 1 line 
x.view(12)
Output 
tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])
Python 1 line 
xt = x.t()      # transpose: a view, no data moved
Inspect a value 1 line 
xt
Output 
tensor([[ 0,  4,  8],
        [ 1,  5,  9],
        [ 2,  6, 10],
        [ 3,  7, 11]])
Python 2 lines 
print("x contiguous? ", x.is_contiguous())print("x.t() contiguous? ", xt.is_contiguous())
Output 
x contiguous?  True
x.t() contiguous?  False
Inspect a value 1 line 
xt.view(12)
Error raised
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[70], line 1
----> 1 xt.view(12)

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
Inspect a value 1 line 
xt.reshape(12)
Output 
tensor([ 0,  4,  8,  1,  5,  9,  2,  6, 10,  3,  7, 11])
Python 5 lines 
try:    xt.view(12)except RuntimeError as e:    print("\nview on a non-contiguous tensor fails:")    print(" ", str(e)[:95], "...")
Output 
view on a non-contiguous tensor fails:
  view size is not compatible with input tensor's size and stride (at least one dimension spans a ...
Python 2 lines 
print("\nreshape handles it by copying:", xt.reshape(12))print("or make it explicit:            ", xt.contiguous().view(12))
Output 
reshape handles it by copying: tensor([ 0,  4,  8,  1,  5,  9,  2,  6, 10,  3,  7, 11])
or make it explicit:             tensor([ 0,  4,  8,  1,  5,  9,  2,  6, 10,  3,  7, 11])

Adding and removing length-1 axes

unsqueeze inserts an axis of length 1 at a position; squeeze removes axes of length 1. You will use these constantly to reconcile a model output with a loss function's expectations.

Python 2 lines 
v = torch.tensor([1.0, 2.0, 3.0])v
Output 
tensor([1., 2., 3.])
Inspect a value 1 line 
describe("v", v)
Output 
v            shape=(3,)             dtype=torch.float32    device=cpu    ndim=1  numel=3
Inspect a value 1 line 
v.unsqueeze(0)
Output 
tensor([[1., 2., 3.]])
Python 1 line 
describe("v.unsqueeze(0)", v.unsqueeze(0))    # (3,) -> (1, 3): a batch of one row
Output 
v.unsqueeze(0) shape=(1, 3)           dtype=torch.float32    device=cpu    ndim=2  numel=3
Inspect a value 1 line 
v.unsqueeze(1)
Output 
tensor([[1.],
        [2.],
        [3.]])
Python 1 line 
describe("v.unsqueeze(1)", v.unsqueeze(1))    # (3,) -> (3, 1): a column
Output 
v.unsqueeze(1) shape=(3, 1)           dtype=torch.float32    device=cpu    ndim=2  numel=3
Python 2 lines 
col = v.unsqueeze(1)describe("col.squeeze(1)", col.squeeze(1))    # back to (3,)
Output 
col.squeeze(1) shape=(3,)             dtype=torch.float32    device=cpu    ndim=1  numel=3
Python 7 lines 
# squeeze() with no argument removes EVERY length-1 axis. Convenient and dangerous:# if your batch size happens to be 1, it silently removes the batch axis too.risky = torch.randn(1, 1, 5)print()describe("risky", risky)describe("risky.squeeze()", risky.squeeze())        # lost both axesdescribe("risky.squeeze(1)", risky.squeeze(1))      # only the one you meant
Output 
risky        shape=(1, 1, 5)        dtype=torch.float32    device=cpu    ndim=3  numel=5
risky.squeeze() shape=(5,)             dtype=torch.float32    device=cpu    ndim=1  numel=5
risky.squeeze(1) shape=(1, 5)           dtype=torch.float32    device=cpu    ndim=2  numel=5

That last pair is a real production bug. A batch of one — the last incomplete batch of an epoch, or a single request in a deployed service — turns squeeze() into something that removes the batch axis. Always name the axis: squeeze(1), never bare squeeze().

Reordering axes: permute and transpose

Python 3 lines 
img = torch.randn(4, 32, 32, 3)     # what PIL / matplotlib hand you: (N, H, W, C)describe("as loaded (N,H,W,C)", img)img
Output 
as loaded (N,H,W,C) shape=(4, 32, 32, 3)   dtype=torch.float32    device=cpu    ndim=4  numel=12288
Python 3 lines 
torch_layout = img.permute(0, 3, 1, 2)      # -> (N, C, H, W)describe("permuted (N,C,H,W)", torch_layout)torch_layout
Output 
permuted (N,C,H,W) shape=(4, 3, 32, 32)   dtype=torch.float32    device=cpu    ndim=4  numel=12288
Python 1 line 
print("contiguous after permute?", torch_layout.is_contiguous())
Output 
contiguous after permute? False
Python 4 lines 
# transpose swaps exactly two axes; permute reorders all of them.seq = torch.randn(2, 10, 64)        # (N, L, E)describe("seq", seq)seq
Output 
seq          shape=(2, 10, 64)      dtype=torch.float32    device=cpu    ndim=3  numel=1280
Output 
tensor([[[-2.5744e-01,  8.3295e-04,  2.0257e-01,  ..., -2.6973e-01,
          -4.1540e-01,  4.1612e-01],
         [ 1.6719e+00,  9.3445e-01,  9.3036e-01,  ..., -3.7335e-01,
           2.9129e-01, -1.4553e+00],
         [ 6.7020e-01,  2.0534e-01, -4.4031e-01,  ...,  9.1655e-01,
          -1.0710e+00, -9.0602e-01],
         ...,
         [ 1.4881e-01, -1.0629e+00, -3.5009e-01,  ...,  1.4352e+00,
          -1.1908e+00,  7.3657e-01],
         [ 2.5693e+00,  3.2579e-01, -1.2760e-01,  ..., -2.6638e-01,
          -9.7265e-01,  1.1840e+00],
         [ 4.5051e-01, -2.4164e+00, -1.3880e-01,  ..., -1.5625e+00,
           1.6677e+00,  2.3402e+00]],

        [[ 9.5596e-01,  3.6468e-01,  4.1431e-01,  ..., -3.4242e-01,
           2.0233e-01, -1.7589e+00],
         [ 5.5581e-01, -7.1602e-01, -1.5441e+00,  ..., -6.4152e-01,
          -5.7506e-02,  1.3140e+00],
         [-1.1513e-01,  1.4627e-01,  1.8633e+00,  ...,  7.4607e-01,
           1.4400e+00,  1.6266e+00],
         ...,
         [-1.3137e+00,  7.5883e-02,  1.5082e+00,  ..., -7.0180e-01,
          -1.7468e+00,  2.0861e+00],
         [-5.2964e-01,  1.1322e-02, -7.6960e-01,  ..., -1.3436e+00,
           3.3995e-01,  1.3885e-01],
         [ 5.2937e-02,  1.0773e-01,  1.2909e+00,  ..., -3.0001e-02,
           9.7417e-01, -1.1895e+00]]])
Python 1 line 
describe("seq.transpose(1, 2)", seq.transpose(1, 2))   # (N, E, L)
Output 
seq.transpose(1, 2) shape=(2, 64, 10)      dtype=torch.float32    device=cpu    ndim=3  numel=1280

permute names the new order of all axes; transpose swaps a specific pair. Neither moves data, so both leave you non-contiguous — call .contiguous() if a later view complains.

Check your understandingQuestions to sit with
  1. x.view(4, -1) and x.reshape(4, -1) return the same values here. Under what circumstance do they differ in cost, and how would you detect that in your own code?
  2. You load a batch of images and get shape (16, 224, 224, 3). Write the one-line fix. Now explain what your model would have computed if you had passed it unfixed and PyTorch had not raised an error.
Checkpoint Before you move on from §3
1

Why can x.t().view(-1) fail while x.t().reshape(-1) works?

2

Why is bare squeeze() a production bug waiting to happen?

Explain it back · two sentences, in your own words

A (4, 3, 32, 32) tensor: which of these reshapes is impossible, (4, 3072), (12, 1024) or (4, 3000), and what rule decides it?

4

Broadcasting, and the bug it hides

Broadcasting lets PyTorch combine tensors of different shapes by virtually stretching length-1 axes. It is what makes per-feature standardisation a one-liner. It is also the mechanism behind the worst category of deep learning bug: the one that runs.

The rule. Align shapes from the right. For each axis pair, they are compatible if they are equal, or if one of them is 1. A missing axis on the left counts as 1.

   (8, 5)        (8, 5)        (8, 5)         (8, 5)
   (   5)   OK   (8, 1)   OK   (1, 5)   OK    (   8)   FAILS
-> (8, 5)     -> (8, 5)     -> (8, 5)         5 vs 8, neither is 1
Four shape examples show (8,5)+(5,) yields (8,5); (8,5)+(8,1) yields (8,5); (8,5)+(8,) is incompatible; and (6,)+(6,1) yields an unintended (6,6) grid.
Figure 2.1.2. Three dimension checks and a silent broadcasting trap. A (6,) vector and a (6,1) column produce a (6,6) grid.
Interactive

Broadcasting rules, visualised

Pick shapes for A and B. Dashed ghost cells are the virtual copies broadcasting creates (no memory is copied). Try (4,) with (4,1).

Loading interactive…
Python 2 lines 
batch = torch.arange(40, dtype=torch.float32).reshape(8, 5)batch
Output 
tensor([[ 0.,  1.,  2.,  3.,  4.],
        [ 5.,  6.,  7.,  8.,  9.],
        [10., 11., 12., 13., 14.],
        [15., 16., 17., 18., 19.],
        [20., 21., 22., 23., 24.],
        [25., 26., 27., 28., 29.],
        [30., 31., 32., 33., 34.],
        [35., 36., 37., 38., 39.]])
Python 2 lines 
per_feature = torch.tensor([0., 10., 100., 1000., 10000.])   # (5,)per_feature
Output 
tensor([    0.,    10.,   100.,  1000., 10000.])
Python 1 line 
print("(8,5) + (5,)   ->", tuple((batch + per_feature).shape), " broadcast across rows")
Output 
(8,5) + (5,)   -> (8, 5)  broadcast across rows
Inspect a value 1 line 
batch + per_feature
Output 
tensor([[0.0000e+00, 1.1000e+01, 1.0200e+02, 1.0030e+03, 1.0004e+04],
        [5.0000e+00, 1.6000e+01, 1.0700e+02, 1.0080e+03, 1.0009e+04],
        [1.0000e+01, 2.1000e+01, 1.1200e+02, 1.0130e+03, 1.0014e+04],
        [1.5000e+01, 2.6000e+01, 1.1700e+02, 1.0180e+03, 1.0019e+04],
        [2.0000e+01, 3.1000e+01, 1.2200e+02, 1.0230e+03, 1.0024e+04],
        [2.5000e+01, 3.6000e+01, 1.2700e+02, 1.0280e+03, 1.0029e+04],
        [3.0000e+01, 4.1000e+01, 1.3200e+02, 1.0330e+03, 1.0034e+04],
        [3.5000e+01, 4.6000e+01, 1.3700e+02, 1.0380e+03, 1.0039e+04]])
Python 9 lines 
per_row = torch.arange(8, dtype=torch.float32).unsqueeze(1)  # (8,1)print("(8,5) + (8,1)  ->", tuple((batch + per_row).shape), " broadcast across columns")bad = torch.arange(8, dtype=torch.float32)                   # (8,)try:    batch + badexcept RuntimeError as e:    print("\n(8,5) + (8,)   -> error, which is exactly what you want:")    print("  ", str(e)[:100])
Output 
(8,5) + (8,1)  -> (8, 5)  broadcast across columns

(8,5) + (8,)   -> error, which is exactly what you want:
   The size of tensor a (5) must match the size of tensor b (8) at non-singleton dimension 1

Using it on purpose: per-feature standardisation

This is the operation you will perform on every tabular dataset for the rest of the course, and broadcasting is what makes it two lines instead of a loop.

Python 13 lines 
X = torch.randn(200, 4) * torch.tensor([1.0, 50.0, 0.01, 300.0]) + torch.tensor([0., 100., 5., -20.])print("Before standardisation")print("  mean per feature:", X.mean(dim=0).numpy().round(3))print("  std  per feature:", X.std(dim=0).numpy().round(3))mu = X.mean(dim=0)            # (4,)  -- collapse the batch axissd = X.std(dim=0)             # (4,)X_std = (X - mu) / sd         # (200,4) - (4,) broadcasts across all 200 rowsprint("\nAfter standardisation")print("  mean per feature:", X_std.mean(dim=0).numpy().round(6))print("  std  per feature:", X_std.std(dim=0).numpy().round(6))
Standardisation with broadcasting

Compute one mean and one standard deviation per feature, then use broadcasting to apply them to every row at once.

  1. 1

    dim=0 collapses the batch axis, leaving one number per feature: shape (4,).

  2. 2

    (200, 4) - (4,): the per-feature vector is stretched across all 200 rows.

Output 
Before standardisation
  mean per feature: [  0.178 104.835   5.001 -30.663]
  std  per feature: [1.05200e+00 5.04490e+01 1.00000e-02 2.74218e+02]

After standardisation
  mean per feature: [ 0.0e+00 -0.0e+00  2.3e-05 -0.0e+00]
  std  per feature: [1. 1. 1. 1.]

Note dim=0. Reducing along axis 0 collapses the batch and leaves one statistic per feature — which is what standardisation means. Reducing along dim=1 would collapse the features and give one statistic per row, which is meaningless here. Getting this backwards produces a tensor of plausible shape and no error.

The silent broadcast

Here is the bug. Your model outputs (N,) and your labels are (N, 1) — or the reverse. Neither shape is wrong on its own. Together they broadcast to (N, N), and the loss function dutifully averages over an N x N matrix of every prediction compared against every label.

Python 19 lines 
import torch.nn.functional as Fnimport warningstargets = torch.tensor([[1.0], [3.0], [5.0], [7.0], [9.0], [11.0]])   # (6, 1)preds   = targets.squeeze(1) + 0.05 * torch.randn(6)                  # (6,)  -- nearly perfectdescribe("preds", preds)describe("targets", targets)correct = Fn.mse_loss(preds, targets.squeeze(1))with warnings.catch_warnings():    warnings.simplefilter("ignore")    broadcast = Fn.mse_loss(preds, targets)print(f"\nShape of (preds - targets): {tuple((preds - targets).shape)}   <- the tell")print(f"Correct loss   : {correct.item():.6f}")print(f"Broadcast loss : {broadcast.item():.6f}")print(f"Ratio          : {broadcast.item() / correct.item():.0f}x too large")
Output 
preds        shape=(6,)             dtype=torch.float32    device=cpu    ndim=1  numel=6
targets      shape=(6, 1)           dtype=torch.float32    device=cpu    ndim=2  numel=6

Shape of (preds - targets): (6, 6)   <- the tell
Correct loss   : 0.001768
Broadcast loss : 23.243324
Ratio          : 13147x too large

The predictions sit within about 0.05 of the targets. The correct loss is roughly 0.0018. The broadcast loss is over 23 — four orders of magnitude larger — and your training run will look like a model that refuses to learn, when in fact it learned fine and you measured it wrong.

PyTorch does emit a UserWarning here, which we suppressed to show the number. In a real run that warning scrolls past in the first epoch and is never seen again. Do not rely on it.

And do not assume the guard is uniform. As of torch 2.14 the binary cross-entropy losses were hardened to raise on a shape mismatch, while the regression losses still broadcast with only a warning. Run the cell below to see where the line currently sits on your version — the answer is a property of the release you installed, not a law.

The defence is an assertion, written once, at the top of your training loop:

Interactive

The silent broadcast, made visible

Each square compares one prediction with one target. The green diagonal is what you meant to measure; everything else is noise the broadcast added. Slide the model quality from nearly perfect to mediocre and compare the two losses.

Loading interactive…
Python 29 lines 
def check_loss_shapes(preds: torch.Tensor, targets: torch.Tensor) -> None:    """Fail loudly if predictions and targets do not have identical shapes.    Mismatched shapes such as ``(N, 1)`` and ``(N,)`` silently broadcast to    ``(N, N)`` in some loss functions. Calling this just before computing the loss    turns that silent bug into an immediate, readable error on the first step.    Args:        preds: Model output tensor.        targets: Target tensor that will be compared with ``preds``.    Raises:        ValueError: If the shapes differ. The message states both shapes and what            they would broadcast to.    """    if preds.shape != targets.shape:        raise ValueError(            f"Shape mismatch: preds {tuple(preds.shape)} vs targets {tuple(targets.shape)}. "            f"These will broadcast to {tuple(torch.broadcast_shapes(preds.shape, targets.shape))}."        )try:    check_loss_shapes(preds, targets)except ValueError as e:    print("Caught before it cost you an afternoon:\n ", e)check_loss_shapes(preds, targets.squeeze(1))print("\nMatching shapes pass silently.")
What this code does

A guard that fails loudly on the first training step instead of letting a broadcast corrupt the loss for fifty epochs.

  1. 1

    Compare shapes exactly; don't rely on each loss function's own checks.

  2. 2

    Reports what the mismatch would have become, e.g. (6, 6), which makes the error message self-explanatory.

Output 
Caught before it cost you an afternoon:
  Shape mismatch: preds (6,) vs targets (6, 1). These will broadcast to (6, 6).

Matching shapes pass silently.
Python 24 lines 
# Which losses protect you, and which do not? Check, do not assume.import warningsp = torch.randn(6, 1)t_reg = torch.randn(6)t_bin = torch.randint(0, 2, (6,)).float()losses = [    ("MSELoss", nn.MSELoss(), t_reg),    ("L1Loss", nn.L1Loss(), t_reg),    ("SmoothL1Loss", nn.SmoothL1Loss(), t_reg),    ("BCEWithLogitsLoss", nn.BCEWithLogitsLoss(), t_bin),]print(f"Feeding preds {tuple(p.shape)} against targets of shape (6,)\n")for name, fn, target in losses:    try:        with warnings.catch_warnings(record=True) as caught:            warnings.simplefilter("always")            value = fn(p, target)        flag = "warned" if caught else "SILENT"        print(f"  {name:<20} ran, loss={value.item():8.4f}   ({flag})")    except (ValueError, RuntimeError) as e:        print(f"  {name:<20} raised {type(e).__name__}  <- protected")
Output 
Feeding preds (6, 1) against targets of shape (6,)

  MSELoss              ran, loss=  3.3967   (warned)
  L1Loss               ran, loss=  1.5244   (warned)
  SmoothL1Loss         ran, loss=  1.0952   (warned)
  BCEWithLogitsLoss    raised ValueError  <- protected

The regression losses run. The binary loss refuses. There is no principle behind that split — it is the accumulated history of which bug reports got acted on. Which is the argument for check_loss_shapes: your own assertion behaves the same way on every loss and every version.

Check your understandingQuestions to sit with
  1. nn.MSELoss on a batch of 256 with a model that outputs (256, 1) and targets of shape (256,). What matrix does the loss actually average over, and how many entries does it have? Roughly what fraction of those entries compare a prediction to its own target?
  2. You standardise using X.mean(dim=1, keepdim=True) by mistake. The code runs. Describe, in terms of what each row now represents, what your model is being asked to learn.
Code exercise

Standardise per feature, and refuse a broadcast

Python · runs in your browser

NumPy broadcasts exactly like PyTorch. Complete standardise (one mean and spread per column) and safe_mse, which must raise a ValueError when the shapes differ instead of silently comparing every prediction with every target.

Checkpoint Before you move on from §4
1

Predictions of shape (256, 1) and targets of shape (256,) go into nn.MSELoss. What does it average over?

2

Which pair of shapes is not broadcastable?

Explain it back · two sentences, in your own words

Why does the lesson call broadcasting the mechanism behind 'the worst category of bug', and what is the defence?

5

Matrix multiplication is the layer

In Lesson 1.2 you multiplied nine parameters by hand. nn.Linear is that operation at scale, and nothing more. Proving this to yourself removes the last of the magic from the phrase "dense layer".

The shape rule for A @ B: the inner dimensions must match, and the outer ones survive.

(N, in) @ (in, out) -> (N, out)
     ^______^  must match
  ^                ^  these survive
Interactive

Matrix multiply: inner dimensions meet, outer ones survive

A is the batch of inputs (N, in), B is the weight matrix (in, out). Hover a result cell to see which row and column produced it.

Loading interactive…
Python 2 lines 
A = torch.randn(8, 5)B = torch.randn(5, 3)
Inspect a value 1 line 
A
Output 
tensor([[ 8.8873e-01,  5.1824e-01,  4.7037e-01, -7.1866e-01,  1.2050e+00],
        [-1.3539e-01, -2.8148e-01, -3.0537e-01, -1.9027e+00, -3.0323e-01],
        [ 3.7933e-01,  1.5163e-01,  4.0593e-01, -4.0637e-01,  7.4333e-05],
        [ 8.6651e-01, -5.2250e-01,  9.1628e-01, -1.9131e-01,  2.3943e+00],
        [-2.7164e-01, -1.3006e+00,  6.2479e-01,  1.5261e+00, -3.8543e-01],
        [-5.5999e-01,  2.9853e-01,  2.5498e-01,  1.9409e+00, -1.1855e-01],
        [ 4.2360e-01, -4.4879e-01,  8.2307e-01,  8.9556e-01, -7.8797e-01],
        [-1.1197e-01, -1.0768e-01,  5.4267e-01,  1.6382e+00, -4.6443e-01]])
Inspect a value 1 line 
A.shape
Output 
torch.Size([8, 5])
Inspect a value 1 line 
B
Output 
tensor([[-1.0112,  0.0038,  1.3225],
        [ 0.2166, -0.7708,  1.3247],
        [ 0.2447,  0.6156, -0.9525],
        [-0.0114,  1.0010, -0.4248],
        [ 0.9434,  1.2993, -0.1211]])
Inspect a value 1 line 
B.shape
Output 
torch.Size([5, 3])
Inspect a value 1 line 
A@B
Output 
tensor([[ 0.4737,  0.7398,  1.5732],
        [-0.2632, -2.2702,  0.5839],
        [-0.2467, -0.2722,  0.4885],
        [ 1.4959,  3.8897, -0.6277],
        [-0.2352,  2.4129, -3.2789],
        [ 0.5593,  1.7137, -1.3982],
        [-1.0778,  0.7269, -1.1033],
        [-0.2341,  1.4531, -1.4473]])
Inspect a value 1 line 
(A@B).shape
Output 
torch.Size([8, 3])
Python 11 lines 
print("A @ B ->", tuple((A @ B).shape))try:    B @ Aexcept RuntimeError as e:    print("\nB @ A fails, as it must:", str(e)[:80])# Element-wise multiply is a completely different operation. Do not confuse * with @.C_ = torch.randn(8, 5)print("\nA * C_ (element-wise) ->", tuple((A * C_).shape))print("A @ B  (matrix)       ->", tuple((A @ B).shape))
Output 
A @ B -> (8, 3)

B @ A fails, as it must: mat1 and mat2 shapes cannot be multiplied (5x3 and 8x5)

A * C_ (element-wise) -> (8, 5)
A @ B  (matrix)       -> (8, 3)

Rebuilding nn.Linear by hand

PyTorch stores a linear layer's weight with shape (out_features, in_features) — transposed relative to the multiply. That ordering is a memory-layout decision, and it is why you see x @ W.T rather than x @ W.

Python 18 lines 
import torch.nn as nntorch.manual_seed(SEED)layer = nn.Linear(in_features=5, out_features=3)describe("layer.weight", layer.weight)describe("layer.bias", layer.bias)x = torch.randn(8, 5)official = layer(x)manual = x @ layer.weight.T + layer.bias     # (8,5) @ (5,3) -> (8,3), then + (3,) broadcastsdescribe("official", official)describe("manual", manual)print("\nMax absolute difference:", (official - manual).abs().max().item())print("Identical within float32 tolerance:", torch.allclose(official, manual, atol=1e-6))
nn.Linear, rebuilt by hand

Proof that a dense layer is exactly x @ W.T + b.

  1. 1

    The weight is stored as (out, in) = (3, 5), which is why the formula transposes it.

  2. 2

    (8, 5) @ (5, 3) → (8, 3), then the bias (3,) broadcasts across the 8 rows.

Output 
layer.weight shape=(3, 5)           dtype=torch.float32    device=cpu    ndim=2  numel=15
layer.bias   shape=(3,)             dtype=torch.float32    device=cpu    ndim=1  numel=3
official     shape=(8, 3)           dtype=torch.float32    device=cpu    ndim=2  numel=24
manual       shape=(8, 3)           dtype=torch.float32    device=cpu    ndim=2  numel=24

Max absolute difference: 0.0
Identical within float32 tolerance: True

Zero difference, or near enough that only floating-point ordering separates them. nn.Linear is x @ W.T + b. Everything else the class provides — parameter registration, device movement, gradient tracking — is bookkeeping around that one line.

This is worth pausing on, because it collapses the whole of Section 5 into a single sentence you can carry: a layer is a matrix multiply that reshapes the last axis from in_features to out_features, and leaves every other axis untouched.

That last clause matters more than it looks:

Python 13 lines 
# nn.Linear operates on the LAST axis only. Every leading axis is treated as batch.proj = nn.Linear(64, 16)flat  = torch.randn(32, 64)          # (N, E)      tabular-styleseq   = torch.randn(4, 10, 64)       # (N, L, E)   transformer-styledeep  = torch.randn(2, 3, 5, 64)     # arbitrary leading axesdescribe("flat -> ", proj(flat))describe("seq  -> ", proj(seq))describe("deep -> ", proj(deep))print("\nThe layer never asked how many leading axes there were.")print("This is exactly why the same nn.Linear works inside a Transformer on Day 6.")
Output 
flat ->      shape=(32, 16)         dtype=torch.float32    device=cpu    ndim=2  numel=512
seq  ->      shape=(4, 10, 16)      dtype=torch.float32    device=cpu    ndim=3  numel=640
deep ->      shape=(2, 3, 5, 16)    dtype=torch.float32    device=cpu    ndim=4  numel=480

The layer never asked how many leading axes there were.
This is exactly why the same nn.Linear works inside a Transformer on Day 6.

Counting parameters

Parameter count is the first number anyone asks about a model. It is fully determined by the shapes.

Interactive

Parameter & memory calculator

Type any stack of layer widths, or pick a preset. Use it to answer this section's first question.

Loading interactive…
Python 9 lines 
model = nn.Sequential(    nn.Linear(50, 64),    nn.ReLU(),    nn.Linear(64, 32),    nn.ReLU(),    nn.Linear(32, 1),)model
Output 
Sequential(
  (0): Linear(in_features=50, out_features=64, bias=True)
  (1): ReLU()
  (2): Linear(in_features=64, out_features=32, bias=True)
  (3): ReLU()
  (4): Linear(in_features=32, out_features=1, bias=True)
)
Python 2 lines 
for name in model.named_parameters():    print(name)
Python 11 lines 
total = 0for name, p in model.named_parameters():    n = p.numel()    total += n    print(f"{name:<12} {str(tuple(p.shape)):<12} {n:>6,}")print(f"{'TOTAL':<12} {'':<12} {total:>6,}")# Verify by hand: (in*out + out) for each linear layer.by_hand = (50 * 64 + 64) + (64 * 32 + 32) + (32 * 1 + 1)print(f"\nComputed by hand: {by_hand:,}  ->  matches: {by_hand == total}")
Output 
0.weight     (64, 50)      3,200
0.bias       (64,)            64
2.weight     (32, 64)      2,048
2.bias       (32,)            32
4.weight     (1, 32)          32
4.bias       (1,)              1
TOTAL                      5,377

Computed by hand: 5,377  ->  matches: True
Check your understandingQuestions to sit with
  1. A layer nn.Linear(1000, 1000) has how many parameters? Now stack ten of them. At float32, how many megabytes do the weights alone occupy?
  2. proj(deep) above accepted a 4-axis tensor. Which axis did it transform, and what would you have to do to apply a linear transformation to a different axis?
Checkpoint Before you move on from §5
1

nn.Linear(64, 16) receives a tensor of shape (2, 3, 5, 64). What comes out?

2

How is nn.Linear's weight stored, and what does the layer compute?

Explain it back · two sentences, in your own words

How many parameters does nn.Linear(1000, 1000) have, and roughly how much memory do ten of them take in float32?

6

dtype and device discipline

Two more clauses of the contract. Both produce errors that are easy to fix and impossible to guess if you have not seen them once.

Labels must be long for cross-entropy

nn.CrossEntropyLoss expects class indices, not floats and not one-hot vectors. Indices are integers, and PyTorch wants them as int64 — which it calls long.

Python 3 lines 
ce = nn.CrossEntropyLoss()logits = torch.randn(4, 3)              # 4 examples, 3 classes, raw scoreslogits
Output 
tensor([[-0.4001, -0.5472,  0.4396],
        [ 0.8689,  0.7260,  0.6413],
        [-0.6592,  0.6288,  1.8414],
        [-1.5752,  0.7442, -1.4613]])
Python 2 lines 
labels_float = torch.tensor([0.0, 1.0, 2.0, 0.0])labels_float
Output 
tensor([0., 1., 2., 0.])
Inspect a value 1 line 
ce(logits, labels_float)
Python 2 lines 
labels_long = torch.tensor([0, 1, 2, 0]) labels_long
Output 
tensor([0, 1, 2, 0])
Inspect a value 1 line 
ce(logits, labels_long)
Output 
tensor(1.3457)
Python 8 lines 
try:    ce(logits, labels_float)except RuntimeError as e:    print("float labels rejected:", str(e)[:90])labels_long = torch.tensor([0, 1, 2, 0])          # already int64 by defaultprint("\nlabels_long dtype:", labels_long.dtype)print("loss:", ce(logits, labels_long).item())
Output 
float labels rejected: expected target dtype to be Long or Byte, but got Float

labels_long dtype: torch.int64
loss: 1.345724105834961
Python 4 lines 
# The reverse rule for binary: BCEWithLogitsLoss expects FLOAT targets, same shape as output.bce = nn.BCEWithLogitsLoss()binary_logits = torch.randn(4, 1)binary_targets = torch.tensor([[0.], [1.], [1.], [0.]])
Inspect a value 1 line 
binary_logits
Output 
tensor([[ 1.3813],
        [ 0.3461],
        [ 0.0496],
        [-0.3879]])
Inspect a value 1 line 
binary_targets
Output 
tensor([[0.],
        [1.],
        [1.],
        [0.]])
Python 1 line 
print("BCE loss:", bce(binary_logits, binary_targets).item())
Output 
BCE loss: 0.83174729347229

The two conventions pull in opposite directions and you will mix them up at least once:

Loss Model output shape Target shape Target dtype
CrossEntropyLoss (N, C) raw logits (N,) long (int64)
BCEWithLogitsLoss (N, 1) raw logits (N, 1) float32

Both take raw logits, not probabilities. Neither wants you to apply a softmax or sigmoid first — they do it internally, in a numerically stable way. Lesson 2.3 explains why that matters.

Interactive

Loss-function contract checker

Choose a loss, an output shape, a target shape and a target dtype. The verdicts were checked against current PyTorch, and the orange ones are the dangerous kind: they run and are wrong.

Loading interactive…

Everything must live on one device

Python 18 lines 
cpu_tensor = torch.randn(4, 5)print("Created on:", cpu_tensor.device)moved = cpu_tensor.to(DEVICE)print("Moved to  :", moved.device)if DEVICE.type != "cpu":    other = torch.randn(5, 3)     # left on CPU    try:        moved @ other    except RuntimeError as e:        print("\nCross-device matmul fails:", str(e)[:110])    print("Fixed:", (moved @ other.to(DEVICE)).shape)else:    print("\nOn CPU-only hardware there is no mismatch to demonstrate.")    print("The error you would see on GPU hardware reads:")    print("  RuntimeError: Expected all tensors to be on the same device, "          "but found at least two devices, cuda:0 and cpu!")
Output 
Created on: cpu
Moved to  : mps:0

Cross-device matmul fails: Tensor for argument #2 'mat2' is on CPU, but expected it to be on GPU (while checking arguments for mm)
Fixed: torch.Size([4, 3])

The rule that prevents this entirely: move the model once, move each batch as it arrives.

model = model.to(DEVICE)              # once, before training

for xb, yb in loader:                 # every batch, inside the loop
    xb, yb = xb.to(DEVICE), yb.to(DEVICE)
    ...

Note the asymmetry. model.to(DEVICE) mutates the model in place and returns it. tensor.to(...) does not mutate — it returns a new tensor, and if you discard the return value nothing happens. This asymmetry is a genuine wart in the API and it catches everyone once.

.to() on a module vs a tensor

model.to(DEVICE)

Moves the parameters in place and returns the model. A bare call on its own line works.

tensor.to(DEVICE)

Returns a new tensor and leaves the original where it was. A bare call does nothing: write x = x.to(DEVICE).

Python 10 lines 
t = torch.randn(3)t.to(DEVICE)                    # return value discarded -- no effectprint("after bare t.to(DEVICE), t is still on:", t.device)t = t.to(DEVICE)                # correct: rebindprint("after t = t.to(DEVICE), t is on:", t.device)m = nn.Linear(3, 2)m.to(DEVICE)                    # modules DO move in placeprint("\nafter bare m.to(DEVICE), weight is on:", m.weight.device)
Output 
after bare t.to(DEVICE), t is still on: cpu
after t = t.to(DEVICE), t is on: mps:0

after bare m.to(DEVICE), weight is on: mps:0
Check your understandingQuestions to sit with
  1. Your labels come from pandas as int64 and you are using BCEWithLogitsLoss. Which cast do you need, and where in the pipeline should it happen so it happens exactly once?
  2. Why does tensor.to() return a new tensor rather than moving in place, when Module.to() does the opposite? Think about what a tensor might be shared with.
Checkpoint Before you move on from §6
1

Which targets does nn.CrossEntropyLoss expect for a batch of N examples?

2

t.to(DEVICE) on its own line leaves t on the CPU, but model.to(DEVICE) moves the model. Why?

7

A shape-debugging clinic

Five broken cells. For each one:

  1. Read the code and write down the error you expect, or "no error" if you think it runs.
  2. Run it.
  3. Fix it.

The point is the prediction. Reading a traceback teaches you nothing you did not already know; predicting one tells you exactly where your model of the system is wrong.

Each case below is one you will meet again in a real run this week.

Python 6 lines 
# --- CASE 1 ---------------------------------------------------------------# A model expecting 10 features, given a batch of 20-feature rows.# Predict: does this error? What does the message say about which shapes?model_1 = nn.Linear(20, 4)batch_1 = torch.randn(32, 20)
Inspect a value 1 line 
model_1(batch_1)
Python 5 lines 
try:    out = model_1(batch_1)    print("CASE 1 ran. Output:", tuple(out.shape))except RuntimeError as e:    print("CASE 1 error:", str(e)[:120])
Output 
CASE 1 error: mat1 and mat2 shapes cannot be multiplied (32x20 and 10x4)

Case 1 diagnosis. mat1 and mat2 shapes cannot be multiplied (32x20 and 10x4). PyTorch reports the weight already transposed, which is why you see 10x4 and not 4x10. The number that must match your input's last axis is the first number in the second pair. Fix: either build nn.Linear(20, 4), or find out why your feature count changed — usually a one-hot encoder fitted on a different split.

Python 3 lines 
# --- CASE 1 FIX ---model_1_fixed = nn.Linear(20, 4)print("Fixed:", tuple(model_1_fixed(batch_1).shape))
Output 
Fixed: (32, 4)
Predict before you run

Case 2. nn.Linear(5, 1) receives torch.randn(5), a single row with no batch axis. What happens?

Python 9 lines 
# --- CASE 2 ---------------------------------------------------------------# A single example passed to a model trained on batches.# Predict: error, or silent success?model_2 = nn.Linear(5, 1)one_row = torch.randn(5)          # NOT (1, 5)out_2 = model_2(one_row)out_2
Output 
tensor([-1.6324], grad_fn=<ViewBackward0>)
Python 2 lines 
describe("CASE 2 output", out_2)print("No error raised. Output has no batch axis.")
Output 
CASE 2 output shape=(1,)             dtype=torch.float32    device=cpu    ndim=1  numel=1
No error raised. Output has no batch axis.

Case 2 diagnosis. No error. nn.Linear transforms the last axis and does not care that there is nothing in front of it, so you get (1,) back instead of (1, 1). On its own this is harmless. Combined with a loss function, or with code downstream that indexes out[i] per example, it becomes Section 4's silent broadcast. Fix: one_row.unsqueeze(0).

Python 2 lines 
# --- CASE 2 FIX ---describe("CASE 2 fixed", model_2(one_row.unsqueeze(0)))
Output 
CASE 2 fixed shape=(1, 1)           dtype=torch.float32    device=cpu    ndim=2  numel=1
Predict before you run

Case 3. nn.Conv2d(in_channels=3, …) receives images shaped (4, 32, 32, 3). What happens?

Python 12 lines 
# --- CASE 3 ---------------------------------------------------------------# Image batch loaded in the layout PIL and matplotlib use.# Predict: which axis does the convolution think is the channel axis?conv = nn.Conv2d(in_channels=3, out_channels=8, kernel_size=3)images_3 = torch.randn(4, 32, 32, 3)      # (N, H, W, C) -- wrong for PyTorchtry:    out = conv(images_3)    print("CASE 3 ran. Output:", tuple(out.shape))except RuntimeError as e:    print("CASE 3 error:", str(e)[:130])
Output 
CASE 3 error: Given groups=1, weight of size [8, 3, 3, 3], expected input[4, 32, 32, 3] to have 3 channels, but got 32 channels instead

Case 3 diagnosis. The convolution reads axis 1 as channels, finds 32 where it wanted 3, and raises. This is the friendly outcome — if your image happened to be 3x3 pixels it would have run and produced garbage. Fix: images_3.permute(0, 3, 1, 2).

Python 5 lines 
# --- CASE 3 FIX ---images_3_fixed = images_3.permute(0, 3, 1, 2)describe("CASE 3 input fixed", images_3_fixed)describe("CASE 3 output", conv(images_3_fixed))print("Note 32 -> 30: a 3x3 kernel with no padding loses one pixel on each side.")
Output 
CASE 3 input fixed shape=(4, 3, 32, 32)   dtype=torch.float32    device=cpu    ndim=4  numel=12288
CASE 3 output shape=(4, 8, 30, 30)   dtype=torch.float32    device=cpu    ndim=4  numel=28800
Note 32 -> 30: a 3x3 kernel with no padding loses one pixel on each side.
Predict before you run

Case 4. CrossEntropyLoss receives logits (16, 3) and labels (16,) holding 0.0/1.0/2.0 as floats. What happens?

Python 12 lines 
# --- CASE 4 ---------------------------------------------------------------# Classification labels that came out of pandas as floats.# Predict: which of the two loss calls fails?logits_4 = torch.randn(16, 3)labels_4 = torch.rand(16).mul(3).floor()      # values 0.0, 1.0, 2.0 -- but floatdescribe("labels_4", labels_4)try:    nn.CrossEntropyLoss()(logits_4, labels_4)except RuntimeError as e:    print("CASE 4 error:", str(e)[:100])
Output 
labels_4     shape=(16,)            dtype=torch.float32    device=cpu    ndim=1  numel=16
CASE 4 error: expected target dtype to be Long or Byte, but got Float
Python 4 lines 
# --- CASE 4 FIX ---labels_4_fixed = labels_4.long()describe("labels_4_fixed", labels_4_fixed)print("CASE 4 loss:", nn.CrossEntropyLoss()(logits_4, labels_4_fixed).item())
Output 
labels_4_fixed shape=(16,)            dtype=torch.int64      device=cpu    ndim=1  numel=16
CASE 4 loss: 1.1663914918899536
Predict before you run

Case 5. MSELoss receives predictions (64, 1) and targets (64,). What shape does preds - targets have?

Python 29 lines 
# --- CASE 5 ---------------------------------------------------------------# The nastiest one: it runs, produces a number, and the number is wrong.# A regression head this time. Predict: what shape does (preds - targets) have?import warningsmodel_5 = nn.Linear(4, 1)X_5 = torch.randn(64, 4)targets_5 = torch.randn(64)                        # (64,)preds_5 = model_5(X_5)                             # (64, 1)describe("preds_5", preds_5)describe("targets_5", targets_5)with warnings.catch_warnings():    warnings.simplefilter("ignore")    wrong = nn.MSELoss()(preds_5, targets_5)right = nn.MSELoss()(preds_5.squeeze(1), targets_5)print(f"\nBroadcast shape: {tuple(torch.broadcast_shapes(preds_5.shape, targets_5.shape))}"      f"  -> {64*64:,} comparisons instead of 64")print(f"Loss as written : {wrong.item():.6f}")print(f"Loss corrected  : {right.item():.6f}")# The same mistake with a binary loss is caught on this version. Do not depend on that.try:    nn.BCEWithLogitsLoss()(preds_5, (targets_5 > 0).float())except (ValueError, RuntimeError) as e:    print(f"\nSame mistake, BCEWithLogitsLoss: {type(e).__name__} raised -- protected here.")
Output 
preds_5      shape=(64, 1)          dtype=torch.float32    device=cpu    ndim=2  numel=64
targets_5    shape=(64,)            dtype=torch.float32    device=cpu    ndim=1  numel=64

Broadcast shape: (64, 64)  -> 4,096 comparisons instead of 64
Loss as written : 1.360844
Loss corrected  : 1.455775

Same mistake, BCEWithLogitsLoss: ValueError raised -- protected here.

Case 5 diagnosis. No exception from MSELoss. The loss is computed over a 64x64 matrix — 4,096 comparisons where you wanted 64 — and every prediction is scored against every target. The two losses agree to within about half a percent, which is precisely what makes this dangerous: the number looks reasonable, the training curve looks plausible, and the model is optimising the wrong objective. Section 4's example diverged wildly only because the predictions there were nearly perfect; when the model is merely mediocre — as it is at the start of every real run — the wrong loss and the right loss are almost indistinguishable, and nothing ever alerts you.

The binary loss caught the identical mistake. That asymmetry is not something to rely on — it is a property of the version you happen to have installed, and it does not exist for the regression losses at all.

Two fixes, and you should pick one and apply it consistently across a project:

  • squeeze the model output: preds.squeeze(1) to match (N,) labels;
  • or unsqueeze the labels: labels.unsqueeze(1) to match (N, 1) output.

Lesson 2.3 uses the second, because keeping an explicit (N, 1) everywhere makes the batch axis visible in every print statement.

Check your understandingQuestions to sit with
  1. Of the five cases, three raised errors and two ran. Rank all five by how expensive they would be to discover in a real project, and justify the ordering.
  2. Case 3 would have run silently if the images were 3 pixels tall. Write an assertion you could put in your data pipeline that would catch the layout problem regardless of image size.
Checkpoint Before you move on from §7
1

nn.Linear(10, 4) receives a (32, 20) batch. The error says (32x20 and 10x4). What must match?

2

Of the five clinic cases, which kind is the most expensive to discover in a real project?

Explain it back · two sentences, in your own words

Write (in words) an assertion for your data pipeline that catches images in (N, H, W, C) layout regardless of image size.

8

Conclusion

The tensor contract has three clauses, and nearly every error in this course is a violation of one of them:

  • shape — axis 0 is the batch; a layer transforms the last axis; conservation of elements governs every reshape;
  • dtype — float32 for inputs and BCE targets, long for cross-entropy labels, cast once at the NumPy boundary;
  • device — the model moves in place, tensors do not, and everything in a single operation must agree.

The specific habits worth keeping:

  • call describe() (or print(t.shape, t.dtype, t.device)) whenever you are unsure — it costs nothing and it is faster than reasoning;
  • prefer reshape over view unless you have a reason, and never call bare squeeze();
  • assert that predictions and targets have equal shapes before every loss computation;
  • treat a UserWarning about broadcasting as an error, because in a long training run it is one.

The single most important idea here is that the dangerous bugs are the ones that run. A traceback is a gift. A loss that is five thousand times too large, computed silently over an N x N matrix, is what actually costs people days.

Transition to Lesson 2.2

You can now describe any tensor and fix any shape error in one. What you cannot yet do is get real data into that shape reliably, at scale, batch after batch, without leaking information between your training and validation sets.

That is the job of Dataset and DataLoader, and Lesson 2.2 builds both — first over the UCI Bank Marketing dataset (45,211 rows, 11.7% positive), then over a directory of image files on disk, so you see that Dataset is a protocol rather than a tabular convenience.

We will also settle a question this lesson deliberately left open: why bother with mini-batches at all, when the entire Bank Marketing training set occupies 7 MB and fits in memory a thousand times over. The answer is not the one most tutorials give, and the measurement is more interesting than the folklore.

Wrap-up

Key takeaways
  1. Every tensor has a shape, a dtype and a device; almost every error violates one of them. Inspect all three whenever you're unsure.
  2. Axis 0 is the batch. Tabular (N, F), images (N, C, H, W) (channels first in PyTorch), tokens (N, L), embeddings (N, L, E).
  3. An integer index drops an axis; a slice keeps it. A single example must still be a batch of one: (1, F).
  4. Reshape conserves the number of elements and never moves data. view needs contiguous memory; reshape copies when it must. Never use bare squeeze().
  5. Broadcasting aligns shapes from the right; a size-1 axis stretches. It powers (X - mu) / sd, and it turns (N, 1) vs (N,) into a silent (N, N) bug.
  6. Assert preds.shape == targets.shape before every loss: loss functions don't guard you consistently.
  7. A dense layer is x @ W.T + b, applied to the last axis only. Parameters = in × out + out.
  8. CrossEntropyLoss: logits (N, C) + long targets (N,). BCEWithLogitsLoss: logits + float targets of identical shape.
  9. Move the model once (in place); move every tensor by reassigning (x = x.to(DEVICE)).
  10. The dangerous bugs are the ones that run. A traceback is a gift.
Knowledge check

Quick check

1

batch has shape (8, 5). What is the shape of batch[:, 2:3]?

2

Which pair of shapes is not broadcastable?

3

x.t().view(-1) raises an error. Why?

4

Predictions (256, 1) and targets (256,) go into MSELoss. What does the loss average over?

5

How many parameters does nn.Linear(64, 16) have?

6

Labels for CrossEntropyLoss should be…

7

What does t.to(DEVICE) on a line by itself do when t is a tensor?

Questions and discussion

Ask about this lesson, share what you found in a lab, or help someone else. Posts are public on the course's GitHub Discussions: anyone can read them, and you need a GitHub account to post.

Opening it loads giscus from giscus.app.