Connecting Image and Text Representations

A modality is a type of data, defined by how it is recorded: images, text, audio, video, sensor readings. Each modality has its own structure. Images are 2D grids of pixels where neighbours matter; text is a sequence of discrete symbols where order matters. That is why we used different encoders for them: a CNN for images, an embedding layer for text.

Each encoder produces a vector, but there is no reason for the vectors from two separately trained encoders to be comparable. Imagine two people describing the same building, one in French using metres and one in Japanese using feet. Each description is accurate, but you cannot compare them number by number until you agree on a shared language and shared units. A shared representation space is that agreement: a single space in which an image of a bag and the text "a photo of a bag" land in the same place.

Figure 5.5.1. Separate encoders do not automatically align their coordinates. Joint contrastive training can make projected image and text vectors comparable in a shared space. Cluster positions here are schematic.

We start by loading the images. We will create the text ourselves in Section 2.

A shared space is learned from pairs: an image together with text that describes it. Real systems learn from enormous collections of images and captions gathered from the internet. We will make simple captions ourselves from the class names, using four sentence patterns called templates, so that each class can be described in several different ways.

Each class gets 4 captions, which makes 40 distinct captions in total, built from a vocabulary of only 21 words. The four captions for class 7 are shown: "a photo of a sneaker", "a picture of a sneaker", "an image showing a sneaker" and "a sneaker on a plain background". During training, every image will be paired with one of its class's captions, chosen at random.

The recipe for a shared space has three parts:

1. An image encoder that turns an image into a vector. We use the same small CNN as in Lesson 5.2.
2. A text encoder that turns a caption into a vector. We use the bag-of-embeddings idea from Lesson 5.3: look up each word's embedding and average them.
3. A projection at the end of each encoder: a linear layer that maps the encoder's output into the shared space. Both projections output vectors of the same length, 32 numbers, and we normalise each vector to length 1, so that their dot product equals their cosine similarity.

The CNN on its own produces 1,568 numbers per image, while the text encoder's word embeddings have 32 numbers. Those two cannot be compared at all. After the projections, both produce vectors of shape (batch, 32), and each has length 1.0. Now an image and a caption can be compared, with a single dot product.

But being comparable is not the same as being meaningful. Let's test the untrained encoders. For each validation image, we compute its similarity with the 10 descriptions "a photo of a t-shirt", "a photo of a trouser" and so on, and pick the most similar. If the space were meaningful, this would classify images correctly using only text.

With untrained encoders, the accuracy is 0.100: exactly what random guessing among 10 classes would give. The two encoders produce vectors of the same size, but they have no reason to agree. We need to train them so that matching images and captions end up close together.

Key terms

• Modality: a type of data, such as images, text or audio.
• Shared representation space: a single vector space in which data from different modalities can be compared directly.
• Projection: a layer, usually linear, that maps an encoder's output into the shared space.
• Normalise (to unit length): divide a vector by its length so that its length becomes 1; then dot product equals cosine similarity.

How do we train two encoders so that matching pairs are close? We cannot simply tell the network "the correct vector for a bag is this one", because nobody knows what the right vectors are. What we do know is which image goes with which caption. Contrastive learning uses exactly that.

Take a batch of images, each with its own caption. Compute the similarity of every image with every caption in the batch. The training goal is:

• pull together each image and its own caption (the positive pair);
• push apart each image and all the other captions in the batch (the negative pairs).

Think of a seating chart at a wedding. You don't know the exact coordinates where each guest should sit, but you know who belongs with whom. Moving partners together and strangers apart, again and again, gradually produces a sensible arrangement of the whole room.

Figure 5.5.2. In this toy batch each image has one designated matching caption. Contrastive loss raises matched scores relative to other batch entries; semantically similar off-diagonal pairs can be false negatives.

Here is the loss on a tiny made-up example with 4 images and 4 captions. S[i, j] is the similarity between image i and caption j, and the correct matches are on the diagonal.

We read the matrix row by row: each row asks "which of the 4 captions belongs to this image?". That is a 4-way classification problem, so we can use the ordinary softmax and cross-entropy loss from classification, with the correct answer being the diagonal entry. We then do the same column by column ("which image belongs to this caption?") and average the two. Before the softmax, the similarities are multiplied by a scale, here 10, which controls how sharply the softmax separates the scores. (In CLIP it is described as dividing by a temperature, which is the same thing.)

In this example, the diagonal entries are clearly the largest in each row, so after scaling and softmax, each image puts almost all its probability on its own caption: 0.999, 0.996, 0.977 and 0.973. The loss is correspondingly small: 0.014 for the rows, 0.012 for the columns, and 0.013 on average. If an off-diagonal similarity were higher than the diagonal one, the probability on the correct caption would drop and the loss would rise sharply. That is the signal that drives learning.

The contrastive_loss function below does exactly this for a whole batch. In every training step, we draw 128 random training images, give each one a random caption from its own class, embed both, and minimise the contrastive loss. We train for 400 steps.

The training function also records a floor for the loss, which needs explaining. Each batch contains about 12.8 images of each class, and many of their captions are identical or nearly so. Our model is supposed to map all bag images to the same place, so it cannot tell which of the roughly 13 bag images in the batch belongs to a particular bag caption. The best it can do is spread the probability evenly among them, and that gives a loss of about the logarithm of their number. These same-class "negatives" are called false negatives: pairs that the loss treats as wrong, even though they are a perfectly good match.

The starting loss is close to ln(128) = 4.852, which is what you get when every one of the 128 captions looks equally likely: the untrained model has no idea which caption fits. Over the first 20 steps the loss averages 4.886, and over the last 20 steps it has fallen to 2.934. The floor caused by same-class items is 2.584, so the model has closed most of the possible gap. The plot shows the loss approaching, but not reaching, the floor.

Now we repeat the test from Section 2: match each of the 10,000 validation images to the most similar of the 10 descriptions "a photo of a ...". No classifier head was trained for this; we are classifying images purely by comparing them with text.

The accuracy is now 0.875, against 0.100 before training. Classes with distinctive shapes are easiest: bags (0.974), trousers (0.973) and sandals (0.960). Coats are hardest (0.671), followed by pullovers (0.790) and shirts (0.796), the same family of similar upper-body garments that troubled the CNN in Lesson 5.2.

The text side is genuinely flexible, within limits. Because the text encoder averages word embeddings, any combination of words it knows produces a valid description. Let's try different wordings:

All four training templates give similar accuracies, from 0.868 to 0.875, and even the bare class name '{}', a description the model never saw on its own during training, scores 0.865. But the prompt "this is a {}" fails completely, because "this" is not one of the 21 words in the vocabulary. Our text encoder can only understand words it was trained on, a limitation we return to in Section 4.

To see the shared space more directly, we average the vectors of all validation images of each class and compare each average with each of the 10 class descriptions. A meaningful shared space should produce a bright diagonal.

Images and the description of their own class have an average cosine of 0.888; images and other classes' descriptions average 0.203. The brightest off-diagonal cells all involve the hard upper-body classes: T-shirt images against "a photo of a shirt" (0.71), coat images against "shirt" (0.69), pullover images against "shirt" (0.68), and coat images against "pullover" (0.61). The model's confusions in the space are the confusions you would expect.

Finally, a picture of the space itself. We project 1,000 validation images (dots) and the 10 descriptions (stars) together onto two dimensions with PCA. This 2D picture keeps only 50% of the variation in the 32-dimensional vectors, so treat it as an illustration of the numbers above, not a measurement.

The images form clusters by class, and each description sits near its own class's cluster. Two details are worth noticing. The stars lie near their clusters rather than in their centres: images and text occupy slightly different regions of the shared space, which is consistent with the matching cosine of 0.888 being high but below 1. And the "coat" and "shirt" stars sit almost on top of each other in the crowded upper-body region, which matches the high coat–shirt similarity of 0.69.

Common mistake: expecting the contrastive loss to go to zero. With false negatives in every batch, it cannot. Judge a contrastive model by what it can do (matching accuracy, retrieval quality), not by how low the loss goes.

Key terms

• Contrastive learning: training by pulling matching pairs together and pushing non-matching pairs apart.
• Positive / negative pair: a matching image–caption pair / a non-matching one.
• Contrastive loss: cross-entropy over the image–caption similarity matrix, in both directions, with the matches on the diagonal.
• Temperature (scale): a number that controls how sharply the softmax separates similarity scores.
• False negative: a pair treated as non-matching by the loss even though it is actually a valid match.

What we have built is a miniature version of CLIP (Contrastive Language–Image Pre-training), a model released by OpenAI that showed how powerful this idea becomes at scale. CLIP has exactly the same structure as our toy:

• an image encoder (in CLIP, a large CNN or a Vision Transformer, which you will meet in Module 6);
• a text encoder (in CLIP, a Transformer that reads the whole sentence, not an average of word vectors);
• projections into a shared space, with normalised vectors;
• the contrastive loss over large batches of image–caption pairs, with a learned temperature.

The big difference is the data. Our toy learned from 40 template captions about 10 kinds of clothing. CLIP learned from an enormous collection of real images paired with the text that appeared next to them on the internet: captions, titles, descriptions. That variety is what gives CLIP its most famous ability, zero-shot classification: classifying images into categories it was never explicitly trained on, simply by writing a description of each category.

Figure 5.5.3. Training aligns image–caption pairs. At inference, compare a new image with candidate text descriptions. Bars are illustrative similarities, not measured confidence or calibrated probabilities.

Our model did classify images using text, but only for the 10 classes whose captions it was trained on. Is that really "zero-shot"? Let's test it properly. We train a fresh model without any sneaker images or sneaker captions, then ask it to recognise sneakers using the description "a photo of a sneaker".

The model trained without sneakers still reaches 0.796 overall on the validation images, but it recognises none of the sneakers (0.000). It matches 721 of them to "sandal", 233 to "ankle boot" and 1 to "bag": all visually similar shoes, or nearly. It has learned sensible things about the images, since sneakers land near other footwear, but the word "sneaker" never appeared in any training caption, so its embedding is still random. Text it has never learned from carries no meaning.

This is precisely why CLIP's scale matters. CLIP can recognise categories that no one explicitly labelled because, during training, it saw those words in many real captions, alongside many images. Its text encoder has learned what words like "sneaker", "zebra" or "lighthouse" refer to. Zero-shot classification is not magic: it is reuse of what was learned from text during training.

A shared image–text space is useful far beyond classification. Because images and text are now comparable, we can search in either direction.

To search a collection of images with a sentence, embed every image once and store the vectors. To answer a query, embed the text and rank all the stored images by cosine similarity. Here are the top 8 validation images for three queries, with the cosine similarity above each one:

Every result in all three rows is from the requested class, with cosine similarities between 0.91 and 0.93. Let's measure this more broadly: for each class, we search with "a photo of a ..." and check what fraction of the top 100 results really belong to that class. This fraction is called precision at 100.

Precision at 100 is 1.00 for trousers, sandals and ankle boots, and at least 0.95 for every class, averaging 0.976. Even coats, the hardest class to classify, reach 0.95: the very best matches for a query are much more reliable than the average match.

Notice something about this search: no image was ever tagged with keywords. The ranking came entirely from the shared space. That is how modern photo libraries let you search your own pictures by typing what you remember.

The same shared space powers several other kinds of system:

• Image-to-text retrieval: find the captions, products or documents that best match an image (the reverse direction of the search above).
• Zero-shot classification: as in Section 4, classify images into categories described in words, without training a new classifier.
• Duplicate and similarity search: find images similar to a given image by comparing image vectors directly.
• Guiding image generation: some text-to-image generators use a CLIP-style shared space to check how well a generated image matches the prompt.
• Multimodal assistants: models that answer questions about images often begin with an image encoder trained in a shared image–text space, whose vectors are then passed to a language model.

You will load a real pretrained CLIP model and use it for zero-shot classification and search in Module 7.

Key terms

• Retrieval: finding the items in a collection that best match a query.
• Precision at k: the fraction of the top k results that are correct.
• Zero-shot classification: classifying into categories using descriptions only, without training on labelled examples of those categories.

Images and text are different modalities, and separately trained encoders produce vectors that cannot be compared. Matching images to class descriptions with untrained encoders scored 0.100, pure chance.

To build a shared space, we gave each modality its own encoder (a CNN for images, averaged word embeddings for text) followed by a projection into the same 32-dimensional space, with normalised vectors. We trained both encoders together with contrastive learning: compute the similarity of every image with every caption in a batch, and use cross-entropy in both directions to make the diagonal (the true pairs) win. The loss fell from about ln(128) = 4.852 towards a floor of 2.584 set by false negatives, same-class items the loss cannot tell apart.

The resulting space works: matching 10,000 validation images to text descriptions reached 0.875, matching classes had cosine 0.888 against 0.203 for the rest, and text-to-image search achieved an average precision at 100 of 0.976. Its limits are just as instructive: it failed on words outside its 21-word vocabulary, and when sneakers were left out of training it recognised none of them (0.000).

CLIP is the same recipe, with Transformer encoders and a vast collection of real image–caption pairs from the internet. That scale is what turns "matching known classes with text" into genuine zero-shot classification and open-ended search.

Figure 5.5.4. Each encoder is projected into the same 32-dimensional, unit-length space. The contrastive objective trains the projections and encoders; image–text comparison then supports retrieval or classification. The clusters are schematic.