What is a Multimodal Model?

A multimodal model is an AI that handles more than one kind of input, such as text, images, audio and video, inside a single system. It is why you can photograph a page and ask a question about it in the same breath. Earlier models could only read text, so anything else had to be described to them first.

Older models could only read text, so anything else had to be described in words first, and most things do not survive that translation. A multimodal model removes that step by turning every kind of input into the same internal format before it processes anything. A sentence, a photograph and a few seconds of speech all become long lists of numbers. Once they are numbers the model treats them alike, which is what lets a question about an image sit in the same request as the image itself. So a photograph of a receipt or an error screen goes straight in, and it can answer things needing both at once, such as what is wrong with this chart. The cost is size and speed, since images and audio become far more numbers than a sentence does. That eats into the context window and makes each answer slower and more expensive. Being able to take an input is not the same as handling it well, so a model that reads a typed page reliably may still misread a hand-drawn diagram.

Talking someone through a computer problem over the phone. Everything you know arrives through their description, so if they never mention the flashing red light, it does not exist as far as you are concerned. Stand next to the machine and you read the error yourself, spot the unplugged cable and hear the clicking. You did not get smarter, you got access to information that was there the whole time.