Text-to-video or image-to-video: which should you start from?

Every video model offers the same two doors. Through one you type a description and get a clip. Through the other you hand it a picture and get that picture, moving. New users read these as a convenience choice, as if one were the quick way and the other the careful way, and pick whichever is nearer the cursor.

They are not a convenience choice. They are a decision about who composes the shot: you, or the model. Make it deliberately and most shots take two attempts. Make it by accident and the same shots take six.

What text-to-video is actually doing

A text prompt asks the model to invent everything at once: the subject, the framing, the light, the lens, the colours, the motion, and the way all of those change over five seconds. It does this well, in the sense that the result is nearly always a plausible piece of video. It does it unpredictably, in the sense that the result is nearly never the piece of video you had in your head.

That unpredictability is not a flaw you can prompt your way out of. A sentence underdetermines a picture. "A woman reads by a window at dusk" is compatible with ten thousand frames, and the model picks one. If you had no particular frame in mind, that is fine, and often better than fine, because the model's choice may be more interesting than yours. If you did have one in mind, you are now spending a generation per guess, and a video generation is the most expensive guess in the building.

So text-to-video is the right door when the picture is open. Exploring a look. Generating texture and atmosphere shots where any good frame will do. Discovering what the model does with an idea before you commit to it. The prompting guide covers how to get the most from it, and the short version is: describe motion, describe light, stop describing things the model will decide for itself anyway.

What image-to-video is actually doing

Hand the model a still and the problem it has to solve shrinks by a large margin. The subject, framing, lens, palette and light are decided; the only thing left to invent is how the frame moves. That is a far smaller space, and the model lands in it far more reliably.

The consequence is that the quality of an image-to-video clip is mostly decided before you press Generate. A still that is composed well, lit well and sharp animates into a clip that is composed well, lit well and sharp. A still with a wrong hand animates into a clip where the wrong hand waves at the camera. The model adds motion; it does not add judgement. (Why image-to-video clips look wrong goes through the specific failures and their fixes.)

So the control you gain is real, and so is the responsibility. Image-to-video is the right door when the picture is decided: a product, a character, a location that has to look a particular way, and anything that has to match a shot you already have.

The cost argument, which settles most cases

Here is the arithmetic that experienced users run without thinking about it.

A video generation costs a lot and takes a minute or two. An image generation costs very little and takes seconds. If a shot has any chance of needing more than one attempt, it is cheaper to make the attempts as images.

Generate the still in a batch of four. Pick the best. If none is right, adjust the prompt and run four more; you are still well inside the cost of one video. When one is right, edit it rather than regenerate it: the image editor takes reference images, so a wrong prop or a wrong colour is a targeted fix rather than a fresh roll of the dice. Only when the frame is exactly what you want does it go to the video generator as the seed for Animate.

That ladder, text to image, image to image, image to video, is what the two doors are really for. Text-to-video is the short-cut you take when a shot is not worth the ladder. Image-to-video is the last rung of it.

When text-to-video still wins

Three cases, and they are genuine.

The shot is about motion rather than composition. Water, smoke, crowds, traffic at night, anything where the movement is the point and the exact frame is not. Text-to-video models are trained on more video than any still model has seen, and the motion they invent from a prompt is often better than what they add to a static image.

You want to be surprised. Early in a project, before the look is settled, a batch of text-to-video clips is a cheap way to see several answers to a question you cannot yet phrase. Pick the one you like and make a still from its best frame; the frame anchor in the generate form captures it. From then on you are on the ladder.

The subject must move in a way a still cannot imply. A walk cycle, a turn, a gesture with a beginning and an end. A still shows one moment and the model has to guess the rest; a prompt can say "she turns from the window and walks toward the camera" and get it.

When image-to-video wins

Everything with continuity in it. Any character who appears in more than one shot, any product, any location that must match itself across a cut. (Keeping a character consistent is a still-first workflow for exactly this reason.) Any shot where a client, a brand or your own taste has already decided what the frame looks like. And any shot you have already generated three times from text and are now about to try a fourth; at that point the still is the cheaper way to say what you mean.

It also wins on quality more often than people expect, for a dull technical reason. Image models produce more detail, at higher resolution, with better hands and better text, than video models do. A still generated well and then animated starts from a better frame than a video model would have invented on its own, and a 2× upscale on the result makes the difference permanent.

The mistakes on each side

On the text side, the mistake is over-description: a two-hundred-word prompt that pins down every object and leaves the model no room, producing a cluttered frame that moves badly. Prompts for video want the subject, the motion, the light and the mood, in that order, and not much else.

On the image side, the mistake is the wrong kind of still. A frame that is beautiful as a photograph can be a poor seed for motion: a subject at the very edge, nowhere for the camera to go, a tight crop that leaves the model no space to invent in. The best seeds have room in them. Wider than the final framing, subject off the edges, a direction the motion can take.

The shared mistake is asking either door for a long clip. Generated clips are a few seconds long and that is where they are best; the longer story is built on the timeline, out of several short shots each made by whichever door suited it.

The order that works

1. Decide whether the picture is open or decided. Open: text-to-video. Decided: still first.
2. Make the attempts as images, four at a time, and edit the near-miss rather than regenerating it.
3. Animate the still only when the frame is right. Describe the motion, not the content; the content is already there.
4. Use text-to-video for motion shots and exploration, and capture the best frame as a still if the shot becomes one you need to match.
5. Upscale the keeper and build the sequence on the timeline, one short shot at a time.