Generating and Understanding Images, Audio, and Video on Bedrock

The situation

A media and operations team has landed two projects in the same sprint, and because both say the word “images” in the brief, someone has filed them under one ticket. The first project is a marketing pipeline: given a product name and a short brief, produce on-brand hero images and a few seconds of promotional video, at volume, without a photoshoot. The second is a claims-intake pipeline: customers upload a soup of PDFs, phone-camera photos of receipts, voicemails, and short video clips, and the business wants structured records out of all of it, fields, tables, transcripts, and summaries, so downstream systems can process a claim without a human retyping anything.

They are both “working with media on AWS”, and that is where the resemblance ends. One is a generation problem, where the model produces the artefact. The other is an understanding problem, where the model reads an artefact and hands back structure. The tools barely overlap, the failure modes are opposite, and the provenance question (can we prove which images our system generated?) lands on only one of them.

The instinct to solve both with “a model on Bedrock” is not wrong, it is just too coarse. The useful first cut is not which model, it is which of the two jobs you are doing, and after that, which modality and what shape you need out.

What actually matters

The dividing line that decides everything else is direction of travel. Generation goes text-to-media: a prompt in, an image or a video out. Understanding goes media-to-structure: an image, document, audio, or video in, and fields, transcripts, or a reasoned answer out. Almost every downstream choice follows from which way the arrow points, so naming that first saves picking a service that solves the other problem beautifully.

On the generation side, the properties worth weighing are modality and control. Image generation, image editing (inpainting, background removal, variation), and short-form video are distinct capabilities, and not every model does all of them. Beyond raw quality there is a governance property that is easy to miss until legal asks: provenance. Some generators embed an invisible, machine-detectable watermark in every image, which is what makes “did our system make this?” answerable later from the pixels alone. Others do not, and then provenance is something your pipeline has to record for itself. Either way it is a property you are choosing at the point you pick a model, not a detail to sort out afterwards.

The generation side also has a property the understanding side mostly does not: the models turn over. Image and video families arrive, get superseded, and are marked legacy in the catalogue on a cycle measured in months rather than years, and a legacy model eventually stops answering. That makes lifecycle status and region availability real design inputs rather than trivia, and it makes the invocation pattern (a synchronous call for a still, an asynchronous job for a clip) the part of the design worth building against, because the pattern outlives whichever model is currently best at the job.

On the understanding side, the property that matters most is what shape you need out, and it splits three ways. Sometimes you need structured fields and tables extracted from mixed media at volume through one managed pipeline; that is Amazon Bedrock Data Automation. Sometimes you need deep, fine-grained control over a single modality, the best available OCR on a specific document layout, speaker-diarised transcription, object and moderation labels on video; that is the purpose-built services, Amazon Textract, Amazon Transcribe, Amazon Rekognition. And sometimes you do not want extraction at all, you want a model to look at an image or a document and reason about it in open language, answer a question, judge it, explain it; that is a multimodal foundation model such as Nova Lite, Nova Pro, or Claude, reading the media directly in a prompt.

Those three understanding answers are genuinely different tools, and the trap is reaching for a multimodal model’s free-form reasoning when you needed reliable structured fields, or standing up three separate single-purpose services when one managed multimodal pipeline would have covered the mix. Structured-versus-reasoned, and unified-versus-single-modality, are the two axes that separate them.

What we’ll filter on

1. Direction: are you generating media from a prompt, or understanding media into structure or an answer?
2. Modality and availability: image, video, audio, or document, and is a model for it still active in the catalogue and callable from your region?
3. Output shape (understanding only): strict structured fields and tables, single-modality precision, or free-form reasoning?
4. Breadth: one mixed stream of many media types, or one modality you want fine control over?
5. Provenance (generation only): do you need to prove which images your own system produced, and does the model give you anything to prove it with?
6. Operational fit: managed pipeline with blueprints, a direct model call, or wiring into a Knowledge Base as a parser?

The multi-modal landscape

The generation half of this landscape has already turned over twice, so it pays to read the catalogue before believing any list of models, including this one. Every entry ListFoundationModels returns carries a modelLifecycle.status, and the values that matter are ACTIVE and LEGACY. Legacy is not a soft deprecation notice: a legacy model refuses invocation for any account that has not called it in the last thirty days, and the refusal arrives as a ResourceNotFoundException whose message says the model is marked by the provider as legacy. An account with a long-running pipeline keeps working; a new project trying to start on the same model gets a hard no.

Image generation. The active image generators on Bedrock are Stability AI’s: stability.stable-image-core-v1:1 for volume work, stability.stable-image-ultra-v1:1 when the output has to be the best the platform can do, and stability.sd3-5-large-v1:0 for a different look and more control. All three generate in us-west-2. Alongside them sits a set of editing operations, each its own model id rather than a mode of the generator: inpaint, outpaint, erase object, search-and-replace, search-and-recolor, style guide, style transfer, sketch and structure control, three upscalers, and remove background. Those are active in both us-east-1 and us-west-2, so the editing half of a pipeline can live next to the rest of your stack even when generation cannot.

Amazon Nova Canvas is the sunsetting incumbent here. One model covered generation and the same kinds of edit, and it embedded the invisible provenance watermark on everything it produced, but it is marked legacy in the catalogue now, which means an account without recent usage cannot call it at all, and it carries an end-of-life date of 30 September 2026, when it is withdrawn for the accounts still calling it too. Amazon Titan Image Generator, the model before it, has gone further and no longer appears in the catalogue in us-east-1 or us-west-2. Both still show up in training material and in older architecture diagrams, so recognising the names is worth something; starting a new pipeline on either is not.

Video generation. Luma Ray 2, luma.ray-v2:0, is the active video generator, in us-west-2 only. It produces a five- or nine-second clip at 540p or 720p from a text prompt of up to five thousand characters, in any of seven aspect ratios, and it takes optional keyframes: frame0 sets the first frame from an image you supply, frame1 sets the last, and a loop flag asks for a clip that repeats cleanly. Rendering takes a few minutes, so the call is asynchronous by necessity rather than by preference. Amazon Nova Reel, in both its versions, is the sunsetting incumbent on this side, marked legacy the same way Canvas is.

Bedrock Data Automation (BDA). A managed service that takes unstructured multimodal input, documents, images, audio, and video, and returns structured insights: fields and tables lifted from documents, transcripts and summaries from audio and video, captions and detected content from images. You configure what comes out with blueprints, which describe the fields and structure you want for a given document or media type, so the same pipeline can handle a receipt one way and a claim form another. BDA is one API surface across all four modalities, and it plugs into a Bedrock Knowledge Base as the parser that turns raw uploads into indexable content, so a retrieval system can ingest PDFs and media without a bespoke extraction stage in front of it.

Purpose-built AI services. Amazon Textract reads documents (text, forms, tables, signatures, queries) with fine control over a single modality. Amazon Transcribe turns speech into text with speaker labels, custom vocabulary, and language options. Amazon Rekognition analyses images and video for objects, faces, text, and moderation. Each is deep in its one lane, tunable, and long-established. They are covered in their own right elsewhere; here they are the alternative to BDA when you want single-modality precision rather than one pipeline across everything.

Multimodal foundation models. Nova Lite, Nova Pro, and Claude on Bedrock accept an image or a document alongside the text prompt and reason over it directly: “what is wrong with this diagram”, “does this receipt match this policy”, “summarise what the person in this photo is doing”. No blueprint, no fixed schema, just language in and language out over the media. This is understanding of a third kind, open-ended reasoning rather than structured extraction, and it is the right tool when the question is fuzzy or one-off rather than a repeatable field-extraction job.

Side by side

The picks in depth

The marketing pipeline is a generation job, and the picks are Stable Image Core plus Luma Ray 2. Product stills at volume from a text brief go to Core, the workhorse of the three generators and the one to reach for when the designer wants twenty options rather than one. The hero shot that ends up on the campaign page is worth spending Ultra on. Editing matters as much as the raw generation, and that work is a set of separate calls rather than a mode of the generator: remove-background to drop the product onto the clean canvas the brand guidelines want, inpaint to fix a detail, style-guide or style-transfer to hold a house look across a set. The few seconds of motion come from Luma Ray 2, which takes the still as frame0 and animates from it, so the clip and the still stay on-brand together instead of being two independent guesses at the same product.

The region shape is part of the pick. Generation is us-west-2 only, for both the Stability generators and Ray 2, while the editing operations are available in us-east-1 as well. If the rest of the pipeline lives in us-east-1, the practical layout is a us-west-2 client for the stills and the clips, an S3 bucket in us-west-2 for the video output, and editing wherever the assets already are. That is a small amount of plumbing, and it is much cheaper to design in at the start than to retrofit when the first hero shot is due.

Provenance is no longer free, so decide who records it. The invisible watermark was a feature of the Amazon-built generators, and those are the models on their way out; the current generators are third-party, and nothing says their output carries anything you can read back. If the brand needs to answer “did our system make this?” a year from now, the pipeline has to answer it from its own records: log the model id, the prompt, the seed, the region, and a hash of the bytes at the moment each asset is produced, and keep that ledger next to the asset library. Content credentials and metadata written into the file help, though metadata is the first thing a crop-and-recompress round trip destroys, which is why the ledger and the hash are the part that actually holds up. This is the requirement to raise before legal does, because it changes the pipeline and not just the model call.

The claims-intake pipeline is an understanding job, and the pick is Bedrock Data Automation. The defining feature of the input is that it is mixed: PDFs, photos, voicemails, and video clips arriving together, all needing to become records. That breadth is precisely what BDA is for, one managed API across all four modalities instead of a routing layer that sniffs each upload and dispatches it to a different service. Blueprints let you say what “a claim form” and “a receipt” should yield, so the structure is consistent and the downstream system gets the same fields every time. And because BDA slots into a Knowledge Base as a parser, the same extraction that structures a claim can also make the whole document corpus retrievable, so a support assistant can answer questions grounded in the uploads without a separate ingestion pipeline.

When to walk away from BDA toward a purpose-built service. If the claims stream were actually a single modality with an exacting requirement, the answer flips. A pipeline that is only scanned forms with awkward layouts and needs the strongest possible OCR, query-based field extraction, and signature detection is a Textract job. Audio that needs speaker diarisation, custom vocabulary, and per-channel transcription is a Transcribe job. Content moderation and object detection on a video library is a Rekognition job. BDA earns its place by being one pipeline across many modalities; the purpose-built services earn theirs by being deeper in one. The wrong move is standing up all three single-purpose services to reconstruct what BDA gives you in one call, or forcing a genuinely single-modality precision task through a generalist pipeline.

When the answer is neither, and you want a model to just look. If the requirement is not “extract these fields every time” but “read this and tell me something”, the tool is a multimodal foundation model reasoning over the media in the prompt. Does this uploaded receipt match the policy the customer quoted? Is the damage in this photo consistent with the described incident? That is open-ended judgement, not schema-shaped extraction, and Nova Lite, Nova Pro, or Claude reading the image directly answers it in language. Trying to force that kind of fuzzy question into BDA’s structured output, or into Rekognition’s fixed label set, is fighting the tool; the free-form reasoning of a multimodal model is a different understanding capability, and it is the right one for the one-off, human-shaped question.

A worked example: two uploads, two paths

A single web form now feeds both pipelines, and two items arrive in the same minute. The marketing team submits a brief, “matte black insulated flask, studio lighting, plain background, plus a five-second hero clip”. The claims team’s customer uploads a phone photo of a damaged flask, a PDF claim form, and a ten-second voicemail describing what happened.

The brief goes to generation. Stable Image Core produces the still from the text prompt, the remove-background operation drops the flask onto the clean canvas the brand guidelines want, and the resulting image becomes the first frame Luma Ray 2 animates from. The pipeline writes its own provenance record as it goes, because nothing in the output will tell it later. Nothing here reads media; it all writes it.

The still is a plain synchronous invoke, and the shape of the response is worth noticing.

still = bedrock_runtime.invoke_model(   # bedrock-runtime in us-west-2
    modelId="stability.stable-image-core-v1:1",
    body=json.dumps({
        "prompt": "matte black insulated flask, studio lighting, plain background",
        "aspect_ratio": "16:9",
        "output_format": "png",
        "seed": 7,
    }),
)

result = json.loads(still["body"].read())
if result["finish_reasons"][0] is not None:
    raise RuntimeError(f"filtered: {result['finish_reasons'][0]}")
png = base64.b64decode(result["images"][0])

finish_reasons is the part people miss. A generation the content filter caught comes back as a successful call, and the only signal that you got nothing usable is a non-null entry in that list, so the check belongs in the code rather than in whoever reviews the batch later. The response also hands back seeds, which is how a designer who likes one of the twenty options gets that exact frame again.

The clip is where the invocation shape changes underneath you. A few seconds of video takes minutes to render, so there is no synchronous response to wait on: video generation runs as an asynchronous job. You start it, you poll it, and the MP4 lands in your bucket.

clip = bedrock_runtime.start_async_invoke(
    modelId="luma.ray-v2:0",
    modelInput={
        "prompt": "Slow dolly toward the matte black flask, studio lighting",
        "aspect_ratio": "16:9",
        "duration": "5s",
        "resolution": "720p",
        "loop": True,
        "keyframes": {
            "frame0": {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": base64.b64encode(png).decode(),
                },
            },
        },
    },
    outputDataConfig={"s3OutputDataConfig": {"s3Uri": "s3://brand-assets-generated"}},
)

job = bedrock_runtime.get_async_invoke(invocationArn=clip["invocationArn"])
if job["status"] == "Completed":
    print(job["outputDataConfig"]["s3OutputDataConfig"]["s3Uri"])

Three things in that shape carry over to any async model on Bedrock. start_async_invoke lives on the same bedrock-runtime client as Converse but hands back only an invocation ARN; get_async_invoke (or list_async_invokes) is how you learn what happened to it; and the output never travels through the API at all, it is written to the S3 location you named.

Budget a few minutes for a five-second clip and longer for a nine-second one. That makes the poll interval a patient one, and it means a pipeline of any size wants the job to be a step in a state machine rather than a thread sitting in a request handler. The keyframe is what ties the clip to the still: frame0 takes the image base64-encoded, anywhere from 512 to 4096 pixels on a side, frame1 can pin the last frame the same way, and loop asks for a clip that comes back round to where it started, which is what a looping hero on a product page needs.

The claims upload goes to understanding, and it is mixed media, so it goes to BDA. One call takes the photo, the PDF, and the voicemail: a blueprint pulls the claimant, policy number, and itemised loss from the form; the voicemail comes back as a transcript and a summary; the photo comes back with a caption and detected content. The structured result lands in the downstream claims system as fields, and because BDA is wired as the Knowledge Base parser, the same upload becomes retrievable, so the support assistant can later answer “what did the customer say happened?” from the voicemail transcript. If a supervisor then asks the harder question, “does the damage in the photo match the described incident?”, that last step is not extraction at all; it hands the photo and the transcript to a multimodal model and asks for a judgement. Same intake form, three different tools, chosen by direction first and output shape second.

What’s worth remembering

1. The first cut is not which model, it is which job: generation writes media from a prompt, understanding reads media into structure or an answer, and almost everything else follows from that.
2. Active image generation is the Stability family in us-west-2, Core for volume, Ultra for the shot that has to be perfect, SD3.5 Large for a different look; the editing operations (remove background, inpaint, outpaint, upscale, style transfer) are separate model ids and run in us-east-1 as well.
3. Active video generation is Luma Ray 2 in us-west-2: five or nine seconds, 540p or 720p, optionally keyframed from a still you generated a moment earlier.
4. Nova Canvas and Nova Reel are marked legacy in the catalogue, a legacy model refuses invocation for any account that has not called it in the last thirty days, and both reach end of life on 30 September 2026, so recognise the names in older material without starting anything new on them.
5. Generation families turn over fast, from Titan to Nova to the current stack, so anchor the design to the invocation pattern, a synchronous invoke for a still and an asynchronous job for a clip, rather than to a model id, and read modelLifecycle in the catalogue before you build.
6. Provenance is not free any more: the invisible watermark belonged to the Amazon-built generators, so with the current stack the pipeline has to keep its own record of the model, prompt, seed, and a hash of the bytes.
7. Video generation is asynchronous: start_async_invoke returns a job ARN, get_async_invoke reports its status, and the MP4 lands in your S3 bucket rather than in the response.
8. Bedrock Data Automation is the one managed pipeline across documents, images, audio, and video, configured with blueprints and plugging into a Knowledge Base as the parser; reach for it when the input is mixed media needing consistent fields at volume.
9. Reach for a purpose-built service, Textract, Transcribe, or Rekognition, when the job is one modality that needs deep control, and do not rebuild BDA out of three of them.
10. Reach for a multimodal foundation model when you want open-ended reasoning over an image or document; forcing a fuzzy, one-off question into schema-shaped extraction fights the tool.