Benchmark · run 28 July 2026

Ask for the same thing three times

Eight architectural briefs, three public models, three runs each. Not one of the 72 images repeats another: floor plans vary a third less than photorealistic scenes, and a tightly written brief steadies the framing.

Open run.json
Results 24 triples · 72 images
By mode
Floor plans
Exterior
Masterplan
Interiors
VarianceMean pixel difference between two runs. Higher: more different
Floor plans 34.77
Exterior 52.73
Masterplan 52.84
Interiors 55.62
Composition overlapEdge-map overlap, 0 to 1. 1 would be the same framing
Floor plans 0.077
Exterior 0.073
Masterplan 0.061
Interiors 0.075
By model
Gemini 3.1 Flash Lite ImageGoogle, called through the Vercel AI Gateway
Gemini 3.1 Flash ImageGoogle, called through the Vercel AI Gateway
FLUX.2 proBlack Forest Labs, called through fal
VarianceMean pixel difference between two runs. Higher: more different
Gemini 3.1 Flash Lite Image 50.8
Gemini 3.1 Flash Image 50.36
FLUX.2 pro 45.8
Composition overlapEdge-map overlap, 0 to 1. 1 would be the same framing
Gemini 3.1 Flash Lite Image 0.072
Gemini 3.1 Flash Image 0.070
FLUX.2 pro 0.073
By brief
Open brief
Brief with constraints
VarianceMean pixel difference between two runs. Higher: more different
Open brief 51.33
Brief with constraints 46.65
Composition overlapEdge-map overlap, 0 to 1. 1 would be the same framing
Open brief 0.066
Brief with constraints 0.078
Across all 24 brief-and-model combinations the composition overlap between two runs averages 0.072 out of 1.0. The by-model figure is indicative rather than exact: the models delivered different pixel grids, so frames were resampled to a common long edge of 640 px first.
What we found

Nothing here is a repeat.

  1. No model repeats itself. Composition overlap between two runs of the same brief averages 0.07 out of 1.0: you do not get the same picture back with small differences, you get a different picture. If a frame is worth keeping, keep it; you cannot re-roll your way back to it.
  2. The mode matters more than the model. Floor plans vary a third less than photorealistic work (34.77 against 52.73 for exteriors, 55.62 for interiors and 52.84 for masterplans). Line drawings converge on a similar answer; a furnished, weathered scene has far more room to differ.
  3. Writing a tighter brief is the one lever that works. Briefs carrying six to eight explicit requirements came back at 46.65 against 51.33 for open one-liners, with composition agreement 0.078 against 0.066.
  4. The gap between models is smaller than the gap between a floor plan and an interior (45.8 to 50.8 against 34.8 to 55.6). Pick a model for its character; none of them offers repeatability.
See it Three runs of one brief, one model
Steadiest triple in the runFloor plan, programme given · FLUX.2 provariance 27.61 · composition 0.089
Floor plan, programme given, FLUX.2 pro, run 1
Run 1
Floor plan, programme given, FLUX.2 pro, run 2
Run 2
Floor plan, programme given, FLUX.2 pro, run 3
Run 3
Wildest triple in the runInterior, open brief · Gemini 3.1 Flash Lite Imagevariance 63.48 · composition 0.069
Interior, open brief, Gemini 3.1 Flash Lite Image, run 1
Run 1
Interior, open brief, Gemini 3.1 Flash Lite Image, run 2
Run 2
Interior, open brief, Gemini 3.1 Flash Lite Image, run 3
Run 3
Every triple All 72 images
Gemini 3.1 Flash Lite Image
Exterior, open briefvariance 58.92 · composition 0.059
Exterior, open brief, Gemini 3.1 Flash Lite Image, run 1
Run 1
Exterior, open brief, Gemini 3.1 Flash Lite Image, run 2
Run 2
Exterior, open brief, Gemini 3.1 Flash Lite Image, run 3
Run 3
Exterior, eight constraintsvariance 49.86 · composition 0.069
Exterior, eight constraints, Gemini 3.1 Flash Lite Image, run 1
Run 1
Exterior, eight constraints, Gemini 3.1 Flash Lite Image, run 2
Run 2
Exterior, eight constraints, Gemini 3.1 Flash Lite Image, run 3
Run 3
Interior, open briefvariance 63.48 · composition 0.069
Interior, open brief, Gemini 3.1 Flash Lite Image, run 1
Run 1
Interior, open brief, Gemini 3.1 Flash Lite Image, run 2
Run 2
Interior, open brief, Gemini 3.1 Flash Lite Image, run 3
Run 3
Interior, material disciplinevariance 52.67 · composition 0.086
Interior, material discipline, Gemini 3.1 Flash Lite Image, run 1
Run 1
Interior, material discipline, Gemini 3.1 Flash Lite Image, run 2
Run 2
Interior, material discipline, Gemini 3.1 Flash Lite Image, run 3
Run 3
Floor plan, open briefvariance 32.59 · composition 0.072
Floor plan, open brief, Gemini 3.1 Flash Lite Image, run 1
Run 1
Floor plan, open brief, Gemini 3.1 Flash Lite Image, run 2
Run 2
Floor plan, open brief, Gemini 3.1 Flash Lite Image, run 3
Run 3
Floor plan, programme givenvariance 32.3 · composition 0.094
Floor plan, programme given, Gemini 3.1 Flash Lite Image, run 1
Run 1
Floor plan, programme given, Gemini 3.1 Flash Lite Image, run 2
Run 2
Floor plan, programme given, Gemini 3.1 Flash Lite Image, run 3
Run 3
Masterplan, open briefvariance 60.72 · composition 0.055
Masterplan, open brief, Gemini 3.1 Flash Lite Image, run 1
Run 1
Masterplan, open brief, Gemini 3.1 Flash Lite Image, run 2
Run 2
Masterplan, open brief, Gemini 3.1 Flash Lite Image, run 3
Run 3
Masterplan, site rulesvariance 55.84 · composition 0.072
Masterplan, site rules, Gemini 3.1 Flash Lite Image, run 1
Run 1
Masterplan, site rules, Gemini 3.1 Flash Lite Image, run 2
Run 2
Masterplan, site rules, Gemini 3.1 Flash Lite Image, run 3
Run 3
Gemini 3.1 Flash Image
Exterior, open briefvariance 51.12 · composition 0.073
Exterior, open brief, Gemini 3.1 Flash Image, run 1
Run 1
Exterior, open brief, Gemini 3.1 Flash Image, run 2
Run 2
Exterior, open brief, Gemini 3.1 Flash Image, run 3
Run 3
Exterior, eight constraintsvariance 51.92 · composition 0.089
Exterior, eight constraints, Gemini 3.1 Flash Image, run 1
Run 1
Exterior, eight constraints, Gemini 3.1 Flash Image, run 2
Run 2
Exterior, eight constraints, Gemini 3.1 Flash Image, run 3
Run 3
Interior, open briefvariance 50.58 · composition 0.065
Interior, open brief, Gemini 3.1 Flash Image, run 1
Run 1
Interior, open brief, Gemini 3.1 Flash Image, run 2
Run 2
Interior, open brief, Gemini 3.1 Flash Image, run 3
Run 3
Interior, material disciplinevariance 53.27 · composition 0.081
Interior, material discipline, Gemini 3.1 Flash Image, run 1
Run 1
Interior, material discipline, Gemini 3.1 Flash Image, run 2
Run 2
Interior, material discipline, Gemini 3.1 Flash Image, run 3
Run 3
Floor plan, open briefvariance 41.26 · composition 0.063
Floor plan, open brief, Gemini 3.1 Flash Image, run 1
Run 1
Floor plan, open brief, Gemini 3.1 Flash Image, run 2
Run 2
Floor plan, open brief, Gemini 3.1 Flash Image, run 3
Run 3
Floor plan, programme givenvariance 38.35 · composition 0.073
Floor plan, programme given, Gemini 3.1 Flash Image, run 1
Run 1
Floor plan, programme given, Gemini 3.1 Flash Image, run 2
Run 2
Floor plan, programme given, Gemini 3.1 Flash Image, run 3
Run 3
Masterplan, open briefvariance 61.63 · composition 0.061
Masterplan, open brief, Gemini 3.1 Flash Image, run 1
Run 1
Masterplan, open brief, Gemini 3.1 Flash Image, run 2
Run 2
Masterplan, open brief, Gemini 3.1 Flash Image, run 3
Run 3
Masterplan, site rulesvariance 54.78 · composition 0.059
Masterplan, site rules, Gemini 3.1 Flash Image, run 1
Run 1
Masterplan, site rules, Gemini 3.1 Flash Image, run 2
Run 2
Masterplan, site rules, Gemini 3.1 Flash Image, run 3
Run 3
FLUX.2 pro
Exterior, open briefvariance 56.58 · composition 0.066
Exterior, open brief, FLUX.2 pro, run 1
Run 1
Exterior, open brief, FLUX.2 pro, run 2
Run 2
Exterior, open brief, FLUX.2 pro, run 3
Run 3
Exterior, eight constraintsvariance 47.96 · composition 0.084
Exterior, eight constraints, FLUX.2 pro, run 1
Run 1
Exterior, eight constraints, FLUX.2 pro, run 2
Run 2
Exterior, eight constraints, FLUX.2 pro, run 3
Run 3
Interior, open briefvariance 59.16 · composition 0.068
Interior, open brief, FLUX.2 pro, run 1
Run 1
Interior, open brief, FLUX.2 pro, run 2
Run 2
Interior, open brief, FLUX.2 pro, run 3
Run 3
Interior, material disciplinevariance 54.57 · composition 0.083
Interior, material discipline, FLUX.2 pro, run 1
Run 1
Interior, material discipline, FLUX.2 pro, run 2
Run 2
Interior, material discipline, FLUX.2 pro, run 3
Run 3
Floor plan, open briefvariance 36.49 · composition 0.070
Floor plan, open brief, FLUX.2 pro, run 1
Run 1
Floor plan, open brief, FLUX.2 pro, run 2
Run 2
Floor plan, open brief, FLUX.2 pro, run 3
Run 3
Floor plan, programme givenvariance 27.61 · composition 0.089
Floor plan, programme given, FLUX.2 pro, run 1
Run 1
Floor plan, programme given, FLUX.2 pro, run 2
Run 2
Floor plan, programme given, FLUX.2 pro, run 3
Run 3
Masterplan, open briefvariance 43.45 · composition 0.065
Masterplan, open brief, FLUX.2 pro, run 1
Run 1
Masterplan, open brief, FLUX.2 pro, run 2
Run 2
Masterplan, open brief, FLUX.2 pro, run 3
Run 3
Masterplan, site rulesvariance 40.61 · composition 0.056
Masterplan, site rules, FLUX.2 pro, run 1
Run 1
Masterplan, site rules, FLUX.2 pro, run 2
Run 2
Masterplan, site rules, FLUX.2 pro, run 3
Run 3
How it was run
Question

We generated the same eight briefs three times each, through three public models, and measured how much the repeats differ from one another. The question is narrow and practical: if you generate something and it is nearly right, can you ask again and expect a near-miss variation, or something else entirely?

Models and access
  • Gemini 3.1 Flash Lite Image — Google, called through the Vercel AI Gatewayai-gateway.vercel.sh/v1/chat/completions · google/gemini-3.1-flash-lite-image
  • Gemini 3.1 Flash Image — Google, called through the Vercel AI Gatewayai-gateway.vercel.sh/v1/chat/completions · google/gemini-3.1-flash-image
  • FLUX.2 pro — Black Forest Labs, called through falfal.run/fal-ai/flux-2-pro · flux-2-pro, image_size preset
Measures
For every brief-and-model triple all three pairs of runs were compared two ways: the mean absolute pixel difference (variance) and the overlap of their edge maps (composition). Both are computed by script; nothing on this page is judged by eye.
System prompt
Every request carried the same fixed system prompt (architectural visualisation, photographic style, follow the user’s text). It is not shipped in the kit; the harness reads one from a file you supply.
Reproduce
Measurements, scripts and briefs are published under CC BY 4.0. The frames are outputs of third-party models.
Scope

A dated snapshot of three public models, executed on 28 July 2026. Providers update models under the same name; figures are left exactly as the run produced them. Providers update models without renaming them, so the date, endpoint and version of everything tested are stated on the page.

The eight briefs, verbatim
exterior
A two-storey family house on a gentle slope, timber and white render, large windows facing the valley, late afternoon light.Exterior, open brief
16:9
exterior
A single-storey courtyard house in a dry mediterranean landscape. Flat roof with a deep overhang on the south side. Walls in warm lime render, window frames in dark bronze. One olive tree in the courtyard. Gravel ground, no lawn. Morning light from the left. No cars, no people. Photographed at eye level from the entrance side.Exterior, eight constraints
16:9
interiors
A calm living room with a plaster fireplace, oak floor, linen sofa, and tall windows with sheer curtains.Interior, open brief
3:2
interiors
A kitchen in a converted stone barn. Exactly three materials: pale oak, honed black stone, and lime plaster. A single long island with no upper cabinets. One pendant light over the island. Original timber roof trusses visible. Overcast daylight from a window on the right. No decoration on the surfaces.Interior, material discipline
3:2
floor-plan
A clean architectural floor plan of a three-bedroom single-storey house, black lines on white, room labels, furniture indicated.Floor plan, open brief
4:3
floor-plan
An architectural floor plan, black line drawing on white. Programme: entrance hall, open kitchen and dining, separate living room, three bedrooms, one bathroom plus one WC, utility room, and a covered terrace on the south side. Single storey, roughly rectangular footprint. Label every room. Show wall thickness and door swings. No colour, no shading.Floor plan, programme given
4:3
masterplan
An aerial masterplan of a small residential cluster of eight houses around a shared green, with parking at the edge.Masterplan, open brief
1:1
masterplan
An aerial site plan of a hillside development. Twelve dwellings in four terraced rows following the contours. One access road entering from the north-west, no through traffic. Communal parking in two pockets at the road edge. A pedestrian path linking all rows to a viewpoint at the south end. Existing trees kept along the eastern boundary. Top-down orthographic view.Masterplan, site rules
1:1
Questions
Can I regenerate an AI image and get the same result again?

Not from the same prompt with these models. Across 24 brief-and-model combinations, composition overlap between repeat runs was 0.07 out of 1.0: effectively a different photograph each time, not a variation of the first. A generated image you like is a one-off; save it and branch from it rather than trying to reproduce it from the prompt.

Do more detailed prompts make AI image output more consistent?

Measurably, yes. It is the one thing that reliably narrows the spread. Briefs carrying six to eight explicit requirements came back at 46.65 variance against 51.33 for open one-line briefs, with composition agreement 0.078 against 0.066. They still do not make two runs the same picture.

Which AI images are most reproducible: plans or renderings?

Plans, by a wide margin. Floor-plan briefs varied at 34.77 against 52.73 for exteriors, 55.62 for interiors and 52.84 for masterplans. A black-line plan has a narrow space of plausible answers, so repeat runs converge; a photorealistic scene has to re-decide materials, vegetation and weather each time.

How was repeatability measured?

Eight briefs across four modes were each generated three times by each of three models, 72 images. For every brief-and-model triple we compared all three pairs two ways: the mean absolute pixel difference, and the intersection-over-union of their edge maps, which measures whether the composition is the same regardless of colour. Frames were resampled to a common long edge first so resolution is not a factor. The script and the raw run are published.

More runs
Three image models, one set of briefsThe same frames, a different question: speed, adherence, aspect ratio.
Sequential editing: what eight edits do to one imageThree editing engines, eight edits in a row.