Terminal image passthrough

(feature: terminal-images)

Programs running inside a Terminal pane can draw pictures. TerminalScreen reads the Kitty graphics protocol out of the PTY byte stream, decodes it, and the renderer paints the result over the pane's text.

tui-lipan = { version = "0.30", features = ["terminal-images"] }

cargo run --example terminal_images --features terminal-images

It does not depend on the host terminal

The framework parses the child's graphics commands and retains their source images. It gives each pane separate host image IDs and clips placements to the visible pane and uncovered UI cells. On Kitty hosts, eligible natural-size images are uploaded once and placed as pixel-cropped patches. The host blends their z planes, avoiding pane-side composition and retransmission of unchanged pixels.

Scaled images, mismatched or unmeasured host cell sizes, overlapping images at the same z, and backdrop effects use pixel composition. An overlapping later widget branch also uses composition so opaque text and filled blocks cover the image. Nonpainting Canvas hosts and inactive Animated wrappers preserve native eligibility; their children are checked individually. Unfilled frame borders outside terminal content also preserve native rendering. Overlap checks include ancestor clipping and separately rendered modal and portal subtrees. Transparent overlays containing painted content also select composition, so positive-z host images cannot cover their text. Overlay geometry uses Animated offsets only within its own paint layer, so translations above a portal do not move its separately rendered content. Eligibility checks each portal subtree once in paint order; an eligible terminal inside a portal can still forward native patches. The fallback emits individual image cells in overlapping branches; batched placeholder rows and ordinary placements cannot cover a later widget whose anchor leaves the image anchor intact. Opaque text backgrounds exclude the pixels they cover. Composed image trees use full tree paints when another pane reports damage. Native image trees can repaint text rows independently. A source upload restores every visible region of that image, including regions outside the text damage. Overlay changes recalculate image occlusion across its complete visible area. If a partial paint cannot establish that area, the existing native image remains, and the runner schedules a full paint to present the replacement. Terminals already using composition continue to draw their damaged image rows through the composed path. Composed placements outside the repaint regions remain visible with the same host IDs. A repaint inside a placement preserves the untouched parts, and an upload that replaces its host image restores all retained rows. Scrolling a UI region containing terminal graphics uses a full paint. Each source retains its upload state until the source and pending host output are released, independently of render-cache eviction. When a complete paint leaves a composed image without placements, it is uploaded again the next time it is placed, because the host may have evicted it. Large sources that cannot use file or shared-memory transport stay on the asynchronous composed path; only inline uploads up to 128 KiB are prepared directly. Other host protocols still use the same encoding path as the Image widget: iTerm2, sixel, or half-blocks. A pane in a plain xterm shows pictures.

For a controlled comparison or to diagnose host graphics behavior, launch the app with TUI_LIPAN_TERMINAL_IMAGES=composed to disable native patch placements. Captures always compose retained source images, so PNG output and replay do not depend on the host's image storage.

Run cargo run --example terminal_patch_layers --features terminal-images to compare image-only, text-overlay, and filled-block scenes on a real host. The overlays leave the image anchor uncovered. Compare with the composed environment setting and resize the host in both modes. Host resize notifications do not pause image rendering by default. The optional TUI_LIPAN_IMAGE_RESIZE_PAUSE_MS setting applies to both terminal pictures and Image widgets; leave it unset or set it to 0 to avoid a fixed delay after resizing. See the Image environment settings. Screen erase commands used during resize invalidate native host image uploads. The next paint restores unchanged pictures with their existing image and placement IDs, even when the producer has stopped sending updates. TUI_LIPAN_PATCH_LAYER=t, b, m or p selects an initial overlay in that example. The m key opens a backdrop-free transparent modal over a positive-z image; its text must remain visible in both native and composed modes. The p key hosts the terminal inside a root portal; its image should still use native placements when eligible.

Setting the cell size

A terminal deals in cells; a program drawing a picture needs pixels. It learns the conversion from the PTY's TIOCGWINSZ pixel fields or by asking with CSI 14 t. Give both ends the same answer:

use tui_lipan::prelude::*;

let cell = host_cell_size();

let mut screen = TerminalScreen::new(rows, cols, scrollback);
screen.set_cell_size(cell);

let config = TerminalPtyConfig::default().cell_size(cell);

ManagedTerminal does this for you. A raw TerminalScreen defaults to a 10x20 cell, which is a guess: a mismatch shows up as images that overlap the text below them or leave a gap, because the child reserved a different number of rows than the pane drew.

Pass the cell size to later resizes too, with TerminalPty::resize_with_cell_size; plain resize keeps the last one it was given.

Images placed at the cursor honor the X/Y pixel offsets within that cell. Natural-size images keep their pixel dimensions rather than stretching to fill the final cell. Explicit c and r size an ordinary placement to that cell box, stretching each axis. With only one axis specified, the other preserves the source aspect ratio. Per-cell senders such as Yazi can fill each cell even when their source crops differ by one pixel. With ordinary placements on a Kitty host, cropped placements of one source share a single upload of that source, and each crop is placed from it. A lone crop uses that upload only when the source is at most four times the crop's area; a smaller crop of a large image uploads just the crop. Overlapping images blend in pixel space within each exact z plane, so a small patch preserves the rest of the cell. On Kitty hosts, mixed-z and negative-z images use ordinary placements that leave text cells intact. Negative z values remain below text; values below INT32_MIN/2 also remain below non-default cell backgrounds. The backend keeps unchanged placements and deletes regions that disappear, including when the pane closes. Composed single-plane images at non-negative z use Unicode placeholders. Native patches use ordinary pixel-cropped placements at every z. Unchanged regions reuse cached pixels and encodings. Composed tiles retain their upload identity when their resulting pixels stay the same, even if another part of the source image changes. Cursor placement capacity scales with the terminal grid; stale-screen and off-screen placements are evicted before visible ones. Natural-size RGB and RGBA patches copy contiguous pixel rows; opaque RGBA rows use a bulk copy, while transparent rows retain source-over blending. Scaled images use the resampling path.

Quiet raw uploads decode once per stored image, shared by the screen and its snapshots. The upload buffer is consumed by that decode rather than retained beside a second copy of the pixels. Replay preserves both uploads that have not been decoded and images already drawn.

Once a screen and its last snapshot are dropped, its queued and cached terminal encodings are released, and results from encodes still in flight are discarded. Retired native patches also delete their host image data after a successful output flush. Shared-memory upload slots expire after a second without use; the idle encoder sweeps the pool every five seconds. A slot the host has not finished reading is retained until safe to unmap. Releasing image buffers does not guarantee an immediate drop in process RSS, since the heap allocator can keep freed memory for reuse.

Captures use the same composition. Terminal CapturedImage values set fill_cell_box so changing the screenshot font cannot open gaps between tiles. When constructing CapturedImage with a struct literal, add fill_cell_box: false for the previous aspect-preserving behavior; CapturedImage::new already uses that default.

Where images live

Two ways an image gets placed

At the cursor (a=T without U=1) is what icat and friends do: the image lands where the cursor is, the cursor moves past it, and the placement is anchored to that scrollback line.

Through placeholder cells (a=T,U=1) is what terminal UI toolkits do, ratatui-image among them: the transmission draws nothing, and the program then writes the placeholder character U+10EEEE into the cells the image should cover, tagging them with the image id (in the cell's foreground colour) and the position inside the image (in combining marks). Those placements are read back off the grid on every snapshot rather than stored, so they scroll, clear, and reflow with the cells holding them, for free. A cell may leave any of that out and inherit it from its left-hand neighbour — including the high byte of the image id — which is what keeps a row of placeholders down to a single escape sequence.

Cursor placements

A cursor placement is anchored to an absolute scrollback line, in the same space as OSC 133 semantic marks. That is what makes it behave like the text it was drawn against:

Event	What happens
Output scrolls	The image scrolls with it, cropped row by row as it leaves the viewport
Scrolling back	It reappears at the line it was drawn on
A line falls out of scrollback	The image's remaining rows stay; it goes once its last row is evicted
Screen erased (ED 2)	The image scrolls into history with the text above it, as the screen does
Scrollback erased (ED 3, as clear sends)	Images in the erased history go with it
Alternate screen	Placements made there are dropped when the child leaves it
Column resize	Kept, unless the change actually rewraps text — then the anchor stops naming what it named, and placements are dropped
RIS / TerminalScreen::reset	Everything is cleared

Each event applies to the placements that exist when it happens, so the outcome does not depend on how the stream was split into process_bytes calls. clear followed by an image leaves the image on screen whether the two arrive in one write or two, which matters to a multiplexer that relays a pane's output in larger chunks than the program wrote it.

A placeholder placement needs none of that bookkeeping: it is the text, so it does whatever the cells do, and it is gone the moment they are.

Each placement carries the image_id its transmission used. A renderer must key its encoding on that and not on the pixels alone: a host drawing through Kitty identifies a placement by the id of its encoding, so two placements sharing one encoding are a single placement to it — and two copies of one picture would collapse, the second silently not drawn.

TerminalRenderSnapshot::images carries the placements overlapping the visible rows, back to front by Kitty z-index. Rows and columns are viewport-relative, and a cursor placement's may be negative when the image starts above or to the left of the pane.

A transmission may name the cell box it is meant to fill (c= columns, r= rows), and a virtual placement is then mapped through that box rather than through the cell grid: each placeholder cell takes the slice of the image its position in the box calls for. This is what lets a child draw at whatever resolution it likes — twice the cell size on a HiDPI screen, say — and still have the picture scaled into the cells it asked for instead of cropped to its top-left corner. The box belongs to the image id, so a sender animating by re-transmitting under the same id declares it once.

Memory

Decoded pixels are capped per screen, 96 MiB by default, and evicted least-recently-used once the cap is passed — placed images included, since leaving old plots on screen must not pin memory. One image larger than the whole budget is kept anyway; that is a budget set too low, not a picture that should silently fail to appear.

screen.set_image_budget(16 * 1024 * 1024);

Server-side terminal mirrors that forward the original PTY stream to another rendering client can avoid decoding every image twice:

screen.set_image_storage_enabled(false);

That mode still validates command metadata, answers protocol queries, and applies image-implied cursor movement. It retains dimensions only; render snapshots contain no images. Raw RGB/RGBA payloads also skip base64 decoding, while PNG payloads are still decoded far enough to obtain their dimensions.

screen.has_images() is whether any of those pixels are still retained: true after a child has transmitted an image that has not been deleted or evicted, including when it has scrolled out of view. Visible placements this frame are snapshot.images. A host that needs the class of pane whose image layer does not follow a widget shrink or fade should look at the flag, not at the process in front of the PTY.

Payloads are bounded before decoding as well: 64 MiB per transmission, 16384 pixels per axis. A single unterminated APC sequence is still bounded to 32 MiB; larger inline uploads must be chunked. Named-file frames and their inline replay share the transmission limit, so a retained high-DPI frame survives reconnect.

Out-of-band payloads

A child redrawing a full window every frame can leave its pixels somewhere and name them in the escape sequence instead of base64-encoding several megabytes through the PTY: t=f a file it leaves in place, t=t a temporary file, t=s a POSIX shared-memory object. This is the difference between a hundred bytes on the wire and a compress/encode/decode/inflate round trip per frame, and it is what lets a program in a pane animate at the host's own frame rate.

The last two are consumed by reading - the file is deleted, the object unlinked - so exactly one reader may claim them. Readers of an unrelayed stream shared by several clients decline consuming media and keep only re-readable t=f:

screen.set_image_media_policy(GraphicsMediaPolicy::SHARED);

A canonical server can instead accept consuming media and relay them to every client, as described below.

GraphicsMediaPolicy::NONE declines all three, so every picture arrives inline. Refusals are the protocol's own EBADF/ENOTSUPP reports, which is what makes a child fall back to t=d rather than lose the image. A t=t path must be temporary or say in its own name what it is for, t=s names are resolved as shared-memory objects rather than paths, and both reads are capped by the same transmission budget as an inline payload.

A path only means something on the machine that wrote it, so a pane attached from elsewhere should decline: a remote client reading /tmp/... from its own filesystem is the one failure mode here that is not a clean error.

Relay out-of-band frames

A server that records a screen's output for other parsers can read out-of-band sources itself and hand those parsers something they can read in place of the source's name:

screen.set_image_media_policy(GraphicsMediaPolicy::default());
screen.set_image_file_storage_enabled(true);
screen.set_image_relay_policy(GraphicsRelayPolicy::NamedAsFiles);
screen.set_output_transaction_recording(true);

GraphicsRelayPolicy::NamedAsFiles relays successfully snapshotted raw t=f, t=t and t=s transmissions as t=f commands naming immutable server files. Enable file storage and transaction recording before processing input. Consuming sources are read once by the server. Other consuming payloads and queries relay inline; local re-readable sources without an eligible snapshot keep their producer representation. Retained replay always carries pixels.

Each TerminalOutputTransaction owns its rewritten files in media. Every TerminalOutputMedia binds a TerminalImmutableFile to the exact complete command_range within bytes. Preserve the handles through synchronized updates, queue merging and coalescing, and retain them until each reader acquires or irrevocably discards that output. Concatenating transactions requires rebasing each command range by the preceding byte length. Never split a bound command. Dropping the last handle unlinks the file; there is no timed retention window. Reading only bytes and dropping its transaction does not provide a file-lifetime guarantee.

On a trusted local transport, call TerminalImmutableFile::acquire(path, byte_len, format) while the sender still owns the file. This takes a private hard link, validates its regular-file type, exact length and raw pixel format, and copies into private storage if linking is unavailable. Keep every imported handle with its bound bytes, including deferred or orphan output. Call TerminalOutputMedia::validate_command(bytes) before acknowledging acquisition. The sender may then release its handle. Acknowledgements permit deletion, never inode reuse.

To apply acquired resources, call screen.prepare_immutable_media_input(bytes, bindings), then screen.process_prepared_media_input(prepared). These use the same screen and input-revision checks as worker preparation. Ordinary process_bytes() never treats a producer path as an immutable shared resource. Remote transports must not open server paths on the local filesystem. Acquisition failures require replay or connection termination, rather than skipping an image.

Applications constructing TerminalOutputTransaction literals must supply media: Vec::new() for output without files. Replace the unreleased ConsumingAsFiles policy with NamedAsFiles and carry its transaction handles through the transport until acquisition is acknowledged.

GraphicsRelayPolicy::Consuming replaces t=t and t=s with inline pixels, including capability queries. Choose All while an attachment cannot open the server's files; this also replaces t=f with inline pixels. None preserves source references. Use one policy for every reader of a shared output stream. Inline raw pixels are compressed unless already compressed or sent as PNG. Failed relayed acquisitions emit no source reference; the authoritative screen answers the child. Chunked out-of-band transmissions are rejected. Direct t=d chunking remains supported.

For large images, call screen.prepare_media_input(bytes) before ordinary processing. It returns None for input that needs no media work. Otherwise, send the returned TerminalMediaWork to a bounded worker queue and call its run() there. This acquires and validates the source, keeps raw pixels in a file snapshot when the screen stores images as files and decodes otherwise, and builds the relayed output. The work item stops after one named image even when the input contains a burst of references.

On completion, read prepared.consumed_bytes(), then pass the result to screen.process_prepared_media_input(prepared). Feed the original input's remaining tail afterward, using the same preparation path. Keep that stream's input budget charged while work is pending. Do not feed later bytes, resize or reset that screen until its result is committed. Other screens can continue. A result for another screen or one superseded by later input or a resize/reset is returned unapplied. The owning screen uses its ordinary parser and output classifier, so text, cursor queries, synchronized frames and replacement dependencies stay in order.

If attachments change during preparation, compare prepared.relay_policy() to the desired policy. Run prepared.with_relay_policy(policy) on the worker before committing. It reuses acquired pixels and never opens a consumed source again. Apply the matching relay policy to the screen before committing the result.

Retain streamed frames in files

screen.set_image_file_storage_enabled(true) opts into immutable file snapshots for quiet raw t=f uploads, and for every raw out-of-band upload prepared with prepare_media_input. Each snapshot copies a validated regular file, or the consumed temporary file or shared-memory object, into an application-owned temporary file. A consuming source is read straight into storage reserved before it is opened, without an intermediate pixel buffer. Producer rewrites and deletion cannot change a stored frame. Snapshots and replay share the file instead of allocating another pixel buffer; capture and composition read pixels on demand. The normal image budget still counts the raw pixel size. Snapshot files also have a process-wide 256 MiB limit. If a snapshot cannot be made, the ordinary in-memory path remains available. Each screen may keep up to two retired files, capped at 32 MiB and its image budget, to reuse allocated pages for equal-sized frames. These spares count against the same 256 MiB limit and are released on screen reset, file-storage disable, or screen drop. When the process-wide limit blocks a new snapshot, admission evicts idle spares across screens and retries. Only retired allocations are eligible; live readers prevent reuse. A snapshot handed to a host or named in relayed output is never recycled, even after its handoff path is removed: the host or a reader may still have the file open. Snapshot and import names live in private process-owned temporary directories. Handles keep their directory alive until its files are released. On Unix, startup cleanup removes only private, current-user directories whose owner process is confirmed dead. Other platforms preserve stale directories when liveness cannot be established conservatively.

A cell-aligned frame can be uploaded directly to a local Kitty host that passed the shared-memory probe. The host receives an immutable temporary hard link with t=t and removes it after reading. At most 32 outstanding links, totaling 64 MiB, are admitted. Unread links expire after 10 seconds; idle encoder workers check them every five seconds. This path does not map frame pixels into the application and does not allocate a shared-memory upload slot. Cropping, overlapping patches, modal dimming, captures and other host protocols retain their existing pixel rendering behavior.

This is useful for a producer sending complete frames under one image ID. Patches that overlap or land between cells still need composition, and therefore need decoded pixels on the rendering client. File snapshots reduce heap allocations, but their pages still use the filesystem cache. Compare total memory and frame pacing as well as application RSS when evaluating this mode.

Kitty uploads remain pending until the native backend flushes their escape sequence from the final cell diff without failed or discarded output. If buffering sends a temporary-file or shared-memory name before a later write fails or is discarded, the retry creates a fresh name from retained immutable pixels. The terminal consuming the first name cannot invalidate the retry. An offscreen capture, clipped frame or startup effect cannot consume the upload. If an effect removes its original cell, the backend moves the upload before the first surviving image row in the final diff. No placeholder for a new image reaches the host ahead of its upload, and a relocated upload is sent only once. Uploads staged for a frame survive encoding-cache eviction through the backend write. A subsequent paint releases staging references from discarded frames. File and shared-memory handoff lifetimes begin after successful flush, so a static image can be presented after startup or client reattachment without requiring another frame from its producer.

Out the other side, to the host

The same reasoning applies to what this framework writes to the terminal it is running in, and the cost is the same one twice over: a pane whose child sends a full window of pixels every frame would otherwise have them deflated, base64-encoded, chunked and written down stdout, and the host would undo all of it.

So a frame goes into a POSIX shared-memory object and the escape sequence carries its name. Nothing configures this. At startup the host is asked - with a t=s query it can only answer by reading a real object - and only a terminal that answers OK is handed frames that way; everything else keeps the inline path, including a terminal reached through tmux, whose reader is not the terminal. Objects are unlinked by the host as it reads them, and by this process for any frame the host was never told about.

The question is only put to a host that could answer it, and that is judged from the environment before anything is written. A terminal on the other end of an ssh connection is never asked: it cannot resolve a name in this machine's memory whatever protocol it speaks, and TERM survives the hop, so it would otherwise look local and capable. Neither is a terminal that is not recognizably one of the implementations - kitty, Ghostty, WezTerm and Konsole, by TERM, TERM_PROGRAM, or the variable each one sets.

That matters because the query is an APC string sent before the alternate screen is entered. A terminal is supposed to consume an APC it does not implement, and most do; the ones that do not print it as text, and the mess lands in the scrollback and stays there. Not asking is cheap - an unasked host keeps the inline path, which is what an unsupported host does anyway.

For a host judged wrongly, TUI_LIPAN_GRAPHICS_SHM settles it: 1 asks a terminal that implements the protocol under a TERM this does not know, and 0 never asks, for a local terminal that mangles the question. The host must still answer OK either way - the variable controls the asking, not the result.

On Linux the objects are pooled, which is why /dev/shm holds a handful of tui-lipan-pool-<pid>-* names for as long as a pane is drawing. A fresh object per frame is a fresh allocation per frame - every byte written lands on a page that does not exist yet - so each slot is created once and written again every frame, and what the host is handed is a fresh hard link to it. That link is the part the host unlinks, and its disappearance is the only signal that the host is finished: a slot whose link is still there is never reused, and a frame that finds every slot busy allocates its own object exactly as it used to. Pool objects left behind by a run that was killed are removed by the next run, which can tell because the name carries the process that made it.

Windows has no POSIX shared-memory namespace, so frames there are always inline.

What is supported

Key	Support
a=t, a=T, a=p, a=d, a=q	Transmit, transmit-and-display, display, delete, query
t=d	Direct transmission, chunked with m=1
Payload	Base64, with or without = padding (Kitty's icat sends it unpadded)
t=f, t=t, t=s	Out-of-band transmission: a file left in place, a temporary file, a POSIX shared-memory object
f=24, f=32, f=100	RGB, RGBA, PNG
o=z	zlib-compressed payloads
s, v	Pixel dimensions for the raw formats
i, I, p	Image id, image number, placement id
x, y, w, h	Source rectangle to display
c, r	Explicit placement size in cells
z	Stacking order against the text layer
C=1	Leave the cursor where it is
q=1, q=2	Suppress success reports / all reports; a later chunk's q= applies to the rest of its run
d=	a/A, i/I, n/N, c/C, z/Z, p/P, x/X, y/Y
U=1	Virtual placements shown through Unicode placeholder cells

Not supported, and answered with the protocol's own ENOTSUPP report so a child that probes first gets a clean answer rather than silence:

• The protocol's animation frames (a=a, a=f, a=c). A sender that animates by re-transmitting under the same image id — which is what ratatui-image does for GIFs — works regardless: the new pixels replace the old and the placeholders keep pointing at them.
• Relative placements (parent references).

A payload that is not base64 is answered with EINVAL, under the command's id and q= like any other report, and nothing in it is acted on. A bad chunk, or one that takes a run past the size cap (EFBIG), abandons the whole m=1 run it belongs to: the error goes out once, under the run's id, and the run's remaining chunks up to its m=0 are consumed without a reply.

Sixel input from the child is a separate protocol and is not read.

Under a modal backdrop

A root-portal overlay's backdrop (Modal::backdrop_style, CommandPalette::backdrop_style) recolors the cells behind it. An image is not cells, and the host has no per-image opacity, so the renderer recolors the image's pixels instead, before encoding them: the backdrop's fill, dim_by, tint_by, and background transform, applied exactly as they apply to a cell background, so a picture dims to the same color as the cells beside it. Alpha is kept. Foreground transforms do not apply, because a picture reads as surface, not as text.

Layers inside the tree that recolor cells reach the pixels too, in the order the renderer applies them. Each runs the renderer's own cell code on the pixel's color, so the match is exact:

• An EffectScope whose effects depend only on a cell's own color: dim_by, lighten_by, tint_by, transform_bg, VisualEffect::Monochrome, and VisualEffect::PaletteQuantize, including under Channels and a rectangular Clipped. EffectScope::cells_only() opts a scope out: its effects recolor the cells and leave every image under it at its own pixels.
• An Animated opacity toward a color, opacity_target, unless it is opacity_fg_only.
• A Canvas or Center stacked over the image whose style dims, tints, or transforms without painting a background. That is how a Local-scope modal's backdrop draws. Its dialog hides the image under it, as a root dialog does.

A layer reaches an image only if it applies after the image draws: an EffectScope or Animated around it, or a surface stacked above it. A layer inside an overlay reaches only the images in that overlay.

• Only covered cells dim. Root backdrops span the whole viewport, so in practice that is the whole visible image; the part under the dialog itself is hidden as before. A layer in the tree dims the cells it covers after its clip.
• One encode per image on open. The dimmed pixels are cached as their own variant beside the undimmed ones, and a Kitty stream transmits them under a separate image id. Closing the dialog over pixels that have not changed switches back to the undimmed image the host already holds. Pixels that changed while it was open, such as a live plot, have only dimmed encodes, so closing encodes the current frame undimmed.
• Never the wrong dimming. While an image re-encodes, its previous frame normally stays on screen. When the only previous frame is dimmed the other way, because the dialog just opened or closed, the renderer encodes the new frame during that paint instead, for every protocol. That costs one synchronous encode per image at each open or close, and none on later frames.
• Live pictures pay per frame. A picture that keeps changing under a dialog, such as a browser or a video, is recolored on every new frame. Dims, tints, and fills cost a few milliseconds for a full 1080p frame. transform_bg(ColorTransform::Elevate(_)), Monochrome, and PaletteQuantize mix a color's channels, so a photographic frame under them costs more; past the first 16,384 colors, each is computed from the nearest color at 64 levels per channel, at most two levels away. cargo bench --bench image_backdrop --features terminal-images measures both.
• Full strength at once. Images take the backdrop at its full strength from its first frame, and an Animated fade at the opacity it ends at, from the frame it starts. Following a fade would re-encode every image under it on every frame of the fade.
• Half blocks are left to the backdrop. They are cells, which the backdrop already recolors.
• Kitty image ids survive. A placeholder cell that carries its image id in its foreground keeps that foreground under a backdrop, an EffectScope, an Animated fade, or a surface in a layer that draws images; only the pixels dim.

Known limits

• Some layers leave the pixels alone. Effects that depend on where a cell is or on time (Scanlines, Gradient, RainbowWave, RetroCrt, Ripple), ContrastPolicy, custom effects, mask clips, and an Animated opacity without opacity_target recolor cells without reaching the pixels under them. So does an EffectScope that wraps an overlay's portal rather than sitting inside the overlay.
• A painted surface covers an image. A Canvas or Center whose style paints a background, even a translucent one, replaces the cells it covers, so the image stops showing there instead of dimming.
• Reattaching to a session loses images drawn before the attach. export_replay_bytes is a text replay stream and does not re-emit image payloads.
• A partly visible image is cropped, not scaled. That is what makes scrolling look right, but it also means an image wider than its pane shows its left part rather than shrinking to fit.
• Encoding is asynchronous. The first image has no pixels until its encode lands. Later updates keep the displayed frame in place while the replacement is encoded. Its transmission is emitted before its native placeholders in one paint, so the host switches only after it has the pixels.

Captures

A capture holds images beside its cells rather than in them. TestBackend::capture_frame(), a UiSnapshot, and TerminalScreen::capture_frame() all fill CapturedFrame::images: one CapturedImage per image, with its RGBA pixels, the cells it is laid out over, and which of those cells still show it.

• What covers an image hides it. A UI capture records each image as the renderer draws it, then checks each of its cells after everything else has drawn. An overlay, a border, a toast, or a pane above takes the cells it covers, exactly as on the host. CapturedImage::shows(x, y) answers per cell.
• Negative z planes stay below text. PNG captures draw these images before glyphs, so image pixels remain visible between glyph strokes, including wide glyphs and antialiased text. Planes below INT32_MIN / 2 also stay behind non-default cell backgrounds. Text and ANSI captures preserve the text in these cells and approximate images in blank cells with half blocks.
• The cells get a half-block stand-in. Image cells hold ▀ in the colors of their top and bottom halves, so plain_text() marks where an image is, to_ansi_text() shows a coarse version in any terminal, and cell assertions read colors straight out of the grid. A cell the image leaves fully transparent keeps what it held. CapturedImage::underlying_cells preserves the original glyphs and styles for PNG output; backgrounds records the backgrounds before the stand-in.
• A PNG draws the pixels. to_png() scales each image into its cells at the PNG's own cell size, keeping its aspect ratio from the top-left corner as a terminal does, and draws only the cells it still shows in. Partially transparent upper z planes keep lower images visible in the same cell, so their pixels blend in draw order. CapturedImage::z_index controls whether the image draws before or after text. Transparent pixels reveal lower images, glyphs, and the recorded background. Half-block output remains a coarse approximation of the stack.
• One image on its own. CapturedImage::to_png() encodes just rgba, at width x height with its alpha, for a serializer that reports images beside the cells. A terminal capture has already cropped an image that runs past the viewport, so these are the pixels inside it.
• A backdrop dims the pixels. Under an open modal backdrop a capture records the recolored pixels, so rgba, the half-block stand-ins, and to_png() all show the image dimmed like the cells around it. So do the layers in the tree that reach pixels; the rest leave the image at full brightness where it still shows.
• A capture does not wait for an encode. It never encodes for a host, so the first capture after an image arrives already has it.
• TerminalScreen::capture_frame() crops each placement to the viewport the way the renderer does, and sizes its half blocks with cell_size().

Testing

Render through TestBackend and read capture_frame(): its images for pixels and visibility, its cells for the half-block colors. See tests/terminal_images_render.rs.