tile: stop the fullscreen backdrop from masking window translucency

[joshsymonds]

Summary

The fullscreen backdrop is currently painted as a single rect covering the entire tile, including the area under the window. This silently overrides user-set window transparency: a fullscreen kitty with background_opacity 0.85 (or mpv with --alpha=yes, etc.) composites its alpha against the opaque backdrop instead of what's behind the tile, so the user's transparency setting becomes invisible.

This PR clips the backdrop to tile − window, preserving its documented role of covering screen-size padding when the window is smaller than the tile (e.g. fullscreening a fixed-size dialog) while no longer masking the window's own alpha channel. Opaque fullscreen windows render identically — the change is opt-in by content alpha, so no config knob is needed.

Implementation

• New helper backdrop_clip_rects(tile, window, tile_corner_radius) decomposes tile − window into up to 4 axis-aligned rects (left bar, right bar, top middle, bottom middle), each with a CornerRadius whose non-zero corners coincide with tile-outer corners and zero corners butt against the window. Zero-area strips are dropped.
• Steady-state path (fullscreen_progress >= 1.0) emits one SolidColorRenderElement per strip via a new from_buffer_at helper that takes an explicit geometry instead of using the buffer's natural size.
• Animation path (fullscreen_progress < 1.0) emits one BorderRenderElement per strip. The rounded-corner shrink animation works the same; the backdrop just doesn't paint under the window at any frame.
• When an L/R bar is absent (e.g. letterbox-shaped fullscreen), the corresponding top/bottom middle strip inherits the tile's outer corners on the relevant edge, so the rounded outer corners survive the clip.

Test plan

• cargo test -p niri --lib backdrop_clip — 8 geometry tests covering the standard 4-strip decomposition, 2-strip / 3-strip degenerate cases, window-equals-tile, window-outside-tile, and corner-radius routing including the letterbox case.
• cargo +nightly fmt --all -- --check and cargo clippy --all --all-targets clean.
• Visually verified on a real desktop with a translucent kitty fullscreen — wallpaper shows through; opaque windows render identically.

[Uzlkav]

If I understand them correctly, the Wayland protocol linked to in the docs you reference say that you should also draw behind the window itself?

If the fullscreened surface is not opaque, the compositor must make sure that other screen content not part of the same surface tree (made up of subsurfaces, popups or similarly coupled surfaces) are not visible below the fullscreened surface.

[joshsymonds]

@Uzlkav fair point — you're right that the protocol clause covers the non-opaque case too, and a blanket clip diverges from spec. I've updated the PR to make this opt-in: a new clip-fullscreen-backdrop-to-window window rule, off by default. Default behavior matches upstream (single full-tile backdrop, spec-compliant); only windows that explicitly set the rule get the tile-minus-window clip.

The 4-strip decomposition + its 8 geometry tests are unchanged; both render paths (steady-state SolidColor and the un/fullscreen-animation BorderRenderElement) now branch on rules.clip_fullscreen_backdrop_to_window == Some(true).

If people want it they can opt-in; if maintainers absolutely don't want toggles for non-spec Wayland features, no harm no foul, close this PR and I'll just keep it on my local fork. It seems obvious to me that if someone adds alpha to their full screen window they mean it, but maybe that's wrong.

[joshsymonds]

Closing this — keeping it on my fork.

@Uzlkav, thanks for the actual review. Your point on the xdg-shell non-opaque clause was right and made the patch better.