Skip to content

perf(windows): build a texture-native D3D11 recording and composition pipeline #173

Open

Description

@petercr

Summary

Replace the Windows system-memory recording path with a texture-native D3D11 pipeline inspired by the architecture used by Cap. Keep captured frames on the GPU through scaling, live composition where required, NV12 conversion, and Media Foundation H.264 submission.

This is the architectural follow-up to #156. Issue #156 proves the encoded bridge, hardware selection, diagnostics, and truthful OpenH264 fallback. This issue removes the remaining GPU-to-CPU readbacks, CPU scene composition, and per-frame Media Foundation uploads that prevent the encoded path from reaching real-time cadence at higher resolutions.

Current evidence

Physical Windows acceptance on win11-x64-i5-8400-gtx1650-super reproduces the performance problem after the recent Windows performance work:

  • 4K30 hardware Media Foundation recording produced 15.37 fps, missed cadence by 48.8%, and stretched duration by 1.95x.
  • 1440p60 hardware Media Foundation recording produced 31.76 fps and missed cadence by 47.1%.
  • Earlier acceptance measured Media Foundation submit p95 between 40 and 52 ms for the system-memory NV12 path.
  • At 1080p30, FFmpeg averaged 92.7% CPU and the Rust backend averaged 62.4%; Electron main averaged only 0.06%. The recording bottleneck is the media graph, not normal renderer or Electron-main work.

The present Windows graph performs several full-frame system-memory operations:

  1. FFmpeg ddagrab captures on the GPU and immediately executes hwdownload,format=bgra.
  2. Raw BGRA frames cross an FFmpeg stdout pipe into the backend.
  3. The Windows compositor renders the scene on the CPU and performs per-pixel RGB-to-I420 conversion.
  4. Different recording and streaming profiles cause a second complete CPU scene render.
  5. The opt-in Media Foundation adapter allocates and copies an input buffer for every I420 frame, with CPU I420-to-NV12 conversion when required.
  6. Windows still defaults to raw YUV/OpenH264 until perf(windows): replace the raw-video bottleneck with a verified encoded path #156 completes its physical fallback proof and default-promotion gate.

A single 4K BGRA frame is 31.6 MiB. At 30 fps, the first raw capture pipe alone approaches 949 MiB/s before composition, preview, encoder upload, and output transport.

Reference architecture

Cap keeps Windows capture frames as ID3D11Texture2D, applies cadence control before conversion or queue work, sends frames through a bounded nonblocking encoder queue, performs scale and NV12 conversion with the D3D11 VideoProcessor, and passes the resulting texture to Media Foundation with MFCreateDXGISurfaceBuffer.

Cap Studio also records screen, camera, microphone, and system audio as separate editable tracks. That allows ordinary record-only sessions to avoid live full-resolution composition entirely.

Reference snapshot: CapSoftware/Cap commit f8fbbc097242156a2e4dad7ac652277fa24a7a54.

Proposed work

Phase 1: texture-native capture and direct recording fast path

  • Add an in-process Windows Graphics Capture source that retains frames as ID3D11Texture2D with scheduled capture timestamps.
  • Apply the requested cadence before scaling, conversion, or channel submission.
  • Use bounded latest-wins or nonblocking queues so encoder pressure drops frames with explicit diagnostics instead of stalling capture.
  • Add a dedicated D3D11/Media Foundation encoder thread.
  • Reuse D3D11 VideoProcessor input and NV12 output textures.
  • Submit NV12 textures through MFCreateDXGISurfaceBuffer; do not upload full CPU I420 frames on the supported hardware path.
  • For record-only sessions that do not require burned-in composition, record screen and camera as separate synchronized tracks and preserve audio/cursor metadata for later editing.
  • Retain the existing composed path when live overlays, burn-in, or streaming require it.

Phase 2: D3D11 live compositor

  • Implement the existing Videorc scene semantics on a Windows GPU compositor: contain/crop, transforms, mirroring, masks, rounded corners, opacity, chroma key, backgrounds, captions, highlights, camera, and screen.
  • Keep source textures on the GPU from capture through composition.
  • Render the scene once for same-profile record and stream sessions.
  • For split profiles, derive the auxiliary output with GPU scaling and only the leg-specific overlay work; do not repeat the entire CPU scene render.
  • Produce reusable NV12 encoder textures with correct BT.709/video-range behavior.
  • Preserve an explicit, diagnosed CPU fallback for unsupported devices.

Phase 3: Windows preview presentation

  • Replace or supplement the CPU-resized uncompressed BMP proof transport with a GPU-backed presentation path suitable for Electron on Windows.
  • Preserve the existing proof-surface identity and lifecycle contracts until a replacement has its own first-frame, liveness, focus, teardown, and acceptance proof.
  • Production PNG transport remains forbidden.

Phase 4: default promotion and cleanup

  • Complete the supported-device and natural-fallback gates from perf(windows): replace the raw-video bottleneck with a verified encoded path #156.
  • Promote the texture-native Media Foundation path to the Windows per-session default only after both proofs pass.
  • Keep raw YUV/OpenH264 as an explicit fallback and developer escape hatch.
  • Remove redundant raw-video stages only after packaged recording, streaming, preview, and fallback acceptance is green.

Diagnostics

Add per-session diagnostics sufficient to prove the actual path rather than inferring it from the output codec:

  • capture backend and source texture format;
  • GPU compositor backend and fallback reason;
  • D3D11 adapter identity;
  • texture frames captured, admitted, dropped, composed, and submitted;
  • active encoder count and per-output frames/bytes;
  • capture-to-compose, compose, GPU conversion, encoder submit, and output-write p95;
  • CPU readback/upload frame and byte counters, which must remain zero on the supported texture-native path;
  • direct-record versus live-composed mode;
  • effective bridge output and exact fallback reason.

Acceptance gates

Run all repository, build, test, smoke, and GitHub commands from Windows PowerShell 7. Do not overlap Cargo operations.

Packaged physical Windows proof must cover:

  • record-only 1080p30/60, 1440p30/60, and 4K30;
  • portrait twins for shipping profiles;
  • screen-only and screen-plus-camera direct-record sessions;
  • live-composed layouts with captions/highlights;
  • same-profile record plus local RTMP;
  • split 4K30 recording plus 1080p30 local RTMP;
  • one healthy and one unavailable stream target;
  • clean stop during encoder activity;
  • a naturally unsupported or failing second Windows device.

The supported-device path must assert:

  • target cadence remains real-time without duration stretch;
  • capture GPU readback frames/bytes are zero;
  • compositor CPU I420 frames are zero for GPU-composed sessions;
  • Media Foundation CPU upload frames/bytes are zero;
  • raw FIFO copied frames are zero;
  • record/stream failure isolation is unchanged;
  • final artifacts pass codec, dimensions, FPS, BT.709/video-range, H.264 level, keyframe cadence, duration, A/V synchronization, and stop-tail checks;
  • preview remains live and correctly identified.

The unsupported-device path must assert the exact fallback reason, claim no texture-native Media Foundation output, remain real-time for currently supported fallback shapes, and pass final-artifact analysis.

STOP conditions

  • Stop if the design requires a second FFmpeg raw-video sidecar or another full-frame CPU staging path on supported hardware.
  • Stop if it weakens record/stream failure isolation, timestamps, color tags, keyframe cadence, A/V synchronization, or clean stop.
  • Stop if the direct-record optimization silently removes requested live overlays or burn-in behavior.
  • Stop if hardware failure is hidden or mislabeled rather than selecting the documented fallback.

Out of scope

  • Rewriting Electron/Tauri or unrelated UI architecture. Current evidence shows media processes dominate recording CPU.
  • Enabling experimental 4K60 legacy-path eligibility.
  • Changing macOS VideoToolbox or Metal behavior.
  • Weakening acceptance gates to make an overloaded path appear successful.

Measurable performance completion criteria

Texture retention alone is foundation work, not a performance result. This issue remains open until a packaged physical-device run proves that the retained texture is consumed without returning to the existing CPU media graph.

A supported-device performance slice must prove all of the following on the same hardware, profile, source, layout, and run duration as its checked-in baseline:

  • the tested product session actually selects the texture-native path; d3d11TextureAvailable=true by itself is insufficient;
  • average recording cadence is at least 95% of the requested FPS and FFmpeg/output speed is at least 0.95x;
  • final duration is within 2% of the scheduled recording window, with no stretched-duration substitute for dropped cadence;
  • 1080p30 reaches at least 28.5 fps, 1440p60 reaches at least 57 fps, and 4K30 reaches at least 28.5 fps on win11-x64-i5-8400-gtx1650-super;
  • capture GPU readback frames/bytes are zero for frames admitted to recording;
  • CPU BGRA-to-I420 compositor frames are zero for direct-record sessions;
  • Media Foundation CPU upload frames/bytes and raw-video FIFO copied frames are zero;
  • preview-only readback, if temporarily required by the Electron proof surface, is independently cadence-limited and cannot block or backpressure recording;
  • backend and FFmpeg per-role CPU/RSS are compared with the pre-change baseline, and the report identifies every remaining full-frame copy;
  • the final artifact still passes dimensions/FPS, BT.709/video-range, H.264 level, keyframe cadence, duration, A/V sync, and stop-tail gates.

No PR may claim a Windows recording performance improvement solely because capture produced or retained a D3D11 texture. The acceptance record must include before/after FPS, speed, duration stretch, CPU/RSS, readback/upload/copy counters, and the effective capture/compositor/encoder path. Opt-in foundation slices may merge, but they do not satisfy or close this issue.

Activity

  1. petercr commented on Jul 30, 2026

    @petercr
    CollaboratorAuthor

    Direct GPU recording implementation is now underway on issue-173-d3d11-direct-record.

    The first exact-eligibility slice is working on the target physical host (win11-x64-i5-8400-gtx1650-super): record-only, Screen Only, matching-aspect, no composed scene/captions/highlights.

    Packaged hardware results (2026-07-30):

    • 1080p30: 303 frames / 10.10 s (~30.0 fps)
    • 1080p60: 489 frames / 8.15 s (~60.0 fps)
    • 1440p60: 485 frames / 8.08 s (~60.0 fps)
    • 4K30: 245 frames / 8.17 s (~30.0 fps)

    Every run asserted:

    • �ncodeBackend=hardware-media-foundation
    • �ncoderBridgeEncodedOutputInputSubtype=NV12-D3D11
    • retained WGC D3D11 texture
    • zero raw-FIFO copied frames
    • positive encoded frames/bytes and zero encoded-output errors
    • final codec, BT.709/video-range, keyframe, FPS, duration, and decode checks

    The implemented hot path is WGC BGRA texture -> D3D11 VideoProcessor NV12 surface pool -> MFCreateDXGISurfaceBuffer -> asynchronous hardware H.264 MFT -> existing MPEG-TS/copy topology. The CPU compositor is not started for this eligible shape, and BMP preview readback is isolated to 10 fps while direct recording is active.

    Other layouts/topologies continue to use the existing compositor/fallback path. Broader eligibility, explicit fallback proof, reviewed CPU/RSS comparisons, and the second-device proof remain open completion work.

  2. petercr commented on Jul 31, 2026

    @petercr
    CollaboratorAuthor

    Camera completion gate added for the Windows performance work:

    • A screen-only result is not sufficient. Physical screen-camera recordings must pass at both 1080p30 and 1080p60.
    • The screen/recording clock must remain at the requested 30/60 FPS even when a physical webcam supplies fewer unique frames (the current C920 is observed near 15 FPS). Camera source FPS must be reported separately; repeated camera frames must not throttle or falsify output cadence.
    • The accepted path must be WGC retained D3D11 screen texture + independently refreshed camera overlay -> D3D11 video processor -> NV12 D3D11 surface -> Media Foundation H.264.
    • Diagnostics must show hardware-media-foundation, effective windows-media-foundation-h264-mpegts, NV12-D3D11, positive encoded frames/bytes, zero raw-FIFO copied frames, zero encoded-output errors, and zero CPU-compositor fallback frames.
    • Windows camera capture must explicitly prefer the advertised MJPEG resolution/rate before exact-rate and negotiated-default fallbacks, because many webcams only expose 720p30 in MJPEG.
    • Finished recordings must still pass codec, dimensions/FPS, color, keyframe, duration, A/V, and clean-stop analysis.

    Current implementation work adds the independently refreshed D3D11 camera overlay, explicit DirectShow MJPEG-first negotiation, focused Rust tests, and maintained packaged 1080p30/60 screen+camera smoke cases. Issue #173 remains open until those physical packaged cases produce accepted diagnostics/artifacts.

  3. petercr commented on Nov 21, 2026

    @petercr
    CollaboratorAuthor

    What still leaves the D3D11 media path on Windows (read from main dbda7f43d, 2026-11-21). Research note from an overnight Windows sweep. I read the code and ran nothing for this list.

    select_windows_d3d11_session (windows_d3d11_session.rs:557 on) names a fallback for each case below. recording.rs:5306-5387 builds its request. A session that takes one of these runs on the CPU compositor bridge, with the reason in Health (windows_d3d11_fallback_diagnostics).

    Fallback code Trigger Notes
    windows-d3d11-media-layout-unsupported CameraOnly / VerticalCameraOnly (windows_d3d11_display_camera_layout, recording.rs:20575-20586) Every camera-only (talking-head) take uses the CPU path. The plan needs a DXGI monitor (screen-missing otherwise), so a camera-only D3D11 plan would need a camera-only canvas with no capture worker.
    windows-d3d11-media-scene-features-unsupported: dual-orientation-simulcast params.simulcast.is_some() (recording.rs:5330-5335) A vertical leg needs its own scene. The compositor also refuses mixed-aspect outputs (windows_d3d11_compositor.rs:776-790).
    …scene-features-unsupported: test-pattern A visible test pattern, a source or a scene layer (windows_d3d11_unmapped_scene_features) Mostly smokes and first runs.
    …scene-features-unsupported: window-source A visible scene Window layer Windows has no window sources yet. The plan for them is #1627 (plan 256).
    windows-d3d11-media-window-capture-unsupported window_id set Same as above.
    windows-d3d11-media-primary/auxiliary-output-unqualified Long side above 1920 or short side above 1080 (windows_d3d11_output_fits_qualified_envelope) #1610 lifts this to 1440p and 4K30.
    windows-d3d11-media-foundation-not-selected The encoder isn't the MF bridge, or the MFT passed only with system-memory input (allows_d3d11_textures) OpenH264 and system-memory MFTs.
    windows-d3d11-media-camera-source-unavailable A screen-camera layout with no live camera frame source

    One more for the "frames never cross to the CPU" goal: the Preview's own capture. When the session falls back, windows_graphics_capture.rs reads every frame back to the CPU (:600-640) for the CPU compositor bridge.

    If this issue is the umbrella, camera-only (the most common shape after screen + camera) and the vertical simulcast leg look like the next two eligibility slices after #1610 and #1622. I haven't measured either on the UHD 620.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions