Repository navigation
Description
Summary
Replace the Windows system-memory recording path with a texture-native D3D11 pipeline inspired by the architecture used by Cap. Keep captured frames on the GPU through scaling, live composition where required, NV12 conversion, and Media Foundation H.264 submission.
This is the architectural follow-up to #156. Issue #156 proves the encoded bridge, hardware selection, diagnostics, and truthful OpenH264 fallback. This issue removes the remaining GPU-to-CPU readbacks, CPU scene composition, and per-frame Media Foundation uploads that prevent the encoded path from reaching real-time cadence at higher resolutions.
Current evidence
Physical Windows acceptance on win11-x64-i5-8400-gtx1650-super reproduces the performance problem after the recent Windows performance work:
- 4K30 hardware Media Foundation recording produced 15.37 fps, missed cadence by 48.8%, and stretched duration by 1.95x.
- 1440p60 hardware Media Foundation recording produced 31.76 fps and missed cadence by 47.1%.
- Earlier acceptance measured Media Foundation submit p95 between 40 and 52 ms for the system-memory NV12 path.
- At 1080p30, FFmpeg averaged 92.7% CPU and the Rust backend averaged 62.4%; Electron main averaged only 0.06%. The recording bottleneck is the media graph, not normal renderer or Electron-main work.
The present Windows graph performs several full-frame system-memory operations:
- FFmpeg ddagrab captures on the GPU and immediately executes hwdownload,format=bgra.
- Raw BGRA frames cross an FFmpeg stdout pipe into the backend.
- The Windows compositor renders the scene on the CPU and performs per-pixel RGB-to-I420 conversion.
- Different recording and streaming profiles cause a second complete CPU scene render.
- The opt-in Media Foundation adapter allocates and copies an input buffer for every I420 frame, with CPU I420-to-NV12 conversion when required.
- Windows still defaults to raw YUV/OpenH264 until perf(windows): replace the raw-video bottleneck with a verified encoded path #156 completes its physical fallback proof and default-promotion gate.
A single 4K BGRA frame is 31.6 MiB. At 30 fps, the first raw capture pipe alone approaches 949 MiB/s before composition, preview, encoder upload, and output transport.
Reference architecture
Cap keeps Windows capture frames as ID3D11Texture2D, applies cadence control before conversion or queue work, sends frames through a bounded nonblocking encoder queue, performs scale and NV12 conversion with the D3D11 VideoProcessor, and passes the resulting texture to Media Foundation with MFCreateDXGISurfaceBuffer.
Cap Studio also records screen, camera, microphone, and system audio as separate editable tracks. That allows ordinary record-only sessions to avoid live full-resolution composition entirely.
Reference snapshot: CapSoftware/Cap commit f8fbbc097242156a2e4dad7ac652277fa24a7a54.
Proposed work
Phase 1: texture-native capture and direct recording fast path
- Add an in-process Windows Graphics Capture source that retains frames as ID3D11Texture2D with scheduled capture timestamps.
- Apply the requested cadence before scaling, conversion, or channel submission.
- Use bounded latest-wins or nonblocking queues so encoder pressure drops frames with explicit diagnostics instead of stalling capture.
- Add a dedicated D3D11/Media Foundation encoder thread.
- Reuse D3D11 VideoProcessor input and NV12 output textures.
- Submit NV12 textures through MFCreateDXGISurfaceBuffer; do not upload full CPU I420 frames on the supported hardware path.
- For record-only sessions that do not require burned-in composition, record screen and camera as separate synchronized tracks and preserve audio/cursor metadata for later editing.
- Retain the existing composed path when live overlays, burn-in, or streaming require it.
Phase 2: D3D11 live compositor
- Implement the existing Videorc scene semantics on a Windows GPU compositor: contain/crop, transforms, mirroring, masks, rounded corners, opacity, chroma key, backgrounds, captions, highlights, camera, and screen.
- Keep source textures on the GPU from capture through composition.
- Render the scene once for same-profile record and stream sessions.
- For split profiles, derive the auxiliary output with GPU scaling and only the leg-specific overlay work; do not repeat the entire CPU scene render.
- Produce reusable NV12 encoder textures with correct BT.709/video-range behavior.
- Preserve an explicit, diagnosed CPU fallback for unsupported devices.
Phase 3: Windows preview presentation
- Replace or supplement the CPU-resized uncompressed BMP proof transport with a GPU-backed presentation path suitable for Electron on Windows.
- Preserve the existing proof-surface identity and lifecycle contracts until a replacement has its own first-frame, liveness, focus, teardown, and acceptance proof.
- Production PNG transport remains forbidden.
Phase 4: default promotion and cleanup
- Complete the supported-device and natural-fallback gates from perf(windows): replace the raw-video bottleneck with a verified encoded path #156.
- Promote the texture-native Media Foundation path to the Windows per-session default only after both proofs pass.
- Keep raw YUV/OpenH264 as an explicit fallback and developer escape hatch.
- Remove redundant raw-video stages only after packaged recording, streaming, preview, and fallback acceptance is green.
Diagnostics
Add per-session diagnostics sufficient to prove the actual path rather than inferring it from the output codec:
- capture backend and source texture format;
- GPU compositor backend and fallback reason;
- D3D11 adapter identity;
- texture frames captured, admitted, dropped, composed, and submitted;
- active encoder count and per-output frames/bytes;
- capture-to-compose, compose, GPU conversion, encoder submit, and output-write p95;
- CPU readback/upload frame and byte counters, which must remain zero on the supported texture-native path;
- direct-record versus live-composed mode;
- effective bridge output and exact fallback reason.
Acceptance gates
Run all repository, build, test, smoke, and GitHub commands from Windows PowerShell 7. Do not overlap Cargo operations.
Packaged physical Windows proof must cover:
- record-only 1080p30/60, 1440p30/60, and 4K30;
- portrait twins for shipping profiles;
- screen-only and screen-plus-camera direct-record sessions;
- live-composed layouts with captions/highlights;
- same-profile record plus local RTMP;
- split 4K30 recording plus 1080p30 local RTMP;
- one healthy and one unavailable stream target;
- clean stop during encoder activity;
- a naturally unsupported or failing second Windows device.
The supported-device path must assert:
- target cadence remains real-time without duration stretch;
- capture GPU readback frames/bytes are zero;
- compositor CPU I420 frames are zero for GPU-composed sessions;
- Media Foundation CPU upload frames/bytes are zero;
- raw FIFO copied frames are zero;
- record/stream failure isolation is unchanged;
- final artifacts pass codec, dimensions, FPS, BT.709/video-range, H.264 level, keyframe cadence, duration, A/V synchronization, and stop-tail checks;
- preview remains live and correctly identified.
The unsupported-device path must assert the exact fallback reason, claim no texture-native Media Foundation output, remain real-time for currently supported fallback shapes, and pass final-artifact analysis.
STOP conditions
- Stop if the design requires a second FFmpeg raw-video sidecar or another full-frame CPU staging path on supported hardware.
- Stop if it weakens record/stream failure isolation, timestamps, color tags, keyframe cadence, A/V synchronization, or clean stop.
- Stop if the direct-record optimization silently removes requested live overlays or burn-in behavior.
- Stop if hardware failure is hidden or mislabeled rather than selecting the documented fallback.
Out of scope
- Rewriting Electron/Tauri or unrelated UI architecture. Current evidence shows media processes dominate recording CPU.
- Enabling experimental 4K60 legacy-path eligibility.
- Changing macOS VideoToolbox or Metal behavior.
- Weakening acceptance gates to make an overloaded path appear successful.
Measurable performance completion criteria
Texture retention alone is foundation work, not a performance result. This issue remains open until a packaged physical-device run proves that the retained texture is consumed without returning to the existing CPU media graph.
A supported-device performance slice must prove all of the following on the same hardware, profile, source, layout, and run duration as its checked-in baseline:
- the tested product session actually selects the texture-native path; d3d11TextureAvailable=true by itself is insufficient;
- average recording cadence is at least 95% of the requested FPS and FFmpeg/output speed is at least 0.95x;
- final duration is within 2% of the scheduled recording window, with no stretched-duration substitute for dropped cadence;
- 1080p30 reaches at least 28.5 fps, 1440p60 reaches at least 57 fps, and 4K30 reaches at least 28.5 fps on win11-x64-i5-8400-gtx1650-super;
- capture GPU readback frames/bytes are zero for frames admitted to recording;
- CPU BGRA-to-I420 compositor frames are zero for direct-record sessions;
- Media Foundation CPU upload frames/bytes and raw-video FIFO copied frames are zero;
- preview-only readback, if temporarily required by the Electron proof surface, is independently cadence-limited and cannot block or backpressure recording;
- backend and FFmpeg per-role CPU/RSS are compared with the pre-change baseline, and the report identifies every remaining full-frame copy;
- the final artifact still passes dimensions/FPS, BT.709/video-range, H.264 level, keyframe cadence, duration, A/V sync, and stop-tail gates.
No PR may claim a Windows recording performance improvement solely because capture produced or retained a D3D11 texture. The acceptance record must include before/after FPS, speed, duration stretch, CPU/RSS, readback/upload/copy counters, and the effective capture/compositor/encoder path. Opt-in foundation slices may merge, but they do not satisfy or close this issue.
Activity
petercr commentedon Jul 30, 2026
Direct GPU recording implementation is now underway on issue-173-d3d11-direct-record.
The first exact-eligibility slice is working on the target physical host (win11-x64-i5-8400-gtx1650-super): record-only, Screen Only, matching-aspect, no composed scene/captions/highlights.
Packaged hardware results (2026-07-30):
Every run asserted:
The implemented hot path is WGC BGRA texture -> D3D11 VideoProcessor NV12 surface pool -> MFCreateDXGISurfaceBuffer -> asynchronous hardware H.264 MFT -> existing MPEG-TS/copy topology. The CPU compositor is not started for this eligible shape, and BMP preview readback is isolated to 10 fps while direct recording is active.
Other layouts/topologies continue to use the existing compositor/fallback path. Broader eligibility, explicit fallback proof, reviewed CPU/RSS comparisons, and the second-device proof remain open completion work.
petercr commentedon Jul 31, 2026
Camera completion gate added for the Windows performance work:
Current implementation work adds the independently refreshed D3D11 camera overlay, explicit DirectShow MJPEG-first negotiation, focused Rust tests, and maintained packaged 1080p30/60 screen+camera smoke cases. Issue #173 remains open until those physical packaged cases produce accepted diagnostics/artifacts.
test(windows): name the renamed level 5.1 selector tests in the D3D11…
docs(acceptance): 1440p and 4K30 on D3D11 on an Intel UHD 620 laptop (#…
petercr commentedon Nov 21, 2026
What still leaves the D3D11 media path on Windows (read from
maindbda7f43d, 2026-11-21). Research note from an overnight Windows sweep. I read the code and ran nothing for this list.select_windows_d3d11_session(windows_d3d11_session.rs:557on) names a fallback for each case below.recording.rs:5306-5387builds its request. A session that takes one of these runs on the CPU compositor bridge, with the reason in Health (windows_d3d11_fallback_diagnostics).windows-d3d11-media-layout-unsupportedCameraOnly/VerticalCameraOnly(windows_d3d11_display_camera_layout,recording.rs:20575-20586)screen-missingotherwise), so a camera-only D3D11 plan would need a camera-only canvas with no capture worker.windows-d3d11-media-scene-features-unsupported: dual-orientation-simulcastparams.simulcast.is_some()(recording.rs:5330-5335)windows_d3d11_compositor.rs:776-790).…scene-features-unsupported: test-patternwindows_d3d11_unmapped_scene_features)…scene-features-unsupported: window-sourceWindowlayerwindows-d3d11-media-window-capture-unsupportedwindow_idsetwindows-d3d11-media-primary/auxiliary-output-unqualifiedwindows_d3d11_output_fits_qualified_envelope)windows-d3d11-media-foundation-not-selectedallows_d3d11_textures)windows-d3d11-media-camera-source-unavailableOne more for the "frames never cross to the CPU" goal: the Preview's own capture. When the session falls back,
windows_graphics_capture.rsreads every frame back to the CPU (:600-640) for the CPU compositor bridge.If this issue is the umbrella, camera-only (the most common shape after screen + camera) and the vertical simulcast leg look like the next two eligibility slices after #1610 and #1622. I haven't measured either on the UHD 620.