Reject unmasked client WebSocket frames (RFC 6455 §5.1)

[ianegordon]

Problem

The server currently processes unmasked client-to-server WebSocket frames as if they were valid. RFC 6455 §5.1:

"a client MUST mask all frames that it sends to the server. ... The server MUST close the connection upon receiving a frame that is not masked. In this case, a server MAY send a Close frame with a status code of 1002 (protocol error)"

The frame decoder read the mask and unmasked the payload, but discarded the mask — so no downstream code could tell a masked frame from an unmasked one.

Change

Masking rules are enforced at the transport boundary in HTTPConnection.switchToWebSocket, so they hold for every WSHandler (the built-in MessageFrameWSHandler and custom implementations alike):

• Inbound: a frame without a mask fails the connection — a single close frame with status 1002 is written and the connection is torn down, even if a custom handler suppresses input-stream errors (§7.1.7: "An endpoint SHOULD send a Close frame with an appropriate status code before closing the underlying connection"). Masks are stripped before frames reach handlers, so handlers can safely echo frames.
• Outbound: frames are always sent with the mask cleared (§5.1: "A server MUST NOT mask any frames that it sends to the client.").
• WSFrameEncoder.decodeFrame now preserves the decoded mask on the returned frame (payload remains stored unmasked) so the boundary can detect the violation.

No public API changes; handlers continue to observe frames with mask == nil exactly as before. The close frame is written only after the server-output task has been cancelled and drained, so it can never interleave with an in-flight write (ThrowingTaskGroup.waitForAll() stores the first error without cancelling siblings, so the coordination waits on next(), cancels, drains, then writes).

Tests

Four integration tests in HTTPConnectionTests drive a real socket pair:

• unmasked client frame → 1002 close + connection failure (default handler)
• same, with a custom raw handler that swallows input errors and never finishes its output
• masked client frames are delivered to handlers unmasked, with the payload correctly decoded
• a deliberately masked server frame leaves the wire unmasked

Plus a decode round-trip assertion pinning mask preservation. Full suite passes (456 tests).

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.18182% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 92.96%. Comparing base (ac24c58) to head (fd03cd0).

Files with missing lines	Patch %	Lines
FlyingFox/Sources/HTTPConnection.swift	98.03%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #232      +/-   ##
==========================================
+ Coverage   92.83%   92.96%   +0.12%     
==========================================
  Files          71       71              
  Lines        3743     3795      +52     
==========================================
+ Hits         3475     3528      +53     
+ Misses        268      267       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[swhitty]

I'm wondering whether HTTPConnection.swift is the best place for this validation.

Presently, decoding errors from WSFrameEncoder.decodeFrame(from:) and protocol validation errors from incoming client frames are surfaced through the AsyncThrowingStream received by WSHandler. The handler is responsible for terminating its outgoing server-frame stream and may optionally send a Close frame first.

It seems simpler to perform the incoming mask validation in WSFrameEncoder, perhaps by adding a server-specific decoding method:

static func decodeClientFrame(
    from bytes: some AsyncBufferedSequence<UInt8>
) async throws -> WSFrame {
    let frame = try await decodeFrame(from: bytes)
    guard frame.mask != nil else {
        throw Error("Incoming client frames must be masked")
    }
    return frame
}

This requires decodeFrame(from:) to preserve the decoded mask, as you have in this PR. The client-frame stream could then use something like decodeClientFrame(from:), and the WSHandler will receive the validation error through its existing input stream.

MessageFrameWSHandler, which is the higher-level handler I expect most users to use, should already catch this error, emit a protocol-error Close frame, and terminate its outgoing frame stream. This appears to avoid the violation side-channel and task-group coordination currently being added to HTTPConnection.

[ianegordon]

Hmmmmmmmmmm..... let me give some thought and check on any related TODOs.
I'll take another pass.