Merge pull request from GHSA-q36x-r5x4-h4q6

Motivation

The HTTP2FrameDecoder is a complex object that was written early in the
development of swift-nio-http2. Its logical flow is complex, and it
hasn't been meaningfully rewritten in quite some time, so it's difficult
to work with and understand.

Annoyingly, some bugs have crept in over the years. Because of the
structure of the code it can be quite difficult to understand how the
parser actually works, and fixing a given issue can be difficult.

This patch aims to produce a substantial change to the HTTP2FrameDecoder
to make it easier to understand and maintain in the long run.

Modifications

This patch provides a complete rewrite of HTTP2FrameDecoder. It doesn't
do this by having a ground-up rewrite: instead, it's more like a
renovation, with the general scaffolding kept. The rewrite was performed
incrementally, keeping the existing test suite passing and writing new
tests when necessary.

The following major changes were made:

1. New states and edges were added to the state machine to handle
   padding.

   Prior to this change, padding was handled as part of frame payload
   decoding. This is not totally unreasonable, but it dispersed padding
   management widely and made it easy to have bugs slip in. This patch
   replaces this with a smaller set of locations.

   Padding is now handled in two distinct ways. For HEADERS and
   PUSH_PROMISE frames, trailing padding is still stripped as part of
   frame payload decode, but it's done so generically, and the padding
   bytes are never exposed to the individual frame parser. For DATA,
   there is a new state to handle trailing padding removal, which
   simplifies the highly complex logic around synthesised data frames.

   For all frames, the leading padding byte is handled by a new
   dedicated state which is used unconditionally, instead of attempting
   to opportunistically strip it. This simplifies the code flow.

   As a side benefit, this change means we can now accurately report the
   padding used on HEADERS and PUSH_PROMISE frames, even when they are
   part of a CONTINUATION sequence.

2. The synthesised DATA frame logic has been substantially reworked.

   With the removal of the padding logic from the state, we now know
   that so long as we have either got a byte of data to emit _or_ the
   DATA frame is zero length, we will always emit a frame. This has made
   it simpler to understand the control flow when synthesising DATA
   frames.

3. The monolithic state switch has been refactored into per-state
   methods.

   This helps manage the amount of state that each method can see, as
   well as to logically split them up. In addition, it allows us to
   recast state transformations as (fairly) pure functions.

   Additionally, this allowed the larger methods to be refactored with
   smaller helpers that are more obviously correct.

4. The frame payload parsers have been rewritten.

   The main goal here was to remove preflight length checks and unsafe
   code. The preflight length checks cause trouble when they disagree
   with the parsing code, so we now rely on the parsing code being
   correct with regard to length.

   Relatedly, we previously had two separate places where we
   communicated length: a frame header length and a ByteBuffer length.
   This was unnecessary duplication of information, so we instead use a
   ByteBuffer slice to manage the length. This ensures that we cannot
   over-parse a message.

   Finally, in places that used unsafe code or separate integer reads,
   we have refactored to stop using that unsafe code and to use combined
   integer reads.

5. Extraneous serialization code has been extracted.

   The HTTP2FrameEncoder was unnecessarily in this file, which took a
   large file and made it larger. I moved this out.

Result

The resulting parser is clearer and safer. Complex logic has been broken
out into smaller methods with less access to global data. The code
should be generally clearer.

Author: Lukasa

FuzzTesting/FailCases/clusterfuzz-testcase-minimized-ServerFuzzer-release-6530582404792320

@@ -21,7 +21,7 @@ let package = Package(
         .library(name: "NIOHTTP2", targets: ["NIOHTTP2"]),
     ],
     dependencies: [
-        .package(url: "https://github.com/apple/swift-nio.git", from: "2.32.0")
+        .package(url: "https://github.com/apple/swift-nio.git", from: "2.35.0")
     ],
     targets: [
         .target(

Package.swift

@@ -152,8 +152,8 @@ public final class NIOHTTP2Handler: ChannelDuplexHandler {
     }
 
     public func channelRead(context: ChannelHandlerContext, data: NIOAny) {
-        var data = self.unwrapInboundIn(data)
-        self.frameDecoder.append(bytes: &data)
+        let data = self.unwrapInboundIn(data)
+        self.frameDecoder.append(bytes: data)
 
         // Before we go in here we need to deliver any pending user events. This is because
         // we may have been called re-entrantly.

Sources/NIOHTTP2/HTTP2ChannelHandler.swift

@@ -1438,6 +1438,13 @@ internal enum InternalError: Error {
     case attemptedToCreateStream
 
     case codecError(code: HTTP2ErrorCode)
+
+    // Used to record that an impossible situation occured. Crashes in debug mode, errors in
+    // release mode.
+    static func impossibleSituation(file: StaticString = #file, line: UInt = #line) -> InternalError {
+        assertionFailure(file: file, line: line)
+        return .codecError(code: .internalError)
+    }
 }
 
 extension InternalError: Hashable { }

Sources/NIOHTTP2/HTTP2Error.swift

@@ -0,0 +1,250 @@
+//===----------------------------------------------------------------------===//
+//
+// This source file is part of the SwiftNIO open source project
+//
+// Copyright (c) 2017-2022 Apple Inc. and the SwiftNIO project authors
+// Licensed under Apache License v2.0
+//
+// See LICENSE.txt for license information
+// See CONTRIBUTORS.txt for the list of SwiftNIO project authors
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+//===----------------------------------------------------------------------===//
+
+import NIOCore
+import NIOHPACK
+
+struct HTTP2FrameEncoder {
+    var headerEncoder: HPACKEncoder
+
+    // RFC 7540 § 6.5.2 puts the initial value of SETTINGS_MAX_FRAME_SIZE at 2**14 octets
+    var maxFrameSize: UInt32 = 1<<14
+
+    init(allocator: ByteBufferAllocator) {
+        self.headerEncoder = HPACKEncoder(allocator: allocator)
+    }
+
+    /// Encodes the frame and optionally returns one or more blobs of data
+    /// ready for the system.
+    ///
+    /// Returned data blobs would include anything of potentially flexible
+    /// length, such as DATA payloads, header fragments in HEADERS or PUSH_PROMISE
+    /// frames, and so on. This is to avoid manually copying chunks of data which
+    /// we could just enqueue separately in sequence on the channel. Generally, if
+    /// we have a byte buffer somewhere, we will return that separately rather than
+    /// copy it into another buffer, with the corresponding allocation overhead.
+    ///
+    /// - Parameters:
+    ///   - frame: The frame to encode.
+    ///   - buf: Destination buffer for the encoded frame.
+    /// - Returns: An array containing zero or more additional buffers to send, in
+    ///            order. These may contain data frames' payload bytes, encoded
+    ///            header fragments, etc.
+    /// - Throws: Errors returned from HPACK encoder.
+    mutating func encode(frame: HTTP2Frame, to buf: inout ByteBuffer) throws -> IOData? {
+        // note our starting point
+        let start = buf.writerIndex
+
+//      +-----------------------------------------------+
+//      |                 Length (24)                   |
+//      +---------------+---------------+---------------+
+//      |   Type (8)    |   Flags (8)   |
+//      +-+-------------+---------------+-------------------------------+
+//      |R|                 Stream Identifier (31)                      |
+//      +=+=============================================================+
+//      |                   Frame Payload (0...)                      ...
+//      +---------------------------------------------------------------+
+
+        // skip 24-bit length for now, we'll fill that in later
+        buf.moveWriterIndex(forwardBy: 3)
+
+        // 8-bit type
+        buf.writeInteger(frame.payload.code)
+
+        // skip the 8 bit flags for now, we'll fill it in later as well.
+        let flagsIndex = buf.writerIndex
+        var flags = FrameFlags()
+        buf.moveWriterIndex(forwardBy: 1)
+
+        // 32-bit stream identifier -- ensuring the top bit is empty
+        buf.writeInteger(Int32(frame.streamID))
+
+        // frame payload follows, which depends on the frame type itself
+        let payloadStart = buf.writerIndex
+        let extraFrameData: IOData?
+        let payloadSize: Int
+
+        switch frame.payload {
+        case .data(let dataContent):
+            if dataContent.paddingBytes != nil {
+                // we don't support sending padded frames just now
+                throw NIOHTTP2Errors.unsupported(info: "Padding is not supported on sent frames at this time")
+            }
+
+            if dataContent.endStream {
+                flags.insert(.endStream)
+            }
+            extraFrameData = dataContent.data
+            payloadSize = dataContent.data.readableBytes
+
+        case .headers(let headerData):
+            if headerData.paddingBytes != nil {
+                // we don't support sending padded frames just now
+                throw NIOHTTP2Errors.unsupported(info: "Padding is not supported on sent frames at this time")
+            }
+
+            flags.insert(.endHeaders)
+            if headerData.endStream {
+                flags.insert(.endStream)
+            }
+
+            if let priority = headerData.priorityData {
+                flags.insert(.priority)
+                var dependencyRaw = UInt32(priority.dependency)
+                if priority.exclusive {
+                    dependencyRaw |= 0x8000_0000
+                }
+                buf.writeInteger(dependencyRaw)
+                buf.writeInteger(priority.weight)
+            }
+
+            try self.headerEncoder.encode(headers: headerData.headers, to: &buf)
+            payloadSize = buf.writerIndex - payloadStart
+            extraFrameData = nil
+
+        case .priority(let priorityData):
+            var raw = UInt32(priorityData.dependency)
+            if priorityData.exclusive {
+                raw |= 0x8000_0000
+            }
+            buf.writeInteger(raw)
+            buf.writeInteger(priorityData.weight)
+
+            extraFrameData = nil
+            payloadSize = 5
+
+        case .rstStream(let errcode):
+            buf.writeInteger(UInt32(errcode.networkCode))
+
+            payloadSize = 4
+            extraFrameData = nil
+
+        case .settings(.settings(let settings)):
+            for setting in settings {
+                buf.writeInteger(setting.parameter.networkRepresentation)
+                buf.writeInteger(setting._value)
+            }
+
+            payloadSize = settings.count * 6
+            extraFrameData = nil
+
+        case .settings(.ack):
+            payloadSize = 0
+            extraFrameData = nil
+            flags.insert(.ack)
+
+        case .pushPromise(let pushPromiseData):
+            if pushPromiseData.paddingBytes != nil {
+                // we don't support sending padded frames just now
+                throw NIOHTTP2Errors.unsupported(info: "Padding is not supported on sent frames at this time")
+            }
+
+            let streamVal: UInt32 = UInt32(pushPromiseData.pushedStreamID)
+            buf.writeInteger(streamVal)
+
+            try self.headerEncoder.encode(headers: pushPromiseData.headers, to: &buf)
+
+            payloadSize = buf.writerIndex - payloadStart
+            extraFrameData = nil
+            flags.insert(.endHeaders)
+
+        case .ping(let pingData, let ack):
+            withUnsafeBytes(of: pingData.bytes) { ptr -> Void in
+                _ = buf.writeBytes(ptr)
+            }
+
+            if ack {
+                flags.insert(.ack)
+            }
+
+            payloadSize = 8
+            extraFrameData = nil
+
+        case .goAway(let lastStreamID, let errorCode, let opaqueData):
+            let streamVal: UInt32 = UInt32(lastStreamID) & ~0x8000_0000
+            buf.writeInteger(streamVal)
+            buf.writeInteger(UInt32(errorCode.networkCode))
+
+            if let data = opaqueData {
+                payloadSize = data.readableBytes + 8
+                extraFrameData = .byteBuffer(data)
+            } else {
+                payloadSize = 8
+                extraFrameData = nil
+            }
+
+        case .windowUpdate(let size):
+            buf.writeInteger(UInt32(size) & ~0x8000_0000)
+            payloadSize = 4
+            extraFrameData = nil
+
+        case .alternativeService(let origin, let field):
+            if let org = origin {
+                buf.moveWriterIndex(forwardBy: 2)
+                let start = buf.writerIndex
+                buf.writeString(org)
+                buf.setInteger(UInt16(buf.writerIndex - start), at: payloadStart)
+            } else {
+                buf.writeInteger(UInt16(0))
+            }
+
+            if let value = field {
+                payloadSize = buf.writerIndex - payloadStart + value.readableBytes
+                extraFrameData = .byteBuffer(value)
+            } else {
+                payloadSize = buf.writerIndex - payloadStart
+                extraFrameData = nil
+            }
+
+        case .origin(let origins):
+            for origin in origins {
+                let sizeLoc = buf.writerIndex
+                buf.moveWriterIndex(forwardBy: 2)
+
+                let start = buf.writerIndex
+                buf.writeString(origin)
+                buf.setInteger(UInt16(buf.writerIndex - start), at: sizeLoc)
+            }
+
+            payloadSize = buf.writerIndex - payloadStart
+            extraFrameData = nil
+        }
+
+        // Confirm we're not about to violate SETTINGS_MAX_FRAME_SIZE.
+        guard payloadSize <= Int(self.maxFrameSize) else {
+            throw InternalError.codecError(code: .frameSizeError)
+        }
+
+        // Write the frame data. This is the payload size and the flags byte.
+        buf.writePayloadSize(payloadSize, at: start)
+        buf.setInteger(flags.rawValue, at: flagsIndex)
+
+        // all bytes to write are in the provided buffer now
+        return extraFrameData
+    }
+}
+
+extension ByteBuffer {
+    fileprivate mutating func writePayloadSize(_ size: Int, at location: Int) {
+        // Yes, this performs better than running a UInt8 through the generic write(integer:) three times.
+        var bytes: (UInt8, UInt8, UInt8)
+        bytes.0 = UInt8((size & 0xff_00_00) >> 16)
+        bytes.1 = UInt8((size & 0x00_ff_00) >>  8)
+        bytes.2 = UInt8( size & 0x00_00_ff)
+        withUnsafeBytes(of: bytes) { ptr in
+            _ = self.setBytes(ptr, at: location)
+        }
+    }
+}
+