Add a client-side encrypting S3 service

README.md

@@ -76,7 +76,9 @@ It's way more work, and way more hassle. This is great for sensitive files, and
 
 ## What this gem _cannot_ do
 
-This gem does not provide an E2E encrypted solution. The file still gets encrypted by your cloud provider, and decrypted by your cloud provider. While it offers a strong protection _at rest_ it does not offer extra protection _in transit._ If you need that level of protection, you may want to look into [S3 client encryption](https://ankane.org/activestorage-s3-encryption) or other similar tech.
+The `EncryptedGCSService` and the `EncryptedS3Service` do not provide an E2E encrypted solution. The file still gets encrypted by your cloud provider, and decrypted by your cloud provider. While it offers a strong protection _at rest_ it does not offer extra protection _in transit._ If you need that level of protection on S3-compatible storage, use the [ClientSideEncryptedS3Service](#clientsideencrypteds3service---s3-compatible-storage-encrypted-in-your-application), which never hands the provider anything but ciphertext.
+
+Nothing here protects you against an attacker who has your running application, since the application is what holds the keys.
 
 ## Encrypted Service implementations
 
@@ -115,6 +117,32 @@ Implementation details:
 
 While S3 allows the `x-amz-server-side-encryption-customer-key-MD5` to be added to the signed URL for PUT, the value of that header gets removed from the signature due to the process called "hoisting" - which occurs during the signing of the URL. So your client _may_ override the encryption key you give it forcibly, by replacing the `x-amz-server-side-encryption-customer-key` and `x-amz-server-side-encryption-customer-key-MD5`. This can produce Blobs encrypted with a key you do not have. If you want to exclude the possibility of this, you need to perform an integrity check on your uploads. The integrity check will fail if the encryption key has been overridden in this manner, and you can then destroy the Blob. This problem has been reported to AWS.
 
+### ClientSideEncryptedS3Service - S3-compatible storage, encrypted in your application
+
+Where the `EncryptedS3Service` asks S3 to encrypt for us (SSE-C), this service encrypts inside your application and hands the bucket ciphertext only. The provider never holds the plaintext, at rest or in transit, and neither does anything between you and it. If your reason for encrypting is that you do not want to trust your storage provider with the contents, this is the service to use.
+
+```yaml
+# storage.yml
+encrypted_s3:
+  service: ClientSideEncryptedS3
+  bucket: my-bucket
+  region: eu-central-1
+  private_url_policy: stream
+```
+
+Implementation details:
+
+* It uses the same encryption scheme as the `EncryptedDiskService`, and the same `block_cipher_kit` code path, rather than introducing a second scheme. The schemes only need an IO to read from, and `ClientSideEncryptedS3Service::SeekableObjectIO` gives them one backed by ranged `GET` requests. So random access - being able to serve a byte range of a large video without downloading all of it - survives the move from a local disk to a bucket.
+* Reads are buffered with a window that starts at 64 KB and doubles up to 5 MB for as long as reads stay sequential, resetting after a seek. A one-byte `download_chunk` costs one small request; a full download quickly reaches the large window.
+* Each object begins with a five byte header: the ASCII `ASEC`, then one byte of scheme version. An object in a bucket has no filename to hang a version on (which is how the `EncryptedDiskService` does it) and asking for object metadata costs a request, so the ciphertext names its own format. Reading an object without that header raises `ActiveStorageEncryption::UnknownCiphertextFormat`, which is also how a blob written by a stock `S3Service` announces itself.
+* `private_url_policy: require_headers` is refused at configuration time. A presigned URL can only ever serve ciphertext, and the client has no key. Use `stream` or `disable`.
+* Direct uploads do not go to the bucket - the browser has no key, so a `PUT` there would store plaintext. They are routed to `EncryptedBlobsController` exactly as the `EncryptedDiskService` routes them: the application receives the plaintext, encrypts it, and puts it in the bucket. No bucket CORS configuration is needed.
+* SSE-C is not used, so S3-compatible providers which do not implement it (R2, Minio, Ceph...) work with this service.
+* The checksum ActiveStorage computes is of the plaintext, so it cannot be sent to S3 as a `Content-MD5` - S3 would be checking it against our ciphertext. The plaintext is digested as it streams into the cipher instead, and the object is deleted if the digests differ.
+* `#compose` downloads, decrypts and re-encrypts, as it does for GCS: the bucket cannot splice objects it cannot read.
+
+Note what streaming decryption can and cannot promise, since it is easy to assume more. GCM verifies its authentication tag only after the whole ciphertext has been read, so a `download` which yields chunks to a block will have yielded tampered plaintext before it raises, and `download_chunk` reads a range with no authentication at all (see the note on random access below). If you need the guarantee that no altered byte reaches a caller, read the blob with the non-block form of `download`, which buffers and raises before it returns anything.
+
 ### EncryptedDiskSevice - Filesystem
 
 Can be used instead of the cloud services in development, or on the server if desired. The service will use AES-256-GCM encryption, with a way to switch to a different/more modern encryption scheme in the future.

lib/active_storage/service/client_side_encrypted_s3_service.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+# Needed so that Rails can find our service definition. It will perform the following
+# steps. Given a "ClientSideEncryptedS3" value of the `service:` key in the YAML, it will:
+#
+# * Force-require a file at "active_storage/service/client_side_encrypted_s3", from any path on the $LOAD_PATH
+# * Instantiate a class called "ActiveStorage::Service::ClientSideEncryptedS3Service"
+require_relative "../../active_storage_encryption"
+class ActiveStorage::Service::ClientSideEncryptedS3Service < ActiveStorageEncryption::ClientSideEncryptedS3Service
+end

lib/active_storage_encryption.rb

@@ -10,6 +10,7 @@ module ActiveStorageEncryption
   autoload :EncryptedDiskService, __dir__ + "/active_storage_encryption/encrypted_disk_service.rb"
   autoload :EncryptedMirrorService, __dir__ + "/active_storage_encryption/encrypted_mirror_service.rb"
   autoload :EncryptedS3Service, __dir__ + "/active_storage_encryption/encrypted_s3_service.rb"
+  autoload :ClientSideEncryptedS3Service, __dir__ + "/active_storage_encryption/client_side_encrypted_s3_service.rb"
   autoload :EncryptedGCSService, __dir__ + "/active_storage_encryption/encrypted_gcs_service.rb"
   autoload :Overrides, __dir__ + "/active_storage_encryption/overrides.rb"
 
@@ -19,6 +20,12 @@ class IncorrectEncryptionKey < ArgumentError
   class StreamingDisabled < ArgumentError
   end
 
+  # Raised when an object in a bucket does not carry the header a client-side encrypting
+  # service writes ahead of its ciphertext - it was written by another service, or is not
+  # encrypted at all.
+  class UnknownCiphertextFormat < StandardError
+  end
+
   class StreamingTokenInvalidOrExpired < ActiveSupport::MessageEncryptor::InvalidMessage
   end
 

lib/active_storage_encryption/client_side_encrypted_s3_service.rb

@@ -0,0 +1,277 @@
+# frozen_string_literal: true
+
+require "block_cipher_kit"
+require "active_storage/service/s3_service"
+
+module ActiveStorageEncryption
+  # Stores ActiveStorage blobs on S3-compatible storage, encrypting them inside the application
+  # process. Where `EncryptedS3Service` asks the storage provider to encrypt for us (SSE-C, so the
+  # provider does hold the plaintext at the moment of the write), this service hands the provider
+  # ciphertext and nothing else. The bytes on the wire and the bytes at rest are both encrypted
+  # with a key the provider never sees.
+  #
+  # It reuses the encryption scheme of `EncryptedDiskService` rather than introducing a second one.
+  # That is possible because the schemes only need an IO to read from - see `SeekableObjectIO`,
+  # which turns ranged GET requests into the `read`/`seek`/`pos`/`size` they expect. So random
+  # access - the property that lets a large video be scrubbed rather than downloaded whole -
+  # survives the move from a local disk to a bucket.
+  #
+  # Configure it like so:
+  #
+  #   encrypted_s3:
+  #     service: ClientSideEncryptedS3
+  #     bucket: my-bucket
+  #     region: eu-central-1
+  #     private_url_policy: stream
+  #
+  # Two things behave differently from the SSE-C service, both of them consequences of the app
+  # being the only party that can encrypt or decrypt:
+  #
+  # * A presigned URL can only ever yield ciphertext, so `private_url_policy: require_headers`
+  #   is refused at configuration time. Use `stream` (through the controller in this gem, or
+  #   through one of your own) or `disable`.
+  # * Direct uploads from a browser cannot go to the bucket, since the browser has no key. They
+  #   are routed to `EncryptedBlobsController` instead, exactly as `EncryptedDiskService` does:
+  #   the app receives the plaintext, encrypts it, and puts it in the bucket.
+  #
+  # Bear in mind what streaming decryption can and cannot promise. GCM verifies its authentication
+  # tag only once the whole ciphertext has been read, so a `download` that yields chunks to a block
+  # will have yielded tampered plaintext before it raises, and `download_chunk` reads a range
+  # without any authentication at all (see `V2Scheme`). If you need a guarantee that no altered
+  # byte can reach a caller, read the whole blob with the non-block form of `download`, which
+  # buffers and raises before returning anything.
+  class ClientSideEncryptedS3Service < ActiveStorage::Service::S3Service
+    include ActiveStorageEncryption::PrivateUrlPolicy
+
+    autoload :SeekableObjectIO, __dir__ + "/client_side_encrypted_s3_service/seekable_object_io.rb"
+
+    # Unlike a file on disk, an object in a bucket has no filename we can hang a scheme version
+    # on, and asking the bucket for object metadata costs a request. So the ciphertext names its
+    # own format: a magic string (which also tells us an object was written by this service at
+    # all, rather than by a stock S3 service) followed by one byte of scheme version.
+    CIPHERTEXT_MAGIC_BYTES = "ASEC"
+    CIPHERTEXT_HEADER_BYTE_SIZE = CIPHERTEXT_MAGIC_BYTES.bytesize + 1
+
+    # The version byte matches the scheme version of EncryptedDiskService, so that the same number
+    # always means the same bytes on the wire regardless of where a blob is stored.
+    SCHEME_VERSIONS = {
+      2 => "ActiveStorageEncryption::EncryptedDiskService::V2Scheme"
+    }
+    CURRENT_SCHEME_VERSION = 2
+
+    # This lets the Blob encryption key methods know that this
+    # storage service _must_ use encryption
+    def encrypted? = true
+
+    def initialize(public: false, **options_for_s3_service_and_private_url_policy)
+      raise ArgumentError, "encrypted files cannot be served via a public URL or a CDN" if public
+      super
+      if private_url_policy == :require_headers
+        raise ArgumentError, "private_url_policy: require_headers is not available for #{self.class.name}, " \
+          "because a presigned URL would serve ciphertext which the client has no key to decrypt"
+      end
+    end
+
+    def service_name
+      # ActiveStorage::Service::DiskService => Disk
+      # Overridden because in Rails 8 this is "self.class.name.split("::").third.remove("Service")"
+      self.class.name.split("::").last.remove("Service")
+    end
+
+    def upload(key, io, encryption_key:, checksum: nil, filename: nil, content_type: nil, disposition: nil, custom_metadata: {}, **)
+      instrument :upload, key: key, checksum: checksum do
+        # The checksum ActiveStorage gives us is of the plaintext, so it cannot be handed to S3 as
+        # a Content-MD5 - S3 would be checking it against our ciphertext. We digest the plaintext
+        # as it streams into the cipher instead, which verifies the same thing without reading the
+        # object back out of the bucket afterwards.
+        plaintext_io = checksum ? PlaintextChecksumIO.new(io) : io
+        content_disposition = content_disposition_with(type: disposition, filename: filename) if disposition && filename
+
+        # Build the scheme before opening the upload. An unusable encryption key must raise here,
+        # where the caller can see the reason, rather than inside the block - the SDK would bury
+        # it in a MultipartUploadError, and we would have left a dangling upload behind.
+        scheme = scheme_for(CURRENT_SCHEME_VERSION, encryption_key)
+
+        object_for(key).upload_stream(
+          content_type: content_type,
+          content_disposition: content_disposition,
+          part_size: MINIMUM_UPLOAD_PART_SIZE,
+          metadata: custom_metadata,
+          **upload_options
+        ) do |ciphertext_io|
+          ciphertext_io.binmode
+          ciphertext_io.write(ciphertext_header)
+          scheme.streaming_encrypt(into_ciphertext_io: ciphertext_io, from_plaintext_io: plaintext_io)
+        end
+
+        ensure_integrity_of(key, checksum, plaintext_io) if checksum
+      end
+    end
+
+    def download(key, encryption_key:, &block)
+      if block_given?
+        instrument :streaming_download, key: key do
+          stream(key, encryption_key, &block)
+        end
+      else
+        instrument :download, key: key do
+          (+"").b.tap do |buf|
+            stream(key, encryption_key) { |chunk| buf << chunk }
+          end
+        end
+      end
+    end
+
+    def download_chunk(key, range, encryption_key:)
+      instrument :download_chunk, key: key, range: range do
+        open_ciphertext(key, encryption_key) do |ciphertext_io, scheme|
+          scheme.decrypt_range(from_ciphertext_io: ciphertext_io, range: inclusive(range))
+        end
+      end
+    end
+
+    def compose(source_keys, destination_key, source_encryption_keys:, encryption_key:, filename: nil, content_type: nil, disposition: nil, custom_metadata: {})
+      if source_keys.length != source_encryption_keys.length
+        raise ArgumentError, "With #{source_keys.length} keys to compose there should be exactly as many source_encryption_keys, but got #{source_encryption_keys.length}"
+      end
+      content_disposition = content_disposition_with(type: disposition, filename: filename) if disposition && filename
+      scheme = scheme_for(CURRENT_SCHEME_VERSION, encryption_key)
+
+      object_for(destination_key).upload_stream(
+        content_type: content_type,
+        content_disposition: content_disposition,
+        part_size: MINIMUM_UPLOAD_PART_SIZE,
+        metadata: custom_metadata,
+        **upload_options
+      ) do |ciphertext_io|
+        ciphertext_io.binmode
+        ciphertext_io.write(ciphertext_header)
+        scheme.streaming_encrypt(into_ciphertext_io: ciphertext_io) do |plaintext_writable|
+          source_keys.zip(source_encryption_keys).each do |(source_key, source_encryption_key)|
+            stream(source_key, source_encryption_key) { |chunk| plaintext_writable.write(chunk) }
+          end
+        end
+      end
+    end
+
+    def url_for_direct_upload(key, expires_in:, content_type:, content_length:, checksum:, encryption_key:, custom_metadata: {})
+      # A browser has no encryption key, so a PUT straight to the bucket would store plaintext.
+      # The upload goes to this gem's own controller instead, which encrypts it on the way in.
+      instrument :url, key: key do |payload|
+        upload_token = ActiveStorage.verifier.generate(
+          {
+            key: key,
+            content_type: content_type,
+            content_length: content_length,
+            encryption_key_sha256: Digest::SHA256.base64digest(encryption_key),
+            checksum: checksum,
+            service_name: name
+          },
+          expires_in: expires_in,
+          purpose: :encrypted_put
+        )
+
+        # Unlike the DiskService, an S3 service has no url_options of its own to build absolute
+        # URLs from - the ones ActiveStorage sets for the current request are what we have.
+        url_options = ActiveStorage::Current.url_options
+        raise ArgumentError, "Cannot generate a direct upload URL because ActiveStorage::Current.url_options is not set" if url_options.blank?
+
+        url_helpers = ActiveStorageEncryption::Engine.routes.url_helpers
+        url_helpers.encrypted_blob_put_url(upload_token, **url_options).tap do |generated_url|
+          payload[:url] = generated_url
+        end
+      end
+    end
+
+    def headers_for_direct_upload(key, content_type:, encryption_key:, checksum:, **)
+      {
+        "Content-Type" => content_type,
+        "x-active-storage-encryption-key" => Base64.strict_encode64(encryption_key),
+        "content-md5" => checksum
+      }
+    end
+
+    def headers_for_private_download(key, **)
+      # Nothing to send: the bytes in the bucket are of no use without the key, and the key
+      # never leaves the application.
+      {}
+    end
+
+    private
+
+    def ciphertext_header
+      (CIPHERTEXT_MAGIC_BYTES + CURRENT_SCHEME_VERSION.chr).b
+    end
+
+    def scheme_for(version, encryption_key)
+      scheme_class_name = SCHEME_VERSIONS.fetch(version) do
+        raise ActiveStorageEncryption::UnknownCiphertextFormat, "Unknown encryption scheme version #{version.inspect}"
+      end
+      Object.const_get(scheme_class_name).new(encryption_key.b)
+    end
+
+    # Opens the object, reads the header which tells us how the ciphertext after it was written,
+    # and yields an IO positioned at the start of that ciphertext together with the matching scheme.
+    def open_ciphertext(key, encryption_key)
+      ciphertext_io = SeekableObjectIO.new(object_for(key))
+      header = ciphertext_io.read(CIPHERTEXT_HEADER_BYTE_SIZE)
+      if header.nil? || header.byteslice(0, CIPHERTEXT_MAGIC_BYTES.bytesize) != CIPHERTEXT_MAGIC_BYTES
+        raise ActiveStorageEncryption::UnknownCiphertextFormat,
+          "Object #{key.inspect} in #{name} was not written by #{self.class.name} (no #{CIPHERTEXT_MAGIC_BYTES} header)"
+      end
+
+      scheme = scheme_for(header.getbyte(CIPHERTEXT_MAGIC_BYTES.bytesize), encryption_key)
+      yield ciphertext_io.rebase(CIPHERTEXT_HEADER_BYTE_SIZE), scheme
+    rescue Aws::S3::Errors::NoSuchKey
+      raise ActiveStorage::FileNotFoundError
+    end
+
+    def stream(key, encryption_key, &blk)
+      open_ciphertext(key, encryption_key) do |ciphertext_io, scheme|
+        scheme.streaming_decrypt(from_ciphertext_io: ciphertext_io, &blk)
+      end
+    end
+
+    # ActiveStorage passes exclusive ranges in some places (`0...4.kilobytes` when identifying a
+    # blob) and inclusive ones in others, while the schemes only understand inclusive ranges.
+    def inclusive(range)
+      range.exclude_end? ? (range.begin..(range.end - 1)) : range
+    end
+
+    def ensure_integrity_of(key, checksum, plaintext_io)
+      return if plaintext_io.base64digest == checksum
+
+      delete key
+      raise ActiveStorage::IntegrityError
+    end
+
+    def private_url(key, **options)
+      # :require_headers is refused in the constructor, and :disable raises inside this call,
+      # so streaming through a controller is all that is left.
+      private_url_for_streaming_via_controller(key, **options)
+    end
+
+    def public_url(key, **)
+      raise "This should never be called"
+    end
+
+    # Passes plaintext through to the cipher while digesting it, so that the checksum
+    # ActiveStorage computed over the same bytes can be verified after the upload.
+    class PlaintextChecksumIO
+      def initialize(io)
+        @io = io
+        @digest = OpenSSL::Digest.new("MD5")
+      end
+
+      def read(n_bytes = nil, outbuf = nil)
+        bytes_read = outbuf ? @io.read(n_bytes, outbuf) : @io.read(n_bytes)
+        @digest << bytes_read if bytes_read
+        bytes_read
+      end
+
+      def base64digest
+        @digest.base64digest
+      end
+    end
+  end
+end