feat: add Amazon Data Firehose + S3 telemetry mirror

[CoreyLeath-code]

Summary

• add an optional, bounded Firehose telemetry mirror to the Go ingestion service
• publish accepted inference logs as newline-delimited JSON with AWS SDK for Go v2
• keep the AWS mirror fail-open so telemetry outages do not take down the primary ingestion path
• provision private/versioned/encrypted S3 storage, Amazon Data Firehose, CloudWatch delivery logs, and least-privilege IAM with Terraform
• expire current and noncurrent S3 object versions using the configured telemetry retention window
• add explicit producer and Terraform validation CI
• document deployment, IAM, observability, failure semantics, and cost controls
• replace the misnamed terraform/main,TF with valid Terraform files

Validation

• Go unit tests cover disabled-by-default configuration, required stream configuration, and NDJSON PutRecord payloads
• CI runs go mod tidy and fails on module-file drift before running go test ./...
• Docker Compose continues to default FIREHOSE_ENABLED=false
• Terraform CI verifies formatting, initializes providers without a backend, and runs terraform validate
• Firehose's S3 delivery role follows AWS's documented S3 destination permission set, scoped to the SentinelAI telemetry bucket

Honest scope

This PR adds a best-effort observability mirror, not a transactional dual-write guarantee. PostgreSQL remains the default local persistence path. The Firehose queue is bounded and in-memory; producer errors/drops are surfaced through ingestion_firehose_records_total and logs. No Firehose/S3 throughput benchmark or deployed-production claim is added by this PR.

Summary by CodeRabbit

• New Features

  • Added an optional AWS telemetry mirror that forwards accepted inference events to Amazon Data Firehose and encrypted S3 storage.
  • Added Terraform configuration for provisioning telemetry storage, delivery, permissions, retention, and outputs.
  • Added configurable Firehose settings, including enablement, delivery stream, queue size, and AWS region.
  • Telemetry mirroring is disabled by default and uses best-effort delivery.

• Documentation

  • Added setup, configuration, monitoring, failure-handling, and cost-control guidance.

• Tests & Validation

  • Added automated Firehose tests and CI validation for the ingestion service and infrastructure configuration.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fc8a132b-a414-4651-a4c8-e6ec1720b9f3

📥 Commits

Reviewing files that changed from the base of the PR and between 9129500 and 93215f3.

⛔ Files ignored due to path filters (1)

• ingestion-service/go.sum is excluded by !**/*.sum

📒 Files selected for processing (14)

• .env.example
• .github/workflows/aws-telemetry.yml
• README.md
• docker-compose.yml
• docs/aws-firehose-s3.md
• ingestion-service/Dockerfile
• ingestion-service/firehose.go
• ingestion-service/firehose_test.go
• ingestion-service/go.mod
• ingestion-service/main.go
• terraform/main,TF
• terraform/main.tf
• terraform/outputs.tf
• terraform/variables.tf

---

📝 Walkthrough

Walkthrough

Adds an optional, fail-open AWS Data Firehose mirror for accepted inference telemetry. It provisions Firehose, S3, IAM, CloudWatch, and ECR resources with Terraform, adds runtime configuration and metrics, documents the path, and adds Go and Terraform CI validation.

Changes

AWS telemetry mirror

Layer / File(s)	Summary
Terraform telemetry infrastructure terraform/main.tf, terraform/variables.tf, terraform/outputs.tf	Adds configurable S3, Firehose, CloudWatch, ECR, IAM role, IAM writer policy, lifecycle, security, delivery, and output definitions.
Firehose dispatcher and publishing ingestion-service/firehose.go, ingestion-service/firehose_test.go, ingestion-service/go.mod	Adds configuration validation, a bounded non-blocking queue, asynchronous PutRecord publishing, NDJSON encoding, timeout handling, Prometheus status metrics, AWS SDK dependencies, and unit tests.
Ingestion integration and deployment configuration ingestion-service/main.go, docker-compose.yml, .env.example, ingestion-service/Dockerfile, README.md, docs/aws-firehose-s3.md	Initializes the optional mirror, enqueues records after warehouse writes, documents fail-open behavior and AWS setup, adds environment variables, uses UTC default timestamps, and updates the Go build image.
Automated Go and Terraform validation .github/workflows/aws-telemetry.yml, README.md	Adds CI checks for Go module tidiness, Go tests, Terraform formatting, provider initialization, and Terraform validation. Updates the documented validation boundary and CI badge.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant logHandler
  participant Warehouse
  participant firehoseDispatcher
  participant AmazonDataFirehose
  participant AmazonS3

  Client->>logHandler: Submit inference log
  logHandler->>Warehouse: Persist accepted entry
  Warehouse-->>logHandler: Write succeeds
  logHandler->>firehoseDispatcher: Enqueue telemetry
  firehoseDispatcher->>AmazonDataFirehose: PutRecord with NDJSON
  AmazonDataFirehose->>AmazonS3: Deliver GZIP-compressed object

Loading

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/aws-firehose-s3

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74b689636c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid making the chart's reused latest tag immutable

With the default project_name, this is the same sentinelai-repo that the checked Helm deployment references as sentinelai-repo:latest (helm/sentinel/templates/deployment.yaml:17). ECR will reject overwriting that tag after the first push, so subsequent releases using the chart's fixed image reference cannot be published. Use versioned tags or digests in the chart, or keep the repository mutable while latest is reused.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Pass AWS credentials into the Compose service

When the mirror is enabled through the documented local docker compose up flow, this environment block forwards the feature settings and region but none of the AWS credential/profile variables, and it does not mount the host's shared AWS configuration. Consequently, credentials available through the host's normal SDK chain are invisible inside the container and every PutRecord fails while /log still returns accepted. Provide a secure credential or workload-role forwarding mechanism, or document the required Compose override.

Useful? React with 👍 / 👎.