feat(docker): modular Docker Compose with layered -f files

[puckpuck]

Closes #2195
Supersedes #3107 (takes a -f layering approach instead of profiles)

Summary

Replaces the monolithic docker-compose.yml with four composable files:

File	Contents
compose.yaml	Core/minimal services — always the base
compose.full.yaml	Kafka group (accounting, fraud-detection, kafka)
compose.observability.yaml	Jaeger, Prometheus, OpenSearch, Grafana
compose.extras.yaml	Empty stub for vendor/fork customizations

The OTel Collector config is split into similar layers (otelcol-config.yml, otelcol-config-full.yml, otelcol-config-observability.yml, otelcol-config-extras.yml) and loaded via --config flags in the same order.

Four make targets cover the combinations:

make start                  # full + observability (default)
make start-minimal          # core + observability
make start-no-o11y          # full, no observability stack
make start-minimal-no-o11y  # core only

Forks can update the otherwise empty compose.extras.yaml and otelcol-config-extras.yml overrides to add custom o11y backends or demo services without touching upstream files.

[julianocosta89]

Please, let's not have 4 different collector config files and 4 different compose files.
This will be a mess maintaining and configuring.

Why can't we use docker profiles?
We can keep it all in one single compose file and just configure which profile will be executed via make.

[puckpuck]

The issue is when you have the same service that is run multiple different ways (collector, checkout, recommendation, etc) based on which profile it belongs to. To make this work with profiles you still need multiple -f files, so with profiles we are now mixing the -f CLI option and the profile construct.

This approach just doesn't use profiles at all and stays with -f, but in a cleaner way than we have today.

[rogercoll]

Please, let's not have 4 different collector config files and 4 different compose files. This will be a mess maintaining and configuring.

Why can't we use docker profiles? We can keep it all in one single compose file and just configure which profile will be executed via make.

I would also prefer not having 4 different compose files, but we might require them either way with Docker Profiles. Profiles can't patch services across boundaries, thus services like checkout needs KAFKA_ADDR and depends_on: kafka only in full mode, and otel-collector needs depends_on: jaeger/opensearch only with the observability mode. Profiles select which services to start, but they can't conditionally add env vars or dependencies to a service that's always on.

See that in #3107 a new compose.full.yaml was needed for the full profile.

[osullivandonal]

Looks like we need to add these env vars to the front-end proxy to get this to work:

  - TELEMETRY_DOCS_HOST
  - TELEMETRY_DOCS_PORT

Without the above when I run any make start-* the front-end proxy restarts with this error:

: Proto constraint validation failed (BootstrapValidationError.StaticResources: embedded message failed validation | caused by StaticResourcesValidationError.Clusters[9]: embedded message failed validation | caused by ClusterValidationError.LoadAssignment: embedded message failed validation | caused by ClusterLoadAssignmentValidationError.Endpoints[0]: embedded message failed validation | caused by LocalityLbEndpointsValidationError.LbEndpoints[0]: embedded message failed validation | caused by LbEndpointValidationError.Endpoint: embedded message failed validation | caused by EndpointValidationError.Address: embedded message failed validation | caused by AddressValidationError.SocketAddress: embedded message failed validation | caused by SocketAddressValidationError.Address: value length must be at least 1 characters)

[osullivandonal]

Even with the added maintenance burden of multiple compose files, I do like the idea of the compose.extras.yaml file to allow forks to easily specify there backends, o11y setup. Makes it a lot easier than the current setup.