update: Migrate Open Source Observability lab to the CloudWatch Observability add-on for AMP (OTel, drop ADOT)

[JaviGG02]

What this PR does / why we need it:

Migrates the EKS Open Source Observability lab
(open-source-metrics) to the current recommended setup
(part of #1895, item 2): metrics are now collected by the
Amazon CloudWatch Observability EKS add-on — the
CloudWatch agent, which is built on the open-source
OpenTelemetry
collector
and Fluent Bit — instead of a manually-deployed ADOT
collector. The lab still stores metrics in Amazon
Managed Service for Prometheus (AMP) and visualizes
them in Grafana; only the collection layer changes. ADOT
is no longer referenced — the content explains the
CloudWatch agent is open source under the hood and notes
that any OpenTelemetry-compatible collector could
remote-write to AMP. This PR updates both the content
and the lab preparation (Terraform/cleanup/tests) so
the module is consistent and passes automated testing.

Content

• adot.md (retitled "Collecting metrics with the
  CloudWatch agent"): replaced the hand-deployed
  OpenTelemetryCollector custom resource with the
  amazon-cloudwatch-observability add-on. The CloudWatch
  agent is granted AMP remote-write permissions via EKS
  Pod Identity — an IAM role trusting
  pods.eks.amazonaws.com with
  AmazonPrometheusRemoteWriteAccess (+
  CloudWatchAgentServerPolicy), associated with the
  cloudwatch-agent service account — then the add-on is
  installed with an OTel pipeline (Prometheus receiver →
  prometheusremotewrite exporter + sigv4auth extension).
  The pipeline is scoped to the single-replica
  cluster-scraper agent so AMP doesn't receive duplicate
  samples from the node DaemonSet. The page shows the add-on
  pods and includes an info box noting any OTel-compatible
  collector could be used.
• index.md: updated the "Before you start" summary
  and intro to describe the CloudWatch add-on (OTel + Fluent
  Bit). Also corrected prose that claimed "Amazon Managed
  Grafana" — the lab deploys self-managed Grafana.
• application-metrics.md: de-branded; replaced the
  kubectl get opentelemetrycollector inspection with the
  kubernetes-pods scrape job from the add-on config.
• kubernetes-dashboards.md: de-branded (metrics now
  "collected by the CloudWatch agent").
• amp-ws.md and accessing-grafana.md are unchanged (no
  ADOT branding). Refreshed the AMP datasource screenshot.
• accessing-grafana.md: added an info box on
  Amazon Managed Grafana for production — a fully
  managed Grafana (in collaboration with Grafana Labs) with
  no servers to operate, Grafana Enterprise plugins via AWS
  without a separate Grafana Labs license, IAM Identity
  Center/SAML sign-in, and native AWS data sources. The lab
  keeps self-managed Grafana for simplicity.

Lab preparation (Terraform / cleanup / tests)

• Terraform: prepare-environment now installs the
  eks-pod-identity-agent add-on (aws_eks_addon,
  resolve_conflicts_on_create=OVERWRITE) and keeps the AMP
  workspace, self-managed Grafana and dashboard ConfigMap;
  the OpenTelemetry operator, cert-manager and the ADOT IRSA
  role are removed. Added
  alb.ingress.kubernetes.io/healthcheck-path: /api/health
  to the Grafana ingress so the ALB target is healthy —
  Grafana redirects / to 302, which fails the ALB's
  default 200 health check. Dropped
  operator_chart_version; outputs.tf drops
  ADOT_IAM_ROLE (keeps AMP_ENDPOINT).
• cleanup.sh: deletes the observability add-on, the
  Pod Identity association and the cloudwatch-agent IAM
  role so the module is re-runnable; dropped the
  opentelemetrycollectors cleanup.
• tests: hook-install.sh waits for the
  eks-pod-identity-agent pods (was: the
  opentelemetrycollectors CRD); hook-deploy-adot.sh
  polls AMP for the up metric.
• The add-on config lives in a repo file
  (cwagent-amp/cloudwatch-agent-amp.yaml) referenced via
  ::yaml and applied with envsubst (restricted to
  $AMP_ENDPOINT $AWS_REGION) rather than an in-lab
  heredoc. It scrapes kubelet + cAdvisor (direct, role: node, static metrics_path — no capture-group
  replacement, which the CloudWatch agent expands as an env
  var) and application pods → AMP, feeding both the Order
  Service Metrics and Kubernetes Cluster Monitoring
  dashboards.

Note: the old adot/ manifests (and the ADOT_IAM_ROLE
entry in website/.kustomize-env) are intentionally kept
because the auto-generated ja translations still
reference them (removing them would fail the website
build). They'll be removed in a follow-up once the
translation pipeline regenerates ja from the updated
English content.

Which issue(s) this PR fixes:

Part of #1895

Testing

Ran make test module="observability/open-source-metrics"
→ 4 passing / 1 failing. The passing cases cover
prepare-environment, Collecting metrics with the
CloudWatch agent (add-on → AMP), Storing metrics with
AMP, and Application Metrics. The single failure is the
Accessing Grafana hook, which curls the internet-facing
Grafana ALB — my test environment's egress cannot reach
the ALB's public IPs (corporate network), so the check
times out. This is an environment limitation, not a
content issue: Grafana is healthy and returns 200, and
the ALB target is healthy once the /api/health
health-check path is applied. I verified end-to-end that
watch_orders_total, cAdvisor and kubelet metrics land in
AMP with no duplicate-sample errors.

Quality checks

• My content adheres to the style guidelines
• I ran make test module="observability/open-source-metrics" (see
  [automated_tests.md](https://github.com/aws-samples/eks-wo
  rkshop-v2/blob/main/docs/automated_tests.md)) — see the
  Testing note above regarding the Grafana ALB hook
• The PR has a meaningful title and description of the
  changes that will be included in the workshop release
  notes

By submitting this pull request, I confirm that you can
use, modify, copy, and redistribute this contribution,
under the terms of your choice.

[netlify]

✅ Deploy Preview for eks-workshop ready!

Name	Link
🔨 Latest commit	a908ef0
🔍 Latest deploy log	https://app.netlify.com/projects/eks-workshop/deploys/6a7c7f38d11b3700087272ec
😎 Deploy Preview	https://deploy-preview-1905--eks-workshop.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.